# Web Crawler intructions for all web servers # Thursday, 18 May 2006 16:25:15 # Exclude Files from all Robots: User-agent: * Disallow: /adminTacX32_rumpC/ Disallow: /DBS_RUMFORD Disallow: /Connections Disallow: /QuB Disallow: /FX_DataCounter User-agent: ia_archiver Disallow: / User-agent: YahooSeeker Disallow: / # End File