# robots.txt file User-agent: * Disallow: /cgi-vb/ # These guys seem courteous enough. Letting them keep crawling. 2007-12-14 User-agent: twiceler Disallow: