# robots.txt for http://cto.homelinux.net/ and ftp://cto.homelinux.net/ # see also http://www.conman.org/people/spc/robots2.html # and http://web.nexor.co.uk/mak/doc/robots/norobots.html # # put recently-added features last in case they cause the reader to stop # reading. # # Visit-time: is a GMT time range of allowed access. # Request-rate: for us is a maximum of one page per 5 seconds, at all hours. # Crawl-delay: is apparently only for msnbot. User-agent: * Visit-time: 2100-2300 Request-rate: 1/10 Crawl-delay: 10 Disallow: /apache2-default/ Disallow: /cgi-bin/ Disallow: /gkacd/ Disallow: /gkgui/ Disallow: /html/ Disallow: /images/ Disallow: /linux/ Disallow: /misc/ Disallow: /nut/ Disallow: /pub/ Disallow: /squid-reports/ Disallow: /usage/ Disallow: /users/ Disallow: /wp-admin Disallow: /yate/