# www.inwap.com welcomes web-crawling robots. User-agent: * Disallow: /web-hints/rgb2gray.cgi # For information on how the robots.txt file works, check out # http://info.webcrawler.com/mak/projects/robots/faq.html # http://info.webcrawler.com/mak/projects/robots/norobots.html # # The line will stop individual # documents from being indexed, # will make it stop looking for HREFs pointing to other documents.