Robots, Spiders, Crawlers and Wanderers - Highway Robbers on your Web Site


	About Products Free Trial Purchase Support Tech Library Neat Stuff Privacy	Search SiteMap Contact

	HTML Info E-Commerce Web History Computer Humor Anti-Spam Info

Robots, Spiders, Crawlers and Wanderers - Highway Robbers on your Web Site

Robots that visit your website and cull e-mail addresses from your pages and other robots, know as websuckers or sitesuckers, that download your entire website in violation of your copyright are a major problem. These robots are assisting people in spamming you and copying your website. In addition, they consume enormous amounts of bandwidth that you pay for and contribute to your site going over hit quotas. The following resources help you identify the culprits and tell you what they do. You'll also find information about how to exclude them from your website. Please note that not all robots are bad. You do want to encourage the major search engines to send their robots to your site to index your pages and include the pages in their databases. It is mainly the private robots used by individuals that are abused and violate your privacy.

BotSpot: The Spot for all Bots & Intelligent Agents
Crawling the Web: A guide to robots, spiders, and other shadowy denizens of the Web
Robot Exclusion Standard - Implementation and Use
Robot Exclusion Standard Revisisted - Complete description of the /robots.txt file
Web Robots Database - List of all Active Robots

Web Robots FAQ
Try a Free Evaluation Copy of our Product: LinkScan

LinkScan is a website quality assurance tool that checks links and produces two types of SiteMaps using multi-threaded simultaneous processing that provides reports on HTML pages readable from any browser on any platform. Get a free evaluation copy.