About  Products  Free Trial  Purchase  Support  Tech Library  Neat Stuff  Privacy 
Search  SiteMap  Contact   
  HTML Info  E-Commerce  Web History  Computer Humor  Anti-Spam Info     


Robots, Spiders, Crawlers and Wanderers - Highway Robbers on your Web Site

Robots that visit your website and cull e-mail addresses from your pages and other robots, know as websuckers or sitesuckers, that download your entire website in violation of your copyright are a major problem. These robots are assisting people in spamming you and copying your website. In addition, they consume enormous amounts of bandwidth that you pay for and contribute to your site going over hit quotas. The following resources help you identify the culprits and tell you what they do. You'll also find information about how to exclude them from your website. Please note that not all robots are bad. You do want to encourage the major search engines to send their robots to your site to index your pages and include the pages in their databases. It is mainly the private robots used by individuals that are abused and violate your privacy.

  • BotSpot: The Spot for all Bots & Intelligent Agents
  • Crawling the Web: A guide to robots, spiders, and other shadowy denizens of the Web
  • Robot Exclusion Standard - Implementation and Use
  • Robot Exclusion Standard Revisisted - Complete description of the /robots.txt file
  • Web Robots Database - List of all Active Robots
  • Web Robots FAQ

    Try a Free Evaluation Copy of our Product: LinkScan

    LinkScan is a website quality assurance tool that checks links and produces two types of SiteMaps using multi-threaded simultaneous processing that provides reports on HTML pages readable from any browser on any platform. Get a free evaluation copy.


      © Copyright 1997-2010 Electronic Software Publishing Corporation (Elsop)
    LinkScan™ and Elsop™ are Trademarks of Electronic Software Publishing Corporation