Support Bot shareware, demo, freeware, software downloads, downloadable, downloading page 2 -

C# Web Scraping Library

... In addition, it can obey robots.txt directives including bot specific crawl rates and limitation. The exact urls ... facilities and for statistical competitive analysis Full documentation, support and downloadable DLLS for the C# Web Scraper ...