About Dolphin Crawler

This page is addressed to web masters who have noticed presence of an unknown robot and follow the link provided in 'from' HTTP header.
Dolphin Crawler is a web-crawler for search engine. Its main functions are:

• downloading a web page;
• extracting links from it;
• extracting plain text from it.

Dolphin Crawler is not a kind of 'smart agent', it just receives a bunch of links and tries to load their content. For now, it accepts only content, which is marked as "text/html" in HTTP header. It skips other types of content. Maximum size of content supported is 1 MB.

Politeness

We did our best to make Dolphin Crawler really 'polite':

• It complies with robots.txt rules and never attempts to visit restricted areas of a site.
• It requests compressed content that helps (if available) to reduce traffic.
• It requests only one page from the IP address at the moment. 1 request per second by default.
• It loads HTTP header first, then checks �Content-Type� field and stops loading a web page if such type of content isn�t supported by the crawler.

You can reduce Dolphin Crawler by placing this line to robots.txt: or disable it

Contact us

Feel free to contact us at crawler@telehouse.ru if you have any additional questions or in case of any problems you meet with Dolphin Crawler.