TERMS & CONDITIONS
TERMS & CONDITIONS
Thank you for contacting us!
We will perform an analysis on
your website and exclude from crawling
as soon as possible.
Click here if you don't want us
to crawl your website
If you've come to this page, then you're probably interested in learning more about
our web crawler, identified as user-agent CrazyWebCrawler-Spider".
CrazyWebCrawler-Spider runs on a grid computing platform that consists of several
thousand computers, which is why you may see our web crawler access your site
from many different IP addresses.
CrazyWebCrawler-Spider is the user-agent used by CrazyWebCrawler, a web
crawling service provider. It allows its users to design and run custom web crawls.
So, if Spider is crawling your website, it means that one or more of our users created
a web crawl that went (eventually) to your website.
People use CrazyWebCrawler for a variety of reasons, including providing data to
their own search engines, monitoring trends in online opinions, and other
If you feel that CrazyWebCrawler-Spider is crawling your website too quickly, please let
us know what an appropriate crawl rate is. If you'd like us to stop crawling your website,
the best thing to do is to block our web crawler using the robots.txt specification.
If you block CrazyWebCrawler-Spider using robots.txt, you will see crawl requests die
down gradually, rather than immediately. This happens because of our distributed
architecture. Our computers only periodically receive robots.txt information for domains
they are crawling.
Blocking our web crawler by IP address will not work. Due to the distributed nature
of our infrastructure, we have thousands of constantly changing IP addresses. We
strongly recommend you don't try to block our web crawler by IP address, as you'll
most likely spend several hours of futile effort and be in a very bad mood at the end
of it. You really should just include us in your robots.txt or contact us directly..