What does the IndexifierBot do?

IndexifierBot is our web crawler which is a very important tool in our sitemap campaigns - since our service is a website urls indexing and crawling, we need to automatically parse third-party sites to verify their urls and status. It is an indispensable part of our technology that aggregates site data into a brief url profile for every site - that usually includes querying your site for metadata, favicons and making a screenshot of its homepage and some other actions. Such site profiles support the way our service works, so in order to rebuff all concerns we need to say that IndexifierBot never harvests any e-mail addresses or content that is not related to sitemap campaigns.

We want our crawler to be as 'polite' as possible (sending a minimal number of queries to your site) but in case it causes you any issues, please let us know via contact form and provide us with all info that might appear helpful.

How do I prevent the IndexifierBot from visiting my site?

Our crawler will obey any standard-conforming rule you provide in your robots.txt file. To disallow IndexifierBot visiting and parsing your site, you can put the following lines into your robots.txt file:

                                            
                                                User-agent: IndexifierBot
                                                Disallow: /
                                            
                                        

This will result in our crawler visiting your site only once and not returning anytime soon (accessing just this one file to execute your robots policy). IndexifierBot generally obeys such robots.txt directives as: allow, disallow, crawl-delay, host. You may look for specifications of these directives on http://www.robotstxt.org.

IndexifierBot keeps crawling your site despite the robot.txt instructions

If IndexifierBot visits your website too frequently and ignores the robot.txt commands, please contact us.