Yahoo! Search blog has announced today the move of their web crawler. With the acquisition of Inktomi a couple of years ago the crawler was set at inktomisearch.com. So everybody interested could see in the log data a visit of Yahoo crawler to their site under this domain name. However, for further the crawler will be removed to crawl.yahoo.net domain.

There are some reasons that could make you keep an eye on Yahoo bot migration process as citing Yahoo blog:

The user-agent will continue to be Yahoo! Slurp, hence you do not need to make any changes to your robots.txt file. Nor are we changing the actual IP addresses of the crawler infrastructure during this transition. However, if you do any reverse DNS checks for the crawler identity or have any network access rules to allow inktomisearch.com, please also update them to allow for crawl.yahoo.net.

The full transition from the old crawler address to the new one will start immediately and will take a few weeks.

Leave a Reply

You must be logged in to post a comment.