Twitter has a bot to crawl webpages: Twitterbot. I noticed it in the server logs. The user agent is Twitterbot/1.0. The IP address is Why is twitter crawling web pages?

Google’s spiders regularly crawl the web to rebuild our index. Crawls are based on many factors such as PageRank, links to a page, and crawling constraints such as the number of parameters in a URL. Any number of factors can affect the crawl frequency of individual sites.

Related Posts:

  • No Related Posts
This entry was posted in Marketing, Web Development and tagged , . Bookmark the permalink.

One Response to Twitterbot

Leave a Reply

Your email address will not be published. Required fields are marked *