Twitter has a bot to crawl webpages: Twitterbot. I noticed it in the server logs. The user agent is Twitterbot/1.0. The IP address is Why is twitter crawling web pages?

Google’s spiders regularly crawl the web to rebuild our index. Crawls are based on many factors such as PageRank, links to a page, and crawling constraints such as the number of parameters in a URL. Any number of factors can affect the crawl frequency of individual sites.

Related Posts:

  • No Related Posts
This entry was posted in Marketing, Web Development and tagged , . Bookmark the permalink.

One Response to Twitterbot

  1. Flash says:

    Yes is crawling our site but we do post a lot of twitter links so maybe they are checking the links are real.

    Google ranking is based on profit, nothing more, nothing less and you can have the best site in the world but unless you pay then don’t think Google will send much traffic your way

Leave a Reply

Your email address will not be published. Required fields are marked *