Tag Archives: robot
Twitter has a bot to crawl webpages: Twitterbot. I noticed it in the server logs. The user agent is Twitterbot/1.0. The IP address is 22.214.171.124. Why is twitter crawling web pages? Google’s spiders regularly crawl the web to rebuild our index. Crawls are based on many factors such as PageRank, links to a page, and crawling constraints such as the number of parameters in a URL. Any number of factors can affect the crawl frequency of individual sites.