At the end of April, Google quietly updated their web crawler list to include a new addition – GoogleOther. Google is constantly updating, with daily changes to the algorithms, improvements to security and search systems and more, so it’s no surprise that this update has come without much in the way of announcement beyond a LinkedIn post by Analyst Gary Illyes, stating:
“We added a new crawler, GoogleOther to our list of crawlers that ultimately take some strain off of Googlebot. This is a no-op change for you, but it’s interesting nonetheless I reckon. As we optimize how and what Googlebot crawls, one thing we wanted to ensure is that Googlebot’s crawl jobs are only used internally for building the index that’s used by Search. For this we added a new crawler, GoogleOther, that will replace some of Googlebot’s other jobs like R&D crawls to free up some crawl capacity for Googlebot.”
GoogleOther was added to the list of crawlers and is designed to relieve the strain the other crawling bots, particularly Googlebot, are under. As one of the most well-known search engines, Google processes over 99,000 searches every second, or over 8.5 billion searches a day and with that level of traffic, the strain on its crawlers to effectively and efficiently index millions of websites to cater to these searches is growing each day. With new AI-based algorithms and machine learning, Googlebot is under more strain and pressure than ever to provide accurate and thorough crawling of websites.
GoogleOther has been released as a way to reduce the strain, offering a generic crawl bot that works along the same directives and protocols as the main Googlebot crawler but frees up resources for the main crawlers. The listing on the Developers’ site lists Googlebot as:
“Generic crawler that may be used by various product teams for fetching publicly accessible content from sites. For example, it may be used for one-off crawls for internal research and development.”
It has the same limitations and features, including robots.txt, host load limitations, the same HTTP protocol version and fetch sizes, effectively making this crawler the same as Googlebot, but under a different name. For webmasters, this means that there is very little that will change or needs to be done. You’ll simply see a new bot appear from time to time.
For more information about how Google views your website or for help setting up your website to be optimised for not just Google, but other search engines too, we are on hand to help. Simply get in touch to find out more.
