Google has launched a new Googlebot, Google web crawler, named GoogleOther. GoogleOther will be used internally at Google by various project teams to crawl the public web, said Gary Illyes of Google on LinkedIn. It respects the same directives and protocols of the main Googlebot crawler and will free up some resources for the main Googlebot crawlers.
GoogleOther. Google added GoogleOther to the Google crawlers list documentation, it says, “Generic crawler that may be used by various product teams for fetching publicly accessible content from sites. For example, it may be used for one-off crawls for internal research and development. The GoogleOther crawler always obeys robots.txt rules for its user agent token and the global user agent (*), and uses the same IP ranges as Googlebot.”
User Agent. The user agent token and full user agent string is simple, it is just “GoogleOther”.
Why the new bot. This new bot will “ultimately will take some strain off of Googlebot,” Gary Illyes explained. “As we optimize how and what Googlebot crawls, one thing we wanted to ensure is that Googlebot’s crawl jobs are only used internally for building the index that’s used by Search. For this we added a new crawler, GoogleOther, that will replace some of Googlebot’s other jobs like R&D crawls to free up some crawl capacity for Googlebot,” Gary Illyes added.
Why we care. Many of you check your crawling activities and bot activity on your website and in your log files. If you see this new GoogleOther bot, do not be alarmed, it is a real Googlebot. It is unclear what would happen if you try to block this new bot. Google said they use it for internal purposes, does that mean it is being used for Bard? It is unclear.
But I suspect you should keep an eye on it and if it is causing too much stress on your server, you can decide if you want to allow or deny access to your site for this new Googlebot.
