Google has now added new details that explain the three categories its Google crawlers fall into, they include Googlebot, special-case crawlers and user-triggered fetchers.
In addition, Google now lists a JSON formatted file containing the list of IP addresses each of these different crawler types use.
Types of Google crawlers. At the top of this Googlebot page, Google listed these three crawler types:
IP addresses. Google also listed the IP address ranges and reverse DNS mask for each type:
What is new. Here is the section of the page that was updated; the rest of the page is mostly unchanged.
Why we care. I believe Google made this change after they saw some of the reactions to the GoogleOther robot they announced the other day. This now explains how Google crawlers act, when they respect the robots.txt and how to identify them better.
Now, if you want not to block Google’s main crawler, Googlebot, but you decide to block the others, you can better identify those crawlers more accurately.
Web directories, once tools for discovering websites in the early days of the Internet, have…
What Is Quality Content?Quality content typically refers to content that is useful, accurate, reliable, and…
Social media monitoring helps you protect your brand’s image on social platforms. And the right…
Here is a recap of what happened in the search forums today, through the eyes…
Google has provided details about the capabilities and limitations of its AI image generation tools…
On March 5, 2024, Google announced the launch of the March 2024 Core Update. The…
This website uses cookies.
Leave a Comment