Google has fixed a typo in their crawler documentation that inadvertently misidentified one of their crawlers. In general, this is a minor issue but it’s a major issue for SEOs and publishers ...

The issue began on August 8, 2025 and is now being resolved. It is unclear what percentage of sites were impacted.

Google may be crawling the web with a new crawler, a new Googlebot, named GoogleProducer. This useragent is not listed on the official Google crawlers page but maybe it is too new to be listed yet?

Google updated the list of their official crawlers by adding the name and information for a relatively unknown crawler that publishers have been seeing now and then but no documentation for it ...

Google has launched GoogleOther, a web crawler that various internal teams within the company will use to crawl the public web, Google's Gary Illyes wrote in a post.

Google announced the launch of a new version of its web crawler Googlebot on Tuesday at the Google I/O developer conference. Googlebot will now be “evergreen,” which means the crawler will ...

Google has updated its crawler help documentation to add a new section for HTTP caching, which explains how Google's crawlers handle cache control headers. Google also posted a blog post begging ...

The web is hostile to upstart search engine crawlers, and most websites only allow Google's crawler.

Fastly report reveals Meta, Google, and OpenAI accounted for 95% of AI crawler requests between April and July ...

Google has added a new section to its crawler and fetcher documentation for HTTP caching, which clarifies how Google’s crawlers handle cache control headers. With that, Gary Illyes from Google ...

One of the cornerstones of Google's business (and really, the web at large) is the robots.txt file that sites use to exclude some of their content from the search engine's web crawler, Googlebot ...