Future InTech

Crawl access and indexing · FIELD GUIDE

Google crawler classes: avoid one policy for every request

Separate common crawlers, special-case crawlers and user-triggered fetchers before interpreting robots behavior.

By Future InTech · Published
Evidence cutoff: 2026-11-02T12:01:04.822040+00:00 · UTC

What the evidence can establish

The first decision is which documented client category the request belongs to. Google distinguishes automatic common crawlers, product-specific special-case crawlers and user-triggered fetchers. These categories are not interchangeable promises about robots behavior. A name in a user-agent string is a classification clue, not verification of the sender. Keep that uncertainty visible before changing an access rule.

Hypothetical triage: an analyst puts a Googlebot request and a Google Site Verifier request in one “search crawling” total. Split them into automatic crawling and a user-triggered verification action. The two requests may have different purposes even when they reach the same URL; neither record establishes an indexing outcome.

Identify the claimed client; Assign its documented category; Check product-specific policy; Record response separately
Original explanatory diagram based on the dated sources below. Any example values are hypothetical; this is not a screenshot, customer measurement or AI platform result.

The saved evidence used here

This guide interprets dated official documentation. It does not report traffic from a customer or a measured AI answer. Each material claim is mapped to a source below.

c1 · Explanation and verification steps
Common Google crawlers respect robots.txt during automatic crawls. Source 1
c2 · Explanation and verification steps
Special-case crawlers can operate under product-specific arrangements, including the documented AdsBot example. Source 1
c3 · Explanation and verification steps
Google Site Verifier is an example of a user-triggered fetcher. Source 1

The diagram explains the investigation or a clearly labeled hypothetical example. No measured counts or platform outcomes are added.

AI answer visibility: Not measured. No original platform answers or verified citation records were collected for this guide. A request count is not an audience, ranking or citation count.

What to do next

  1. Record the exact user agent, requested path, time and response from the existing access logs; remove identifiers unnecessary for the investigation.
  2. Match the documented client to its category. Common crawlers respect robots for automatic crawls; special-case clients can have product-specific arrangements.
  3. For an unexpected client, inspect the product-specific documentation and retain an unclassified state when identification is unresolved. Do not infer ownership from the name alone.
  4. Create a policy table with separate columns for client category, documented robots behavior, desired access and observed response. Change only the rule for the relevant purpose.

How to check the result

The handoff should identify the client category and its evidence, preserve unknown identity separately, and explain why the selected policy fits that category. A successful fetch is a separate observation from search indexing.

Limits of this conclusion

This guide classifies policy expectations; it does not verify a real requester. Product-specific arrangements must be checked before applying the common-crawler rule to another category.

This is a saved, dated edition. Source documentation can change. This edition retains the original retrieval dates and claim mapping; follow the sources before applying version-specific guidance.

Sources and publication checks

Automated checks validate source availability, supported fields, duplicate wording, numeric figures and page configuration. This guide was prepared with AI assistance and checked against saved source passages in the same authoring session. The batch importer checks source fingerprints, quotation locations, wording overlap and required fields. No paid generation API or separate model-review pass was used for this batch. These checks can miss errors and are not expert or human review.

Overview of Google crawlers and fetchers (user agents)

Obtained: 2026-11-02T12:01:04.822040+00:00
Location: Overview of Google crawlers and fetchers (user agents) / b2; Overview of Google crawlers and fetchers (user agents) / b3; Overview of Google crawlers and fetchers (user agents) / b4

Saved source fingerprint059e62e33d0840a7dfbbf0a7fa04eb77b20ae12649bf68b283d34110a6af7b26

Download this edition's evidence and action record (JSON)