Crawl access and indexing · FIELD GUIDE
Google crawler classes: avoid one policy for every request
Separate common crawlers, special-case crawlers and user-triggered fetchers before interpreting robots behavior.
What the evidence can establish
The first decision is which documented client category the request belongs to. Google distinguishes automatic common crawlers, product-specific special-case crawlers and user-triggered fetchers. These categories are not interchangeable promises about robots behavior. A name in a user-agent string is a classification clue, not verification of the sender. Keep that uncertainty visible before changing an access rule.
Hypothetical triage: an analyst puts a Googlebot request and a Google Site Verifier request in one “search crawling” total. Split them into automatic crawling and a user-triggered verification action. The two requests may have different purposes even when they reach the same URL; neither record establishes an indexing outcome.
The saved evidence used here
This guide interprets dated official documentation. It does not report traffic from a customer or a measured AI answer. Each material claim is mapped to a source below.
- c1 · Explanation and verification steps
- Common Google crawlers respect robots.txt during automatic crawls. Source 1
- c2 · Explanation and verification steps
- Special-case crawlers can operate under product-specific arrangements, including the documented AdsBot example. Source 1
- c3 · Explanation and verification steps
- Google Site Verifier is an example of a user-triggered fetcher. Source 1
The diagram explains the investigation or a clearly labeled hypothetical example. No measured counts or platform outcomes are added.
AI answer visibility: Not measured. No original platform answers or verified citation records were collected for this guide. A request count is not an audience, ranking or citation count.
What to do next
- Record the exact user agent, requested path, time and response from the existing access logs; remove identifiers unnecessary for the investigation.
- Match the documented client to its category. Common crawlers respect robots for automatic crawls; special-case clients can have product-specific arrangements.
- For an unexpected client, inspect the product-specific documentation and retain an unclassified state when identification is unresolved. Do not infer ownership from the name alone.
- Create a policy table with separate columns for client category, documented robots behavior, desired access and observed response. Change only the rule for the relevant purpose.
How to check the result
The handoff should identify the client category and its evidence, preserve unknown identity separately, and explain why the selected policy fits that category. A successful fetch is a separate observation from search indexing.
Limits of this conclusion
This guide classifies policy expectations; it does not verify a real requester. Product-specific arrangements must be checked before applying the common-crawler rule to another category.
This is a saved, dated edition. Source documentation can change. This edition retains the original retrieval dates and claim mapping; follow the sources before applying version-specific guidance.
Sources and publication checks
Automated checks validate source availability, supported fields, duplicate wording, numeric figures and page configuration. This guide was prepared with AI assistance and checked against saved source passages in the same authoring session. The batch importer checks source fingerprints, quotation locations, wording overlap and required fields. No paid generation API or separate model-review pass was used for this batch. These checks can miss errors and are not expert or human review.
Obtained: 2026-11-02T12:01:04.822040+00:00
Location: Overview of Google crawlers and fetchers (user agents) / b2; Overview of Google crawlers and fetchers (user agents) / b3; Overview of Google crawlers and fetchers (user agents) / b4
Saved source fingerprint
059e62e33d0840a7dfbbf0a7fa04eb77b20ae12649bf68b283d34110a6af7b26