Crawl access and indexing · FIELD GUIDE
Google robots rules: diagnose group and path precedence
Test user-agent group selection separately from URL-path matching when robots behavior seems inconsistent.
What the evidence can establish
There are two decisions to inspect: the applicable user-agent group and the applicable path rule. Google selects the most specific matching user-agent group; moving a global group above another group is not a reliable repair. Within path matching, the most specific path length matters, and Google uses the less restrictive rule when matching rules conflict. Keep these stages separate in a test table.
Hypothetical file: a global group blocks /private/, while a Googlebot-specific group contains only a rule for /draft/. Do not assume the global block is inherited into the selected specific group. This is a configuration example, not a recommendation to expose /private/; genuinely private pages still need access control.
The saved evidence used here
This guide interprets dated official documentation. It does not report traffic from a customer or a measured AI answer. Each material claim is mapped to a source below.
- c1 · Explanation and verification steps
- Google uses the most specific matching user-agent group; group order is irrelevant. Source 1
- c2 · Explanation and verification steps
- Google merges multiple groups relevant to the same specific user agent. Source 1
- c3 · Explanation and verification steps
- Google resolves path rules by specificity and uses the less restrictive rule in conflicts. Source 1
- c4 · Explanation and verification steps
- Google does not merge global and user-agent-specific robots groups. Source 1
The diagram explains the investigation or a clearly labeled hypothetical example. No measured counts or platform outcomes are added.
AI answer visibility: Not measured. No original platform answers or verified citation records were collected for this guide. A request count is not an audience, ranking or citation count.
What to do next
- Save the exact public robots.txt bytes, retrieval time and host. Do not analyze a local editor copy as if it were the deployed response.
- For each crawler token being tested, identify the matching group and record why it is more specific. Check relevant repeated groups rather than relying on their visual order.
- For each representative URL, list the matching path rules and their lengths. Include an allowed path, a blocked path and a conflict case.
- After editing, fetch the deployed file again and repeat the same table. Keep private endpoints behind authentication independently of robots behavior.
How to check the result
The test record should show both the chosen group and the decisive path rule for every URL. Reordering groups alone should not be accepted as evidence that the intended policy changed.
Limits of this conclusion
These are Google’s documented interpretation rules. They do not establish how an unidentified crawler behaves, and robots.txt is not a substitute for authorization.
This is a saved, dated edition. Source documentation can change. This edition retains the original retrieval dates and claim mapping; follow the sources before applying version-specific guidance.
Sources and publication checks
Automated checks validate source availability, supported fields, duplicate wording, numeric figures and page configuration. This guide was prepared with AI assistance and checked against saved source passages in the same authoring session. The batch importer checks source fingerprints, quotation locations, wording overlap and required fields. No paid generation API or separate model-review pass was used for this batch. These checks can miss errors and are not expert or human review.
Obtained: 2026-11-02T11:46:07.171405+00:00
Location: Examples / b100; Order of precedence for rules / b132; Order of precedence for user agents / b89; Order of precedence for user agents / b90
Saved source fingerprint
3db2f2c94597a0afbfde20b320a65102dac3c30f5fa3b4562a373a8e6e00fe18