Crawl access and indexing · FIELD GUIDE
A blocked page with noindex: check what Google can read
Separate crawl access from noindex processing before repairing a public page or removing a URL from search.
What the evidence can establish
A noindex instruction and a crawl prohibition solve different problems. Google needs to retrieve the page to see a robots meta tag. Blocking that retrieval can prevent the instruction from being processed. For a public acquisition page that should be searchable, inspect both the crawl policy and the actual HTML or response header; fixing only one layer can leave the other conflict in place.
Hypothetical incident: /product has an unintended noindex tag and a robots disallow rule. Removing the tag from a template does not demonstrate what Google previously fetched or what the deployed route now returns. Save the anonymous response and compare it with the available URL Inspection evidence, labeling any missing Google-side record as not obtained.
The saved evidence used here
This guide interprets dated official documentation. It does not report traffic from a customer or a measured AI answer. Each material claim is mapped to a source below.
- c1 · Explanation and verification steps
- Google processes noindex through a robots meta tag or an HTTP response header when crawling. Source 1
- c2 · Explanation and verification steps
- Google does not support noindex in robots.txt. Source 1
- c3 · Explanation and verification steps
- A robots.txt block can prevent Google from seeing a page’s noindex tag. Source 1
- c4 · Explanation and verification steps
- URL Inspection and indexing reports can help investigate the received instruction. Source 1
The diagram explains the investigation or a clearly labeled hypothetical example. No measured counts or platform outcomes are added.
AI answer visibility: Not measured. No original platform answers or verified citation records were collected for this guide. A request count is not an audience, ranking or citation count.
What to do next
- State the intended outcome for the exact URL: public and eligible for indexing, public but excluded, or private behind access control.
- Retrieve the anonymous page and record its status, robots meta tags and X-Robots-Tag response header. Inspect the applicable robots rule separately.
- For an eligible public page, remove unintended exclusions only from that route or template scope. For a page excluded through noindex, ensure the chosen approach permits Google to read the instruction.
- Use URL Inspection to examine the fetched HTML when access is available, then monitor the relevant indexing report. Keep retrieval, index eligibility and actual indexing as separate results.
How to check the result
The result should include the deployed response, applicable crawl rule and intended indexing policy. Do not mark “indexed” solely because the page returns 200 without noindex.
Limits of this conclusion
Noindex does not protect confidential information. This guide does not establish Google’s current indexed state for any particular website.
This is a saved, dated edition. Source documentation can change. This edition retains the original retrieval dates and claim mapping; follow the sources before applying version-specific guidance.
Sources and publication checks
Automated checks validate source availability, supported fields, duplicate wording, numeric figures and page configuration. This guide was prepared with AI assistance and checked against saved source passages in the same authoring session. The batch importer checks source fingerprints, quotation locations, wording overlap and required fields. No paid generation API or separate model-review pass was used for this batch. These checks can miss errors and are not expert or human review.
Obtained: 2026-11-03T04:30:54.149624+00:00
Location: Block Search indexing with noindex / b1; Debugging noindex issues / b16; Debugging noindex issues / b17; Implementing noindex / b3
Saved source fingerprint
b4f6481216462fc08c61ec534e89fbc319f013c2f665ccfbf30128f00a27ee15