Crawler access and indexing · FIELD GUIDE
An allowed AI crawler is blocked: inspect WAF rule order
Trace the enforcing rule when Cloudflare AI Crawl Control settings and observed crawler responses appear inconsistent.
What the evidence can establish
Cloudflare states that an AI Crawl Control allow choice does not remove the effect of other security rules that execute earlier. It also documents ways earlier skip, redirect or transform rules can affect a crawler block. This makes rule order and request evidence central to the investigation. Changing the crawler setting again without identifying the enforcing rule can obscure the actual cause.
Hypothetical example: an intended crawler is allowed in the crawler panel but a broader country rule blocks the same request earlier. The useful evidence is the matching security event and its rule, not a screenshot of the allow setting alone. Conversely, an earlier skip rule may affect whether a later block runs. These examples describe documented interactions, not the current rules on your zone.
The saved evidence used here
This guide interprets dated official documentation. It does not report traffic from a customer or a measured AI answer. Each material claim is mapped to a source below.
- c1 · Explanation and verification steps
- Upstream WAF rules can still block crawlers marked allowed in AI Crawl Control. Source 1
- c2 · Explanation and verification steps
- Those rules may affect traffic without appearing in AI Crawl Control analytics. Source 1
- c3 · Explanation and verification steps
- Earlier rules can bypass or affect the crawler block. Source 1
- c4 · Explanation and verification steps
- Skip rules that bypass WAF custom rules are one documented interaction. Source 1
The diagram explains the investigation or a clearly labeled hypothetical example. No measured counts or platform outcomes are added.
AI answer visibility: Not measured. No original platform answers or verified citation records were collected for this guide. A request count is not an audience, ranking or citation count.
What to do next
- Preserve one affected request with its time, path, response and available security-event reference. Verify the request classification using the evidence your platform actually provides.
- Inspect the matching WAF rule and its position relative to the crawler-control rule. Include relevant skip, redirect and transform behavior in the investigation.
- Have the maintainer make the narrowest rule change consistent with the intended access policy. Do not disable the WAF or broadly trust a user-agent string to make a test pass.
- Retest the intended crawler condition and a negative control that should remain blocked. Keep the resulting rule evidence with the response observations.
How to check the result
The enforcing rule explains the original result, and the approved change produces the intended response while the negative control preserves the relevant protection.
Limits of this conclusion
An allow setting, a recognizable user-agent or one successful request does not demonstrate AI recommendation or citation. Available event detail and plan capabilities vary.
This is a saved, dated edition. Source documentation can change. This edition retains the original retrieval dates and claim mapping; follow the sources before applying version-specific guidance.
Sources and publication checks
Automated checks validate source availability, supported fields, duplicate wording, numeric figures and page configuration. This guide was prepared with AI assistance and checked against saved source passages in the same authoring session. The batch importer checks source fingerprints, quotation locations, wording overlap and required fields. No paid generation API or separate model-review pass was used for this batch. These checks can miss errors and are not expert or human review.
Obtained: 2026-11-02T13:15:24.253901+00:00
Location: Troubleshoot allowed bots / b16; Troubleshoot allowed bots / b17; Troubleshoot blocked bots / b18; Troubleshoot blocked bots / b19
Saved source fingerprint
1724eb978a80e84f952d5af3d2a30197358431449cd92da714404deb94cccf04