{
  "acceptance": "The handoff should identify the client category and its evidence, preserve unknown identity separately, and explain why the selected policy fits that category. A successful fetch is a separate observation from search indexing.",
  "automated_checks": true,
  "batch_id": "analytics-editorial-20261104-v1",
  "built_at": "2026-11-04T11:50:28.566595+00:00",
  "claims": [
    {
      "id": "c1",
      "section": "Explanation and verification steps",
      "source_numbers": [
        1
      ],
      "statement": "Common Google crawlers respect robots.txt during automatic crawls."
    },
    {
      "id": "c2",
      "section": "Explanation and verification steps",
      "source_numbers": [
        1
      ],
      "statement": "Special-case crawlers can operate under product-specific arrangements, including the documented AdsBot example."
    },
    {
      "id": "c3",
      "section": "Explanation and verification steps",
      "source_numbers": [
        1
      ],
      "statement": "Google Site Verifier is an example of a user-triggered fetcher."
    }
  ],
  "description": "Separate common crawlers, special-case crawlers and user-triggered fetchers before interpreting robots behavior.",
  "diagram": [
    "Identify the claimed client",
    "Assign its documented category",
    "Check product-specific policy",
    "Record response separately"
  ],
  "draft_hash": "36ac3c040dd72d206353e028e5a8f9023ce122790036ae44c3bcf8ba4748648f",
  "evidence": {
    "observed_at": "2026-11-02T12:01:04.822040+00:00",
    "period": {
      "end": "2026-11-02T12:01:04.822040+00:00",
      "start": "2026-11-02T12:01:04.822040+00:00",
      "timezone": "UTC"
    },
    "rows": [],
    "scope": "Official documentation; no customer or own-site traffic measurement"
  },
  "evidence_kind": "documentation",
  "figure_caption": "Original explanatory diagram based on the dated sources below. Any example values are hypothetical; this is not a screenshot, customer measurement or AI platform result.",
  "figure_kind": "sequence",
  "figure_notes": [
    "Keep claimed identity separate from verified identity.",
    "Distinguish common, special-case and user-triggered clients.",
    "Do not apply one category’s robots promise to all clients.",
    "Fetching and indexing are separate outcomes."
  ],
  "figure_title": "Google crawler classes: avoid one policy for every request",
  "group": "Crawl access and indexing",
  "human_review_claim": false,
  "intent": "google-crawler-class-policy",
  "interpretation": "The first decision is which documented client category the request belongs to. Google distinguishes automatic common crawlers, product-specific special-case crawlers and user-triggered fetchers. These categories are not interchangeable promises about robots behavior. A name in a user-agent string is a classification clue, not verification of the sender. Keep that uncertainty visible before changing an access rule.",
  "keyword_volume": "Not obtained",
  "limits": "This guide classifies policy expectations; it does not verify a real requester. Product-specific arrangements must be checked before applying the common-crawler rule to another category.",
  "paid_api_calls": 0,
  "path": "/ai-analytics/guides/google-crawler-class-policy/",
  "policy": "session-evidence-editorial-v1",
  "preparation_mode": "session_editorial_batch",
  "problem": "A request labeled Google appears to ignore the robots policy expected by the analyst.",
  "product": "/ai-analytics",
  "published_at": "2026-11-05T00:30:02.921474+00:00",
  "reader": "Site owners and analysts investigating a request labeled Google appears to ignore the robots policy expected by the analyst.",
  "related": [
    "crawler-user-agent-verification",
    "robots-directives-vs-access-enforcement"
  ],
  "related_links": [
    {
      "path": "/ai-analytics/guides/crawler-user-agent-verification/",
      "title": "Separate crawler names from verified request identity"
    }
  ],
  "research_document_ids": [
    "698720e2814f074dc55c0f99c8d9dbfd"
  ],
  "slug": "google-crawler-class-policy",
  "source_signature": "5868dc2c7fa0dd1df295b0ae129882a1330aa2b59667d88d5d21d123ea8ab609",
  "sources": [
    {
      "hash": "059e62e33d0840a7dfbbf0a7fa04eb77b20ae12649bf68b283d34110a6af7b26",
      "locator": "Overview of Google crawlers and fetchers (user agents) / b2; Overview of Google crawlers and fetchers (user agents) / b3; Overview of Google crawlers and fetchers (user agents) / b4",
      "observed_at": "2026-11-02T12:01:04.822040+00:00",
      "title": "Overview of Google crawlers and fetchers (user agents)",
      "url": "https://developers.google.com/crawling/docs/crawlers-fetchers/overview-google-crawlers"
    }
  ],
  "sources_required": [],
  "steps": [
    "Record the exact user agent, requested path, time and response from the existing access logs; remove identifiers unnecessary for the investigation.",
    "Match the documented client to its category. Common crawlers respect robots for automatic crawls; special-case clients can have product-specific arrangements.",
    "For an unexpected client, inspect the product-specific documentation and retain an unclassified state when identification is unresolved. Do not infer ownership from the name alone.",
    "Create a policy table with separate columns for client category, documented robots behavior, desired access and observed response. Change only the rule for the relevant purpose."
  ],
  "title": "Google crawler classes: avoid one policy for every request",
  "worked_example": "Hypothetical triage: an analyst puts a Googlebot request and a Google Site Verifier request in one “search crawling” total. Split them into automatic crawling and a user-triggered verification action. The two requests may have different purposes even when they reach the same URL; neither record establishes an indexing outcome."
}