Any client can claim to be Googlebot, OAI-SearchBot, GPTBot, or another crawler. Recrawl reports that trust user agent strings can treat scanners, SEO tools, uptime checks, and malicious traffic as verified discovery. That distorts source-recrawl lag, target-fetch cohorts, cache analysis, and claims about AI visibility.
Use this for WordPress publishers, CDN operators, security teams, AI-search programs, agencies, and anyone joining crawler logs to Search Console or internal-link publishing waves.
Quick answer
Preserve raw edge and origin evidence, then classify each request by documented provider verification. For Google crawlers, use reverse DNS followed by forward DNS or published IP ranges. For providers without a verifiable network method, label the event self-declared rather than verified. Record source, timestamp, IP or protected token, user agent, method, path, status, bytes, cache result, verification method, verification time, and confidence. Exclude spoofed and unresolved traffic from verified cohort metrics without deleting the raw security evidence.
Test scenarios to run
Run the same controlled fixture across these branches. Write down the expected result before testing so a surprising response is easy to identify.
| Scenario | Fixture | Expected result |
|---|
| Verified Googlebot | Reverse then forward DNS agrees | Included in verified Google cohort |
| Spoofed OAI name | Consumer IP and no provider proof | Excluded from verified AI cohort |
| Proxy address | Trusted edge preserves source field | Verification uses approved source evidence |
| Unresolved provider | No supported verification method | Labeled self-declared, not verified |
Diagnostic table
Use this table to connect the observed behavior to evidence and a verification step.
| Action | Evidence to collect | How to verify |
|---|
| Define provider-specific verification and confidence labels. | Inventory every crawler label, provider documentation, verification method, resolver, IP list refresh, and log field used by reports. | Verified cohorts contain only requests that passed documented checks. |
| Preserve trusted source identity through CDN and origin logs. | Create verified, spoofed, unresolved, proxy, stale-IP-list, IPv6, retry, HEAD, range, and rate-limited fixtures. | Spoofed, self-declared, and unresolved requests remain visible but separate. |
| Classify and exclude spoofed or unresolved requests from verified metrics. | Run reverse and forward verification or provider-published range checks with cached results and expiry. | CDN and origin source fields reconcile without trusting arbitrary headers. |
| Join only verified cohorts to owner recrawls and target fetches. | Join verified source recrawl, target fetch, status, canonical, sitemap, internal link, Search Console discovery, impressions, and clicks by publication wave. | Recrawl reports state sample size, method, freshness, exclusions, and limits. |
What to check first
- Inventory every crawler label, provider documentation, verification method, resolver, IP list refresh, and log field used by reports.
- Create verified, spoofed, unresolved, proxy, stale-IP-list, IPv6, retry, HEAD, range, and rate-limited fixtures.
- Run reverse and forward verification or provider-published range checks with cached results and expiry.
- Join verified source recrawl, target fetch, status, canonical, sitemap, internal link, Search Console discovery, impressions, and clicks by publication wave.
- Report verified, self-declared, spoofed, unresolved, and excluded counts with sample size and freshness caveats.
Field notes
- Keep raw security evidence even when the analytics cohort excludes it.
- Cache verification results briefly and refresh published ranges safely.
- Do not claim an AI citation from a fetch log alone.
Useful command or data shape
Adapt paths, IDs, and privacy handling to the site before running commands or storing data on production.
time,claimed_bot,path,status,edge,source_class,verification,confidence,cohort
14:02,Googlebot,/owner,200,hit,provider,dns_both,verified,include
14:07,OAI-SearchBot,/target,200,miss,consumer,none,spoofed,exclude
14:09,GPTBot,/target,429,edge,unknown,unsupported,self_declared,separate
14:12,Googlebot,/target,200,hit,provider,ip_range,verified,include
Why this usually happens
- User agent strings are client-controlled text.
- CDNs and proxies can hide or rewrite the source address used for verification.
- Provider verification capabilities differ, so one confidence label cannot fit every crawler.
Decision rule
Call a request verified only when it passes the provider's documented identity method using trusted source evidence. Keep unsupported identities separate and never convert a user agent claim into proof.
Production verification checklist
- Verified cohorts contain only requests that passed documented checks.
- Spoofed, self-declared, and unresolved requests remain visible but separate.
- CDN and origin source fields reconcile without trusting arbitrary headers.
- Recrawl reports state sample size, method, freshness, exclusions, and limits.
Safe fix order
Use a sequence that makes each result easy to prove. Stop when new evidence changes the scope or owner of the problem.
- Define provider-specific verification and confidence labels.
- Preserve trusted source identity through CDN and origin logs.
- Classify and exclude spoofed or unresolved requests from verified metrics.
- Join only verified cohorts to owner recrawls and target fetches.
- Publish sample size, freshness, exclusions, and causation limits with every report.
Mistakes to avoid
- Changing production before preserving a reproducible fixture, timestamps, and the current result.
- Treating one successful screen, request, or export as proof that every downstream system agrees.
- Removing logs, identifiers, or rollback evidence before the owner and failure boundary are known.
- Testing only an administrator session instead of the roles, devices, consent states, and failure paths users actually have.
Questions teams ask during testing
Can this be tested on production?
Use production for read-only confirmation and a narrow synthetic fixture. Perform destructive, version, cache-policy, queue, or schema changes on staging first, then promote the smallest proven change.
What evidence should be kept?
Keep versions, fixture IDs, UTC timestamps, request or export evidence, expected and actual results, the decision owner, rollback point, and the final clean verification. Redact personal data.
When is the work finished?
Finish when the canonical user path passes, downstream records reconcile, failure cases are understood, monitoring is in place, and an established page links to the new guide with useful context.
What to tell the client or owner
Give the owner the affected versions, exact fixture, stable IDs, UTC timeline, before and after evidence, decision, rollback point, unresolved risks, and next review date.
When HandL WP should help
Bring in help when this affects leads, checkout, search visibility, security, paid media reporting, or a client production site. HandL WP can trace the issue through WordPress, hosting, cache, tracking, and Search Console, then verify the workflow after the technical fix.
If this is active on a production site, audit AI crawler evidence.
Related HandL WP guides
Use these related guides when the same issue touches tracking, security, checkout, or crawler visibility.
Expire crawler verification evidence
Keep identity classes current with the AI crawler verification cache-expiry test, using provider-specific methods, bounded TTLs, refresh failures, confidence degradation, and safe rate limits.
Helpful references