A crawler provider may publish network ranges, use DNS verification, or document a user agent, but an implementation can still verify only IPv4 and reuse that result for IPv6. Address parsing, CIDR matching, DNS libraries, proxy headers, cache keys, and WAF policies can behave differently across address families. A user-agent string alone is not identity proof.
Use this for WordPress security, CDN, WAF, AI search, hosting, crawler analytics, and log-pipeline teams operating dual-stack infrastructure.
Quick answer
For each supported provider and crawler purpose, document the authoritative verification method. Build separate IPv4 and IPv6 positive, negative, boundary, malformed, mapped-address, proxy, DNS-timeout, range-change, and expired-cache fixtures. Cache by provider, crawler class, address family, normalized address, method version, and evidence version. Refresh before expiry. On failure, degrade confidence to a safe rate-limited class instead of inheriting another family's positive result.
Test scenarios to run
Run the same controlled fixture across these branches. Write down the expected result before testing so a surprising response is easy to identify.
| Scenario | Fixture | Expected result |
|---|
| Published IPv4 range | Boundary and interior addresses | Current provider evidence verifies |
| Published IPv6 range | Compressed and expanded forms | Same address normalizes and verifies |
| IPv4-mapped IPv6 | ::ffff representation | Explicit family and policy, no accidental trust |
| Expired IPv6 proof | Refresh times out | Confidence degrades without IPv4 inheritance |
Diagnostic table
Use this table to connect the observed behavior to evidence and a verification step.
| Action | Evidence to collect | How to verify |
|---|
| Define provider-specific methods and separate IPv4 and IPv6 fixture suites. | Inventory crawler classes, provider-published ranges, DNS rules, proxies, trusted forwarding headers, WAF policies, resolver libraries, and owners. | Positive and negative IPv4 and IPv6 boundary fixtures match the current provider evidence. |
| Normalize source addresses and trusted proxy chains before range or DNS checks. | Normalize IPv4, compressed IPv6, expanded IPv6, IPv4-mapped IPv6, zone IDs, ports, brackets, and proxy-chain inputs before matching. | Equivalent IPv6 forms normalize to one cache identity without accepting malformed input. |
| Cache verification by provider, class, family, address, method, and evidence version. | Test positive and negative CIDR boundaries, reverse and forward DNS where required, range removal, stale entries, timeout, conflicting evidence, and user-agent spoofing. | Expired, conflicting, and failed-refresh results move to the documented lower-confidence policy. |
| Use bounded TTLs and explicit confidence degradation on refresh failure. | Record provider, crawler purpose, source, family, normalized address, method, evidence version, verified time, TTL, refresh, confidence, and action. | Allowed crawlers can fetch public content while spoofed traffic remains controlled. |
What to check first
- Inventory crawler classes, provider-published ranges, DNS rules, proxies, trusted forwarding headers, WAF policies, resolver libraries, and owners.
- Normalize IPv4, compressed IPv6, expanded IPv6, IPv4-mapped IPv6, zone IDs, ports, brackets, and proxy-chain inputs before matching.
- Test positive and negative CIDR boundaries, reverse and forward DNS where required, range removal, stale entries, timeout, conflicting evidence, and user-agent spoofing.
- Record provider, crawler purpose, source, family, normalized address, method, evidence version, verified time, TTL, refresh, confidence, and action.
- Compare allow, rate-limit, challenge, block, robots policy, public fetch, sitemap fetch, and recrawl outcomes independently by family.
Field notes
- Trust forwarding headers only from known proxy hops.
- Keep robots preference separate from network identity and WAF access.
- Log normalized addresses and reason classes, but avoid publishing private security details.
Useful command or data shape
Adapt paths, IDs, and privacy handling to the site before running commands or storing data on production.
provider,crawler,family,source,method,evidence,ttl,refresh,confidence,action
OpenAI,OAI-SearchBot,ipv4,198.51.100.4,range,v12,1h,pass,verified,allow
OpenAI,OAI-SearchBot,ipv6,2001:db8::4,range,v12,1h,pass,verified,allow
Spoof,OAI-SearchBot,ipv6,2001:db8:ffff::9,user-agent,none,0,fail,spoofed,limit
Why this usually happens
- Libraries can compare IPv4 and IPv6 CIDRs through different code paths.
- A cache keyed only by user agent or hostname can reuse proof across addresses.
- Proxy headers can contain several address formats or an untrusted client value.
- Positive verification is often cached longer than the provider evidence version.
Decision rule
Treat a dual-stack request as verified only when current evidence for its exact normalized address family passes. Never inherit trust from another family, address, hostname, or user-agent-only result.
Production verification checklist
- Positive and negative IPv4 and IPv6 boundary fixtures match the current provider evidence.
- Equivalent IPv6 forms normalize to one cache identity without accepting malformed input.
- Expired, conflicting, and failed-refresh results move to the documented lower-confidence policy.
- Allowed crawlers can fetch public content while spoofed traffic remains controlled.
Safe fix order
Use a sequence that makes each result easy to prove. Stop when new evidence changes the scope or owner of the problem.
- Define provider-specific methods and separate IPv4 and IPv6 fixture suites.
- Normalize source addresses and trusted proxy chains before range or DNS checks.
- Cache verification by provider, class, family, address, method, and evidence version.
- Use bounded TTLs and explicit confidence degradation on refresh failure.
- Review false positive, false negative, crawl, and WAF cohorts after every provider update.
Mistakes to avoid
- Changing production before preserving exact versions, UTC timestamps, stable fixture IDs, current settings, and a reproducible baseline.
- Treating one successful browser screen as proof that queues, providers, caches, reports, roles, and downstream records agree.
- Deleting logs or identifiers before the failure boundary, business impact, rollback point, and accountable owner are known.
- Testing only an administrator session instead of the devices, roles, consent states, networks, and failure paths real users have.
Questions teams ask during testing
Can this be tested on production?
Use production for read-only confirmation and one narrow synthetic fixture that cannot charge a card, email a customer, expose personal data, or change inventory. Perform destructive repairs, upgrades, cache-policy changes, and schema work on staging first. Promote only the smallest measured change with a current rollback point.
What evidence should the report keep?
Keep exact component versions, UTC timestamps, stable synthetic IDs, expected and actual results, queue or provider identifiers, the decision owner, rollback point, and final verification. Redact customer data, credentials, tokens, message content, addresses, and private infrastructure details before sharing evidence.
When is the task complete?
Complete the task when the primary user path passes, downstream records reconcile, failure branches are understood, monitoring is active, and an established owner page links to the new guide in context. Record any observation window that remains instead of calling a quiet test a permanent fix.
What to tell the client or owner
Give the site owner the affected versions, exact synthetic fixture, UTC timeline, before and after evidence, root cause or current cause class, decision, rollback point, unresolved risks, and next review date. State which measurements prove success and which observation window is still open.
When HandL WP should help
Bring in help when this affects leads, checkout, search visibility, security, paid media reporting, or a client production site. HandL WP can trace the issue through WordPress, hosting, cache, tracking, and Search Console, then verify the workflow after the technical fix.
If this is active on a production site, audit AI crawler verification.
Related HandL WP guides
Use these related guides when the same issue touches tracking, security, checkout, or crawler visibility.
Detect provider range changes before policy promotion
Add the AI crawler provider range ETag change detector to validate metadata, normalize CIDRs, review additions and removals, replay dual-stack fixtures, and retain a last-good rollback version.
Helpful references