Sites often verify a crawler IP once, cache the result, and reuse that decision long after DNS, published ranges, network ownership, or provider guidance changes. A stale positive can grant abusive traffic a trusted class. A stale negative can block a real crawler and reduce search visibility. User-agent text and robots.txt do not authenticate a request.
Use this for WordPress security, CDN, WAF, AI search, hosting, and log-analysis teams that classify search or AI crawlers.
Quick answer
Create confidence classes for verified, self-declared, unresolved, failed-refresh, and spoofed traffic. For each supported provider, document the authoritative verification method, published network evidence, reverse DNS, forward confirmation, verification time, TTL, cache key, refresh window, failure policy, rate limit, and owner. Refresh before expiry with jitter. On refresh failure, degrade confidence and apply a safe rate limit rather than extending a positive forever or blocking every request blindly.
Test scenarios to run
Run the same controlled fixture across these branches. Write down the expected result before testing so a surprising response is easy to identify.
| Scenario | Fixture | Expected result |
|---|
| Verified fresh | DNS and published evidence agree | Trusted class until bounded expiry |
| Positive expired | Refresh succeeds with changed address | Cache updates before decision |
| Refresh failure | DNS timeout or provider unavailable | Confidence degrades and safe rate limit applies |
| Spoofed user agent | No matching network evidence | Never treated as verified |
Diagnostic table
Use this table to connect the observed behavior to evidence and a verification step.
| Action | Evidence to collect | How to verify |
|---|
| Define provider-specific authoritative verification methods and cache keys. | Inventory crawler user agents, providers, published verification methods, network ranges, DNS rules, WAF exceptions, and owners. | Fresh verified fixtures satisfy every required provider check. |
| Apply bounded positive and negative TTLs with refresh jitter. | Record source IP, user agent, reverse result, forward addresses, range match, autonomous-system evidence, verified time, TTL, and cache key. | Expired positive and negative entries refresh according to policy. |
| Degrade confidence safely when refresh evidence is missing or conflicting. | Test positive expiry, negative expiry, DNS timeout, changed forward result, range removal, private address, IPv6, and spoofed user-agent fixtures. | Failures and conflicts degrade confidence without permanent allow or blind block. |
| Separate verified, self-declared, unresolved, and spoofed rate-limit policies. | Define behavior for stale-while-refresh, refresh failure, unresolved identity, and provider evidence that conflicts with DNS. | Logs expose verification age, method, reason, class, and owner without leaking secrets. |
What to check first
- Inventory crawler user agents, providers, published verification methods, network ranges, DNS rules, WAF exceptions, and owners.
- Record source IP, user agent, reverse result, forward addresses, range match, autonomous-system evidence, verified time, TTL, and cache key.
- Test positive expiry, negative expiry, DNS timeout, changed forward result, range removal, private address, IPv6, and spoofed user-agent fixtures.
- Define behavior for stale-while-refresh, refresh failure, unresolved identity, and provider evidence that conflicts with DNS.
- Compare access logs, rate limits, challenge decisions, robots policy, page fetches, and recrawl outcomes by confidence class.
Field notes
- Do not publish raw internal WAF secrets or complete abuse signatures.
- Keep DNS evidence and decision timestamps in UTC.
- Use robots.txt for crawl preference and a separate mechanism for identity and access control.
Useful command or data shape
Adapt paths, IDs, and privacy handling to the site before running commands or storing data on production.
provider,source,method,verified_at,ttl,refresh,result,confidence,action
Google,192.0.2.10,rdns+fdns,14:00Z,1h,pass,match,verified,allow
AI-Bot,192.0.2.44,published-range,13:00Z,1h,timeout,unknown,unresolved,rate-limit
Spoof,198.51.100.7,user-agent,none,0,fail,no-proof,spoofed,limit
Why this usually happens
- Verification data has a lifetime even when application caches do not enforce one.
- Negative and positive results require different refresh treatment.
- Provider methods differ, so one universal hostname suffix is unsafe.
- Operational teams fear blocking search and let stale positives persist indefinitely.
Decision rule
Treat a request as verified only while current provider-specific evidence agrees. An expired or failed-refresh result must move to an explicit lower-confidence policy until verification succeeds.
Production verification checklist
- Fresh verified fixtures satisfy every required provider check.
- Expired positive and negative entries refresh according to policy.
- Failures and conflicts degrade confidence without permanent allow or blind block.
- Logs expose verification age, method, reason, class, and owner without leaking secrets.
Safe fix order
Use a sequence that makes each result easy to prove. Stop when new evidence changes the scope or owner of the problem.
- Define provider-specific authoritative verification methods and cache keys.
- Apply bounded positive and negative TTLs with refresh jitter.
- Degrade confidence safely when refresh evidence is missing or conflicting.
- Separate verified, self-declared, unresolved, and spoofed rate-limit policies.
- Review false positive, false negative, and recrawl cohorts after policy changes.
Mistakes to avoid
- Changing production before preserving the current result, exact versions, timestamps, and a reproducible fixture.
- Treating one successful screen or request as proof that every queue, provider, report, browser, and customer path agrees.
- Removing logs, identifiers, or rollback evidence before the failure boundary and accountable owner are known.
- Testing only an administrator session instead of the roles, devices, consent states, networks, and failure paths users actually have.
Questions teams ask during testing
Can this be tested on production?
Use production for read-only confirmation and one narrow synthetic fixture. Perform destructive changes, upgrades, queue repairs, cache-policy changes, and schema changes on staging first. Promote only the smallest change that has a measured rollback point.
What evidence should be kept?
Keep component versions, stable fixture IDs, UTC timestamps, request or export evidence, expected and actual outcomes, the decision owner, rollback point, and final clean verification. Remove or redact personal data before sharing.
When is the work finished?
Finish when the canonical user path passes, downstream records reconcile, failure cases are understood, monitoring is active, and an established page links to the new guide with useful context.
What to tell the client or owner
Give the owner the affected versions, exact fixture, stable IDs, UTC timeline, before and after evidence, decision, rollback point, unresolved risks, and next review date. State which measurements prove success and which observation window still remains.
When HandL WP should help
Bring in help when this affects leads, checkout, search visibility, security, paid media reporting, or a client production site. HandL WP can trace the issue through WordPress, hosting, cache, tracking, and Search Console, then verify the workflow after the technical fix.
If this is active on a production site, audit AI crawler access.
Related HandL WP guides
Use these related guides when the same issue touches tracking, security, checkout, or crawler visibility.
Verify IPv4 and IPv6 independently
Extend cache-expiry testing with the AI crawler dual-stack IPv6 TTL test, using normalized addresses, provider evidence, family-specific cache keys, refresh failures, and explicit WAF confidence.
Helpful references