OAI-SearchBot can be allowed in robots.txt yet receive stale HTML, an app shell, a managed challenge, or a different cache object because of CDN user-agent variation.
Use this for WordPress, S3, CloudFront, headless, or cached sites where AI crawler visibility depends on consistent public HTML.
Quick answer
OAI-SearchBot CDN Cache Variation Test should be handled with a narrow evidence-first workflow: fetch four agents, compare headers and hashes, check origin, then verify the result before making broader changes.
What to check first
- Fetch one canonical page as a normal browser, OAI-SearchBot, GPTBot, and a plain curl client.
- Capture status, redirect, content type, cache status, Age, ETag, Vary, title, canonical, and body hash.
- Compare the CDN response with the origin or prerendered object.
- Review bot, device, language, and challenge rules that can change the cache key or response.
- Purge only affected paths, warm the canonical page, and repeat the matrix.
Diagnostic table
Use this table to keep the work practical. It connects the symptom to evidence and a verification step.
| Action | Evidence to collect | How to verify |
|---|
| Fetch four agents | Fetch one canonical page as a normal browser, OAI-SearchBot, GPTBot, and a plain curl client. | Allowed crawlers receive the canonical page-specific HTML. |
| Compare headers and hashes | Capture status, redirect, content type, cache status, Age, ETag, Vary, title, canonical, and body hash. | Browser, allowed crawler, and origin hashes agree where policy expects the same content. |
| Check origin | Compare the CDN response with the origin or prerendered object. | Blocked crawlers receive the intentional policy response without poisoning public cache. |
| Fix cache or WAF rule | Review bot, device, language, and challenge rules that can change the cache key or response. | Sitemap, canonical, title, and article metadata remain consistent. |
Why this usually happens
- A Vary or custom cache policy can create separate bot objects.
- WAF challenges can cache an interstitial or fallback.
- A missing object can become a 200 app shell through custom errors.
- Origin and edge invalidations can complete at different times.
Useful command or data shape
Adapt paths, IDs, and privacy handling to the site before running commands or storing data on production.
for agent in 'Mozilla/5.0' 'OAI-SearchBot' 'GPTBot' 'curl/8'; do
curl -A "$agent" -sS -D /tmp/headers.txt https://example.com/blog/guide -o /tmp/body.html
printf '%s ' "$agent"; sha256sum /tmp/body.html; grep -Ei 'HTTP/|age:|etag:|vary:|x-cache:' /tmp/headers.txt
done
Safe fix order
Do the work in a sequence that makes each result easy to prove. Stop if a step produces new evidence that changes the incident scope.
- Fetch four agents
- Compare headers and hashes
- Check origin
- Fix cache or WAF rule
- Purge and retest
What to tell the client or owner
Share the agent matrix, status, cache headers, body hashes, rule owner, invalidated paths, and final comparison.
Production verification checklist
- Allowed crawlers receive the canonical page-specific HTML.
- Browser, allowed crawler, and origin hashes agree where policy expects the same content.
- Blocked crawlers receive the intentional policy response without poisoning public cache.
- Sitemap, canonical, title, and article metadata remain consistent.
Mistakes to avoid
- Do not judge the fix by one browser or the homepage only.
- Do not delete evidence before recording usernames, file paths, timestamps, and response headers.
- Do not add a cache, security, or tracking plugin while the original problem is still unclear.
- Do not leave test users, temporary debug logs, or broad API keys active after verification.
When HandL WP should help
Bring in help when this affects leads, checkout, search visibility, malware risk, paid media reporting, or a client production site. HandL WP can trace the issue through WordPress, hosting, cache, tracking, and Search Console, then verify the workflow after the technical fix.
If this is active on a production site, verify AI crawler access through your CDN.
Related HandL WP guides
Use these related guides when the same issue touches tracking, security, checkout, or crawler visibility.
Helpful references
Distinguish a real article from the app shell
When crawler and browser requests both return 200, use the AI crawler app-shell detection test to compare title, canonical, schema, answer markers, body hashes, origin objects, and missing-route behavior.