OpenAI documents OAI-SearchBot separately from GPTBot and ChatGPT-User. A WordPress or headless site should verify that its intended public paths return crawlable responses while private, admin, and staging paths remain protected.
Use this when a site wants eligible public content available to AI search but needs evidence from CDN, WAF, robots, sitemap, page, and image responses.
Quick answer
OAI-SearchBot Sitemap Fetch Status Code Matrix should be handled with a narrow evidence-first workflow: define expected policy, fetch all paths, compare edge and origin, then verify the result before making broader changes.
What to check first
- Record expected policy for robots.txt, sitemap, homepage, article, canonical redirect, image, admin, and private paths.
- Fetch each path with the intended user agent and capture status, location, content type, and cache headers.
- Compare CDN and origin results to find app-shell fallbacks, managed challenges, and stale redirects.
- Check access logs for 403, 404, 429, timeout, and challenge outcomes.
- Retest after any robots, WAF, cache, or redirect change and date the matrix.
Diagnostic table
Use this table to keep the work practical. It connects the symptom to evidence and a verification step.
| Action | Evidence to collect | How to verify |
|---|
| Define expected policy | Record expected policy for robots.txt, sitemap, homepage, article, canonical redirect, image, admin, and private paths. | Robots.txt, sitemap, canonical articles, and intended images return expected public responses. |
| Fetch all paths | Fetch each path with the intended user agent and capture status, location, content type, and cache headers. | Redirects end at one clean canonical URL without loops or app-shell fallbacks. |
| Compare edge and origin | Compare CDN and origin results to find app-shell fallbacks, managed challenges, and stale redirects. | Admin, staging, and private content remain protected by real access controls. |
| Review logs | Check access logs for 403, 404, 429, timeout, and challenge outcomes. | The dated matrix matches WAF and origin log evidence. |
Why this usually happens
- A crawler can fetch robots.txt but still be blocked from the sitemap or article.
- CDN custom errors can return a 200 app shell for a missing path.
- Blanket bot rules can override a more precise robots policy.
- Images or scripts may be blocked even when HTML returns 200.
Useful command or data shape
Adapt paths, IDs, and privacy handling to the site before running commands or storing data on production.
for url in robots.txt sitemap.xml blog/example-post images/blog/example.webp; do
curl -A 'OAI-SearchBot' -sSI "https://example.com/$url" | grep -Ei 'HTTP/|location:|content-type:|cache-control:'
done
# Test admin and private paths without printing credentials.
Safe fix order
Do the work in a sequence that makes each result easy to prove. Stop if a step produces new evidence that changes the incident scope.
- Define expected policy
- Fetch all paths
- Compare edge and origin
- Review logs
- Retest and date
What to tell the client or owner
Provide a path matrix with expected result, observed status, content type, redirect, cache state, WAF action, and last test time.
Production verification checklist
- Robots.txt, sitemap, canonical articles, and intended images return expected public responses.
- Redirects end at one clean canonical URL without loops or app-shell fallbacks.
- Admin, staging, and private content remain protected by real access controls.
- The dated matrix matches WAF and origin log evidence.
Mistakes to avoid
- Do not judge the fix by one browser or the homepage only.
- Do not delete evidence before recording usernames, file paths, timestamps, and response headers.
- Do not add a cache, security, or tracking plugin while the original problem is still unclear.
- Do not leave test users, temporary debug logs, or broad API keys active after verification.
When HandL WP should help
Bring in help when this affects leads, checkout, search visibility, malware risk, paid media reporting, or a client production site. HandL WP can trace the issue through WordPress, hosting, cache, tracking, and Search Console, then verify the workflow after the technical fix.
If this is active on a production site, verify WordPress access for AI search crawlers.
Related HandL WP guides
Use these related guides when the same issue touches tracking, security, checkout, or crawler visibility.
Helpful references
Compare crawler and browser cache objects
If status codes pass but crawler HTML still looks stale, use the OAI-SearchBot CDN cache variation test to compare headers, hashes, origin content, and bot rules.