OpenAI documents different crawlers and user agents, including OAI-SearchBot and GPTBot. A WordPress site can write a clear robots.txt policy but still block the crawler through WAF, CDN bot rules, rate limits, cache rules, or private-path protections.
Use this when a WordPress site wants public content eligible for AI search visibility while keeping training preferences, private content, staging paths, and admin areas controlled.
Quick answer
OAI-SearchBot robots.txt Fetch Test for WordPress should be handled with a narrow evidence-first workflow: write policy, fetch test, check waf logs, then verify the result before making broader changes.
What to check first
- Review robots.txt rules for OAI-SearchBot, GPTBot, ChatGPT-User, Googlebot, and the sitemap URL.
- Fetch robots.txt, sitemap, homepage, and two public blog posts with the intended user agent.
- Check WAF, CDN, hosting, and security plugin logs for 403, 429, captcha, and managed challenge responses.
- Confirm private paths are protected by authentication or noindex controls, not only robots.txt.
- Document which bots are allowed for search visibility and which are blocked for training preference.
Diagnostic table
Use this table to keep the work practical. It connects the symptom to evidence and a verification step.
| Action | Evidence to collect | How to verify |
|---|
| Write policy | Review robots.txt rules for OAI-SearchBot, GPTBot, ChatGPT-User, Googlebot, and the sitemap URL. | OAI-SearchBot can fetch robots.txt, sitemap, and intended public pages. |
| Fetch test | Fetch robots.txt, sitemap, homepage, and two public blog posts with the intended user agent. | GPTBot and other crawlers match the written training or search policy. |
| Check WAF logs | Check WAF, CDN, hosting, and security plugin logs for 403, 429, captcha, and managed challenge responses. | Private, client, staging, and admin paths are protected beyond robots.txt. |
| Protect private paths | Confirm private paths are protected by authentication or noindex controls, not only robots.txt. | WAF logs show intentional allow or block reasons for each crawler class. |
Why this usually happens
- Robots.txt is a policy file, not a security boundary for private content.
- A WAF can block a crawler before the crawler reads the robots.txt file.
- Different AI crawlers can serve different product purposes, so one blanket rule may not match the business goal.
- 403 or 404 rate limiting can create crawler and indexing problems for legitimate public pages.
Useful command or data shape
Adapt paths, IDs, and privacy handling to the site before running commands or storing data on production.
curl -A 'OAI-SearchBot' -sSI https://example.com/robots.txt
curl -A 'OAI-SearchBot' -sSI https://example.com/sitemap.xml
curl -A 'OAI-SearchBot' -sSI https://example.com/blog/public-guide
# Review CDN and WAF logs for 403, 429, challenge, or bot score blocks.
Safe fix order
Do the work in a sequence that makes each result easy to prove. Stop if a step produces new evidence that changes the incident scope.
- Write policy
- Fetch test
- Check WAF logs
- Protect private paths
- Retest sitemap
What to tell the client or owner
Give the owner a crawler policy table with user agent, business intent, robots rule, WAF rule, and last fetch result.
Production verification checklist
- OAI-SearchBot can fetch robots.txt, sitemap, and intended public pages.
- GPTBot and other crawlers match the written training or search policy.
- Private, client, staging, and admin paths are protected beyond robots.txt.
- WAF logs show intentional allow or block reasons for each crawler class.
Mistakes to avoid
- Do not judge the fix by one browser or the homepage only.
- Do not delete evidence before recording usernames, file paths, timestamps, and response headers.
- Do not add a cache, security, or tracking plugin while the original problem is still unclear.
- Do not leave test users, temporary debug logs, or broad API keys active after verification.
When HandL WP should help
Bring in help when this affects leads, checkout, search visibility, malware risk, paid media reporting, or a client production site. HandL WP can trace the issue through WordPress, hosting, cache, tracking, and Search Console, then verify the workflow after the technical fix.
If this is active on a production site, review AI crawler access for WordPress.
Related HandL WP guides
Use these related guides when the same issue touches tracking, security, checkout, or crawler visibility.
Helpful references