OAI-SearchBot, GPTBot, ChatGPT-User, PerplexityBot, Googlebot, and other crawlers are not the same thing. A WordPress robots.txt policy should separate search visibility, user-requested fetches, training preferences, and private content protection.
Use this when a WordPress site wants AI search visibility but also needs a clear policy for model-training crawlers, security tools, and WAF behavior.
Quick answer
OAI-SearchBot vs GPTBot should be handled with a narrow evidence-first workflow: inventory crawlers, decide policy, update robots.txt, then verify the result before making broader changes.
What to check first
- List which crawler user agents are currently allowed or blocked in robots.txt.
- Decide whether the site wants ChatGPT search visibility, training opt-out, or both.
- Check whether WAF bot rules block allowed crawlers before robots.txt is even read.
- Do not rely on robots.txt to protect private client, membership, staging, or paid content.
- Keep public service pages and blog posts crawlable when they are meant to generate leads.
Diagnostic table
Use this table to keep the work practical. It connects the symptom to evidence and a verification step.
| Action | Evidence to collect | How to verify |
|---|---|---|
| Inventory crawlers | List which crawler user agents are currently allowed or blocked in robots.txt. | The robots.txt policy names each crawler intentionally. |
| Decide policy | Decide whether the site wants ChatGPT search visibility, training opt-out, or both. | Allowed crawlers can fetch public HTML, CSS, JavaScript, and sitemap files. |
| Update robots.txt | Check whether WAF bot rules block allowed crawlers before robots.txt is even read. | Private content is protected by authentication or noindex controls where appropriate. |
| Check WAF rules | Do not rely on robots.txt to protect private client, membership, staging, or paid content. | The WAF access log does not show accidental blocks for allowed crawlers. |
Useful command or data shape
Adapt paths, IDs, and privacy handling to the site before running commands or storing data on production.
User-agent: OAI-SearchBot
Allow: /
User-agent: GPTBot
Disallow: /
User-agent: ChatGPT-User
Allow: /
User-agent: Googlebot
Allow: /
Sitemap: https://example.com/sitemap.xml
Safe fix order
Do the work in a sequence that makes each result easy to prove. Stop if a step produces new evidence that changes the incident scope.
- Inventory crawlers
- Decide policy
- Update robots.txt
- Check WAF rules
- Retest public pages
Production verification checklist
- The robots.txt policy names each crawler intentionally.
- Allowed crawlers can fetch public HTML, CSS, JavaScript, and sitemap files.
- Private content is protected by authentication or noindex controls where appropriate.
- The WAF access log does not show accidental blocks for allowed crawlers.
Mistakes to avoid
- Do not judge the fix by one browser or the homepage only.
- Do not delete evidence before recording usernames, file paths, timestamps, and response headers.
- Do not add a cache, security, or tracking plugin while the original problem is still unclear.
- Do not leave test users, temporary debug logs, or broad API keys active after verification.
When HandL WP should help
Bring in help when this affects leads, checkout, search visibility, malware risk, paid media reporting, or a client production site. HandL WP can trace the issue through WordPress, hosting, cache, tracking, and Search Console, then verify the workflow after the technical fix.
If this is active on a production site, set a WordPress AI crawler policy.