A WordPress site may want to appear in AI search answers while declining model-training crawler access. That requires a policy that treats OAI-SearchBot, GPTBot, ChatGPT-User, Googlebot, Bingbot, and private areas as separate cases.
Use this for public service websites, documentation sites, ecommerce stores, and publishers that want AI search visibility but need a clear stance on training crawlers and private content.
Quick answer
OAI-SearchBot Allowed But GPTBot Blocked WordPress Policy should be handled with a narrow evidence-first workflow: inventory crawlers, set business policy, align waf rules, then verify the result before making broader changes.
What to check first
- List current robots.txt directives for OAI-SearchBot, GPTBot, ChatGPT-User, Googlebot, Bingbot, and PerplexityBot.
- Decide which public pages should be discoverable in search and which paths should stay blocked or noindexed.
- Align WAF and CDN bot rules so allowed crawlers are not challenged before they can read robots.txt.
- Protect private, staging, client, cart, checkout, and account pages with authentication or noindex, not robots.txt alone.
- Fetch robots.txt and sample public pages using crawler user agents to prove the policy works.
Diagnostic table
Use this table to keep the work practical. It connects the symptom to evidence and a verification step.
| Action | Evidence to collect | How to verify |
|---|
| Inventory crawlers | List current robots.txt directives for OAI-SearchBot, GPTBot, ChatGPT-User, Googlebot, Bingbot, and PerplexityBot. | Allowed crawlers can fetch public pages, CSS, images, and the sitemap. |
| Set business policy | Decide which public pages should be discoverable in search and which paths should stay blocked or noindexed. | Blocked crawlers receive the intended robots directive and no accidental public training access. |
| Align WAF rules | Align WAF and CDN bot rules so allowed crawlers are not challenged before they can read robots.txt. | Private paths are protected by authentication, noindex, or application rules where appropriate. |
| Protect private paths | Protect private, staging, client, cart, checkout, and account pages with authentication or noindex, not robots.txt alone. | The policy owner, review date, and reason for each crawler decision are documented. |
Why this usually happens
- Teams block every AI-related user agent and accidentally remove themselves from useful answer engines.
- The robots file says allow, but the WAF returns 403 or a managed challenge first.
- Private pages depend on robots.txt even though robots directives are not access control.
- The site has no record of why each crawler was allowed or blocked.
Field notes
- Keep the policy boring and explicit. The goal is a durable access rule, not a clever robots.txt trick.
- A split policy should be reviewed when public documentation, pricing, lead funnels, or private portals change.
Useful command or data shape
Adapt paths, IDs, and privacy handling to the site before running commands or storing data on production.
User-agent: OAI-SearchBot
Allow: /
User-agent: GPTBot
Disallow: /
User-agent: ChatGPT-User
Allow: /
User-agent: Googlebot
Allow: /
Disallow: /wp-admin/
Disallow: /cart/
Disallow: /checkout/
Sitemap: https://example.com/sitemap.xml
Safe fix order
Do the work in a sequence that makes each result easy to prove. Stop if a step produces new evidence that changes the incident scope.
- Inventory crawlers
- Set business policy
- Align WAF rules
- Protect private paths
- Retest fetches
What to tell the client or owner
Tell the owner which crawler is allowed for search visibility, which is blocked for training preference, and which private paths are protected outside robots.txt.
Production verification checklist
- Allowed crawlers can fetch public pages, CSS, images, and the sitemap.
- Blocked crawlers receive the intended robots directive and no accidental public training access.
- Private paths are protected by authentication, noindex, or application rules where appropriate.
- The policy owner, review date, and reason for each crawler decision are documented.
Mistakes to avoid
- Do not judge the fix by one browser or the homepage only.
- Do not delete evidence before recording usernames, file paths, timestamps, and response headers.
- Do not add a cache, security, or tracking plugin while the original problem is still unclear.
- Do not leave test users, temporary debug logs, or broad API keys active after verification.
When HandL WP should help
Bring in help when this affects leads, checkout, search visibility, malware risk, paid media reporting, or a client production site. HandL WP can trace the issue through WordPress, hosting, cache, tracking, and Search Console, then verify the workflow after the technical fix.
If this is active on a production site, set up a WordPress AI search crawler policy.
Related HandL WP guides
Use these related guides when the same issue touches tracking, security, checkout, or crawler visibility.
Helpful references
Turn crawler policy into fetch evidence
After writing the policy, use the OAI-SearchBot sitemap fetch status matrix to test public and protected paths. Pair it with the AI crawler user-agent verification checklist before creating WAF exceptions.