OpenAI documents different user agents for search, user-requested fetches, and other crawler behavior. A WordPress site can write a clear robots.txt policy and still block useful fetches if the CDN, WAF, hosting firewall, or security plugin challenges the request first.
Use this for WordPress sites that want ChatGPT Search visibility while keeping private, transactional, staging, and admin routes protected.
Quick answer
ChatGPT Search robots.txt and WordPress WAF Alignment should be handled with a narrow evidence-first workflow: decide user agents, write robots policy, read waf logs, then verify the result before making broader changes.
What to check first
- Decide which OpenAI user agents should access public pages and which should not.
- Confirm robots.txt reflects the business policy for public pages, private paths, and sitemap location.
- Check WAF and CDN logs for 403, 429, managed challenge, captcha, and blocked-country responses.
- Test public HTML, sitemap, CSS, JavaScript, and image paths with the intended crawler user agents.
- Protect private content with authentication and access controls, not robots.txt alone.
Diagnostic table
Use this table to keep the work practical. It connects the symptom to evidence and a verification step.
| Action | Evidence to collect | How to verify |
|---|---|---|
| Decide user agents | Decide which OpenAI user agents should access public pages and which should not. | Allowed search crawlers can fetch public HTML and sitemap URLs. |
| Write robots policy | Confirm robots.txt reflects the business policy for public pages, private paths, and sitemap location. | Blocked crawler categories match the written policy. |
| Read WAF logs | Check WAF and CDN logs for 403, 429, managed challenge, captcha, and blocked-country responses. | Private paths remain protected even when robots.txt is public. |
| Test public assets | Test public HTML, sitemap, CSS, JavaScript, and image paths with the intended crawler user agents. | WAF logs confirm intentional allows and blocks. |
Useful command or data shape
Adapt paths, IDs, and privacy handling to the site before running commands or storing data on production.
curl -A 'OAI-SearchBot' -sSI https://example.com/robots.txt
curl -A 'OAI-SearchBot' -sSI https://example.com/sitemap.xml
curl -A 'ChatGPT-User' -sSI https://example.com/blog/example-post
curl -A 'GPTBot' -sSI https://example.com/wp-login.php
Safe fix order
Do the work in a sequence that makes each result easy to prove. Stop if a step produces new evidence that changes the incident scope.
- Decide user agents
- Write robots policy
- Read WAF logs
- Test public assets
- Protect private paths
Production verification checklist
- Allowed search crawlers can fetch public HTML and sitemap URLs.
- Blocked crawler categories match the written policy.
- Private paths remain protected even when robots.txt is public.
- WAF logs confirm intentional allows and blocks.
Mistakes to avoid
- Do not judge the fix by one browser or the homepage only.
- Do not delete evidence before recording usernames, file paths, timestamps, and response headers.
- Do not add a cache, security, or tracking plugin while the original problem is still unclear.
- Do not leave test users, temporary debug logs, or broad API keys active after verification.
When HandL WP should help
Bring in help when this affects leads, checkout, search visibility, malware risk, paid media reporting, or a client production site. HandL WP can trace the issue through WordPress, hosting, cache, tracking, and Search Console, then verify the workflow after the technical fix.
If this is active on a production site, align WordPress AI crawler policy.
Related HandL WP guides
Use these related guides when the same issue touches tracking, security, checkout, or crawler visibility.