A WordPress site should not rely on robots.txt to protect private pages from ChatGPT-User or any other user-requested fetch. Private content needs authentication, noindex where appropriate, and WAF rules that match the business policy.
Use this for membership sites, client portals, staging sites, private docs, paid resources, and service pages that need different AI access rules.
Quick answer
OpenAI ChatGPT-User Private Page Access Policy should be handled with a narrow evidence-first workflow: classify paths, separate agents, protect private pages, then verify the result before making broader changes.
What to check first
- Classify URLs as public, noindex, authenticated, staging, paid, client-private, or blocked.
- Decide how to handle GPTBot, OAI-SearchBot, ChatGPT-User, Googlebot, PerplexityBot, and other agents separately.
- Protect private content with authentication or authorization, not only robots.txt.
- Review WAF logs for public pages that are accidentally blocked or private paths that are publicly reachable.
- Document who can approve changes to AI crawler and user-agent policy.
Diagnostic table
Use this table to keep the work practical. It connects the symptom to evidence and a verification step.
| Action | Evidence to collect | How to verify |
|---|---|---|
| Classify paths | Classify URLs as public, noindex, authenticated, staging, paid, client-private, or blocked. | Public pages return 200 and useful HTML to allowed search bots. |
| Separate agents | Decide how to handle GPTBot, OAI-SearchBot, ChatGPT-User, Googlebot, PerplexityBot, and other agents separately. | Private and staging pages require authentication before content is served. |
| Protect private pages | Protect private content with authentication or authorization, not only robots.txt. | Robots.txt, WAF rules, and sitemap URLs match the written policy. |
| Review WAF logs | Review WAF logs for public pages that are accidentally blocked or private paths that are publicly reachable. | A log sample proves the policy works for representative user agents. |
Why this usually happens
- Teams often group all AI user agents together even though crawlers, search bots, and user-requested fetches do different work.
- Robots.txt is a crawl instruction, not an access-control system.
- A public app shell route can make a private path look public if authorization is handled only in browser JavaScript.
- Security and SEO teams may want different outcomes unless the policy is written down.
Field notes
- Test private URLs while logged out, logged in as a low-permission user, and with representative user agents.
- Keep staging subdomains out of public sitemaps and protect them with authentication.
- Allowing public service pages for AI search is different from exposing client-specific content.
Useful command or data shape
Adapt paths, IDs, and privacy handling to the site before running commands or storing data on production.
path_policy:
/services/: public, indexable, allow_search_bots
/blog/: public, indexable, allow_search_bots
/client-portal/: authenticated, no_public_fetch
/staging/: authenticated, noindex
/wp-admin/: authenticated, block_public
/private-downloads/: signed_url_or_login_required
Safe fix order
Do the work in a sequence that makes each result easy to prove. Stop if a step produces new evidence that changes the incident scope.
- Classify paths
- Separate agents
- Protect private pages
- Review WAF logs
- Document approvals
What to tell the client or owner
Give the owner a simple URL class table. The key message is that private content is protected by access control, while public content policy is an SEO and business decision.
Production verification checklist
- Public pages return 200 and useful HTML to allowed search bots.
- Private and staging pages require authentication before content is served.
- Robots.txt, WAF rules, and sitemap URLs match the written policy.
- A log sample proves the policy works for representative user agents.
Mistakes to avoid
- Do not judge the fix by one browser or the homepage only.
- Do not delete evidence before recording usernames, file paths, timestamps, and response headers.
- Do not add a cache, security, or tracking plugin while the original problem is still unclear.
- Do not leave test users, temporary debug logs, or broad API keys active after verification.
When HandL WP should help
Bring in help when this affects leads, checkout, search visibility, malware risk, paid media reporting, or a client production site. HandL WP can trace the issue through WordPress, hosting, cache, tracking, and Search Console, then verify the workflow after the technical fix.
If this is active on a production site, set a WordPress AI access policy.
Related HandL WP guides
Use these related guides when the same issue touches tracking, security, checkout, or crawler visibility.