WordPress AI crawler policy should separate public content from private paths. Robots.txt is not access control. Admin pages, account pages, staging URLs, checkout sessions, and client documents need authentication or server-side blocks.
Use this when a WordPress site wants AI search visibility while keeping private, customer, admin, staging, and paid content out of crawler access.
Quick answer
Perplexity-User WordPress Private Path Policy should be handled with a narrow evidence-first workflow: define public paths, define private paths, separate robots from auth, then verify the result before making broader changes.
What to check first
- List public pages that should be crawlable by search and answer engines.
- List private paths that require authentication, server-side blocks, or noindex rules.
- Check robots.txt separately from WAF and application access controls.
- Review access logs for PerplexityBot and Perplexity-User fetches.
- Do not use crawler user agents as the only protection for private content.
Diagnostic table
Use this table to keep the work practical. It connects the symptom to evidence and a verification step.
| Action | Evidence to collect | How to verify |
|---|---|---|
| Define public paths | List public pages that should be crawlable by search and answer engines. | Public blog and service pages return normal crawlable HTML. |
| Define private paths | List private paths that require authentication, server-side blocks, or noindex rules. | Private paths require authentication or return the intended protected status. |
| Separate robots from auth | Check robots.txt separately from WAF and application access controls. | Robots.txt matches the business policy without pretending to secure private data. |
| Review WAF logs | Review access logs for PerplexityBot and Perplexity-User fetches. | WAF logs show crawler decisions with enough evidence to audit. |
Useful command or data shape
Adapt paths, IDs, and privacy handling to the site before running commands or storing data on production.
private_path_policy:
public:
- /blog/
- /services/
private:
- /wp-admin/
- /my-account/
- /checkout/order-pay/
- /client-docs/
rule: private_paths_require_authentication
Safe fix order
Do the work in a sequence that makes each result easy to prove. Stop if a step produces new evidence that changes the incident scope.
- Define public paths
- Define private paths
- Separate robots from auth
- Review WAF logs
- Retest sample URLs
Production verification checklist
- Public blog and service pages return normal crawlable HTML.
- Private paths require authentication or return the intended protected status.
- Robots.txt matches the business policy without pretending to secure private data.
- WAF logs show crawler decisions with enough evidence to audit.
Mistakes to avoid
- Do not judge the fix by one browser or the homepage only.
- Do not delete evidence before recording usernames, file paths, timestamps, and response headers.
- Do not add a cache, security, or tracking plugin while the original problem is still unclear.
- Do not leave test users, temporary debug logs, or broad API keys active after verification.
When HandL WP should help
Bring in help when this affects leads, checkout, search visibility, malware risk, paid media reporting, or a client production site. HandL WP can trace the issue through WordPress, hosting, cache, tracking, and Search Console, then verify the workflow after the technical fix.
If this is active on a production site, set a WordPress AI crawler policy.
Related HandL WP guides
Use these related guides when the same issue touches tracking, security, checkout, or crawler visibility.