OpenAI and Perplexity document crawler behavior and IP data differently. A WordPress WAF policy that allows useful AI crawlers should avoid stale manual IP lists, user-agent-only trust, and silent production changes that nobody can roll back.
Use this for WordPress sites with Cloudflare, AWS WAF, Fastly, Akamai, Sucuri, Wordfence, hosting firewall rules, or custom edge code that controls crawler access.
Quick answer
AI Crawler IP JSON Automation for WordPress WAF Rules should be handled with a narrow evidence-first workflow: fetch official data, validate changes, match user agents, then verify the result before making broader changes.
What to check first
- Fetch official crawler documentation and IP JSON sources from the vendor before updating rules.
- Validate JSON shape and compare added or removed ranges before applying a production WAF change.
- Match IP rules with user-agent and path conditions instead of trusting a single signal.
- Use rate limits for public crawl paths and stronger blocks for admin, account, checkout, and staging paths.
- Keep a rollback copy of the previous rule set and monitor 403, 429, and challenge responses after deployment.
Diagnostic table
Use this table to keep the work practical. It connects the symptom to evidence and a verification step.
| Action | Evidence to collect | How to verify |
|---|---|---|
| Fetch official data | Fetch official crawler documentation and IP JSON sources from the vendor before updating rules. | Allowed crawlers can fetch public pages and sitemap URLs. |
| Validate changes | Validate JSON shape and compare added or removed ranges before applying a production WAF change. | Spoofed user agents without expected network signals do not get broad access. |
| Match user agents | Match IP rules with user-agent and path conditions instead of trusting a single signal. | Admin, checkout, account, and staging paths remain protected. |
| Protect private paths | Use rate limits for public crawl paths and stronger blocks for admin, account, checkout, and staging paths. | The WAF change log records source URL, timestamp, diff, and rollback value. |
Useful command or data shape
Adapt paths, IDs, and privacy handling to the site before running commands or storing data on production.
curl -sS https://www.perplexity.com/perplexitybot.json | jq '.prefixes[0:3]'
curl -A 'PerplexityBot' -sSI https://example.com/sitemap.xml
curl -A 'OAI-SearchBot' -sSI https://example.com/blog/example-post
# Compare new ranges to the previous WAF set before applying changes.
Safe fix order
Do the work in a sequence that makes each result easy to prove. Stop if a step produces new evidence that changes the incident scope.
- Fetch official data
- Validate changes
- Match user agents
- Protect private paths
- Monitor and rollback
Production verification checklist
- Allowed crawlers can fetch public pages and sitemap URLs.
- Spoofed user agents without expected network signals do not get broad access.
- Admin, checkout, account, and staging paths remain protected.
- The WAF change log records source URL, timestamp, diff, and rollback value.
Mistakes to avoid
- Do not judge the fix by one browser or the homepage only.
- Do not delete evidence before recording usernames, file paths, timestamps, and response headers.
- Do not add a cache, security, or tracking plugin while the original problem is still unclear.
- Do not leave test users, temporary debug logs, or broad API keys active after verification.
When HandL WP should help
Bring in help when this affects leads, checkout, search visibility, malware risk, paid media reporting, or a client production site. HandL WP can trace the issue through WordPress, hosting, cache, tracking, and Search Console, then verify the workflow after the technical fix.
If this is active on a production site, automate WordPress AI crawler rules.