If a WordPress site wants AI search visibility, public pages need to return useful content to allowed crawlers. A repeatable curl test catches 403s, 429s, captcha challenges, app-shell fallbacks, missing assets, and stale robots.txt rules.
Use this for public blog posts, service pages, documentation, comparison pages, and landing pages that should be visible in AI search and normal organic search.
Quick answer
AI Crawler Allowed Public Pages Test Script should be handled with a narrow evidence-first workflow: pick url set, fetch robots, test agents, then verify the result before making broader changes.
What to check first
- Choose five representative public URLs and two private URLs for a control test.
- Fetch robots.txt, sitemap.xml, HTML, and one image or CSS asset with representative user agents.
- Check status codes, content type, title tag, canonical tag, and a page-specific phrase.
- Review WAF or CDN logs when a public page returns 403, 429, challenge, or an empty shell.
- Document which agents are intentionally allowed or blocked.
Diagnostic table
Use this table to keep the work practical. It connects the symptom to evidence and a verification step.
| Action | Evidence to collect | How to verify |
|---|---|---|
| Pick URL set | Choose five representative public URLs and two private URLs for a control test. | Allowed public pages return 200 with page-specific HTML. |
| Fetch robots | Fetch robots.txt, sitemap.xml, HTML, and one image or CSS asset with representative user agents. | Sitemap and robots.txt are reachable to representative crawlers. |
| Test agents | Check status codes, content type, title tag, canonical tag, and a page-specific phrase. | Private control URLs do not reveal protected content. |
| Review WAF | Review WAF or CDN logs when a public page returns 403, 429, challenge, or an empty shell. | CDN and WAF logs match the intended policy. |
Why this usually happens
- CDN bot rules can block allowed AI search crawlers before robots.txt is read.
- Security plugins can challenge unfamiliar user agents even on public pages.
- A public page can return 200 while the content is only an app shell with no useful body.
- Private pages can accidentally appear in a sitemap if route generation is too broad.
Field notes
- Use a normal browser user agent as a baseline so AI-specific failures are easier to spot.
- Never whitelist by user-agent string alone without rate limits and log review.
- Include one known private URL to verify that the test script does not encourage overexposure.
Useful command or data shape
Adapt paths, IDs, and privacy handling to the site before running commands or storing data on production.
urls=("https://example.com/" "https://example.com/services/" "https://example.com/blog/example-post")
agents=("OAI-SearchBot" "ChatGPT-User" "PerplexityBot" "Googlebot")
for agent in "${agents[@]}"; do
for url in "${urls[@]}"; do
code=$(curl -sS -A "$agent" --max-time 10 -o /tmp/crawler-test.html -w '%{http_code}' "$url")
title=$(grep -Eio '<title>[^<]+' /tmp/crawler-test.html | head -1)
printf '%s\t%s\t%s\t%s\n' "$agent" "$code" "$url" "$title"
done
done
Safe fix order
Do the work in a sequence that makes each result easy to prove. Stop if a step produces new evidence that changes the incident scope.
- Pick URL set
- Fetch robots
- Test agents
- Review WAF
- Document policy
What to tell the client or owner
Share the test output table with the owner. It should show which public URLs are reachable, which private URLs are protected, and which blocked requests were intentional.
Production verification checklist
- Allowed public pages return 200 with page-specific HTML.
- Sitemap and robots.txt are reachable to representative crawlers.
- Private control URLs do not reveal protected content.
- CDN and WAF logs match the intended policy.
Mistakes to avoid
- Do not judge the fix by one browser or the homepage only.
- Do not delete evidence before recording usernames, file paths, timestamps, and response headers.
- Do not add a cache, security, or tracking plugin while the original problem is still unclear.
- Do not leave test users, temporary debug logs, or broad API keys active after verification.
When HandL WP should help
Bring in help when this affects leads, checkout, search visibility, malware risk, paid media reporting, or a client production site. HandL WP can trace the issue through WordPress, hosting, cache, tracking, and Search Console, then verify the workflow after the technical fix.
If this is active on a production site, test WordPress AI crawler access.
Related HandL WP guides
Use these related guides when the same issue touches tracking, security, checkout, or crawler visibility.