A robots.txt file expresses crawler preferences, but an allowed bot can still receive a WAF challenge, 429, stale redirect, empty app shell, blocked asset, or wrong canonical. A blocked crawler can also keep appearing in logs because it checks robots.txt, retries, or ignores the policy. The useful report compares declared rules, delivered responses, crawl identity, index controls, citations, referrals, and business outcomes on the same date range.
Use this for publishers, service businesses, ecommerce sites, security teams, and SEO owners deciding how OAI-SearchBot, GPTBot, ChatGPT-User, Googlebot, Bingbot, Perplexity crawlers, or other AI agents should reach public content.
Quick answer
Version the robots policy, fetch it through the same CDN path a crawler uses, and parse rules by exact user agent. Query edge logs for claimed agents, request paths, status codes, cache results, bytes, and timing, then separate verified identities from claimed strings where verification is available. Join that evidence to canonical pages, noindex controls, citations, ChatGPT referral sessions, leads, and conversions. Treat a user agent alone as a claim, not proof of identity.
What to check first
- Record the robots.txt body, response status, redirect chain, cache headers, ETag, last-modified value, sitemap declaration, applicable meta robots, and CDN or WAF policy at the start of the report window.
- Create an intended-policy table for OAI-SearchBot, GPTBot, ChatGPT-User, Googlebot, Bingbot, Perplexity crawlers, ad-related crawlers if relevant, and unknown automated clients.
- Query edge and origin logs for exact and case-normalized user-agent claims, requested path, query string, status, bytes, cache result, IP or network evidence, request ID, referrer, and date.
- Classify canonical content, sitemap, robots, feeds, assets, redirects, errors, private paths, staging paths, and traps so a high request count does not hide poor coverage of valuable pages.
- Compare allowed pages that received no successful fetch, blocked agents that received content, repeated 403 or 429 responses, citation appearances, referral sessions, assisted leads, and policy changes.
Diagnostic table
Use this table to keep the work practical. It connects the symptom to evidence and a verification step.
| Action | Evidence to collect | How to verify |
|---|
| Version the intended crawler policy | Record the robots.txt body, response status, redirect chain, cache headers, ETag, last-modified value, sitemap declaration, applicable meta robots, and CDN or WAF policy at the start of the report window. | Every named crawler has an owner-approved purpose, allow or disallow rule, effective date, and reason in the policy table. |
| Fetch policy through the edge | Create an intended-policy table for OAI-SearchBot, GPTBot, ChatGPT-User, Googlebot, Bingbot, Perplexity crawlers, ad-related crawlers if relevant, and unknown automated clients. | Edge tests and logs show successful robots, sitemap, canonical HTML, and required asset delivery for allowed search crawlers without broad security bypasses. |
| Query and classify observed requests | Query edge and origin logs for exact and case-normalized user-agent claims, requested path, query string, status, bytes, cache result, IP or network evidence, request ID, referrer, and date. | Claimed user-agent counts are labeled separately from identities supported by published verification methods or stronger network evidence. |
| Reconcile delivery and index controls | Classify canonical content, sitemap, robots, feeds, assets, redirects, errors, private paths, staging paths, and traps so a high request count does not hide poor coverage of valuable pages. | The report shows crawler fetches, errors, citations, ChatGPT referrals, engaged visits, leads, and conversions as separate measures. |
Why this usually happens
- Robots.txt can be correct at the origin while a CDN serves a stale version or redirects the crawler to a different host.
- A WAF can challenge or block the request before the crawler receives the policy or article HTML.
- Any client can claim a known user-agent string, so raw counts can overstate real crawler demand and hide abusive traffic.
- A page can be allowed in robots.txt but still carry noindex, point its canonical elsewhere, return thin JavaScript, or fail on required assets.
Field notes
- OpenAI's publisher FAQ distinguishes search discovery from training controls. Write policy decisions by crawler purpose instead of applying one blanket rule.
- If a page should not appear in search, use an appropriate noindex control and allow supported crawlers to read it. Blocking crawling can prevent the noindex directive from being seen.
- Keep search visibility and business value separate. More crawler requests are not a win unless useful pages are fetched and resulting citations or referrals help users.
Useful command or data shape
Adapt paths, IDs, and privacy handling to the site before running commands or storing data on production.
SELECT date, user_agent, status, cache_result, path, count(*) requests
FROM cloudfront_logs
WHERE lower(user_agent) LIKE '%oai-searchbot%'
OR lower(user_agent) LIKE '%gptbot%'
OR lower(user_agent) LIKE '%chatgpt-user%'
GROUP BY 1,2,3,4,5
ORDER BY requests DESC;
Safe fix order
Do the work in a sequence that makes each result easy to prove. Stop if a step produces new evidence that changes the incident scope.
- Version the intended crawler policy
- Fetch policy through the edge
- Query and classify observed requests
- Reconcile delivery and index controls
- Connect citations and referrals to outcomes
Decision rule
The policy is operating as intended when allowed crawlers receive useful canonical public content with expected status codes, blocked purposes do not receive protected content, noindex directives remain readable where required, and WAF or cache behavior matches the written decision.
What to tell the client or owner
Give the owner the affected versions, exact workflow, observed result, business impact, evidence location, temporary control, named owner, and next review time. Remove credentials and personal data from shared screenshots and logs.
Production verification checklist
- Every named crawler has an owner-approved purpose, allow or disallow rule, effective date, and reason in the policy table.
- Edge tests and logs show successful robots, sitemap, canonical HTML, and required asset delivery for allowed search crawlers without broad security bypasses.
- Claimed user-agent counts are labeled separately from identities supported by published verification methods or stronger network evidence.
- The report shows crawler fetches, errors, citations, ChatGPT referrals, engaged visits, leads, and conversions as separate measures.
Mistakes to avoid
- Do not change several plugins, cache rules, firewall settings, or integrations before preserving a baseline.
- Do not treat one successful browser test as proof for APIs, background jobs, alternate roles, webhooks, or mixed-version fleets.
- Do not paste secrets, personal data, complete production payloads, or customer records into tickets, screenshots, or long-lived logs.
- Do not close the test until the user-visible result and server-side evidence agree.
Questions teams ask during testing
Does allowing OAI-SearchBot allow GPTBot too?
No. OpenAI documents them as separate controls with different purposes, so list each user agent intentionally.
Why can a blocked URL still show a title and link?
OpenAI notes that a page disallowed from crawling may still be surfaced from third-party signals. Use noindex when the goal is exclusion and let the crawler read that directive where supported.
Are all OAI-SearchBot user agents in logs genuine?
No. A user-agent string is easy to copy. Label it as claimed unless stronger verification supports the identity.
When HandL WP should help
Bring in HandL WP when a production checkout, form, editor, firewall, search visibility, or attribution workflow is at risk. We can preserve evidence, isolate the failing layer, make a narrow corrective change, and verify the result across WordPress, connected services, logs, and the user journey.
If this is active on a production site, build an AI crawler evidence report.
Related HandL WP guides
Use these related guides when the same issue touches tracking, security, checkout, or crawler visibility.
Helpful references