CDN custom errors and single-page-app fallbacks can return HTTP 200 for a missing prerendered page. AI crawlers and search engines then receive the same generic shell for many URLs even though a browser later loads page content from an API.
Use this for headless WordPress, React, S3, CloudFront, Lambda prerendering, or any frontend where clean routes and missing files can share a fallback document.
Quick answer
AI Crawler App Shell 200 Detection Test should be handled with a narrow evidence-first workflow: fetch known and missing routes, compare markers and hashes, check origin objects, then verify the result before making broader changes.
What to check first
- Fetch a known article, a deliberately missing slug, and the site shell with browser and allowed crawler user agents.
- Compare HTTP status, content type, title, meta description, canonical, Article schema, H1, answer text, and body hash.
- Check whether the expected clean prerender object exists at origin and whether a duplicate .html object is absent.
- Review CDN custom error mapping, function rewrites, cache keys, and invalidations for the route.
- Add an automated marker test that fails when a published page matches the shell or a missing route returns publishable metadata.
Diagnostic table
Use this table to keep the work practical. It connects the symptom to evidence and a verification step.
| Action | Evidence to collect | How to verify |
|---|
| Fetch known and missing routes | Fetch a known article, a deliberately missing slug, and the site shell with browser and allowed crawler user agents. | Published pages contain page-specific metadata, schema, H1, and main answer in crawler HTML. |
| Compare markers and hashes | Compare HTTP status, content type, title, meta description, canonical, Article schema, H1, answer text, and body hash. | Missing routes return the intended non-indexable status and do not imitate a published article. |
| Check origin objects | Check whether the expected clean prerender object exists at origin and whether a duplicate .html object is absent. | The clean origin object exists and no new .html duplicate competes with it. |
| Fix fallback behavior | Review CDN custom error mapping, function rewrites, cache keys, and invalidations for the route. | Allowed crawler and browser-agent fetches agree on the canonical public content. |
Why this usually happens
- S3 403 or 404 responses can be replaced with index.html and status 200.
- A missing clean object can be hidden by client-side API rendering in a normal browser.
- The CDN can cache the fallback before the prerender Lambda finishes writing the article.
- Default app metadata can make many missing pages appear indexable.
Useful command or data shape
Adapt paths, IDs, and privacy handling to the site before running commands or storing data on production.
for path in /blog/known-page /blog/definitely-missing /index.html; do
curl -A 'OAI-SearchBot' -sS -D /tmp/headers.txt "https://example.com$path" -o /tmp/body.html
printf '%s ' "$path"; sha256sum /tmp/body.html
rg -o '<title>[^<]+|rel="canonical"[^>]+|Article|<h1[^>]*>[^<]+' /tmp/body.html
done
Safe fix order
Do the work in a sequence that makes each result easy to prove. Stop if a step produces new evidence that changes the incident scope.
- Fetch known and missing routes
- Compare markers and hashes
- Check origin objects
- Fix fallback behavior
- Automate shell detection
What to tell the client or owner
Share route matrix, status, content type, title, canonical, schema marker, hashes, origin object status, CDN rule, and automated test result.
Production verification checklist
- Published pages contain page-specific metadata, schema, H1, and main answer in crawler HTML.
- Missing routes return the intended non-indexable status and do not imitate a published article.
- The clean origin object exists and no new .html duplicate competes with it.
- Allowed crawler and browser-agent fetches agree on the canonical public content.
Mistakes to avoid
- Do not judge the fix by one browser or the homepage only.
- Do not delete evidence before recording usernames, file paths, timestamps, and response headers.
- Do not add a cache, security, or tracking plugin while the original problem is still unclear.
- Do not leave test users, temporary debug logs, or broad API keys active after verification.
When HandL WP should help
Bring in help when this affects leads, checkout, search visibility, malware risk, paid media reporting, or a client production site. HandL WP can trace the issue through WordPress, hosting, cache, tracking, and Search Console, then verify the workflow after the technical fix.
If this is active on a production site, verify crawler-visible HTML on a headless WordPress site.
Related HandL WP guides
Use these related guides when the same issue touches tracking, security, checkout, or crawler visibility.
Helpful references