WordPress AI Crawler Log Analysis with CloudFront and Athena

By HandL WP Engineering

2026-08-01T13:20:00Z

WordPress teams often count GPTBot, ClaudeBot, PerplexityBot, Google-Extended, or other labels from a user-agent string and call it AI visibility. CloudFront logs can show requested pages, response codes, cache outcomes, bytes, timing, IPs, and referrers, but the same label can be spoofed and a successful fetch does not prove citation, ranking, or referral traffic.

Use this for publishers, ecommerce teams, agencies, SEO owners, security teams, and infrastructure analysts serving WordPress through CloudFront or another CDN with queryable access logs.

Quick answer

Query by user-agent family, date, URI, status, cache result, edge location, IP, request rate, bytes, and referrer. Normalize URLs before grouping, exclude static assets from content demand, verify published bot IP or reverse-DNS methods when available, and keep separate metrics for claimed agents, verified agents, successful content fetches, blocked requests, citations, referrals, and conversions.

What to check first

Confirm CloudFront logging is enabled, fields match the Athena table, timestamps use UTC, partitions are current, retention is appropriate, sensitive query strings are protected, and analysts can reproduce the query window.
Normalize host, path, trailing slash, percent encoding, query string, locale, pagination, feed, preview, canonical, and static-asset requests before measuring page demand.
Group claimed AI agents by status, cache result, content type, path, IP, autonomous system, edge location, request interval, bytes, time taken, and suspicious user-agent variants.
Verify crawler identity using the provider's supported network method where one exists, flag impossible request rates and spoofed labels, and keep unknown automation outside the verified bot total.
Join successful canonical content fetches to robots policy, sitemap membership, indexability, server errors, referral sessions, assisted conversions, citations, and Search Console trends without implying causation.

Diagnostic table

Use this table to keep the work practical. It connects the symptom to evidence and a verification step.

Action	Evidence to collect	How to verify
Validate logs and partitions	Confirm CloudFront logging is enabled, fields match the Athena table, timestamps use UTC, partitions are current, retention is appropriate, sensitive query strings are protected, and analysts can reproduce the query window.	The same query and partition window reproduce the reported totals.
Normalize content URLs	Normalize host, path, trailing slash, percent encoding, query string, locale, pagination, feed, preview, canonical, and static-asset requests before measuring page demand.	Canonical content, static assets, duplicate parameters, previews, feeds, and error paths are reported separately.
Segment claimed and verified agents	Group claimed AI agents by status, cache result, content type, path, IP, autonomous system, edge location, request interval, bytes, time taken, and suspicious user-agent variants.	Claimed, verified, spoofed, unknown, allowed, blocked, cached, and origin-served requests have distinct counts.
Measure status and cache behavior	Verify crawler identity using the provider's supported network method where one exists, flag impossible request rates and spoofed labels, and keep unknown automation outside the verified bot total.	The dashboard never labels a fetch as a citation, referral, lead, or conversion without corresponding evidence.

Why this usually happens

Any client can send a famous crawler name in its user-agent header.
Static files and duplicate URL variants can dominate raw request counts.
CloudFront status and cache fields can be misread when the distribution uses custom errors or multiple origins.
Google reports AI feature traffic inside Web search, while third-party crawlers have different controls and reporting.

Useful command or data shape

Adapt paths, IDs, and privacy handling to the site before running commands or storing data on production.

SELECT
  date, cs_user_agent, sc_status, x_edge_result_type,
  regexp_replace(cs_uri_stem, '/+
    
    
  

, '') AS canonical_path,
  count(*) AS requests, count(DISTINCT c_ip) AS ips
FROM cloudfront_logs
WHERE from_iso8601_timestamp(concat(date, 'T', time, 'Z'))
  BETWEEN TIMESTAMP '2026-07-01 00:00:00' AND TIMESTAMP '2026-08-01 00:00:00'
  AND lower(cs_user_agent) LIKE '%bot%'
GROUP BY 1,2,3,4,5;

AI crawler request investigation console for WordPress AI Crawler Log Analysis with CloudFront and Athena

Safe fix order

Do the work in a sequence that makes each result easy to prove. Stop if a step produces new evidence that changes the incident scope.

Validate logs and partitions
Normalize content URLs
Segment claimed and verified agents
Measure status and cache behavior
Connect crawl evidence to business signals

Decision rule

Use AI crawler data for content or infrastructure decisions only when the agent is classified, the URL is canonical content, the response is interpretable, the time window is reproducible, and crawl evidence is kept separate from citations, referrals, and revenue.

What to tell the client or owner

Give the owner the affected versions, exact workflow, observed result, business impact, evidence location, temporary control, named owner, and next review time. Remove credentials and personal data from shared screenshots and logs.

Production verification checklist

The same query and partition window reproduce the reported totals.
Canonical content, static assets, duplicate parameters, previews, feeds, and error paths are reported separately.
Claimed, verified, spoofed, unknown, allowed, blocked, cached, and origin-served requests have distinct counts.
The dashboard never labels a fetch as a citation, referral, lead, or conversion without corresponding evidence.

Mistakes to avoid

Do not change several plugins, cache rules, firewall settings, or integrations before preserving a baseline.
Do not treat one successful browser test as proof for APIs, retries, alternate clients, background jobs, or mixed-version fleets.
Do not paste secrets, personal data, complete production payloads, or customer records into tickets, screenshots, or long-lived logs.
Do not close the test until the final user-visible state and the server-side evidence agree.

Questions teams ask during testing

Can CloudFront logs prove an AI answer cited a page?

No. They prove requests reached the distribution. Citation monitoring and referral or conversion evidence are separate.

Should query strings be stored for analysis?

Only when needed and governed. They can contain personal data, tokens, email addresses, or campaign identifiers.

When HandL WP should help

Bring in HandL WP when a production checkout, form, editor, security gate, performance incident, or attribution workflow is at risk. We can preserve evidence, isolate the failing layer, make the smallest corrective change, and verify the result across WordPress, connected services, logs, and the user journey.

If this is active on a production site, build an AI crawler evidence dashboard.

Related HandL WP guides

Use these related guides when the same issue touches tracking, security, checkout, or crawler visibility.