If a WordPress robots.txt edit does not appear at the public URL, first identify what serves that URL. You may be changing WordPress's virtual output while a physical file, host rule, or cached CDN response takes precedence. Repeatedly saving an SEO plugin setting will not fix the wrong owner.
WordPress exposes a do_robots function for virtual output. That proves WordPress can generate a response; it does not prove your production request reaches that function.
Capture the unmodified public response
Use the canonical production host, then inspect other active hostnames that actually serve your site. Save the status, redirects, content type, cache headers, and text. A successful status with a site's HTML shell is not a valid robots policy.
curl -sS -L --max-time 20 -D - 'https://example.com/robots.txt'
Replace the example host. Compare the returned rules with the exact change you expected. A browser refresh may still use a cached response; a query-string variation can use a different cache entry and should not be your only passing test.
Map the possible owners
| Layer |
Evidence to inspect |
| Web root |
An actual robots.txt file and deployment history |
| WordPress |
Virtual output, SEO plugin settings, and filters |
| Hosting or proxy |
A route that returns a generated or fixed response |
| CDN |
Cached body, response age, and relevant behavior |
On staging, add an innocuous comment identifying the proposed owner and verify the public response. Comments should contain no secrets or internal network details. Do not experiment by adding a site-wide Disallow rule on production.
Evidence: Status and body captured. Owner: One source of truth named. Scope: Crawler policy preserved. Public URL: Unmodified path returns new rules. Explanatory checklist, not a customer test result.
Repair ownership before editing policy
If a physical file is intentionally deployed, update its tracked source through the normal deployment path. If WordPress should own the response, have the server maintainer resolve the competing file or route with a rollback copy. Do not delete a file simply because a plugin says it cannot edit it.
If the correct output exists at the origin but not the edge, refresh only the relevant cached path and confirm the unmodified public URL afterward. Do not expose a protected origin to the internet just to compare responses; use the host's authorized diagnostics.
Retest crawler intent separately
Once the text is current, review each user-agent group and path against the business policy. A successful content update does not mean the rules express the intended access. Use the AI crawler policy guide to distinguish search visibility from training preferences.
Google's robots.txt introduction explains the limits of crawl rules. They are not authentication and are not a reliable method for removing an already known URL from search. For public PDF exclusion, check the direct-file noindex header instead.
Keep one named owner for future edits, the verified public response, and the rollback path. HandL WP can trace WordPress and CDN response ownership when a saved setting and the live crawler policy disagree.
References reviewed October 9, 2026. Examples are explanatory, not customer test results.