A URL can be blocked from crawling yet still appear as a bare URL if other pages link to it. Conversely, a noindex directive cannot be observed if robots.txt prevents the crawler from fetching the page.
Start with evidence and the smallest reversible change; work on staging when possible, preserve the logs and rollback material if production is already down, and keep the objective specific: separate crawl control from index control and inspect WordPress visibility, HTTP headers, meta robots, CDN rules, and robots.txt status.
What the symptom narrows down
- Settings → Reading still discourages search engines after staging or launch.
- A theme/plugin emits noindex or X-Robots-Tag on public content.
- robots.txt blocks pages, CSS, or JavaScript needed for rendering.
The branches are ordered to protect the strongest evidence around this possibility: settings → Reading still discourages search engines after staging or launch. The observed scope and logs—not a familiar-looking error screen—decide which one applies.
Crawlability, indexability, and canonical selection are separate; a crawler may discover a URL it cannot fetch, fetch a URL marked noindex, or index a different canonical than the publisher prefers; test the live HTTP response and rendered document before interpreting a report label; in this guide, the practical goal is to separate crawl control from index control and inspect WordPress visibility, HTTP headers, meta robots, CDN rules, and robots.txt status.
Evidence to collect before the fix
- Fetch
/robots.txtand the page headers/body as Googlebot and as a normal browser. - Inspect meta robots and X-Robots-Tag across representative templates.
- Review WordPress visibility settings and CDN/security rules.
- Use Search Console URL Inspection for the live URL.
The sequence moves from observation toward intervention. Preserve the result of the final check—use Search Console URL Inspection for the live URL—because it provides a useful comparison after the repair.
One practical branch through the failure
Begin with the first plausible cause: settings → Reading still discourages search engines after staging or launch. Before changing state, write down what would confirm it and run the first read-only check: fetch /robots.txt and the page headers/body as Googlebot and as a normal browser.
If the result supports that cause, try one bounded repair on staging: remove accidental noindex from pages intended for search. If evidence from “Fetch /robots.txt and the page headers/body as Googlebot and as a normal browser” points elsewhere, keep this layer unchanged and move to the next check. That small decision log is far easier to audit than several simultaneous edits.
Apply fixes in the safest order
- Remove accidental noindex from pages intended for search.
- Limit robots rules to genuinely unneeded crawl spaces; do not use robots.txt as secrecy.
- Allow rendering resources needed to understand public pages.
Before applying “Remove accidental noindex from pages intended for search,” name its rollback point and the evidence that will count as success. Afterward, repeat the original request and specifically check whether you can confirm the live test can fetch and render the canonical URL; a changed symptom at that point is new evidence, not permission to make several more changes at once.
Prove recovery
- Confirm the live test can fetch and render the canonical URL.
- Check both page and attachment/PDF policies intentionally.
- Monitor Page Indexing after recrawl rather than expecting immediate change.
One successful refresh is not closure. Keep the incident open until you can also check both page and attachment/PDF policies intentionally, adjacent paths have not regressed, temporary diagnostics are gone, and another operator can explain what changed.
Prepare a useful escalation if the boundary is outside your control
When escalating, include the exact canonical URL, response chain, robots rules result, meta and X-Robots-Tag values, canonical annotation, sitemap entry, and Search Console live-test date; search changes require recrawling; they are not instant configuration toggles; include the result of this first observation: fetch /robots.txt and the page headers/body as Googlebot and as a normal browser.
State what was tested, including the result of “Fetch `/robots.txt` and the page headers/body as Googlebot and as a normal browser,” and what changed between attempts; evidence tied to that observation is safer and more actionable than granting broad access or sending an unnecessary full database export.
Compare normal and Googlebot-visible directives
Fetch robots.txt and the target page headers with an explicit crawler user agent. Then inspect meta robots in the rendered HTML.
curl -sS https://example.com/robots.txt
curl -sS -D - -o /tmp/page.html \
-A 'Googlebot' https://example.com/public-page/
rg -n -i 'robots|canonical' /tmp/page.htmlInterpretation and safety: User-agent differences can reveal a CDN or security rule. Do not expose private pages merely to make an indexing test pass.
Shortcuts that create a second incident
- Do not alternate robots rules frequently to manipulate crawl budget.
- Do not expose private content merely to fix indexing; use authentication where privacy is required.
Evidence log
Primary references
- Google robots.txt guide — official reference consulted for this guide.
- Google robots specification — official reference consulted for this guide.
Editorial note: The scenario above illustrates how to approach “Remove accidental noindex from pages intended for search”; it is a documented example, not a claim about a reader’s server, so verify the cited documentation, take the appropriate backup, and follow the real environment’s access and change-control rules.