A URL can be blocked from crawling yet still appear as a bare URL if other pages link to it. Conversely, a noindex directive cannot be observed if robots.txt prevents the crawler from fetching the page.
The reliable route is not the longest checklist; establish the failing boundary, keep one clean reproduction, and change one layer at a time until the evidence lets you separate crawl control from index control and inspect WordPress visibility, HTTP headers, meta robots, CDN rules, and robots.txt status.
Read the failure at the right layer
- Settings → Reading still discourages search engines after staging or launch.
- A theme/plugin emits noindex or X-Robots-Tag on public content.
- robots.txt blocks pages, CSS, or JavaScript needed for rendering.
The branches are ordered to protect the strongest evidence around this possibility: settings → Reading still discourages search engines after staging or launch. The observed scope and logs—not a familiar-looking error screen—decide which one applies.
Crawlability, indexability, and canonical selection are separate; a crawler may discover a URL it cannot fetch, fetch a URL marked noindex, or index a different canonical than the publisher prefers; test the live HTTP response and rendered document before interpreting a report label; in this guide, the practical goal is to separate crawl control from index control and inspect WordPress visibility, HTTP headers, meta robots, CDN rules, and robots.txt status.
Build a clean diagnostic record
- Fetch
/robots.txtand the page headers/body as Googlebot and as a normal browser. - Inspect meta robots and X-Robots-Tag across representative templates.
- Review WordPress visibility settings and CDN/security rules.
- Use Search Console URL Inspection for the live URL.
The sequence moves from observation toward intervention. Preserve the result of the final check—use Search Console URL Inspection for the live URL—because it provides a useful comparison after the repair.
One practical branch through the failure
Treat “Settings → Reading still discourages search engines after staging or launch” as a working hypothesis, not a conclusion. Establish a baseline first: fetch /robots.txt and the page headers/body as Googlebot and as a normal browser. Record both the result you expected and the result you actually saw.
A supporting result justifies a staging test of the narrowest repair: remove accidental noindex from pages intended for search. A result that contradicts “Settings → Reading still discourages search engines after staging or launch” is useful too: it rules out one layer without disturbing production and gives the next operator a clean starting point.
Repair the cause—not the message
- Remove accidental noindex from pages intended for search.
- Limit robots rules to genuinely unneeded crawl spaces; do not use robots.txt as secrecy.
- Allow rendering resources needed to understand public pages.
Before applying “Remove accidental noindex from pages intended for search,” name its rollback point and the evidence that will count as success. Afterward, repeat the original request and specifically check whether you can confirm the live test can fetch and render the canonical URL; a changed symptom at that point is new evidence, not permission to make several more changes at once.
Compare normal and Googlebot-visible directives
Fetch robots.txt and the target page headers with an explicit crawler user agent. Then inspect meta robots in the rendered HTML.
curl -sS https://example.com/robots.txt
curl -sS -D - -o /tmp/page.html \
-A 'Googlebot' https://example.com/public-page/
rg -n -i 'robots|canonical' /tmp/page.htmlInterpretation and safety: User-agent differences can reveal a CDN or security rule. Do not expose private pages merely to make an indexing test pass.
Verification checklist
- Confirm the live test can fetch and render the canonical URL.
- Check both page and attachment/PDF policies intentionally.
- Monitor Page Indexing after recrawl rather than expecting immediate change.
One successful refresh is not closure. Keep the incident open until you can also check both page and attachment/PDF policies intentionally, adjacent paths have not regressed, temporary diagnostics are gone, and another operator can explain what changed.
Prepare a useful escalation if the boundary is outside your control
When escalating, include the exact canonical URL, response chain, robots rules result, meta and X-Robots-Tag values, canonical annotation, sitemap entry, and Search Console live-test date; search changes require recrawling; they are not instant configuration toggles; include the result of this first observation: fetch /robots.txt and the page headers/body as Googlebot and as a normal browser.
State what was tested, including the result of “Fetch `/robots.txt` and the page headers/body as Googlebot and as a normal browser,” and what changed between attempts; evidence tied to that observation is safer and more actionable than granting broad access or sending an unnecessary full database export.
Tempting moves to avoid
- Do not alternate robots rules frequently to manipulate crawl budget.
- Do not expose private content merely to fix indexing; use authentication where privacy is required.
Change record
Primary references
- Google robots.txt guide — official reference consulted for this guide.
- Google robots specification — official reference consulted for this guide.
Editorial note: The scenario above illustrates how to approach “Remove accidental noindex from pages intended for search”; it is a documented example, not a claim about a reader’s server, so verify the cited documentation, take the appropriate backup, and follow the real environment’s access and change-control rules.