Google Not Indexing Pages: Diagnose One Exact URL by Index-Status Stage

Confirm the URL is actually missing

A URL that does not appear in search results is not evidence of a single problem. It is evidence of a missing outcome.

Between the moment you publish a URL and the moment it can appear for a query, several distinct things must happen, and each can fail independently: Google must discover the URL, fetch it, decide it is allowed to be indexed, and select it — rather than some other URL — as the canonical representative of that content.

The central question is narrow: why is this specific URL not in Google’s index, and which stage is actually failing? The thesis is equally narrow: non-indexing is not one condition.

The status reported for that URL determines which single check is valid, and acting on the wrong stage wastes the fix.

If you rewrite content because you assume a quality problem while the URL is actually blocked from crawling, you have changed nothing that mattered.

If you remove a noindex directive while the real issue is that Google selected a different canonical, you have also changed nothing that mattered.

This article treats the work as a sequence with dependencies. Each step consumes the result of the previous one.

You do not skip ahead, and you do not run every check at once, because simultaneous unrelated edits destroy your ability to tell which change produced which result.

One boundary applies throughout.

Google’s own documentation on the Page Indexing report states that the report distinguishes indexing exclusion reasons, that not every URL should be indexed, and that duplicates can be correctly excluded — and that an indexed URL is not guaranteed visibility for a particular query (Page indexing report). That means "not indexed" is sometimes the correct state, not a fault. Your job is to determine which case you are in before you touch anything.

Before diagnosing a cause, establish that there is a problem. Readers frequently conclude a URL is missing when it is filtered, substituted, or simply queried incorrectly.

Start by recording the exact URL string you are testing. Write it down character for character, including protocol, host, path, trailing slash, and any parameters. A mismatch can occur when the tested variant is not the one you published.

If your site serves both an HTTP and an HTTPS version, or both a trailing-slash and non-trailing-slash version, you may be inspecting a URL that was never the canonical one.

Next, run a site-restricted query for that exact string. A site-restricted query narrows results to your host, which reduces the chance that you are looking at an unrelated page with similar wording.

If the URL appears, the problem is not absence — it is visibility, ranking, or query mismatch, and this diagnostic does not apply.

If the URL does not appear, you have a candidate for the rest of this process, but you have not yet proven absence, because a site-restricted query is an observation about results, not a statement from Google about index membership.

Distinguish three conditions that look similar from the outside:

1. Absent. The URL is not in the index at all.
2. Suppressed. The URL is indexed but not shown for the query you tried, because another URL on your site is a better match.
3. Substituted. A different URL — a parameter variant, a paginated version, or a duplicate — is being shown in place of the one you expected.

An indexed page with low visibility needs query and ranking analysis. A substituted URL needs canonical inspection. Neither should automatically be diagnosed as an undiscovered page.

Read the Page Indexing status for that URL

Now collect the primary evidence. Open the URL Inspection tool for that exact URL in Search Console and read the reported status.

The Page Indexing report is the documented starting point for diagnosing indexing problems, and the inspection view reports status at the level of the individual URL.

Record the status verbatim. Do not paraphrase it, and do not translate it into your own diagnosis yet. The exact wording matters because different statuses point to different stages, and a loose paraphrase will collapse two distinct stages into one.

Copy the string as shown.

Alongside the status, record the last crawl date if one is displayed. A recent crawl date tells you Google fetched the URL recently, which rules out a pure discovery failure. A missing or old crawl date does not by itself establish a discovery failure; retain it as incomplete or dated evidence.

Also note whether the status appears to be URL-level or site-level. A site-level condition — for example, a property-wide access problem — changes the scope of the fix, and you should not treat it as a single-URL issue.

One important boundary: Google’s documentation on the URL Inspection tool states that indexed data is not a live test, that live testing cannot predict the Google-selected canonical, and that discovery, crawl, indexing allowance, and selected canonical should be reviewed separately (URL Inspection tool). This is the reason the next section exists. You are not looking for one verdict; you are looking for which of several separate conditions is failing.

The same documentation notes that missing referring-page information does not prove that no referring page exists. Do not treat an empty referring-page list as proof that the URL is orphaned.

Map the status to a failure stage

This is the decision point that determines everything downstream. Use the observed status to choose the first diagnostic branch, while retaining other possible blockers. The stages are discovery, crawl access, indexing allowance, and canonical selection.

Use the following mapping logic. It is a classification aid, not a substitute for reading your own reported status.

Two distinctions in this table carry most of the diagnostic weight.

First, discovered and crawled are different states. A URL that has been discovered but not crawled has been seen as a reference — typically through a link or a sitemap — but not fetched. A URL that has been crawled has been fetched.

These are different observations requiring different checks; neither status alone establishes the root cause.

Second, a robots.txt block and a noindex directive are not interchangeable. Google’s documentation on blocking indexing states that Google must be allowed to crawl a page in order to see its noindex directive, and that blocking robots.txt access is not equivalent to a noindex directive (noindex documentation). This matters because a URL blocked in robots.txt will never reveal a noindex tag to Google, and removing the noindex tag will change nothing while the block remains.

Check discovery and crawl access

Run this section only if the mapping assigned the URL to discovery or crawl access. If it did not, skip to the next section. A dated indexing observation does not rule out a new access problem; recheck availability when the page has changed.

The question here is simple and falsifiable: can Google reach and fetch this URL at all?

Check robots.txt against the exact path. Do not check the domain root and assume the answer applies. Robots rules can be path-specific, and a rule that blocks one directory may not block another.

Compare the rule against the exact path you recorded in the first section.

Check for a noindex directive. Look for both a meta robots tag in the page head and an X-Robots-Tag response header. These are separate mechanisms and either can carry the directive. Remember the ordering constraint from the previous section: if robots.txt blocks the URL, Google cannot see the meta tag, so a robots block must be resolved before a noindex check is meaningful.

Check the server response code. A 404 or a 5xx response is a crawl-access failure, not an indexing decision. Confirm the URL returns a successful response to a normal request.

Check internal links pointing to the URL. A URL with no internal links is harder to discover. Note that this is a discovery factor, not proof of cause, and the inspection view’s referring-page list is not exhaustive.

Check for orphan status. If nothing on your site links to the URL and it is absent from any sitemap, discovery is the plausible stage, and the fix belongs here rather than in content.

Check indexing and canonical selection

Run this section only if the URL was fetched and the mapping assigned it to indexing allowance or canonical selection.

The question is now different: given that Google fetched the URL, is it choosing to index this URL, or a different one?

Compare the declared canonical with the URL Google selected. If your page declares one canonical and Google reports a different selected canonical, the two disagree, and that disagreement is the finding. Do not assume your declaration wins.

Check for near-duplicate and parameter variants. Variants of the same content — sort orders, tracking parameters, session identifiers, print versions — compete with each other.

If a variant is indexed and your target URL is not, canonical selection is the stage.

Check whether the page content is thin or substantially duplicated elsewhere. This is a selection factor, and it is the one most often guessed at without evidence. Treat it as a hypothesis to test, not a conclusion to assert.

Check whether the canonical target is itself indexed. A non-indexed target warrants inspection, but its current status does not by itself prove the canonical declaration is wrong. Verify target availability, indexability, content equivalence and the intended representative.

One caution applies to the whole section. The URL Inspection tool’s live test cannot predict the Google-selected canonical, so you cannot resolve a canonical disagreement by running a live test and reading the result.

You resolve it by comparing the declared canonical against the reported selected canonical and, where they differ, deciding which URL should represent the content.

Apply the smallest matching fix

The discipline in this section is subtraction, not addition. Change only the element tied to the confirmed cause.

If the stage was crawl access and the cause was a robots.txt rule, the fix is the rule. If the stage was crawl access and the cause was a server error, the fix is the server response.

If the stage was indexing allowance and the cause was a noindex directive, the fix is that directive — and only after confirming the URL is crawlable, per the ordering constraint above.

If the stage was canonical selection and the cause was a self-referencing canonical that disagreed with Google’s selection, the fix is the canonical declaration or the duplication that produced the disagreement.

Three rules keep the result interpretable:

1. Scope the repair. Avoid unrelated edits. If multiple confirmed blockers coexist, repair them as one documented change set rather than leaving a known blocker in place.
2. Record the change and the date. Write down what you changed and when. Without a date, a later status change cannot be sequenced against the edit.
3. Keep the URL stable. Do not rename, redirect, or restructure the URL during verification. A moving target cannot be re-inspected meaningfully.

If the mapping assigned the URL to a stage but the stage-specific check did not confirm a cause, do not apply a speculative fix. Return to the mapping step and re-read the reported status.

A fix applied without a confirmed cause converts a diagnosable problem into an unattributable one.

Re-inspect and record the observed change

Close the loop with an observation, not an assumption.

Re-run inspection on the same URL. Record the new status and the date you observed it. Compare it against the status you recorded before the fix. That comparison — before and after, same URL, dated — is the only reliable output of this process.

State the result honestly. A status change is an observation, not proof of cause. Other things change over time: Google recrawls on its own schedule, other URLs on your site change, and the index is not static.

A status that moved from one state to another after your edit is consistent with your fix having worked, but it does not establish that your fix caused the change. Report it as an observation.

If the status is unchanged, do not escalate to a larger intervention. Return to the mapping step and re-read the reported status.

A status that has not moved may indicate that you diagnosed the wrong stage, that the fix was incomplete, or that the change has not yet been reflected. Re-classify before you re-fix.

Use the following record for the URL you are testing. Fill it in as you work; the blank fields are the artifact.

Record the first diagnostic branch and any additional observed blockers. Do not force a URL with multiple problems into an exclusive category.

What this process does and does not establish

This workflow produces one thing: a classified URL with a dated status comparison. It does not produce a guarantee of indexing, and it should not be presented as one.

Google’s documentation is explicit that not every URL should be indexed and that duplicates can be correctly excluded, and that an indexed URL is not guaranteed visibility for a particular query.

A URL that remains unindexed after a repair has no confirmed recovery yet. Check whether Google has recrawled the changed page, then reassess content distinctiveness and canonical signals; do not declare the unresolved result a success.

The value of the sequence is that it prevents the most common waste in indexing troubleshooting: applying a fix from the wrong stage. Discovery, crawl access, indexing allowance, and canonical selection are separate conditions with separate checks.

The reported status tells you which one is in play. Read it first, select the next evidence-based check, repair confirmed blockers, and record what you observe.

Frequently asked questions

Does a successful live test mean my URL is indexed?

No. A live test checks current accessibility and some indexability conditions. The indexed report describes Google’s recorded version; neither a passed test nor an indexing request guarantees inclusion. Compare the last crawl with your deployment time.

Can Google index content that is rendered with JavaScript?

Yes, Google can render JavaScript, but rendering can fail or be delayed. Inspect the rendered HTML and resource errors in URL Inspection, not only the initial response. Content requiring a click or login may remain unavailable.

Can I prove crawling without Search Console access?

Verified Googlebot server logs can show specific fetches, including time and response status. Verify crawler identity; a user-agent string alone is insufficient. Logs do not prove indexing or Google’s canonical selection.

Should I submit the same URL every day?

No. After repairing and live-testing a page, request indexing where available and record the date. Repeated requests do not guarantee faster crawling. Recheck when there is new crawl evidence rather than repeatedly changing a stable URL.

What if the page is indexed but has no traffic?

Inspect query impressions and page performance separately from index status. Check intent, competing pages and relevance before treating low traffic as an indexing fault.

Sources: URL Inspection, JavaScript SEO basics, Verifying Googlebot, Request recrawling.