Google SEO HTTP 429 Responses on an Export Product Catalog: Rate Limit or Missing Page?

A 429 on an export catalog URL is the responding server telling the client it has sent too many requests in a given amount of time — a rate-limit answer, not a statement that the product page is gone. Google’s crawler documentation treats 429 as a server-overload signal and groups it with 5xx: crawling slows temporarily, already indexed URLs are preserved but eventually dropped, and received content is ignored. This article separates that behavior from the 4xx codes that do signal non-existence, explains what a Retry-After header does and does not commit the client to, and gives a conditional method for locating which server layer emits the 429. No safe request rate, retry interval, or ranking-recovery timeline is asserted, because the supplied official material documents none.

The 429 on your catalog URL is a server answer, not a page verdict

A product URL in an export catalog starts returning 429 Too Many Requests, and the first instinct is to read it as bad news about the product: the page is gone, the listing was pulled, the CMS lost the record. That reading is wrong, and it is worth correcting before any other investigation begins.

The status code is generated by the server that hosts the site when it responds to a request made by a client, for example a browser or a crawler. That single sentence locates the meaning of the response: it describes the state of the responding server at that moment, not the existence or value of the product behind the URL.

MDN defines the code directly: the HTTP 429 Too Many Requests client error response status code indicates the client has sent too many requests in a given amount of time, and this mechanism of asking the client to slow down the rate of requests is commonly called rate limiting. The message is about request volume, not about content.

Google’s crawler documentation makes the same separation explicit. Google’s crawlers treat the 429 status code as a signal that the server is overloaded, and it is considered a server error. That places 429 in the server-error family rather than the not-found family.

The contrast with the rest of the 4xx range is the part that matters for a catalog operator. All 4xx errors, except 429, are treated the same: Google crawlers inform the next processing system that the content doesn’t exist. And the consequence of that message is documented: Google doesn’t use the content from URLs that return 4xx status codes, and URLs that are already indexed and return a 4xx status code are removed from the index.

So the two families lead to opposite conclusions. A 404 or 410 on a retired product is a removal signal, and the index treatment follows. A 429 is a slow-down-and-retry condition: the server is answering, the URL still resolves to a server that has something to say, and nothing in the response asserts that the product record is absent.

Hypothetical illustration, not a real client, test, or observed outcome. As a hypothetical illustration (not a real client, test, or observed outcome): imagine an export catalog whose product URLs are being fetched by a monitoring client at a pace the origin server considers excessive. The client records 429 for those URLs. Nothing about the product records has changed, and the pages still exist — the server is simply refusing to serve them at that request volume. Treating those 429s as evidence of deletion would send the operator hunting for a content problem that does not exist.

The reader decision that follows is narrow and concrete: a 429 on a catalog URL is a rate-limit answer from the responding server, not a verdict on whether the product page exists. Everything after this point — crawler behavior, the Retry-After header, and which layer is enforcing the limit — depends on holding that distinction.

Source reference

MDN reference

What Google’s crawler does with a 429: slow down, preserve, then eventually drop

Once the 429 is understood as a server answer rather than a removal signal, the next question is what Google’s crawler actually does with it. The documented answer has two parts that are easy to merge into one, and merging them produces the wrong expectation.

The first part is crawl rate. 5xx and 429 server errors prompt Google’s crawlers to temporarily slow down with crawling. The slowdown is a response to the server’s own signal, and it is described as temporary.

The second part is index treatment, and it is where the phrase that matters appears. For Google Search, already indexed URLs are preserved in the index, but eventually dropped. Two things are being said at once: the URL is not removed on the first 429, and the preservation is not unconditional. The word "eventually" is doing real work, and the documentation does not attach a number to it.

There is a third documented behavior that operators often miss. Any content Google receives from URLs that return a 5xx status code is ignored. Google’s guidance groups 429 with 5xx for crawl-rate and index behavior, so a 429 body is not a place to publish a message for search processing. That is a statement about content handling, not about whether the response was fetched or read.

The recovery side is documented in the same direction but without a schedule. Once the server starts responding with a 2xx status code, Google gradually increases the crawl rate for the site. "Gradually" is the whole of the documented commitment. No interval, no number of successful responses, and no ranking outcome is stated, so none should be assumed.

For a catalog operator, the practical reading is this: a burst of 429s on product URLs is a crawl-rate event with a delayed index consequence, not an immediate deindexing event. The correct response is to fix the condition that produces the 429s, not to start rewriting or removing product URLs. If you are separately trying to establish whether a specific URL is being indexed at all, that is a different diagnostic with a different starting point; see Google Not Indexing Pages: Diagnose One Exact URL by Index-Status Stage.

Reading Retry-After: what the header tells the client and what it does not

A 429 response often arrives with a companion header, and the header is frequently read as a promise. It is not one.

MDN states that a Retry-After header may be included to this response to indicate how long a client should wait before making the request again. Two words in that sentence carry the weight: "may be included." The header is optional, so its absence tells you nothing about when to retry, and its presence tells you what the server wants rather than what will happen.

MDN’s worked example shows the mechanism in a concrete form. In that example, server-wide rate limiting is active when a client exceeds a set threshold of requests per minute, and a 429 response is returned with a Retry-After header that indicates requests will be allowed for this client again after 3600 seconds (60 minutes). That 3600-second value is an illustration of the header’s syntax and placement inside a documentation example. It is not a recommended value, not a default, and not a figure to copy into a server configuration.

What the header does not do is guarantee anything. It is a server-side instruction about when the client may try again; it is not a commitment that the retry will succeed, that the limit will have lifted, or that the same client will be treated the same way. A client that waits the stated interval and retries can still receive another 429.

This is also where the temptation to invent a number should be resisted. No safe request rate and no guaranteed retry timing is documented in the supplied official material, and the correct posture for a catalog operator is to read the Retry-After value the server actually sends rather than to assume a standard one. If the header is absent, the response has told you less, not more.

Which layer is issuing the 429: a conditional investigation

Knowing what the 429 means and what Retry-After says still leaves the operational question: which part of the stack is enforcing the limit? The honest starting point is that this cannot be answered from the status code alone, because rate-limiting implementations vary.

MDN describes the variation directly: implementations of rate limiting vary, restrictions may be server-wide or per resource, and they are typically based on a client’s IP but can be specific to users or authorized applications if requests are authenticated or contain a cookie. That range is why a generic answer would be misleading. A limit keyed to an IP address, a limit keyed to a resource path, and a limit keyed to an authenticated identity produce the same status code and require different fixes.

The investigation starts from the one fact that is always true: the status code is generated by the server that’s hosting the site when it responds to a request made by a client. So the first step is to identify which server in the request chain actually produced the response, because that is the component whose configuration is in play.

From there, the checks are conditional. Each one is a question whose answer depends on the site’s own setup, and none of them can be resolved by assumption.

Compare the response headers returned by a request that goes through the full public path against one that reaches the origin directly. If the two responses differ in the identifiers they expose, the layer that differs is a candidate for the limiter. If they do not differ, the evidence points elsewhere.

Check whether the limit is per-IP or per-resource by varying the request path while holding the client constant, and by varying the client while holding the path constant. A limit that follows the path is resource-scoped; a limit that follows the client is client-scoped. MDN’s description of server-wide versus per-resource restrictions is the frame for reading the result.

Check whether authentication or a cookie changes the response. MDN notes that restrictions can be specific to users or authorized applications when requests are authenticated or carry a cookie, so a difference between an anonymous request and an authenticated one is meaningful evidence about which key the limiter uses.

Record whether a Retry-After header is present and what value it carries on each path. The header’s presence and value are properties of the responding layer, so a difference between paths localizes the behavior.

Each of these checks is a way of gathering evidence about a specific site. None of them establishes a universal answer, and the article cannot assert which layer any given catalog uses. The point of running them is to replace a guess about the stack with an observation about the stack.

Should you use 429 to control Googlebot’s crawl rate?

Once an operator understands that 429 slows crawling, the next thought is often to use it deliberately: return 429 to Googlebot and the catalog gets crawled less. The documented mechanics support part of that reasoning, and the rest of it needs to be stated carefully.

The documented effect is real. 5xx and 429 server errors prompt Google’s crawlers to temporarily slow down with crawling. And the contrast with the rest of the 4xx range is explicit: the 4xx status codes, except 429, have no effect on crawl rate. So 429 is one of the few client-error codes that does move crawl rate at all.

Google also documents what not to reach for. Don’t use 401 and 403 status codes for limiting the crawl rate. Those codes are for access control, and using them as a crawl throttle is a documented misuse.

The trade-off is where the instinct needs tempering. The same documentation that describes the slowdown also describes the index consequence: already indexed URLs are preserved in the index, but eventually dropped. A deliberate 429 campaign against Googlebot is therefore not a free throttle. It buys a slower crawl rate with a preservation risk that the documentation describes but does not quantify.

Whether 429 is the right tool for a given catalog is a site-specific judgment, and it depends on facts this article does not have: why the crawl rate is considered a problem, what the server can actually sustain, and whether the catalog’s URLs can tolerate the preservation risk. Google’s own guidance points operators toward its documented crawl-rate controls rather than toward status-code tricks, and that is the safer default to evaluate first.

The narrower conclusion is the one the evidence supports: 429 does affect crawl rate, it is grouped with 5xx for index preservation, and using it as a deliberate crawl-rate lever carries a documented preservation trade-off that should be weighed rather than assumed away.