429 Too Many Requests Example: How Crawlers Should Back Off
I knocked so often at the Sultan's hives that his bees now hand me a numbered ticket through the entrance.
The short answer
- What it is
- A 429 is an HTTP status code that says a client sent too many requests in a given time. The page itself may be fine, and the server is only asking the client to slow down.
- Why it matters
- Rate limits protect servers, but a limit that catches search bots can slow crawling and delay indexing. A crawler that ignores the code can also get its IP address blocked.
- How a crawler should report it
- Report the link as likely blocked, with status 429 or 999 in the status field, rather than as a broken page. Mark the URL as non-indexable for that crawl, and say that the result may change at a slower speed.
- Live examples
/lab/likely-blocked-links/(positive)/lab/likely-blocked-links/near-miss/(near miss)
Slowing down when a server asks a crawler to wait
A 429 too many requests response means a client has passed a rate limit set by the server. Limits are often counted per IP address, per user, or per API key. A web application firewall, a CDN, or the app itself can send the code. Some large social sites answer automated visitors with a nonstandard 999 code that means much the same thing. In both cases, the target page may work well in a normal browser.
A person rarely sees this code, unless they refresh a page again and again. Search engines do see it. Google treats a 429 as a sign that the server is overloaded and handles it like a server error. It slows its crawl. If the code lasts for a long time, URLs that are already indexed can drop out, so a limit that catches Googlebot needs prompt attention.
A well-built crawler should respect the Retry-After header when one is present. If there is none, it should wait, try again, and double the delay after each new 429. It should also cut its request rate for that host. A 429 result should never be counted as a dead link, because the crawler has not learned whether the page exists.
To find the source of the limit, look in your server and firewall logs for the IP addresses and user agents that got the code. To fix it, raise limits for verified search bots, using the IP ranges that the search engines publish. Do not trust the user agent string alone, since anyone can fake it. When you crawl your own site, lower the speed or add your crawler to an allow list.