Links Likely Blocked From Crawling: Example
The guards at Constantinople turned me away 429 times, so I came back as a goose.
The short answer
- What it is
- A likely blocked link is an internal or external link that answers 429 or 999. The server is refusing or limiting the crawler, so the real state of the page stays unknown.
- Why it matters
- Calling these links broken would send people off to fix pages that work. On your own site, a 429 can also mean you are turning away search engine bots.
- How a crawler should report it
- Report these links in their own likely blocked group, with the status, the host, and any Retry-After value. Do not count them as broken pages, though the target can be marked as not indexable for this crawl.
- Live examples
/lab/likely-blocked-links/(positive)/lab/likely-blocked-links/near-miss/(near miss)
Reading 429 and 999 Answers From Busy Servers
Links likely blocked from crawling are links that a server refused to answer in a normal way. Two codes signal this. A 429 Too Many Requests is the standard way to say that a client has sent too many requests. It may include a Retry-After header that says how long to wait. A 999 is not a standard code at all. Some large sites, including LinkedIn, use it to turn away automated visitors.
Rate limits usually come from a firewall, a CDN, or the app itself. A crawl with many threads can trip a limit in a few seconds. Shared hosting plans can have low limits. Data center IP addresses are often treated with suspicion. Social networks are the most common source of these answers, since they block anything that doesn't look like a signed-in person. A link to one of them can work fine in a browser and fail in every audit.
A person who clicks the link usually sees the page, because their request is not throttled. Googlebot reacts to 429 much as it does to a server error. It slows down, and if the code keeps coming, it can drop those URLs from the index. So a 429 on your own site is a serious sign. A 429 from someone else's site is mostly just noise.
To detect this, list every link with a 429 or 999 answer and group it by host. If one host dominates, the cause is likely your crawler's speed or identity. To fix it on your own site, slow the crawl, respect Retry-After, and allow known good bots at the firewall. Be sure to verify them by reverse DNS, since user-agent names are easy to fake. For other sites, test the link by hand once, then move on.