503 Service Unavailable: What It Means for Crawlers
My castle is closed for repairs until the Moon is back where I left it.
The short answer
- What it is
- A 503 is an HTTP status code that says the server cannot handle the request right now. It points to a short-term problem, such as maintenance or too much traffic.
- Why it matters
- A 503 is the correct way to tell search engines that a site will be back soon. Used for a few hours, it protects your indexed pages, but left on for days it can cost you those pages.
- How a crawler should report it
- Report the URL as a broken page with status 503 in the status field, and mark it as non-indexable. Record any Retry-After value, and retry later before you treat the fault as lasting.
- Live examples
/lab/broken-pages/503/(positive), on the first shelf/lab/broken-pages/near-miss/(near miss)
Taking a site offline for maintenance without losing rankings
A 503 service unavailable response means the server is working but cannot serve the page at this moment. It often appears during planned maintenance. It also shows up when traffic is too high, when an app has no free workers, or when a load balancer has no healthy servers behind it. The server can add a Retry-After header, with a number of seconds or a date, to say when to try again.
Browsers show the body of the response, so a short note about the outage works well. Search engines read the code as a pause. Google slows its crawl and keeps indexed pages for a limited time. A 503 that lasts more than a day or two is risky, because URLs can start to drop out. A 503 on robots.txt is worse. Google may stop crawling the whole site until it can read that file again.
Watch for 503 codes in your uptime checks, server logs, and crawl reports. A site that returns them only at peak times has a capacity problem, not a maintenance problem. Run a crawl slowly and then quickly. If the code appears only at high speed, your server or firewall is limiting requests.
For planned work, return a 503 on every page with a short Retry-After value, and keep the window as small as you can. Never serve a maintenance page with a 200 status, since that can replace your real content in the index. Keep robots.txt available during the work. For load problems, add caching, add server capacity, or limit expensive requests, then confirm that the code is gone.