HTTP 404 status code example
I leapt into Mount Etna to see what lay inside, and the mountain answered with a 404, a 500, and a very rude remark from Vulcan.
The short answer
- What it is
- A 404 is an HTTP status code that says the server could not find anything at the requested URL. It does not say whether the page will ever come back.
- Why it matters
- Each 404 that real links point to wastes a visit and sends crawlers to a dead end. Links to that URL also stop passing value to any live page.
- How a crawler should report it
- Report the URL as a broken page with status 404 in the status field, and mark it as non-indexable. List every inbound link to it as a broken link, with the source page beside it.
- Live examples
/lab/broken-pages/gone/(positive), on the first shelf/lab/broken-pages/near-miss/(near miss)
How servers, browsers, and crawlers handle a dead URL
A 404 not found response is the server's way of saying it has nothing at that address. The most common causes are simple. Someone deleted a page and left the links in place. Someone changed a URL and forgot the redirect. Someone typed a link by hand and made a typo. A site move can create thousands of these at once if the old paths are not mapped to the new ones.
Browsers show whatever HTML the server sends with the 404, so a site can offer a helpful custom page with search and top links. That page must still return the 404 status. Search engines read the status, not the words. Google drops a 404 URL from its index after it has seen the code, and it checks back less and less often. A 404 is not a penalty, and a few of them will not hurt the rest of a site.
To find them, run a site crawl and filter by status. Then check your server logs, since logs show the bad URLs that real users and bots ask for. You can test one URL with curl -I and read the first line of the reply. Always note which pages link to each bad URL, because that list is your repair job.
The fix depends on the cause. If the content moved, add a 301 redirect to the new URL and update your internal links. If the link has a typo, correct the link. If the content is gone and has no close match, leave the 404 in place and remove the links. Do not send every dead URL to the home page, since search engines often treat that as a soft error.