Truncated Page Example
I began to read the Sultan the full list of my victories, and the Sultan died, and his son died, and I was only as far as Tuesday.
The short answer
- What it is
- A truncated page is a response that a crawler cut off because the body went past its size limit. In this check, the limit is 10 MiB.
- Why it matters
- The crawler never sees the rest of the page, so its links and text are lost to the audit. A page this big has usually gone wrong somewhere, and search engines read far less.
- How a crawler should report it
- Mark the page as truncated when the body passes 10 MiB and the fetch stops. Report the URL, the bytes read, and the limit, and add the large HTML and Google size warnings.
- Live examples
/lab/truncated-pages/(positive)/lab/truncated-pages/near-miss/(near miss)
Pages Whose Content Stops Before the Final Tag
A truncated page ends early from the point of view of whoever is reading it. Crawlers set a maximum download size so that one huge response cannot use up all their memory. When the body passes that limit, they stop and keep what they have. The rest is never parsed. Each crawler has its own number, so one tool may read a page in full while another stops well before the end.
Pages get this big through mistakes more often than through design. A template loop may repeat without an exit. A product listing may print every item on one URL. A log, data export, or debug dump may land in the HTML by accident. Very large comment threads are another source. A browser will often keep going and may become slow. Google stops much sooner, at 2 MB of HTML for Search, so its copy is cut short too.
To detect truncation, compare the bytes you received with the size limit, and record when the limit stopped the download. A declared Content-Length larger than the amount you read is a strong sign. A document with no closing </html> tag is a weaker hint, since that tag is optional in HTML. A connection failure can also cut a body short. Report that as a network error so that it is not confused with a size limit.
The fix is to find out why the page is so large. View the source and scroll to see which block repeats. Remove any debug output, and repair the loop if a template is at fault. Break long lists into paginated pages with normal links. Move large data sets to a separate download or an API that loads on demand. After the change, measure the page again and make sure the footer and closing tags are all in the response.