Near-Duplicate Content Example
A lion and a crocodile came for me at once, so alike in appetite that I could tell them apart only by the ending.
The short answer
- What it is
- Near duplicate content is a set of pages with almost the same text but not an exact match. A new title or a swapped city name is often the only change.
- Why it matters
- Search engines may treat these pages as one and show just one of them. The other pages then earn little and still take up crawl time.
- How a crawler should report it
- A crawler should pair similar URLs and show a near duplicate warning with the similarity score. This lab warns at 0.9 or higher, based on 3-word shingles, when the hashes differ.
- Live examples
/lab/near-duplicate-pages/a/(positive)/lab/near-duplicate-pages/b/(positive)/lab/near-duplicate-pages/near-miss/(near miss)
When pages differ by only a few words
Near duplicate content is text that is almost, but not exactly, the same on two or more pages. An exact hash will miss it, because one changed word gives a new hash. It often comes from templates. Think of city pages that swap one place name, product pages that differ by color, or blog tags that show the same posts in a new order.
Browsers show each page as usual. Search engines compare the text and can fold similar pages into one cluster. Only one page from the cluster is likely to appear for a query. Which one they pick is up to them. Pages that add almost nothing may also be crawled less often over time, or may be left out of the index.
To detect it, split the visible text of each page into shingles. A shingle is a run of words, such as three in a row. Compare the sets of shingles for two pages and divide the shared count by the total count. That is the Jaccard score, and 1.0 means the sets match. This lab warns at 0.9 or higher. Strip boilerplate first, or a big footer will make every page look the same.
To fix a group of near copies, ask if each page has its own purpose. If it does, write text that is truly specific, such as local details, specs, or reviews. Swapping a noun is not enough. If it does not, merge the pages and use a 301 redirect. For product variants that need their own URLs, add rel="canonical" that points to the main product. Then run the comparison again.