Semantic Content Checks: Similar and Off-Topic Pages
I told the tale of the bear and the two flints in French and again in Latin, and the bear, on hearing both, insisted on royalties.
The short answer
- What it is
- These checks compare the meaning of pages by turning their text into numbers. They group near-duplicates, flag off-topic pages, and can score how well a page answers questions.
- Why it matters
- Pages that say the same thing can compete with each other in search. A page far from the site's main theme may confuse readers or point to hacked or misplaced content.
- How a crawler should report it
- Report similar groups at a similarity of 0.95 or more among pages of at least 200 words. Flag pages over 2 standard deviations below the content centroid, and label AI scores under 50 as simulated.
- Live examples
/lab/answer-readiness-low/(positive)/lab/semantic-similar/a/(positive)/lab/semantic-low/(positive)/lab/semantic-similar/b/(positive)/lab/answer-readiness-low/near-miss/(near miss)/lab/semantic-similar/near-miss/(near miss)
Using Text Embeddings to Spot Twin Pages and Off-Topic Strays
Semantic content checks look at what pages mean and not just at the words they use. A model turns the text of each page into a list of numbers called an embedding. Pages with close meanings get close embeddings. Two pages can therefore match even when they share few exact phrases. This often catches rewritten service pages, city pages built from one template, and blog posts that cover one idea twice.
Google does not publish these scores, so treat them as your own working estimate. Search engines do cluster similar pages and tend to show one from each group. Similar pages can also split links and clicks. An off-topic page has a separate meaning. It sits far from the average of the site, called the centroid. That can be a harmless post, a misfiled article, or a sign that spam pages were injected.
To run the check, extract the main text from each page and leave out menus and footers, since shared boilerplate pulls pages together. Create embeddings, then measure cosine similarity between pairs. Group pairs above a set level, here 0.95. Compute the centroid and flag pages more than 2 standard deviations below it. Results depend on the model you use. Very short pages give weak results, so a minimum of 200 words is sensible.
Review each result by hand before you change anything. If two pages serve one purpose, merge them and redirect the weaker URL. If they serve different needs, rewrite them to make that difference clear. Move or remove pages that do not belong on the site. Answer scores and link suggestions depend on an AI model, so read them as ideas and not as measurements. A custom prompt result is just stored output and is not a defect.