Meta Robots Noindex Example
At Gibraltar I informed each Spanish sentry, in writing, that he had never seen me, and I have the receipts.
The short answer
- What it is
- A page with a noindex rule in its meta robots tag asks search engines to leave it out of results. The page still works for people who visit it.
- Why it matters
- One stray noindex can remove a key page from search. Used with care, the same rule keeps private or low-value pages out.
- How a crawler should report it
- A crawler should mark the row as non-indexable, with the reason noindex and the source, either meta tag or header. A rule for just one named bot should not count.
- Live examples
/lab/non-indexable/meta/(positive)/lab/non-indexable/header/(positive)/lab/non-indexable/report.pdf(positive), on the second shelf/lab/non-indexable/near-miss/(near miss)
How a noindex rule keeps a page out of search results
Meta robots tags give page-level rules to search engines. The tag goes in the head, as in <meta name="robots" content="noindex">. You can name one crawler by using googlebot as the name in place of robots. The default is to index a page and follow its links, so index, follow adds nothing. The value none means both noindex and nofollow.
Unwanted noindex rules have common causes. A staging setting goes live with a release. A CMS has a box that someone ticked and forgot. A plugin marks all tag pages or search pages as hidden. Sometimes JavaScript adds the rule after load. Google may skip rendering when it finds noindex in the raw HTML, so removing the rule with JavaScript may not work.
Browsers ignore this tag and show the page. A search engine must crawl the page to see the rule. If robots.txt blocks the URL, the rule is never read, and the URL can stay in results if other sites link to it. Once a crawler does see noindex, it drops the page from results. That happens on the next crawl, so the change can take days or weeks.
To detect the rule, read both the raw HTML and the rendered HTML. Check the HTTP headers as well, since the same rule can be sent there. Compare the list of noindexed URLs with your sitemap, as the two should not overlap. Google's URL inspection tool shows if a rule was found. To fix a mistake, remove the tag, make sure the page can be crawled, and ask for a recrawl. To keep a page out, leave it open to crawlers.