X-Robots-Tag Example: Noindex in a Response Header
I pinned the "do not call" notice to my hat instead of my door, so only those who saw me coming knew to stay away.
The short answer
- What it is
- X-Robots-Tag is an HTTP response header that carries the same rules as a meta robots tag. A noindex value makes the URL non-indexable, even for files like PDFs.
- Why it matters
- It is the only way to set page rules on files that have no HTML head. It is also easy to miss, because the rule does not appear in the page source.
- How a crawler should report it
- A crawler should mark the row as non-indexable and name the X-Robots-Tag header as the source. A rule scoped to one named bot should be shown but should not count.
- Live examples
/lab/non-indexable/meta/(positive)/lab/non-indexable/header/(positive)/lab/non-indexable/report.pdf(positive), on the second shelf/lab/non-indexable/near-miss/(near miss)
Sending robots rules in the HTTP response instead of the HTML
X-Robots-Tag is an HTTP header that gives robots rules to search engines. A server sends it with a response, as in X-Robots-Tag: noindex. It accepts the same rules as the meta tag, such as nofollow, nosnippet, and unavailable_after. You can aim it at one crawler with a line like X-Robots-Tag: googlebot: noindex. It works on any file type, so it suits PDFs, images, and video.
Teams often set this header in the server config. On nginx, the line is add_header X-Robots-Tag "noindex";. On Apache, it is Header set X-Robots-Tag "noindex", often inside a FilesMatch block. Problems start when a rule is too broad. A staging config reaches production, or a CDN adds the header to every path. Since the rule is not in the HTML, a page review will not catch it.
Browsers ignore this header and show the file as normal. Search engines read it when they fetch the URL. If the URL is blocked in robots.txt, they never see the header, so the rule has no effect. If both the header and the meta tag are present, the most limiting rule wins. A noindex in either place keeps the page out of results.
To detect the header, run curl -I https://example.com/report.pdf, or read the response headers in your crawl data. Test real file types and not just pages. A HEAD request may get a different reply from a GET, so confirm with a GET if something looks odd. To fix a mistake, find the layer that adds the header, such as the server, the app, or the CDN. Remove it there, clear the cache, and test again.