Robots.txt Example: Disallow and Crawl-Delay Rules
I posted orders on the ramparts of Gibraltar in Spanish, Turkish, and blank, and the garrison obeyed the blank one with enthusiasm.
The short answer
- What it is
- A robots.txt file tells crawlers which paths on a host they may fetch. A robots.txt issue is a line the parser can't use, or an internal link that points to a path the file blocks.
- Why it matters
- One bad line can leave a section open that you meant to close, or close one you meant to keep open. Links to blocked pages also send crawlers to places they can't read.
- How a crawler should report it
- Report each bad line as a warning with its line number, its text, and the reason, such as unknown field or relative Sitemap. List blocked link targets once each, with the rule that matched and the user-agent used.
- Live examples
/lab/robots-blocked-urls/private/(positive)/lab/robots-blocked-urls/near-miss/(near miss)
How Disallow, Allow, and Crawl-delay Lines Work
A robots.txt example is the fastest way to learn the format. The file lives at the root of a host, as in https://example.com/robots.txt. Each host and protocol needs its own copy. A group starts with a User-agent line and holds Disallow and Allow rules. So User-agent: * followed by Disallow: /private/ keeps polite bots out of that folder. A Sitemap line can sit anywhere, but it must use a full URL.
Most errors come from hand edits and old advice. A rule placed before any User-agent line belongs to no group. A line with no colon can't be parsed. A path should start with a slash or a wildcard. A Noindex line does nothing at Google, which dropped support for it in September 2019. Google also ignores Crawl-delay, though Bing and some other crawlers honor it. A Host line is a leftover from one search engine and is not part of the standard.
Search engines read this file before they fetch other pages. Google uses the most specific matching rule, so a longer Allow beats a shorter Disallow. A 4xx answer for the file usually means no limits apply. A 5xx answer can pause crawling of the whole site. Keep in mind that blocking is not the same as hiding. A blocked URL can still show up in results if other sites link to it, just without a snippet.
To detect problems, fetch the file and parse it line by line. Then test your internal links against the rules for the user-agent you care about. To fix a page you want out of results, allow crawling and use a noindex meta tag. To fix a link to a blocked page, remove the link or unblock the path. Delete any line that you can't explain.