Block GPTBot and AI Bots in Robots.txt: Example
The Sultan's marksman stands at my gate and shoots the hat off any mechanical reader who asks for this page.
The short answer
- What it is
- A blocked AI bot is a known AI crawler that a page turns away through robots.txt, a meta tag named for the bot, or a bot-scoped X-Robots-Tag header. The block may be on purpose or an accident.
- Why it matters
- A block decides whether AI tools can read your pages for training or for live answers. Many sites block these bots by mistake through a shared template or a host default.
- How a crawler should report it
- Report each page with the bot name and the method that blocked it: robots.txt, meta tag, or header. Treat it as a notice, since blocking can be a valid choice, and ignore rules for bots that are not on your tracked list.
- Live examples
/lab/ai-bots-blocked/robots/(positive)/lab/ai-bots-blocked/meta/(positive)/lab/ai-bots-blocked/header/(positive)/lab/ai-bots-blocked/near-miss/(near miss)
Choosing Which AI Crawlers to Turn Away
A GPTBot robots.txt rule is the simplest way to tell OpenAI's training crawler to stay away. Two lines do the job: User-agent: GPTBot and Disallow: /. Other vendors use their own names. ClaudeBot comes from Anthropic, CCBot from Common Crawl, and PerplexityBot from Perplexity. Google-Extended is a special case. It is a control token, not a separate crawler, and it does not change how pages appear in Google Search.
Companies often run more than one bot for different jobs. OpenAI, for example, lists GPTBot for training, OAI-SearchBot for search, and ChatGPT-User for fetches a person asks for. Blocking one does not block the others. So decide what you want first. You may be fine with search features but not with model training. Write a separate group for each bot, because a named group replaces the * group for that bot.
Blocks can also live outside robots.txt. A page can carry a meta tag named for one bot, such as <meta name="gptbot" content="noindex">. A server can send X-Robots-Tag: gptbot: noindex. Not every AI vendor documents support for these forms, so check each vendor's own notes. Robots rules are also voluntary. A bot that ignores them can only be stopped at the firewall or the CDN, and many CDNs now offer a switch for this.
To detect blocks, parse robots.txt for each bot name, then scan each page's meta tags and response headers. Compare what you find with your policy. Also read your server logs, since they show which bots really visit. To fix an accidental block, remove the rule or turn off the CDN setting. To keep a block, write it down so the next person knows it was a choice.