Bot Access Example: A Page That Treats Bots Differently
I wore a polar bear's skin among the polar bears, and they gave me a much better dinner than they ever gave the Admiral.
The short answer
- What it is
- This issue means a search bot or AI bot gets a blocked, challenge, or different answer than a browser does. The test compares responses on the start page and a second page.
- Why it matters
- A blocked search bot cannot crawl your pages, which can remove them from results. Serving bots different content on purpose can also break Google's spam policies.
- How a crawler should report it
- Report each bot name, its status, its body length, and a result of blocked, challenge, or different, compared with the browser baseline. List search bots apart from AI bots.
- Live examples
/lab/a-bot-access/(positive)
When a Server Gives Crawlers a Different Answer Than People
Bot access problems often begin with a security setting. A firewall or CDN rule may block unfamiliar user agents, or show them a challenge page. Rate limits can catch busy crawlers. Teams also block AI crawlers on purpose, which is a fair choice if it is intended. The risk is a rule that reaches further than planned. A bot that gets a 403 will look like a site that has gone offline.
Search engines read the status code and the body they receive. A 403 or a challenge page gives them nothing useful to index. Long runs of 5xx or 429 responses can make Google slow its crawling. Showing different content to Googlebot than to people is cloaking, and Google treats that as spam when it is done to manipulate rankings. Small differences are often innocent. A shorter body for a bot deserves a closer look.
To test access, request the same URL with several user agents, including Googlebot, bingbot, and GPTBot, and compare each response with a normal browser request. Record the status and the length of the body. A user agent test has limits. Some systems check the IP address too, and a request from your machine cannot prove what happens for the real crawler. Server logs and Google's URL Inspection tool can confirm that.
To fix it, find the rule that fires and narrow it. Allow verified search bots through your firewall, using the reverse DNS check or the IP ranges they publish. Do not trust the user agent alone, because anyone can fake it. For AI crawlers, decide on a policy first, then state it in robots.txt. Serve the same main content to bots and people. Retest after any change to your CDN or security settings.