Server Log Analysis Example: A Sample Crawl Log
Not a single spy has reported on the hole in my ship that I am sitting on, an insult I intend to repay.
The short answer
- What it is
- Log analysis reviews the server's own records of each request to see which bots visited which URLs. It finds pages bots skip, errors they hit, and bots that are fake.
- Why it matters
- A crawl shows what could be reached, while a log shows what was actually fetched. The gap between them explains many indexing and crawl budget problems.
- How a crawler should report it
- Report counts of indexable pages with no bot hits, verified bot 4xx and 5xx responses, IPs failing reverse DNS, 200 pages the crawl never reached, waste by bucket, and rarely crawled pages.
- Live examples
- These checks need an imported access log, so no page here can show them.
What Access Logs Reveal About Real Crawler Visits
Server log analysis starts with the raw access logs from your web server or CDN. Each line usually holds the IP address, the time, the URL, the status code, the bytes sent, and the user agent. Getting the files is often the hard part. Some hosts keep them for only a few days. Others offer them only on paid plans. Sites behind a CDN need the logs from the edge, because the origin sees only part of the traffic.
Logs reveal behavior that nothing else can. You can see which sections Googlebot visits often and which it ignores. You can see redirects, parameter URLs, and old paths taking up crawl activity. A run of 5xx or 429 responses matters most, because Google slows down crawling when a server seems strained. Crawl budget is mainly a concern for very large sites or sites that change quickly. Smaller sites still gain by finding errors.
Never trust the user agent in a log line, since anyone can claim to be Googlebot. To verify a hit, run a reverse DNS lookup on the IP address. Confirm the host name ends in googlebot.com or google.com, and then run a forward lookup to check that it maps back to the same IP. Google and Bing also publish their crawler IP ranges. Filter out unverified hits before you count anything.
Then join the log data to your crawl data by URL. Pages with no hits may need better internal links. A URL that bots fetch and your crawl never reached is an orphan, so link it or retire it. Fix errors that verified bots hit, starting with the most frequent. Cut crawl waste by removing links to redirects and by tidying parameter URLs. Repeat the analysis on a regular basis, as a single week can mislead you.