CSS and JS Blocked by Robots.txt: Example
My best cannon at Gibraltar stands behind a gate I locked and ate the key to, so its effect on the Spanish remains theoretical.
The short answer
- What it is
- A blocked asset is a script, stylesheet, or other site file that robots.txt disallows. A polite crawler never requests it, so its real status stays unknown.
- Why it matters
- Google renders pages to understand them, and a blocked file can hide content or change the layout it sees. The page may then be misread or judged as not mobile friendly.
- How a crawler should report it
- Report the asset URL, the matching robots.txt rule, the user agent tested, and the pages using it. Mark it as blocked and not requested, and do not give it an HTTP status.
- Live examples
/lab/blocked-assets/(positive)/lab/blocked-assets/near-miss/(near miss)
Disallow Rules That Keep Crawlers From Seeing a Page's Styles
CSS and JS blocked by robots.txt is usually a leftover from older advice. Years ago, site owners often blocked folders like /assets/ or /includes/ to save crawl activity. Some CMS setups blocked plugin and theme folders by default. A broad rule such as Disallow: /static/ can also catch more than its author meant. Files on a separate CDN host follow that host's own robots.txt, which is easy to forget.
A browser ignores robots.txt, so visitors see a complete page. A search engine crawler obeys it. When rendering, Google cannot load the blocked file, so it works with what remains. If the file only adds a small effect, little changes. If it loads the main content or the layout, Google sees a very different page. Google's documentation advises against blocking the CSS and JavaScript needed to understand a page.
To detect the problem, collect each asset URL on the page and test it against the rules for your target crawler. Remember that the longest matching rule wins, and that an Allow can override a Disallow of equal length in Google's parser. Check the robots.txt for every host involved. Google's URL Inspection tool lists page resources it could not load, and shows why each one failed.
To fix it, remove the rule, or add a narrower Allow for the files that matter. For example, Allow: /assets/css/ can open one folder while another stays closed. Do not block a file just to hide it from search results, since blocking only stops the fetch. Third-party files on hosts you do not control are out of your hands. If one of those is essential to the content, host a copy yourself.