July 28, 2015, several webmasters received an error from Google Search Console (Webmaster Tools) which read “Googlebot cannot access CSS and JS files on http://www.website.com/”.
The message reads:
To: Webmaster of http://www.website.com/,
Google systems have recently detected an issue with your homepage that affects how well our algorithms render and index your content. Specifically, Googlebot cannot access your JavaScript and/or CSS files because of restrictions in your robots.txt file. These files help Google understand that your website works properly so blocking access to these assets can result in suboptimal rankings.
Screenshot of the Googlebot CSS and JS files error
What does this mean?
Google’s technical guidelines state that they want access to all of your files to thoroughly understand your website. This includes seeing potential advertisements on it and how it renders on a mobile device. To do this they need to crawl CSS and JS files.
How to fix the problem in WordPress
Many WordPress webmasters are inadvertently blocking CSS and JS files.
A few ways WordPress can block these files:
- robots.txt file is blocking crawling of this subfolder /wp-includes
- robots.txt is blocking crawling of JS and CSS files
- A security plugin such as iThemes Security is blocking JS and CSS crawling
To fix the error and allow Googlebot to crawl your JS and CSS files, take the following steps:
Check your robots.txt file. You can FTP in, use Google’s robots.txt tester, or simply visit yourwebsite.com/robots.txt. It should look something like this:
User-agent: * Disallow: /wp-admin/
If you’re using iThemes Security, uncheck the “disable directory browsing” box. Using it blocks access to JS and CSS files via the .htaccess file.
There are other ways these files can become blocked but this should get you started.
Once the issue is resolved, “Fetch and Render” your page with the “Mobile: smartphone” option selected to double-check that Googlebot for smartphones renders your content properly.
Note: Fetch and Render can be used to instantly crawl pages. Read about that here.
Testing for blocked resources
In search console, visit the blocked resources section.
Does blocking JS and CSS files hurt SEO?
Yes, but don’t freak out. I haven’t seen this affect traffic personally, however, it is well documented on the Yoast blog here. In that case study, the site bounced back to normal health once the issue was resolved.
- Google “Pure Spam” Penalty Deindexes Sites March 6 2024 - March 12, 2024
- What Happened to ChicagoNow.com? - August 30, 2022
- The December 2021 Google Local Pack Algorithm Update - December 17, 2021
The simplest thing to do is to chenge your robot file as below:
User-Agent: Googlebot
Allow: .js
Allow: .css