I'm getting a crawl error on Cloudflare cdn-cgi files.

If you are getting a crawl error in Google Webmaster Tools for a cdn-cgi file, please disallow Google from crawling that with a rule in robots.txt for:

Disallow: /cdn-cgi/

An additional option using WordPress:

Solution provided by Ben

You can make modifications in your /wp-includes/functions.php file. to create an additional Disallow rule. A sample of what it would look like is below:

function do_robots() {
header( 'Content-Type: text/plain; charset=utf-8' );

do_action( 'do_robotstxt' );

$output = "User-agent: *\n";
$public = get_option( 'blog_public' );
if ( '0' == $public ) {
$output .= "Disallow: /\n";
} else {
$site_url = parse_url( site_url() );
$path = ( !empty( $site_url['path'] ) ) ? $site_url['path'] : '';
$output .= "Disallow: $path/wp-admin/\n";
$output .= "Disallow: $path/wp-includes/\n";
$output .= "Disallow: $path/cdn-cgi/\n";
}

 

Note: This won't impact the actual crawl for your site at all. The error for /cdn-cgi/ is also not going to impact your rankings at all. Google is attempting to crawl a Cloudflare resource, one that doesn't need to be crawled, and they possibly can't read the DOM or JavaScript in that resource.

Not finding what you need?

95% of questions can be answered using the search tool. This is the quickest way to get a response.

Powered by Zendesk