I'm getting a crawl error on Cloudflare cdn-cgi files.

If you are getting a crawl error in Google Webmaster Tools for a cdn-cgi file, please disallow Google from crawling that with a rule in robots.txt for:

Disallow: /cdn-cgi/

An additional option using WordPress:

Solution provided by Ben

You can make modifications in your /wp-includes/functions.php file. to create an additional Disallow rule. A sample of what it would look like is below:

function do_robots() {
header( 'Content-Type: text/plain; charset=utf-8' );

do_action( 'do_robotstxt' );

$output = "User-agent: *\n";
$public = get_option( 'blog_public' );
if ( '0' == $public ) {
$output .= "Disallow: /\n";
} else {
$site_url = parse_url( site_url() );
$path = ( !empty( $site_url['path'] ) ) ? $site_url['path'] : '';
$output .= "Disallow: $path/wp-admin/\n";
$output .= "Disallow: $path/wp-includes/\n";
$output .= "Disallow: $path/cdn-cgi/\n";
}

 

Note: This won't impact the actual crawl for your site at all. The error for /cdn-cgi/ is also not going to impact your rankings at all. Google is attempting to crawl a Cloudflare resource, one that doesn't need to be crawled, and they possibly can't read the DOM or JavaScript in that resource.

Still not finding what you need?

The Cloudflare team is here to help. 95% of questions can be answered using the search tool, but if you can’t find what you need, submit a support request.

Powered by Zendesk