{"id":2871,"date":"2019-10-23T15:38:31","date_gmt":"2019-10-23T07:38:31","guid":{"rendered":"https:\/\/www.mondoze.com\/guide\/?post_type=kb&p=2871"},"modified":"2022-10-05T08:01:01","modified_gmt":"2022-10-05T00:01:01","slug":"im-getting-a-crawl-error-on-cloudflare-cdn-cgi-files","status":"publish","type":"kb","link":"https:\/\/www.mondoze.com\/guide\/kb\/im-getting-a-crawl-error-on-cloudflare-cdn-cgi-files","title":{"rendered":"I’m getting a crawl error on Cloudflare cdn-cgi files."},"content":{"rendered":"\t\t
\n\t\t\t\t\t\t
\n\t\t\t\t\t\t\t
\n\t\t\t\t\t\t\t
\n\t\t\t\t\t\t
\n\t\t\t\t\t\t\t
\n\t\t\t\t\t
\n\t\t\t
\n\t\t\t\t\t\t\t
\n\t\t\t\t\t\t
\n\t\t\t\t
\n\t\t\t

Crawl Error on Cloudflare cdn-cgi Files<\/h2>\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t\t\t\t
\n\t\t\t\t

If you are getting a crawl error in Google Webmaster Tools for a cdn-cgi file, please disallow Google from crawling that with a rule in robots.txt for:\u00a0<\/p>

Disallow: \/cdn-cgi\/<\/code><\/p>

An additional option using WordPress:<\/strong><\/p>

Solution provided by Ben
<\/strong><\/p>

You can make modifications in your \/wp-includes\/functions.php file. to create an additional Disallow rule. A sample of what it would look like is below:<\/p>

function do_robots() {
header( ‘Content-Type: text\/plain; charset=utf-8’ );<\/p>

do_action( ‘do_robotstxt’ );<\/p>

$output = “User-agent: *\\n”;
$public = get_option( ‘blog_public’ );
if ( ‘0’ == $public ) {
$output .= “Disallow: \/\\n”;
} else {
$site_url = parse_url( site_url() );
$path = ( !empty( $site_url[‘path’] ) ) ? $site_url[‘path’] : ”;
$output .= “Disallow: $path\/wp-admin\/\\n”;
$output .= “Disallow: $path\/wp-includes\/\\n”;
$output .= “Disallow: $path\/cdn-cgi\/\\n”;
}<\/p>

Note: This won’t impact the actual crawl for your site at all. The error for \/cdn-cgi\/ is also not going to impact your rankings at all. Google is attempting to crawl a Cloudflare resource. One that doesn’t need to be crawl, and they possibly can’t read the DOM or JavaScript in that resource.<\/p>\t\t\t\t\t<\/div>\n\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t<\/div>\n\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t<\/section>\n\t\t\t\t\t\t<\/div>\n\t\t\t\t\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t","protected":false},"excerpt":{"rendered":"

Crawl Error on Cloudflare cdn-cgi Files If you are getting a crawl error in Google Webmaster Tools for a cdn-cgi file, please disallow Google from crawling that with a rule in robots.txt for:\u00a0 Disallow: \/cdn-cgi\/ An additional option using WordPress: Solution provided by Ben You can make modifications in your \/wp-includes\/functions.php file. to create an …<\/p>\n

I’m getting a crawl error on Cloudflare cdn-cgi files.<\/span> Read More \u00bb<\/a><\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"closed","ping_status":"closed","template":"","meta":{"site-sidebar-layout":"default","site-content-layout":"","ast-site-content-layout":"","site-content-style":"default","site-sidebar-style":"default","ast-global-header-display":"","ast-banner-title-visibility":"","ast-main-header-display":"","ast-hfb-above-header-display":"","ast-hfb-below-header-display":"","ast-hfb-mobile-header-display":"","site-post-title":"","ast-breadcrumbs-content":"","ast-featured-img":"","footer-sml-layout":"","theme-transparent-header-meta":"","adv-header-id-meta":"","stick-header-meta":"","header-above-stick-meta":"","header-main-stick-meta":"","header-below-stick-meta":"","astra-migrate-meta-layouts":"","ast-page-background-enabled":"default","ast-page-background-meta":{"desktop":{"background-color":"","background-image":"","background-repeat":"repeat","background-position":"center center","background-size":"auto","background-attachment":"scroll","background-type":"","background-media":"","overlay-type":"","overlay-color":"","overlay-gradient":""},"tablet":{"background-color":"","background-image":"","background-repeat":"repeat","background-position":"center center","background-size":"auto","background-attachment":"scroll","background-type":"","background-media":"","overlay-type":"","overlay-color":"","overlay-gradient":""},"mobile":{"background-color":"","background-image":"","background-repeat":"repeat","background-position":"center center","background-size":"auto","background-attachment":"scroll","background-type":"","background-media":"","overlay-type":"","overlay-color":"","overlay-gradient":""}},"ast-content-background-meta":{"desktop":{"background-color":"var(--ast-global-color-5)","background-image":"","background-repeat":"repeat","background-position":"center center","background-size":"auto","background-attachment":"scroll","background-type":"","background-media":"","overlay-type":"","overlay-color":"","overlay-gradient":""},"tablet":{"background-color":"var(--ast-global-color-5)","background-image":"","background-repeat":"repeat","background-position":"center center","background-size":"auto","background-attachment":"scroll","background-type":"","background-media":"","overlay-type":"","overlay-color":"","overlay-gradient":""},"mobile":{"background-color":"var(--ast-global-color-5)","background-image":"","background-repeat":"repeat","background-position":"center center","background-size":"auto","background-attachment":"scroll","background-type":"","background-media":"","overlay-type":"","overlay-color":"","overlay-gradient":""}}},"kbtopic":[53],"kbtag":[110],"mkb_version":[],"_links":{"self":[{"href":"https:\/\/www.mondoze.com\/guide\/wp-json\/wp\/v2\/kb\/2871"}],"collection":[{"href":"https:\/\/www.mondoze.com\/guide\/wp-json\/wp\/v2\/kb"}],"about":[{"href":"https:\/\/www.mondoze.com\/guide\/wp-json\/wp\/v2\/types\/kb"}],"author":[{"embeddable":true,"href":"https:\/\/www.mondoze.com\/guide\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/www.mondoze.com\/guide\/wp-json\/wp\/v2\/comments?post=2871"}],"version-history":[{"count":6,"href":"https:\/\/www.mondoze.com\/guide\/wp-json\/wp\/v2\/kb\/2871\/revisions"}],"predecessor-version":[{"id":18942,"href":"https:\/\/www.mondoze.com\/guide\/wp-json\/wp\/v2\/kb\/2871\/revisions\/18942"}],"wp:attachment":[{"href":"https:\/\/www.mondoze.com\/guide\/wp-json\/wp\/v2\/media?parent=2871"}],"wp:term":[{"taxonomy":"kbtopic","embeddable":true,"href":"https:\/\/www.mondoze.com\/guide\/wp-json\/wp\/v2\/kbtopic?post=2871"},{"taxonomy":"kbtag","embeddable":true,"href":"https:\/\/www.mondoze.com\/guide\/wp-json\/wp\/v2\/kbtag?post=2871"},{"taxonomy":"mkb_version","embeddable":true,"href":"https:\/\/www.mondoze.com\/guide\/wp-json\/wp\/v2\/mkb_version?post=2871"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}