{"id":2791,"date":"2019-10-23T12:56:12","date_gmt":"2019-10-23T04:56:12","guid":{"rendered":"https:\/\/www.mondoze.com\/guide\/?post_type=kb&#038;p=2791"},"modified":"2022-10-05T08:01:37","modified_gmt":"2022-10-05T00:01:37","slug":"im-getting-google-crawler-errors-what-should-i-do","status":"publish","type":"kb","link":"https:\/\/www.mondoze.com\/guide\/kb\/im-getting-google-crawler-errors-what-should-i-do","title":{"rendered":"I&#8217;m getting Google Crawler Errors. What should I do?"},"content":{"rendered":"\t\t<div data-elementor-type=\"wp-post\" data-elementor-id=\"2791\" class=\"elementor elementor-2791\" data-elementor-settings=\"[]\">\n\t\t\t\t\t\t<div class=\"elementor-inner\">\n\t\t\t\t\t\t\t<div class=\"elementor-section-wrap\">\n\t\t\t\t\t\t\t<section class=\"elementor-section elementor-top-section elementor-element elementor-element-59d0a236 elementor-section-boxed elementor-section-height-default elementor-section-height-default\" data-id=\"59d0a236\" data-element_type=\"section\">\n\t\t\t\t\t\t<div class=\"elementor-container elementor-column-gap-default\">\n\t\t\t\t\t\t\t<div class=\"elementor-row\">\n\t\t\t\t\t<div class=\"elementor-column elementor-col-100 elementor-top-column elementor-element elementor-element-6cc7c8aa\" data-id=\"6cc7c8aa\" data-element_type=\"column\">\n\t\t\t<div class=\"elementor-column-wrap elementor-element-populated\">\n\t\t\t\t\t\t\t<div class=\"elementor-widget-wrap\">\n\t\t\t\t\t\t<div class=\"elementor-element elementor-element-1987a12 elementor-widget elementor-widget-heading\" data-id=\"1987a12\" data-element_type=\"widget\" data-widget_type=\"heading.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t<h2 class=\"elementor-heading-title elementor-size-default\">Google Crawler Errors<\/h2>\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-46ade175 elementor-widget elementor-widget-text-editor\" data-id=\"46ade175\" data-element_type=\"widget\" data-widget_type=\"text-editor.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t\t\t\t<div class=\"elementor-text-editor elementor-clearfix\">\n\t\t\t\t<h4><strong>General guidelines<\/strong><\/h4><p>Review the\u00a0following guidelines if you&#8217;re experiencing crawler errors in the Google Webmaster Tools Health dashboard.<\/p><ul><li>Monitor the performance and availability of your website directly to rule out any errors produced by your own web server.\u00a0There are a number of free and paid services which you can use:<ul><li>Pingdom<\/li><li>StatusCake<\/li><li>Monitor.Us<\/li><li>updown<\/li><\/ul><\/li><li>Do not block Google crawler IPs in the Cloudflare Threat Control Panel (you can verify whether an address belongs to Google using the method they mention in\u00a0this article). Blocking IPs in their\u00a0range\u00a0will prevent Google from crawling your website.<\/li><li>Do not block the United States as a country in the Cloudflare Threat Control Panel. Many search engines are based in the United States and this setting could prevent them from crawling your website.<\/li><li>Do not block Google&#8217;s User-Agents in your\u00a0.htaccess, server configuration,\u00a0robots.txt, or web application. Google may use a variety of User-Agents to crawl your website. Please review this\u00a0list of common User-Agents Google uses when crawling\u00a0your website. When making modifications, you can test your robots.txt file on the, &#8220;Blocked URLs page<strong>&#8220;<\/strong>\u00a0under the, &#8220;Health&#8221; section<strong>\u00a0<\/strong>in Google Webmaster Tools.<\/li><li>Do not allow\u00a0crawling of files in the \/cdn-cgi\/ directory. This path is used internally by Cloudflare and Google will encounter errors when crawling it.<\/li><\/ul><h3><strong>Troubleshooting<\/strong><\/h3><p>Troubleshooting steps for the most commonly reported errors are below. For anything not mentioned here, please follow the steps in the additional troubleshooting section and then open a\u00a0support ticket.<\/p><h3><strong>HTTP 4xx Errors<\/strong><\/h3><p><em>Google detected a significant increase in the number of URLs we were blocked from crawling due to authorization permission errors.<\/em><\/p><p>HTTP errors in the\u00a0400 range\u00a0are the most common type of error users will see in Google Webmaster Tools Health Dashboard. Errors in this range indicate a request failure. Cloudflare acts as a messenger and delivers these errors from your web server to Google. These\u00a0errors may be caused by something as simple as a missing page\u00a0on your web server or a malformed link in your HTML. The solution will depend upon the problem encountered. Once you have identified and corrected the problem, Google will make additional attempts to crawl your website and errors will clear automatically.<\/p><h3><strong>HTTP 5xx Errors<\/strong><\/h3><p>HTTP errors in the\u00a0500 range, indicate that Cloudflare or your web server experienced an internal error.\u00a0Unfortunately, the Google Webmaster Tools Health Dashboard provides a limited diagnostic view into these kinds of errors.\u00a0If your web server is producing errors in the 500 range, it can be challenging to isolate these errors from errors that Cloudflare may be producing. The best solution is to monitor the health of your web server to help\u00a0correlate dates of any outages with dates of errors.<\/p><h3><strong>Increased &#8220;Time spent downloading a page&#8221;<\/strong><\/h3><p>File a Cloudflare\u00a0Support Ticket\u00a0if you are experiencing increased page download times after enabling Cloudflare. Please include the diagnostic output that the process below will generate in your inquiry.<\/p><p>1. \u00a0 \u00a0Login to your Google Webmaster Tools account and navigate to the Health section of the affected domain.<\/p><p>2. \u00a0 \u00a0Click,\u00a0<strong>Fetch as Google<\/strong>\u00a0in the left hand navigation.<\/p><p>3. \u00a0 \u00a0Type,\u00a0<em>cdn-cgi\/trace<\/em>\u00a0in lowercase in the form field and then click\u00a0<strong>FETCH<\/strong>.<\/p><p><a class=\"image-enlarge-link\" href=\"https:\/\/s3.amazonaws.com\/cdn.freshdesk.com\/data\/helpdesk\/attachments\/production\/14047646295\/original\/zqKE1uwW2RPfeeHspunbSxG4VoZZQrvWuQ.png?1557886836\" target=\"_blank\" rel=\"noopener noreferrer\"><img decoding=\"async\" class=\"fr-fic fr-dii\" title=\"Image: \/article_attachments\/360020838511\/200169806_Screen_Shot_2013-06-19_at_12.35.49_PM.png\" src=\"https:\/\/s3.amazonaws.com\/cdn.freshdesk.com\/data\/helpdesk\/attachments\/production\/14047646295\/original\/zqKE1uwW2RPfeeHspunbSxG4VoZZQrvWuQ.png?1557886836\" alt=\"Screen_Shot_2013-06-19_at_12.35.49_PM.png\" data-filelink=\"https:\/\/s3.amazonaws.com\/cdn.freshdesk.com\/data\/helpdesk\/attachments\/production\/14047646295\/original\/zqKE1uwW2RPfeeHspunbSxG4VoZZQrvWuQ.png?1557886836\" data-fileid=\"14047646295\" data-uniquekey=\"1557886826716\" \/><\/a><\/p><p>4. \u00a0 \u00a0Click,\u00a0<strong>Success<\/strong>\u00a0under the\u00a0<strong>Fetch Status<\/strong>\u00a0column.<\/p><p><img decoding=\"async\" loading=\"lazy\" class=\"alignnone size-medium wp-image-2796\" src=\"https:\/\/www.mondoze.com\/guide\/wp-content\/uploads\/2021\/03\/dns2-300x32.png\" alt=\"\" width=\"300\" height=\"32\" \/><\/p><p>5. \u00a0 \u00a0Copy all of the text in the grey area and submit this to Cloudflare\u00a0Customer Support.<\/p><h3><strong>DNS Errors<\/strong><\/h3><p><em>Over the last 24 hours, Googlebot encountered a number\u00a0errors while attempting to retrieve DNS information for your site. The overall error rate for DNS queries for your site is\u00a0percent.<\/em><\/p><p>Most DNS failures will be handled gracefully by browsers and your visitors will not be impacted. Steps for further troubleshooting will vary depending on how your Cloudflare service is configured. Cloudflare can be enabled either through usage of a CNAME or through zone delegation. If you are using Cloudflare through a hosting provider, you are likely using a CNAME setup.<\/p><p>To verify which configuration you&#8217;re using, open a terminal and execute the following command. Replace\u00a0<em>www.example.com<\/em>\u00a0with your domain registered with Cloudflare.<\/p><pre>dig +short SOA www.example.com<\/pre><p>If you get output from the command similar to the text below, you are using a CNAME setup. Please proceed to the CNAME section.<\/p><pre>example.com.cdn.cloudflare.net.<\/pre><p>If you get output from the command similar to the text below, you are using Cloudflare through zone delegation. Please proceed to the Delegation section.<\/p><pre>josh.ns.cloudflare.com. dns.cloudflare.com. 2013050901 10000 2400 604800 3600<\/pre><h4><strong>CNAME<\/strong><\/h4><p>In a CNAME setup, your hosting provider is primarily responsible for your DNS service. In order to investigate DNS errors, you will need to contact their technical support team. Provide them with the date that Google encountered DNS errors to help with their investigation. Additionally, you should view the\u00a0Cloudflare System Status\u00a0page and observe any network outages on the date the errors were encountered by Google.<\/p><h4><strong>Delegation<\/strong><\/h4><p>At all plan levels, Cloudflare will provide at least two authoritative name server records for your domain. Browsers and operating system APIs handle failure of all but one of these nameservers gracefully without exposing an error to your visitors. If Google reports DNS errors to you often, please file a Cloudflare\u00a0Support Ticket\u00a0and provide the date and time that Google observed the errors.<\/p><h3><strong>Additional troubleshooting<\/strong><\/h3><p>Follow the following steps below to export crawler errors as a .csv file from your Google Webmaster Tools Dashboard. Include this file\u00a0when reporting errors to Cloudflare Technical Support.<\/p><p>1. \u00a0 \u00a0Login to your Google Webmaster Tools account and navigate to the Health section of the affected domain.<\/p><p>2. \u00a0 \u00a0Click,\u00a0<strong>Crawl Errors<\/strong>\u00a0in the left hand navigation.<\/p><p>3. \u00a0 \u00a0Click, <strong>Download<\/strong>\u00a0to export the list of errors as a .csv file.<br \/><img decoding=\"async\" loading=\"lazy\" class=\"alignnone size-medium wp-image-2798\" src=\"https:\/\/www.mondoze.com\/guide\/wp-content\/uploads\/2021\/03\/dns3-300x41.png\" alt=\"\" width=\"300\" height=\"41\" \/><\/p><p>4. \u00a0 \u00a0Attach the downloaded .csv file to the Cloudflare Support Ticket.<\/p>\t\t\t\t\t<\/div>\n\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t<\/div>\n\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t<\/section>\n\t\t\t\t\t\t<\/div>\n\t\t\t\t\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t","protected":false},"excerpt":{"rendered":"<p>Google Crawler Errors General guidelines Review the\u00a0following guidelines if you&#8217;re experiencing crawler errors in the Google Webmaster Tools Health dashboard. Monitor the performance and availability of your website directly to rule out any errors produced by your own web server.\u00a0There are a number of free and paid services which you can use: Pingdom StatusCake Monitor.Us &hellip;<\/p>\n<p class=\"read-more\"> <a class=\"\" href=\"https:\/\/www.mondoze.com\/guide\/kb\/im-getting-google-crawler-errors-what-should-i-do\"> <span class=\"screen-reader-text\">I&#8217;m getting Google Crawler Errors. What should I do?<\/span> Read More \u00bb<\/a><\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"closed","ping_status":"closed","template":"","meta":{"site-sidebar-layout":"default","site-content-layout":"","ast-site-content-layout":"","site-content-style":"default","site-sidebar-style":"default","ast-global-header-display":"","ast-banner-title-visibility":"","ast-main-header-display":"","ast-hfb-above-header-display":"","ast-hfb-below-header-display":"","ast-hfb-mobile-header-display":"","site-post-title":"","ast-breadcrumbs-content":"","ast-featured-img":"","footer-sml-layout":"","theme-transparent-header-meta":"","adv-header-id-meta":"","stick-header-meta":"","header-above-stick-meta":"","header-main-stick-meta":"","header-below-stick-meta":"","astra-migrate-meta-layouts":"","ast-page-background-enabled":"default","ast-page-background-meta":{"desktop":{"background-color":"","background-image":"","background-repeat":"repeat","background-position":"center center","background-size":"auto","background-attachment":"scroll","background-type":"","background-media":"","overlay-type":"","overlay-color":"","overlay-gradient":""},"tablet":{"background-color":"","background-image":"","background-repeat":"repeat","background-position":"center center","background-size":"auto","background-attachment":"scroll","background-type":"","background-media":"","overlay-type":"","overlay-color":"","overlay-gradient":""},"mobile":{"background-color":"","background-image":"","background-repeat":"repeat","background-position":"center center","background-size":"auto","background-attachment":"scroll","background-type":"","background-media":"","overlay-type":"","overlay-color":"","overlay-gradient":""}},"ast-content-background-meta":{"desktop":{"background-color":"var(--ast-global-color-5)","background-image":"","background-repeat":"repeat","background-position":"center center","background-size":"auto","background-attachment":"scroll","background-type":"","background-media":"","overlay-type":"","overlay-color":"","overlay-gradient":""},"tablet":{"background-color":"var(--ast-global-color-5)","background-image":"","background-repeat":"repeat","background-position":"center center","background-size":"auto","background-attachment":"scroll","background-type":"","background-media":"","overlay-type":"","overlay-color":"","overlay-gradient":""},"mobile":{"background-color":"var(--ast-global-color-5)","background-image":"","background-repeat":"repeat","background-position":"center center","background-size":"auto","background-attachment":"scroll","background-type":"","background-media":"","overlay-type":"","overlay-color":"","overlay-gradient":""}}},"kbtopic":[53],"kbtag":[110],"mkb_version":[],"_links":{"self":[{"href":"https:\/\/www.mondoze.com\/guide\/wp-json\/wp\/v2\/kb\/2791"}],"collection":[{"href":"https:\/\/www.mondoze.com\/guide\/wp-json\/wp\/v2\/kb"}],"about":[{"href":"https:\/\/www.mondoze.com\/guide\/wp-json\/wp\/v2\/types\/kb"}],"author":[{"embeddable":true,"href":"https:\/\/www.mondoze.com\/guide\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/www.mondoze.com\/guide\/wp-json\/wp\/v2\/comments?post=2791"}],"version-history":[{"count":11,"href":"https:\/\/www.mondoze.com\/guide\/wp-json\/wp\/v2\/kb\/2791\/revisions"}],"predecessor-version":[{"id":18314,"href":"https:\/\/www.mondoze.com\/guide\/wp-json\/wp\/v2\/kb\/2791\/revisions\/18314"}],"wp:attachment":[{"href":"https:\/\/www.mondoze.com\/guide\/wp-json\/wp\/v2\/media?parent=2791"}],"wp:term":[{"taxonomy":"kbtopic","embeddable":true,"href":"https:\/\/www.mondoze.com\/guide\/wp-json\/wp\/v2\/kbtopic?post=2791"},{"taxonomy":"kbtag","embeddable":true,"href":"https:\/\/www.mondoze.com\/guide\/wp-json\/wp\/v2\/kbtag?post=2791"},{"taxonomy":"mkb_version","embeddable":true,"href":"https:\/\/www.mondoze.com\/guide\/wp-json\/wp\/v2\/mkb_version?post=2791"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}