Here is the Solution! You have Ad Crawler Errors, Which can result Lost Revenue (2021)

Working on Google AdSense, you will one day or another come across this one of the most common errors You have Ad Crawler Errors, Which can result in Lost Revenue. There are many reasons this error can be caused, so here we will talk about all the ways to solve this error. One more thing, this error can occur for any site be it a WordPress site or a Blogger site. The solutions work fine for each of the cases – as long as it is connected with Google AdSense.

No the most obvious effect of this erros is loss in revenue. But if this error is on large scale or at Site Level, then things can get worse. So better to solve this error right away.

Another common popular variation of this error can be:

“Our crawler was unable to access the pages listed in the ’Blocked URLs’ column to determine the content and display ads. Click the ’plus’ icon beside each domain to see an expanded list of affected URLs. When our crawler can’t access your content we won’t show ads, resulting in lower revenue and coverage. Follow the link in the ’Error’ column to learn how to correct these errors”.

In each of the case or other ad crawling errors, go through this post thoroughly and if you know Hindi/Urdu language, there is the complete tutorial video at the end of the post.

AdSense Ads Crawler

A crawler, also known as a spider or a bot, is the software Google uses to process and index the content of webpages. The AdSense crawler visits your site to determine its content in order to provide relevant ads.

In the words of Google, here are some important facts to know about the AdSense crawler:

• The crawler report is updated weekly. The crawl is performed automatically and we’re not able to accommodate requests for more frequent crawling.
• The AdSense crawler is different from the Google crawler. The two crawlers are separate, but they do share a cache. We do this to avoid both crawlers requesting the same pages, thereby helping publishers conserve their bandwidth. Similarly, the Search Console crawler is separate.
• Resolving AdSense crawl issues will not resolve issues with the Google crawl. Resolving the issues listed on your Crawler access page will have no impact on your placement within Google search results. For more information on your site’s ranking on Google, review our entry on getting included in Google search results.
• The crawler indexes by URL. Our crawler will access site.com and www.site.com separately. However, our crawler will not count site.com and site.com/#anchor separately.
• The crawler won’t access pages or directories prohibited by a robots.txt file. Both the Google and AdSense Mediapartners crawlers honour your robots.txt file. If your robot.txt file prohibits access to certain pages or directories, then they will not be crawled. Note that if you’re serving ads on pages that are being roboted out with the line User-agent: *, then the AdSense crawler will still crawl these pages. To prevent the AdSense crawler from accessing your pages, you need to specify User-agent: Mediapartners-Google in your robots.txt file. Learn more
• The crawler will attempt to access URLs only where our ad tags are implemented. Only pages displaying Google ads should be sending requests to our systems and being crawled.
• The crawler will attempt to access pages that redirect. When you have “original pages” that redirect to other pages, our crawler must access the original pages to determine that a redirect is in place. Therefore, our crawler’s visit to the original pages will appear in your access logs.
• Re-crawling sites At this time, we’re unable to control how often our crawlers index the content on your site. Crawling is done automatically by our bots. If you make changes to a page, it may take up to 1 or 2 weeks before the changes are reflected in our index.

Access Issues

Change in Robots.txt

First thing first, make sure that your site is fully crawlable can help you earn more revenue from your content. If the AdSense ads crawler can’t access your content, refer to the following list of crawler issues to help you fix the problem.

In this case, you’ve blocked the ads crawler in your robots.txt file. All you need to do is to check whether the ads crawler disallowed in your robots.txt file? If yes, then give access to the ads crawler in your robots.txt file. There is information about a robots.txt file but we do not have a custom one and have never had this issue. 

If you’ve modified your site’s robots.txt file to disallow the AdSense crawler from indexing your pages, then Google is not able to serve Google ads on these pages.

To update your robots.txt file to grant Google crawler access to your pages, remove the following two lines of text from your robots.txt file:

User-agent: Mediapartners-Google
Disallow: /

This change will allow Google crawler to index the content of your site and provide you with Google ads. Please note that any changes you make to your robots.txt file may not be reflected in its index until its crawlers attempt to visit your site again.

Your site is behind a login.

It’s possible that the ads crawler is being redirected to a login page, which means it can’t crawl your content. So, do visitors need login details to access your content? If yes, set up a crawler login to display ads on login-protected pages.

After your account has been activated, you can display Google ads on pages behind a login by creating a crawler login. This will give the AdSense ads crawler access to visit your pages and deliver ads.

To do so, you’ll need to provide login details for your site, and verify your site using Search Console. Please follow these steps.

1. Sign in to your AdSense account.
2. Click Account.
3. Click Access and authorization, then click Crawler access.
4. In the “Crawler access” section, click Add login details.
5. In the “Login information” section, provide the login details that our crawler will use to access your site:
   • Restricted directory or URL: The URL that is blocked from our crawler’s access.
   • Login URL: The URL that our crawler should visit to sign in.
   • Login method: The method your site uses to authenticate users who log in. POST and GET are common authentication methods.
   • Login parameters: The credentials that our crawler will use to sign in (for example, your site’s username and password). For GET and POST methods, please enter the URL parameter key-value pairs such that your server will return a cookie for logged-in access.
6. Click Add login.
7. Sign in to Search Console using your Google Account. You must be using your Google Account in order for any changes you make in the Search Console to be reflected in your AdSense account.
8. Add your site to your Search Console account by following these instructions. Make sure you add the site that you want to allow our crawler to access. After you’ve verified your site, the login details you’ve provided should allow the AdSense crawler to reach your pages and provide ads.

Your content is behind a restricted network or IP range.

So, first, you need to check have you put restrictions in place that limit the geographies or IP ranges that can access your content? If yes, then consider removing these restrictions or making your content publicly accessible, so that the ads crawler can crawl your URLs.

WordPress / Blogger Site Access Issues

Your site can’t be found.

If the URL sent to Google points to a page that doesn’t exist (or no longer exists) on a site or results in a 404 (Not Found) error, the ads crawler won’t successfully crawl your content.

So, first you need to check Is your site up and running? Can visitors access your site on the web? If no, then make sure your site is published and live on the web. Since, there can be many reasons for a site can’t be found so the solutions vary accordingly.

Your site has freshly published content.

When you publish a new page, you may make ad requests before Google’s crawlers have had a chance to crawl your content.

If you’re facing issues with any of your recently published content, then allow more time for the ads crawler to crawl your new content. Usually, after the ad request is made on a new URL, the content will get crawled in some time. However, during these initial few minutes, because your content has not yet been crawled, you may experience low ad volume. You can go to your Search Console and using URL Inspection tool index your site right away.

Your site uses multiple redirects.

If your site uses redirects, there’s a risk that the ads crawler might have issues following through them. For example, if you have multiple redirects and intermediate redirects fail, or if important parameters such as cookies get dropped during redirection, this could decrease the quality of crawling.

You need to think does your site use lots of redirects? Have you set up redirects on pages that you’re showing ads? You need to consider minimizing the use of redirects on pages with ad code. Make sure all your redirects are implemented properly.

You’re using personalized page URLs.

Some sites include extra parameters in their URLs that indicate the user who is logged in (e.g., a Session ID), or other information that may be unique to each visit. When this happens, the ads crawler may treat the URL as a new page, even if the content is the same. This could result in a few minutes lag time between the first ad request on the page and when the page gets crawled, as well as an increase in the crawler load on your servers.

Generally, if the content on a page doesn’t change, consider removing the parameters from the URL and persisting that information another way. Having a simpler URL structure can help make your site more easily crawlable.

Possibly, it could be something that is only temporarily or AdSense has a problem with URLs not using Latin letters or emojis.

Issues with Your Hosting

An issue with your site’s name server.

Is your name server set up correctly? Do you have any restrictions on where requests can come from?

Make sure the name server for your domain or subdomain is properly directing the ads crawler to your content. For this purpose, you can contact your hosting providing company immediately to get this problem resolved. If you have a good site with some good traffic and revenue, then consider leaving the shared hosting and moving to cloud hosting.

There’s an issue with your site’s server.

Sometimes when the ads crawler tries to access site content, the site’s server is unable to respond in time. This can happen because the server is down, slow or overloaded by requests.

We recommend ensuring your site is hosted on a reliable server or by a reliable service provider.

So, let me know in comments below which method worked for you and what type of error you’re getting. Besides, keep it in mind that once you’ve done with any of the steps, it takes some time (when your site is crawled) then the error will be removed from your AdSense Account. So, be patient till that time.

Source Google