On this page
Why this issue is happening
Some of your products specify a landing page (via the link
[link]
attribute) that cannot be crawled by Google. This can happen if robots.txt forbids Google's crawler to download the landing page, or if the landing page returns an HTTP error when crawled. These products will remain disapproved and stop showing up in your Shopping ads and free product listings until we are able to crawl the landing page.
How to fix the issue
Update the robots.txt file on your web server to allow Google's crawler to fetch the provided landing pages. The robots.txt file can usually be found in the root directory of the web server (for example, http://www.example.com/robots.txt).
In order for us to access your whole site, ensure that your robots.txt file allows both user-agents 'Googlebot' (used for landing pages) and 'Googlebot-image' (used for images) to crawl your full site.
You can allow a full-site crawl by changing your robots.txt file as follows:
User-agent: Googlebot
Disallow:
User-agent: Googlebot-image
Disallow:
You can learn more about how to configure robots.txt here . You can test your current configuration with the URL Inspection tool .
Fix products one by one directly in Merchant Center
Go to Products
in Merchant Center, then select the Needs attentiontab.
Use the filtericon
above the table to find all products affected by your issue.
Click on the title of the product you want to fix, then click Edit product.
Update the robots.txt file on your web server to allow Google's crawler to fetch the provided landing pages for affected products.
Select Save.
Fix multiple products at once
Go to Products
in Merchant Center, then select the Needs attentiontab.
Use the filtericon
above the table to find all products affected by your issue.
Select the downloadicon
to download a list (.csv) of all affected products for all issues or this particular issue.
Cross reference the downloaded list with your uploaded product data, and update the robots.txt file on your web server to allow Google's crawler to fetch the provided landing pages for affected products.
Reupload your corrected product data to Merchant Center using your chosen upload method.
Next steps
After making the requested changes, check that you’ve fixed the issue by making sure the issue is no longer listed on the “ Needs attention ” page.
Keep in mind: It may take 24-72 hours for your change to be reflected on the “ Needs attention ” page.
If you have fixed these issues and updated your products via a new feed upload or the Content API, the errors you see here should disappear within a couple of days. This time allows us to verify that we can crawl the landing pages that are provided, after which the products will start showing up in your Shopping ads and listings again. If you want to speed up the process you can increase Google's crawl rate .
