Troubleshoot uncrawlable landing pages in a Dynamic Ad Group

Step 1: Locate the uncrawlable URL

Uncrawlable pages can affect your Dynamic Search Ads in 3 ways:

• If you select URL-Equals as your targeting type and your URL is uncrawlable, you'll see the status “Disapproved: Page cannot be crawled” on your dynamic ad target.
• If you use page feeds and a URL in your feed is uncrawlable, you'll see the status “Disapproved: Page cannot be crawled” on the row for that URL in the shared library.
• If you use any other targeting mechanism and a URL isn't or cannot be crawled, Google Ads won’t know about it, so no error message will be displayed. If a URL you’re attempting to target isn't getting traffic, you can troubleshoot by first targeting it through URL-Equals or page feeds. The URL will either get traffic or you'll get an error explaining why it isn’t getting traffic.

Step 2: Make sure that Google AdsBot can crawl the landing page

1. Copy the URL and paste it in a web browser's address bar.
2. Add /robots.txt to the end of the domain name, then press Enter. For example, if your landing page is http://www.example.com/folder1/folder2/ , check what robots are allowed to crawl the domain by visiting the URL: http://www.example.com/robots.txt .
3. Look for the line User-agent: AdsBot-Google .
   • If it's not there, work with the web developer to add it.
   • If the page is blank, work with the web developer to add a robots.txt file to the domain.
   • If User-agent: AdsBot-Google is there, make sure it isn't followed by a line that says Disallow: / . Make sure AdsBot-Google isn't restricted by another line in the file. Learn more About robots.txt files and Google crawlers .
4. If the landing page is a subdomain, check if the subdomain has a separate robots.txt file. For example, https://subdomain.domain.com/robots.txt . Make sure this robots.txt also allows User-agent: AdsBot-Google .
5. Open the source of the URL. In Chrome, you can do this by opening view-source:THE_URL . For example, if your landing page is http://www.example.com/folder1/folder2/ , you can check what robots are allowed to crawl the domain by navigating to view-source:http://www.example.com/folder1/folder2/ .
6. Look for the string “ AdsBot-Google ”.
   • If it’s not there, this is not the source of the issue.
   • If the string is within a tag like <meta name="AdsBot-Google" content="noindex" /> , work with your web developer to remove it. Learn more about Meta tags that Google understands .

Other reasons that a landing page can't be crawled

If the Google AdsBot is allowed to crawl the landing page ( User-agent: AdsBot-Google is already included in the site's robots.txt file), there may be another reason the page can't be crawled:

• The landing page isn't loading. If you open the page with a web browser and see an error (for example, 404 Not Found, Site not reachable, Your session has expired), contact the site's web developer to make sure the landing page loads in any location, environment (mobile and desktop), or browser. Also, make sure the landing page can be crawled by Google AdsBot on both desktop and mobile devices.
• The landing page requires sign-in. All ad landing pages must be publicly accessible. Use another landing page or work with the web developer to remove the sign-in requirement from the page.
• The landing page has too many forwards or redirects. Landing pages must have fewer than 10 redirects. Work with web developer to reduce the number of redirects.