Search Console

    List of Google's common crawlers

    Google's common crawlers are used to find information for building Google's search indexes, perform other product specific crawls, and for analysis. They always obey robots.txt rules when crawling automatically. The general technical properties of Google's crawlers also apply to the common crawlers.

    The common crawlers generally crawl from the IP ranges published in the googlebot.json object, and the reverse DNS mask of their hostname matches crawl-***-***-***-***.googlebot.com or geo-crawl-***-***-***-***.geo.googlebot.com .

    The following list shows the common crawlers, their user agent strings as they appear in the HTTP requests, their user agent tokens for the User-agent: line in robots.txt, and the products that are affected by crawl preferences for the crawler. Some crawlers have more than one user agent token; you need to match only one crawler token for a rule to apply. The list is not exhaustive, it only covers the requestors that are more likely to show up in log files and that we've received questions about.

    User-Agent in HTTP requests
    Googlebot Smartphone
    Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/ W.X.Y.Z 
 
Mobile Safari/537.36 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)
    Googlebot Desktop
    Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; Googlebot/2.1; +http://www.google.com/bot.html) Chrome/ W.X.Y.Z 
 
Safari/537.36

    Rarely:

    • Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)
    • Googlebot/2.1 (+http://www.google.com/bot.html)
    robots.txt
    User-agent token in robots.txt Googlebot
    Example robots.txt group 
    user-agent: Googlebotallow: /archive/1Q84
disallow: /archive
    Affected products
    Crawling preferences addressed to the Googlebot user agent affect Google Search (including Discover and all Google Search features), as well as other products such as Google Images, Google Video, Google News, and Discover.
    User-Agent in HTTP requests 
    Googlebot-Image/1.0
    robots.txt
    User-agent token in robots.txt Googlebot-Image
    Googlebot
    Example robots.txt group 
    user-agent: Googlebot-Imageallow: /archive/1Q84
disallow: /archive/moons.jpg
    Affected products
    Crawling preferences addressed to the Googlebot-Image user agent affect Google Images, Discover, Google Video, and all features in Google Search where images, logos, and favicons are presented.
    User-Agent in HTTP requests 
    Googlebot-Video/1.0
    robots.txt
    User-agent token in robots.txt Googlebot-Video
    Googlebot
    Example robots.txt group 
    user-agent: Googlebot-Videoallow: /archive/1Q84
disallow: /archive/
    Affected products
    Crawling preferences addressed to the Googlebot-Video user agent affect video-related Google Search features and other products dependent on videos.
    User-Agent in HTTP requests
    Googlebot-news doesn't have a separate HTTP request user agent string. Crawling is done with various Googlebot user agent strings .
    robots.txt
    User-agent token in robots.txt Googlebot-News
    Googlebot
    Example robots.txt group 
    user-agent: Googlebot-Newsallow: /archive/1Q84
disallow: /archive/
    Affected products
    Crawling preferences addressed to the Googlebot-News user agent affect the Google News product, including news.google.com and the Google News app.
    User-Agent in HTTP requests
    Desktop agent 
    Mozilla/5.0 (X11; Linux x86_64; Storebot-Google/1.0) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/ W.X.Y.Z 
 
Safari/537.36
    Mobile agent 
    Mozilla/5.0 (Linux; Android 8.0; Pixel 2 Build/OPD3.170816.012; Storebot-Google/1.0) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/ W.X.Y.Z 
 
Mobile Safari/537.36
    robots.txt
    User-agent token in robots.txt Storebot-Google
    Example robots.txt group 
    user-agent: Storebot-Googleallow: /archive/1Q84
disallow: /archive/konbini
    Affected products
    Crawling preferences addressed to the Storebot-Google user agent affect all surfaces of Google Shopping (for example, the Shopping tab in Google Search and Google Shopping ).
    User-Agent in HTTP requests
    Desktop agent 
    Mozilla/5.0 (compatible; Google-InspectionTool/1.0;)
    Mobile agent 
    Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/ W.X.Y.Z 
 
Mobile Safari/537.36 (compatible; Google-InspectionTool/1.0;)
    robots.txt
    User-agent token in robots.txt Google-InspectionTool
    Googlebot
    Example robots.txt group 
    user-agent: Google-InspectionToolallow: /archive/1Q84
disallow: /archive/
    Affected products
    Crawling preferences addressed to the Google-InspectionTool user agent affect Search testing tools such as the Rich Result Test and URL inspection in Search Console. It has no effect on Google Search or other products.
    User-Agent in HTTP requests 
    Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/ W.X.Y.Z 
 
Mobile Safari/537.36 (compatible; GoogleOther)
     
    Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; GoogleOther) Chrome/ W.X.Y.Z 
 
Safari/537.36
    robots.txt
    User-agent token in robots.txt GoogleOther
    Example robots.txt group 
    user-agent: GoogleOtherallow: /archive/1Q84
disallow: /archive/
    Affected products
    Crawling preferences addressed to the GoogleOther user agent don't affect any specific product. GoogleOther is the generic crawler that may be used by various product teams for fetching publicly accessible content from sites. For example, it may be used for one-off crawls for internal research and development. It has no effect on Google Search or other products.
    User-Agent in HTTP requests 
    GoogleOther-Image/1.0
    robots.txt
    User-agent token in robots.txt GoogleOther-Image
    GoogleOther
    Example robots.txt group 
    user-agent: GoogleOther-Imageallow: /archive/1Q84
disallow: /archive/moon.jpg
    Affected products
    Crawling preferences addressed to the GoogleOther-Image user agent don't affect any specific product, similar to GoogleOther. GoogleOther-Image is the version of GoogleOther optimized for fetching publicly accessible image URLs.
    User-Agent in HTTP requests 
    GoogleOther-Video/1.0
    robots.txt
    User-agent token in robots.txt GoogleOther-Video
    GoogleOther
    Example robots.txt group 
    user-agent: GoogleOther-Videoallow: /archive/1Q84
disallow: /archive
    Affected products
    Crawling preferences addressed to the GoogleOther-Video user agent don't affect any specific product, similar to GoogleOther. GoogleOther-Video is the version of GoogleOther optimized for fetching publicly accessible video URLs.
    User-Agent substring in HTTP requests 
    Google-CloudVertexBot
    robots.txt
    User-agent token in robots.txt Google-CloudVertexBot
    Googlebot
    Example robots.txt group 
    user-agent: Google-CloudVertexBotallow: /archive/1Q84
disallow: /archive/
    Affected products
    Crawling preferences addressed to the Google-CloudVertexBot user agent affect crawls requested by the site owners' for building Vertex AI Agents . It has no effect on Google Search or other products.
    User-Agent in HTTP requests
    Google-Extended doesn't have a separate HTTP request user agent string. Crawling is done with existing Google user agent strings; the robots.txt user-agent token is used in a control capacity.
    robots.txt
    User-agent token in robots.txt Google-Extended
    Example robots.txt group 
    user-agent: Google-Extendedallow: /archive/1Q84
disallow: /archive/
    Affected products

    Google-Extended is a standalone product token that web publishers can use to manage whether content Google crawls from their sites may be used for training future generations of Gemini models that power Gemini Apps and Vertex AI API for Gemini and for grounding (providing content from the Google Search index to the model at prompt time to improve factuality and relevancy) in Gemini Apps and Grounding with Google Search on Vertex AI .

    Google-Extended does not impact a site's inclusion in Google Search nor is it used as a ranking signal in Google Search.

    A note about Chrome/ W.X.Y.Z in user agents

    The string Chrome/ W.X.Y.Z in the user agent strings in the list is a placeholder that represents the version of the Chrome browser used by that user agent: for example, 41.0.2272.96 . This version number increases over time to match the latest Chromium release version used by Googlebot .

    If you are searching your logs or filtering your server for a user agent with this pattern, use wildcards for the version number rather than specifying an exact version number.
    Create a Mobile Website
    View Site in Mobile | Classic
    Share by: