Controlling crawling and indexing now documented on code.google.com
Stay organized with collectionsSave and categorize content based on your preferences.
Wednesday, November 24, 2010
Do you know how Google's crawler, Googlebot, handles conflicting rules in your robots.txt
file? Do you know how to prevent a PDF file from being indexed? Do you know Googlebot's favorite
song? The answers to these questions (except for the last one :)), along with lots of other
information about controlling the crawling and indexing of your site, are now available oncode.google.com:
Now site owners have a comprehensive resource where they can learn about robots.txt files,robotsmetatags, andX-Robots-TagHTTP header rules. Please share your
comments, and if you have questions you can post them in ourWebmaster Help Forum.
[[["Easy to understand","easyToUnderstand","thumb-up"],["Solved my problem","solvedMyProblem","thumb-up"],["Other","otherUp","thumb-up"]],[["Missing the information I need","missingTheInformationINeed","thumb-down"],["Too complicated / too many steps","tooComplicatedTooManySteps","thumb-down"],["Out of date","outOfDate","thumb-down"],["Samples / code issue","samplesCodeIssue","thumb-down"],["Other","otherDown","thumb-down"]],[],[[["\u003cp\u003eGoogle has launched a comprehensive resource on \u003ccode\u003ecode.google.com\u003c/code\u003e for controlling how Google crawls and indexes websites.\u003c/p\u003e\n"],["\u003cp\u003eThis resource provides information on robots.txt, robots meta tags, and X-Robots-Tag for managing website visibility in search results.\u003c/p\u003e\n"],["\u003cp\u003eSite owners can learn how Googlebot handles conflicting rules, prevent specific file types from being indexed, and more.\u003c/p\u003e\n"],["\u003cp\u003eFor support, website owners can visit the Webmaster Help Forum to ask questions and share feedback.\u003c/p\u003e\n"]]],["Google launched a resource on `code.google.com` detailing how to manage website crawling and indexing. This resource explains robots.txt files, robots meta tags, and X-Robots-Tag HTTP header rules. Site owners can now learn how Googlebot handles conflicting rules and how to prevent PDF indexing. Questions and comments can be shared via the Webmaster Help Forum. The resource is designed to be a complete guide for site owners to control how their site is crawled and indexed.\n"],null,["# Controlling crawling and indexing now documented on code.google.com\n\nWednesday, November 24, 2010\n\n\nDo you know how Google's crawler, Googlebot, handles conflicting rules in your robots.txt\nfile? Do you know how to prevent a PDF file from being indexed? Do you know Googlebot's favorite\nsong? The answers to these questions (except for the last one :)), along with lots of other\ninformation about controlling the crawling and indexing of your site, are now available on\n`code.google.com`:\n\n[Controlling crawling and indexing](/search/docs/crawling-indexing/robots/robots_txt)\n\n\nNow site owners have a comprehensive resource where they can learn about robots.txt files,\nrobots `meta` tags, and `X-Robots-Tag` HTTP header rules. Please share your\ncomments, and if you have questions you can post them in our\n[Webmaster Help Forum](https://support.google.com/webmasters/community).\n\n\nPosted by\n[Jonathan Simon](/search/blog/authors/jonathan-simon),\nWebmaster Trends Analyst"]]
