Stay organized with collectionsSave and categorize content based on your preferences.
February 13, 2006
We've just fixed a few issues that ourGoogle Groupmembers brought to our attention.
Capitalization in robots.txt lines
Our robots.txt analysis tool didn't correctly interpret lines that include capitalized letters.
This has been fixed. All results with URLs that include capital letters are being processed
correctly. Note that in a robots.txt file, the capitalization of the rules don't matter. For
instance,Disallow:anddisallow:are interpreted in the same way.
However, capitalization of URLsdoesmatter. So, for the following robots.txt file:
User-agent: *
Disallow: /Myfile.html
https://www.example.com/Myfile.htmlis blocked. Buthttps://www.example.com/myfile.htmlis not blocked.
Google User-agents other than Googlebot
Our robots.txt analysis tool didn't correctly process robots.txt files for Google user-agents
other than Googlebot. This has been fixed. The Google user-agents we provide analysis for are:
Googlebot-MediaPartners:crawls pages to determineAdSense content(used only
if you show AdSense ads on your site)
robots.txt files with extra initial characters
Some robots.txt files have extra characters before the start of the first rule. Some text editors
place these characters in the file, but you can't see them with the editor. When the tool
processed these characters, it reported a syntax error. The tool now mimics Googlebot's behavior
and ignores these extra characters.
Capitalization in site URLs
When you add a site, we now convert all the letters in the domain portion of the URL to lowercase,
regardless of how you entered the letters. For instance, if you enterhttps://www.Example.com/, we convert that tohttps://www.example.com/in your account. This applies only to the domain, so for
instance, if you addhttps://www.Example.com/MySite/, we will convert it tohttps://www.example.com/MySite/. If you added sites to your account using
capitalized letters, you'll notice the domain portions have been converted to lowercase. We
made this minor change as part of our efforts to ensure you see all available stats for your site.
[[["Easy to understand","easyToUnderstand","thumb-up"],["Solved my problem","solvedMyProblem","thumb-up"],["Other","otherUp","thumb-up"]],[["Missing the information I need","missingTheInformationINeed","thumb-down"],["Too complicated / too many steps","tooComplicatedTooManySteps","thumb-down"],["Out of date","outOfDate","thumb-down"],["Samples / code issue","samplesCodeIssue","thumb-down"],["Other","otherDown","thumb-down"]],[],[[["\u003cp\u003eThe robots.txt analysis tool has been updated to fix issues with capitalization in lines and site URLs, the handling of Google User-agents other than Googlebot, and files with extra initial characters.\u003c/p\u003e\n"],["\u003cp\u003eCapitalization matters for URLs specified in robots.txt, but not for robots.txt directives like "Disallow".\u003c/p\u003e\n"],["\u003cp\u003eThe domain portion of site URLs added to Google Search Console are now automatically converted to lowercase to ensure accurate statistics.\u003c/p\u003e\n"],["\u003cp\u003eGoogle Search Console supports analysis for various Googlebots, including Googlebot-Mobile, Googlebot-Image, and Googlebot-MediaPartners.\u003c/p\u003e\n"]]],["The robots.txt analysis tool was updated to correctly interpret capitalized letters in URLs and handle Google user-agents beyond Googlebot. The tool now ignores extra initial characters in robots.txt files, mimicking Googlebot's behavior. Additionally, the system now converts the domain portion of site URLs to lowercase upon addition to the account, ensuring consistent data handling and accurate statistics. The capitalization of rules in a robots.txt does not matter but for URLs it does.\n"],null,["# We've fixed a few things\n\n| It's been a while since we published this blog post. Some of the information may be outdated (for example, some images may be missing, and some links may not work anymore).\n\nFebruary 13, 2006\n\n\nWe've just fixed a few issues that our\n[Google Group](https://support.google.com/webmasters/community)\nmembers brought to our attention.\n\nCapitalization in robots.txt lines\n----------------------------------\n\n\nOur robots.txt analysis tool didn't correctly interpret lines that include capitalized letters.\nThis has been fixed. All results with URLs that include capital letters are being processed\ncorrectly. Note that in a robots.txt file, the capitalization of the rules don't matter. For\ninstance, `Disallow:` and `disallow:` are interpreted in the same way.\nHowever, capitalization of URLs *does* matter. So, for the following robots.txt file: \n\n```\nUser-agent: *\nDisallow: /Myfile.html\n```\n\n\n`https://www.example.com/Myfile.html` is blocked. But\n`https://www.example.com/myfile.html` is not blocked.\n\nGoogle User-agents other than Googlebot\n---------------------------------------\n\n\nOur robots.txt analysis tool didn't correctly process robots.txt files for Google user-agents\nother than Googlebot. This has been fixed. The Google user-agents we provide analysis for are:\n\n- `Googlebot-Mobile:` crawls pages for our [mobile index](https://mobile.google.com/mobile_search.html)\n- `Googlebot-Image:` crawls pages for our [image index](https://www.google.com/imghp?tab=wi&q=)\n- `Googlebot-MediaPartners:` crawls pages to determine [AdSense content](https://www.google.com/adsense/) (used only if you show AdSense ads on your site)\n\nrobots.txt files with extra initial characters\n----------------------------------------------\n\n\nSome robots.txt files have extra characters before the start of the first rule. Some text editors\nplace these characters in the file, but you can't see them with the editor. When the tool\nprocessed these characters, it reported a syntax error. The tool now mimics Googlebot's behavior\nand ignores these extra characters.\n\nCapitalization in site URLs\n---------------------------\n\n\nWhen you add a site, we now convert all the letters in the domain portion of the URL to lowercase,\nregardless of how you entered the letters. For instance, if you enter\n`https://www.Example.com/`, we convert that to\n`https://www.example.com/` in your account. This applies only to the domain, so for\ninstance, if you add `https://www.Example.com/MySite/`, we will convert it to\n`https://www.example.com/MySite/`. If you added sites to your account using\ncapitalized letters, you'll notice the domain portions have been converted to lowercase. We\nmade this minor change as part of our efforts to ensure you see all available stats for your site.\n\nPosted by [Vanessa Fox](https://www.vanessafox.com/)"]]
