Unicode was invented to solve that problem: to encode all human languages, from Chinese (ä¸æ–‡) to Russian (руÑÑкий) to Arabic (العربية), and even emoji symbols like
or 
;
it encodes nearly 75,000 Chinese ideographs alone. In the ASCII
encoding, there wasn’t even enough room for all the English punctuation
(like curly quotes), while Unicode has room for over a million
characters. Unicode was first published in 1991, coincidentally the year
the World Wide Web debuted—little did anyone realize at the time they
would be so important for each other. Today, people can easily share
documents on the web, no matter what their language. Every January, we look at the percentage of the webpages in our index that are in different encodings. Here’s what our data looks like with the latest figures*:
*Your mileage may vary: these figures
may vary somewhat from what other search engines find. The graph lumps
together encodings by script. We detect the encoding for each webpage;
the ASCII pages just contain ASCII characters, for example. Thanks again
to Erik van der Poel for collecting the data.
As you can see, Unicode has experienced an 800 percent increase in “market share†since 2006. Note that we separate out ASCII (~16 percent) since it is a subset of most other encodings. When you include ASCII, nearly 80 percent of web documents are in Unicode (UTF-8). The more documents that are in Unicode, the less likely you will see mangled characters (what Japanese call mojibake) when you’re surfing the web.
We’ve long used Unicode as the internal format for all the text Google searches and process: any other encoding is first converted to Unicode. Version 6.1 just released with over 110,000 characters; soon we’ll be updating to that version and to Unicode’s locale data from CLDR 21 (both via ICU). The continued rise in use of Unicode makes it even easier to do the processing for the many languages that we cover. Without it, our unified index it would be nearly impossible—it’d be a bit like not being able to convert between the hundreds of currencies in the world; commerce would be, well, difficult. Thanks to Unicode, Google is able to help people find information in almost any language.
Source:http://googleblog.blogspot.com/2012/02/unicode-over-60-percent-of-web.html
