What's the world's most efficient language?
Edit: This has now made it to the front page of HackerNews. I'm sure a lot of valid criticism will come from it. Do note that I'm more curious in the discussion than the experiment itself (I'm already learning from the comments). I consistently claim here to not be qualified for an analysis, having done this for fun. The "Deriving information" section ends with "...the only thing I'm really measuring is efficiency from the perspective of printer ink. But so be it, I'll measure that.". Also check out Limitations.
A while ago I was sitting on a plane and in a moment of boredom picked up the in-flight magazine.
The magazine had a little travel article, and it was written in English on one page and Thai on the other.
The Thai version was so much shorter that I started to wonder if it was more efficient. In other words, was it able to convey the same exact meaning to the reader with fewer "resources" than the English version?
The topic came up again when I was speaking to a Japanese man at a language exchange meetup. He said:
"You can technically write Japanese without Kanji, but it is a lot less efficient."
So what is language efficiency? And how can we measure it?
Thoughts from a nobody
One can't find a lot resources about "language efficiency" very easily, and the majority of my findings related to spoken language. Here are two good reads on this topic for those interested:
But I was interested in written language, simply because something in me told me I could quantify it, without any formal linguistics knowledge.
As you'd imagine, quantifying the efficiency of a language is a complicated task, one that I'm not at all qualified to explore in a scientifically significant way.
But I thought I'd do a little experiment.
Could I gather the same text in various languages that I'm certain (as certain as you can be) conveys the exact the meaning, and then calculate from the language snippets how much information they contain?
If I could do that, I could arrive at the following measure of efficiency:
meaning / amount of information.
Meaning is supposed to be a constant, thus the more information the body of text contains, the lower its "meaning per piece of information" ratio, making the language less efficient.
In essence, if we take information to be
information = noise + signal, we're looking for the signal density of languages, or, conversely, the noise ratio - how much stuff is in there that doesn't need to be?
So how could I derive that information value?
When brainstorming about this, a few immediate indicators might come to mind, like total number of characters.
Characters, however, are not very uniform - they vary widely within and across languages. Consider:
|我爱咖啡||Minä rakastan kahvia|
Both sentences say "I love coffee".
If we're counting characters, Mandarin blows Finnish out of the water in efficiency.
But most people looking at this will immediately notice that Mandarin characters are much more condensed. They make up for in "complexity" what they sentence is missing in length.
And that "complexity" is what we're looking to measure.
Imagine I gave you a task to try and find a pattern in the following 2 different images. The task is "done" when you either find a pattern or decide you can't come up with one.
Which would take your brain more time to conclude - A or B?
I believe the answer would be B for most of us.
B has more information, more data - in this case, non-background colored "pixels" - that our brain needs to process before it makes sense of what it is seeing.
The same should hold true of characters. When reading, in order for our brain to determine the meaning of the character it is seeing (if it even knows it), it needs to take in all the pixels making up that character and run them across its "database" of known patterns to try and find a hit.
So there's my information metric: pixels.
To derive the amount of information present in a snippet of text, we can count the total "used" pixels i.e. on a black-on-white representation, count the black pixels.
Now, at this point I will mention once again that a true analysis of efficiency should be much more nuanced, and it's not something I'd be able to undertake.
In fact, a friend, upon hearing about this idea, said the only thing I'm really measuring is efficiency from the perspective of printer ink. But so be it, I'll measure that.
This post is bound to be long, so I'll spare many of the details here. But essentially, in order to test this out, I did the following:
1: Selected a snippet of text that I would be likely to find good translations for in various languages
But you must keep in mind the name of this blog: "Sunday Afternoon". Stuff contained here is often done over a single weekend just for the fun of it, as was the case with this, so I needed to keep things simple.
- I could easily scrape it in hundreds of languages
- It needs to convey the exact same meaning in all languages
- It has legal implications, meaning if Google puts it up online in a language, it must have been thoroughly checked
- Google probably knows a thing or two about translations
The exact snippet I picked was:
And I verified it said exactly this in English, Portuguese, Spanish, Finnish, German, and Icelandic (the last 2 with external help). That is, by actually reading it, other languages checked out on Google Translate.
2: Pull and parse the data
I then found the desired paragraph and extracted it into a separate file for each language.
For the more technical readers, the paragraph's CSS selector was consistent across all languages, which is how I managed to extract it. It's easy to confirm you extracted the right thing by popping the snippet into a translator.
3: Map out how many pixels each character takes
Once I had all the clean data, I iterated over every character in the dataset and drew an image for each using Python's Pillow library.
From that image I could then count the total number of black pixels and generate a map of the results.
Here are the basics of how this works:
# Python arial_unicode = ImageFont.truetype('/Library/Fonts/Arial Unicode.ttf', 60) img = Image.new('RGB', (200, 200), 'white') draw = ImageDraw.Draw(img) draw.text((75,0), letter, font=arial_unicode, fill='#000000') pixels = list(img.getdata()) total_black_pixels = len(list(filter(lambda rgb: sum(rgb) == 0, pixels)))
4: Build up the results
Having determined the black pixel value (information) for each character, I could then derive how much information (again, in my limited definition), each language's written representation was using to convey the same meaning.
Some manual intervention here was needed, and I ended up looking through every picture of a character that the script generated to make sure it was valid. Two key things here were removing from the results the languages for which generic squares drawn when the font didn't some or all of its characters (e.g. Amharic), as well as making sure the drawings were containing the full character.
The most efficient language prize in my little child experiment was Gujarati, followed by Hebrew, and then Arabic. Gujarati and Hebrew also had some of the lowest mean pixel/character ratio in the dataset.
The least efficient ones were Japanese, Malay, and Canadian French. You heard that right. Out of all the French dialects included in the dataset, Canadian was the only one with different wording. I'd be curious to hear from someone who speaks French about whether the Canadian version has words that are actually not used elsewhere or if it's just a matter of choice of words.
English, by the way, was eighth on the list.
Particularly interesting to me was comparing results from language/dialects in the same language group, such as the French from France vs. France from Canada comparison mentioned above. Here are a few takeaways from the results:
American English > British English
American English performed slightly better than British English, and this one actually makes intuitive sense to me.
In most cases when there's a variation between the two, British English tends to be the one with additional letters. Think "color" vs. "colour", "traveled" vs. "travelled", etc.
These additional letters are almost decidedly redundant, inefficient. Given that there is a variation of the language without these extra letters, one can infer that they are not very important if the objective is to convey meaning efficiently.
r in cart for instance. Without that
r the word would clash with an existing word - cat, so the letter is significant in establishing meaning. The
u in color is not, however.
The Chinese Language Group
This was a surprising one to me. Simplified Mandarin Chinese was expectedly more efficient than Traditional Mandarin Chinese, but both were beaten out by Cantonese using traditional characters.
Also, if you ever wanted to put a number to the complexity of Chinese characters, they are around 3x more "complex" (pixels/char ratio) than the average of the dataset.
Portuguese from Brazil > Portuguese from Portugal (to my utter delight)
When reading both sentences one can indeed notice inherent language differences, like the lack of the word "você" in Portuguese from Portugal. However, in some cases the differences were merely a word choice thing (e.g. "Entendemos" vs. "Compreendemos").
Spanish from Spain > Spanish from Latin America
The biggest disparity within a language group happened with Spanish, with Spanish from Spain ranking at 14 and Latin American Spanish landing at 32. The vast majority of differences are purely arbitrary word selections, though.
Dialects were collapsed if they used the same exact sentence.
|Position||Language||Total pixels||Total chars||Mean pixels/char|
|8||English (United States)||82897||345||240.28|
|9||English (United Kingdom)||84774||351||241.52|
|32||Spanish (Latin America)||95226||376||253.26|
The limitations of this little Sunday experiment are many. From the size of the snippet, the lack of extensive validation, to the lack of consideration for variations in writing, the use of only one font that may bias towards certain language families, etc.
But perhaps the most interesting discussions regard how counting pixels may be limited as an approach to measuring efficiency and how we could measure it instead.
Maybe (probably) the sheer amount of "information" is not the only factor contributing to our ability to read text efficiently?
Maybe the fact that Chinese characters were originally representative drawings helps association in the brain despite the extra strokes?
Maybe spaces also play a role and are thus are also a form of information?
Maybe my whole approach to quantifying "information" is wrong?
Either way, I'd be curious to explore this topic further, and would love to hear any thoughts others may have. Feel free to send those to
yakko [dot] majuri [at] protonmail [dot] com if you like.
You can find my loose snippets of code used for this analysis on the yakkomajuri/lang GitHub repo.