TL;DR<\/b> If you use language other than English, you pay more per word sent\/received from an LLM. If you don’t use Latin alphabet, you pay way more.<\/p>\n
Here’s why. LLMs don’t charge by word, they charge by token<\/i> received and returned. Tokens are generated statistically, based on the frequency of byte combinations in the training material (explanation link<\/a>).<\/p>\n Tokenizers use UTF-8, and it produces shorter sequences for Latin alphabet. Non-Latin characters are encoded as 2 or even 3 bytes, so for languages like Hebrew and Japanese you may get more tokens than characters<\/b>. In fact, transliterating Russian or Hebrew into Latin alphabet reduces the number of tokens by the factor of 2 in my small example. I used Huggingface tokenizer playground<\/a> with GPT-4 setting. Claude produces different results, but the overall picture is similar.<\/p>\n The cheapest language to use is English: it uses Latin alphabet, its words rarely change, and the majority of the training material is in English. Second go Latin-based languages like Spanish, Icelandic and Polish with tokenization efficiency of about 1\/2 of English. Tokenization efficiency for non-Latin based languages like Hebrew and Russian is ~1\/3 of English. Chinese and Japanese perform approximately like Polish and Icelandic at 1\/2 of English: on one hand they use lots of bytes per character, from the other hand they use fewer characters per sentence, since they are not phonetic. Hebrew shows the worst character-to-token ratio, probably because not a lot of training material is in it. Of course, this is based on a very small selection, and the numbers of large texts may be different, but I assume the general trends will not change.<\/p>\n <\/p>\n","protected":false},"excerpt":{"rendered":"\n\n
\n \nLanguage<\/th>\n Sentence<\/th>\n Characters<\/th>\n UTF8 bytes<\/th>\n Tokens<\/th>\n<\/tr>\n<\/thead>\n \n English<\/td>\n I met a huge dog<\/td>\n 16<\/td>\n 16<\/td>\n 5<\/td>\n<\/tr>\n \n Spanish<\/td>\n Conoc\u00ed a un perro enorme<\/td>\n 24<\/td>\n 25<\/td>\n 8<\/td>\n<\/tr>\n \n Icelandic<\/td>\n \u00c9g hitti risast\u00f3ran hund<\/td>\n 24<\/td>\n 26<\/td>\n 10<\/td>\n<\/tr>\n \n Polish<\/td>\n Spotka\u0142em ogromnego psa<\/td>\n 23<\/td>\n 24<\/td>\n 8<\/td>\n<\/tr>\n \n \n Russian<\/td>\n \u042f \u0432\u0441\u0442\u0440\u0435\u0442\u0438\u043b \u043e\u0433\u0440\u043e\u043c\u043d\u0443\u044e \u0441\u043e\u0431\u0430\u043a\u0443<\/td>\n 26<\/td>\n 49<\/td>\n 14<\/td>\n<\/tr>\n \n Russian Transliterated<\/td>\n Ya vstretil ogromnuyu sobaku<\/td>\n 28<\/td>\n 28<\/td>\n 11<\/td>\n<\/tr>\n \n Hebrew<\/td>\n \u05e4\u05d2\u05e9\u05ea\u05d9 \u05db\u05dc\u05d1 \u05e2\u05e0\u05e7<\/td>\n 13<\/td>\n 24<\/td>\n 16<\/td>\n<\/tr>\n \n Hebrew Transliterated<\/td>\n pgSti klv 3nq<\/td>\n 13<\/td>\n 13<\/td>\n 9<\/td>\n<\/tr>\n \n Japanese<\/td>\n \u5927\u304d\u306a\u72ac\u306b\u51fa\u4f1a\u3063\u305f<\/td>\n 9<\/td>\n 27<\/td>\n 11<\/td>\n<\/tr>\n \n Chinese<\/td>\n \u6211\u9047\u89c1\u4e86\u4e00\u53ea\u5927\u72d7<\/td>\n 8<\/td>\n 24<\/td>\n 11<\/td>\n<\/tbody>\n<\/table>\n