The author Paul McCann (polm) is one of my favourite programmers out there!
He’s done awesome work in the Japanese NLP space over the last decade which has really helped me in my language learning projects.
He maintains a mecab (Japanese tokenizer) wrapper for Python [0], has a book on Japanese NLP written for English speakers [1] and also worked on Spacy at one point [2].
I think there’s evidence found for the origin of “彁” as the result of a poor scan of a newspaper article. Look up “彁 新聞” to find some japanese sources about this.
Ghost characters reveal something about the joint semantic and phonetic nature of most Chinese characters. I know 彁 isn't real: but it still insists on a pronunciation: ka in Japanese, (gē or gé Mandarin). And it hints at meaning: bowed weapon, sound, elder brother; but your guess is as good as mine!
The article mentions "an example of 彁 mistakenly used in a digitized Taisho newspaper due to a faded printing of 彊", but to me that implies the symbol already existed before then.
Isn't that backwards? The nonexistent character is in the -digitized- version so presumably OCR or something got 彊 wrong, that's not saying that 彁 was used in the -print- version.
Indeed, the source link is about exactly this: a crappy scan appears to have 彁 but a better copy reveals it was 彊.
I think you're both saying the same thing: the digitization of the article wouldn't have been the source, since 彁 would have had to exist before the digitization happened in order for the OCR to misread 彊 as 彁.
Most kanji are a combination of several smaller parts called "radicals" in English. If you look at these two kanji through this lens, you will see that it's actually a very simple mistake, one existing radical is replaced by another existing radical. It is very easy to imagine software that was working exactly like that: interpreting kanji as a combination of radicals rather than individual unrelated symbols
It occurs to me that we can use 彊 to mean "a completely unknown concept that cannot be named". For example if you ask, "when Cthulhu rises from its slumber, what is thefirst thing it will do? Probably it will 彊.
Hey, that's today's smbc https://www.smbc-comics.com/comic/r
The problem is, there's no good reason to think the concept of 彊 in your mind is at all similar to the concept of 彊 in my brain. After all, we (probably) don't pronounce it the same, right?
Well, vast swaths of the Kangxi dictionary (which serves as "sources" for probably most of the CJK characters) are such "ghost" characters as described in the article...
The peculiar properties of CJK characters and the philosophy (apparently the Japanese did not like Unicode's tendencies towards Aristotelian essentialism) under which they were implemented in Unicode probably singlehandedly forced unicode to expand beyond the BMP....
> apparently the Japanese did not like Unicode's tendencies towards Aristotelian essentialism
Do you mean Han Unification? I.e. that conceptually equivalent characters which are written differently in Japan and China received only a single unicode code-point, and are rendered the Chinese way by default on most computers?
Yes. As far as I understood it, the result was an incoherent mess.
I think some characters got different code points, while others were merged. And apparently the Japanese complained bitterly over the ones that were merged. If you had read any articles about that, this is probably what you have in mind right now.
And I'm also a bit tilted by the ones that had different code points, because when processing CJK text now we have to deal with characters that are (in my native Cantonese) essentially the same, looks similar (to my undiscriminating eyes), yet having different code points so that things like text search sometimes don't work.
Of course I'm not "blaming" the Japanese, if anything the simplified vs traditional Chinese thing is much more of a practical problem, and the conflicting code points I deal with on a routine basis are more of a Hong Kong vs Taiwan thing, but I was told that the Unified CJK thing adopted a different "philosophy" from the rest of Unicode (which I think really is some kind of Aristotelian essentialism...) mostly due to vocal objections from the Japanese.
Han-unification was the result of Asian scholars, in particular the Ideaographic Research Group [1]
I have no idea what "Aristotelian essentialism" is supposed to mean, or if you are saying that the unification was that.
> And apparently the Japanese complained bitterly over the ones that were merged.
and
> but I was told that the Unified CJK thing adopted a different "philosophy" from the rest of Unicode [...] mostly due to vocal objections from the Japanese.
I think the Aristotelian (though I might have reached for Platonic) idea is that there is an "A" which is essential, and that you map many various "A"s to that one A - which results in weirdness when you leave the Latin-alphabet based languages.
(Arguably even in English we run into the fact that there is not one "A" - there are many "A"s that sound and act completely different, and we collapsed them into one representation.)
Many times I have asked my Hungarian wife whether a sound is "a" or "a with an accent" (A vs Á), but they are different letters in Hungarian, not just variations of one as they are in Italian. We made a mess with "accent unification".
Which is always fun when you think that clearly nobody thinks of "o or o with a leg" (O vs Q).
Sorry, probably not very standard terminology. I somehow picked it up from the OReilly book: Fonts & Encodings: From Advanced Typography to Unicode and Everything in Between
Page 55:
""" there are characters with no glyphs. glyphs that can correspond to a number of different characters according to context. Glyphs that correspond to multiple characters at the same time (with weightings assigned to each), and even more possibilities.
The problem of glyphs and characters is so complex that it has gone beyond the realm of computer specialists and has come to be of interest even to philosophers. For example, the Japanese philosopher Shigeki Moro, who has worked with ideographic characters in Buddhist documents, goes so far in his article Surface or Essence: Beyond Character Model Set [274] as to say that Unicode's approach is Aristotelian essentialist and to recommend supplanting it by an approach inspired by Jacques Derrida's theory of writing [114, 115]. The reader interested in the philosophical aspects of the issue is invited to consult [165,156], in addition to the works cited above. """
I think "essentialist" is probably a good description of the philosophy of how Unicode defines characters as opposed to fonts and glyphs, so I adopted it.
Are there examples of characters with no glyphs and glyphs with multiple characters? Does it mean there is a motion of a certain word but nobody knows how to write it or there are multiple ways to write it? When going from traditional to simplified Chinese many words with different meanings were merged which is kinda weird. Like sprout 發 and hair 髮 now being a single word 发. Is that what its trying to say?
> I was told that the Unified CJK thing adopted a different "philosophy" from the rest of Unicode
What would that philosophy be about? Sounds apocryphal. Unicode has never done "unification" like that for other languages/scripts?
i/ı/i, ö/ø/ø̈/oͤ: Same same, different codepoints.
Search and sorting is a mess everywhere. Depending on your locale, ö sorts either after o or after z. Sometimes it's semantically and phonetically equivalent to o wrt search but moreoften not. https://en.wikipedia.org/wiki/%C3%96
Well, Unicode didn't unify 門/门 either, but the appearance of the second glyph changes a lot depending on whether you use a Chinese font or a Japanese one. Meanwhile they did unify all the graphical variants of Latin characters with identical semantics, it's just that everyone is so used to the result that you hardly notice not being able to control whether g has a hook or a looped tail, and nobody expects to copy some German blackletter text into a French document and have it show up visually distinct, without explicitly setting the font.
I was involved in this before Unicode be a e Unicode when it started as discussions with Xerox and Apple—Xerox at that time had their XCCS standard. Those discussions directly morphed into Unicode. [[I was even there when we picked the name Unicode for what we had home to produce.]]
Conceptually we long debated unifying everything and in an ideal perfect world we would have done it. The reason was one primary goal for a new standard was to make it easily parseable and having unique rather than repeated codes was key to that. Sadly in the end we did not unify everything only to get buy-in from all major countries to support. That’s even why you see the roman/asciii characters repeated within Unicode itself—like as romaji. This was all well in good until we came to CJK and the number characters with semantical overlap was huge that this was more seriously considered—infact we started investigating this at Xerox before even thinking about Unicode and that work predated and influenced and leveraged the work done later.
Is that true? Other than things like not seeing a Ы, Э or Ё in Bulgarian, the modern printed fonts look the same to me, though I am not from either country.
Unicode has done the same unification for cuneiform languages. If you’re dealing with texts in Hittite, Sumerian, or Akkadian, you need to make sure you’re using the right fonts, otherwise the text will look entirely wrong.
Unicode merges some Chinese and Japanese characters into same code points and leave others as they are, on random per-character basis. So Chinese guys gets a ton of Japanese mixed texts and it causes tons of problems.
Japanese users are angry when system chooses Chinese fonts over Japanese, Chinese users apparently suffer with a font mishmash of Chinese and Japanese all the time like a mid-word capitalization. The official sanctioned solution is to just commit a genocide and nuke the offending language out of the system or to attach IVS to every Japanese characters which is like zero padding every single characters by a byte or two.
It's really putting everyone in pain and making absolutely no one happy, which sounds like what a good compromise tends to be, but LLMs seem to be struggling with there being multiple completely separate language families sharing codepoints but not character shapes or syntax. I've seen a smaller LLM model use made up verbs spilling over from the other language. I believe ML guys don't like datasets that do that kind of things. Same apply to image models.
I don't know but last I glanced at it, the Unification was implied to be coming from Chinese pressure than Japanese. Each of Kanji/Hanzi countries already had each own acceptable common use character sets and those could have been just populated into each own spaces. That seemed like the original plan at some point.
There has to be a reason why it happened(as to why Unicode suddenly started insisting it has to fit inside a 2^16 total chars or whatever).
>having different code points so that things like text search sometimes don't work.
Cross language search seems like a hack to me. Searching in Chinese should find Chinese words and searching in Japanese should find Japanese words. Being able to search in Japanese and get Chinese results is not what most users want, unless they don't have a proper keyboard.
I think you underestimate the similarities. For instance Japanese names are not translated to Chinese. Theyre just read with Chinese pronunciations. So Chinese will regularly interact with Japanese content (im guess it happens the other way around too, but i dont have the personal experience)
Meanwhile, Latin a and Cyrillic а, which look identical, have the same origin and are often pronounced identically too, get separate codepoints. CJK unification was not based on any clear preestablished principles but simply an attempt to cling to 16-bit codepoints, Asian languages be damned.
Cling to 16 bit codepoints sounds like a legitimate explanation? Latin and Cyrillic didn't would have saved no relevant amount of space.
It looks like the CJK unified space is over 20,000 characters, so that's a real technical magnitude distinction compared to Latin and Cyrillic. "Asian languages be damned" seems like a bad faith read, compared to "Java and Windows char is 16 bits and that will never change realistically" (and in fact they still haven't, even in 2026 things which rely on UTF16 instead of UCS2 are still commonly bugged unfortunately)
We also have at least 15 copies of Latin alphabet in Unicode, all differently styled/fonted (there is chancery cursive, mathematical cursive, the fraktur version, the full-width version, etc.)
Yes, but most of those are intended to be used exclusively for math, where the different styles do have semantic meaning. Even in introductory textbooks, it's pretty common to see the roman, calligraphic, and double-struck variants of a single letter all used with different meanings.
Plus, these characters were only added 20+ years later, and in the supplemental planes, not the BMP.
Letterlike Symbols is a bit of a weird grab bag of characters, most of which are clearly distinct from their "regular" Latin versions, but I do agree that a few of them are pointless exact copies of regular Latin letters.
The fullwidth/halfwidth stuff is a bit of a mess, but you could argue that these are actually CJK characters that just happen to resemble Latin characters (much like how Greek and Cyrillic both happen to have letters that look a lot like the Latin "A"), since they only exist for compatibility with older CJK encodings. This wouldn't be a very good argument though :)
That’s correct, it was deemed critical to making Unicode 16bits (although there were other good technical reasons but they would apply to unifying other languages as well.)
> The peculiar properties of CJK characters and the philosophy (apparently the Japanese did not like Unicode's tendencies towards Aristotelian essentialism) under which they were implemented in Unicode probably singlehandedly forced unicode to expand beyond the BMP....
Note that Han Unification has a history predating Unicode, and the Unicode work very much followed on from that.
The most notable is CCCII, developed in Taiwan in the early 80s, and then standardised in various places.
That's not to say that it hasn't been controversial (it has!), nor to say that it hasn't caused problems (it has!), but it's also unfair to say that it's just Unicode doing its own special thing.
It is a blunder that Japan could not see that they just copied other characters in to JIS and it’s also why we see duplicated Romaji characters within Unicode. The duplication is frankly bad regardless of number of bits need to encode although with 16 it was indeed a requirement.
Before that, there was the case of ÿ and Ÿ in the base IBM character set (see https://en.wikipedia.org/wiki/Ÿ). The former exists in a handful of proper nouns in French and Hungarian (as in about five proper nouns), but only as a y with a diæresis, which would normally be handled with a Unicode modifier symbol. It is also used in a couple of endangered languages in South America, which would not have influenced the character getting in to a PC character set. It is thought that possibly it was supposed to represent the Dutch "ij" character, but apparently that is never written as "ÿ" in print. Ÿ seems to not be used by any language.
Surely those aforementioned non-initial cases would sometimes find themselves in a piece of all-uppercase text? You'd have found such things in print media even before the advent of computers.
Not everything is upper/lower convertible like ß for example (as far as I know)
At least until 2017, I imagine there are others though.
> In 2017, the Council for German Orthography officially adopted a capital form ⟨ẞ⟩ as an acceptable variant, ending a long debate. https://en.wikipedia.org/wiki/%C3%9F
> Why didn't they simly replace the original bad one?
Starting with version 2.1 they made it so that code points can't ever be unassigned. People were getting really upset with the removals, since they can result in old data being misinterpreted. (Er, nothing actually was removed in 2.1, but it probably didn't become official policy until 3.0… ?)
Besides, it's useful to have the bad characters in there for people who want to write posts like this one discussing them.
But it's even more useful for people who want to use the actual character to be able to use it! And you can discuss using other means, not like drawings or old standard data disappears
Weird absolutism re error preservation instead of striving for correctness
> Not that hard to imagine, OCR existed back then?
How do you train OCR on a character that doesn't exist though? Even these days, OCR often confuses the 52 Latin characters, so I wouldn't expect as good of results with older technology and some 20k CJK characters.
It's listed in the Unicode character database, but I doubt that it was in (m)any fonts 20 years ago, it would have been in zero printed books [0], and nobody knew the meaning of the character, so I'd argue that it's more an artifact of the Unicode compilation process than a "real" character.
[0]: Aside from the "Overview of National Administrative Districts" book mentioned in the article
You don't need (m)any fonts, only need one font used by the book and you can get it the same way it was originally done - gluing some parts of other characters together. Or you could just draw it if the Unicode version is too dissimilar from the book version for a reliable OCR.
(though if the character came from this single book, likely the Unicode reference is its replica?)
I don't know if this is how OCR would work here, but they're made of components called radicals. It's kind of like how letters make words, but in a two dimensional layout, and one letter was wrong and a nonexistent word was made.
For example 彁 is made of parts of 引 and 歌 (the first that popped into my mind).
IIRC there's only like 200-something radicals, though some can be hard to spot due to how they overlap or nest inside each other.
"slow" - wasn't like they were pressed for time. It took them almost 20years to even start the investigation!
Also not certain you needed that much reliability compared to what was available to match one pic to a quality scan to get a reasonable number of candidates
There's some fan fiction that treats the characters as names of ghosts and comes up with back stories and visualizations for them (the article links to a compilation, but it's 403ing for me). Nothing mainstream though.
See also https://en.wikipedia.org/wiki/A_Book_from_the_Sky (a 1988 art project based on intentionally making up ghost characters by hand and printing a whole book with nonexistent but aesthetically plausible Chinese characters). Amusingly, Wikipedia mentions that over a dozen of them (out of about 4,000) were later found to accidentally exist!
The author Paul McCann (polm) is one of my favourite programmers out there!
He’s done awesome work in the Japanese NLP space over the last decade which has really helped me in my language learning projects.
He maintains a mecab (Japanese tokenizer) wrapper for Python [0], has a book on Japanese NLP written for English speakers [1] and also worked on Spacy at one point [2].
[0] https://github.com/polm/fugashi [1] https://www.japanesenlp.com/ [2] https://spacy.io/
I think there’s evidence found for the origin of “彁” as the result of a poor scan of a newspaper article. Look up “彁 新聞” to find some japanese sources about this.
Ghost characters reveal something about the joint semantic and phonetic nature of most Chinese characters. I know 彁 isn't real: but it still insists on a pronunciation: ka in Japanese, (gē or gé Mandarin). And it hints at meaning: bowed weapon, sound, elder brother; but your guess is as good as mine!
> I know 彁 isn't real: but it still insists on a pronunciation: ka in Japanese, (gē or gé Mandarin).
Huh? How do you pronounce 切?
> And it hints at meaning: bowed weapon, sound, elder brother; but your guess is as good as mine!
Why is "elder brother" a meaning hint if you've already assumed that 哥 is the phonophore?
The article mentions "an example of 彁 mistakenly used in a digitized Taisho newspaper due to a faded printing of 彊", but to me that implies the symbol already existed before then.
Isn't that backwards? The nonexistent character is in the -digitized- version so presumably OCR or something got 彊 wrong, that's not saying that 彁 was used in the -print- version.
Indeed, the source link is about exactly this: a crappy scan appears to have 彁 but a better copy reveals it was 彊.
I think you're both saying the same thing: the digitization of the article wouldn't have been the source, since 彁 would have had to exist before the digitization happened in order for the OCR to misread 彊 as 彁.
Most kanji are a combination of several smaller parts called "radicals" in English. If you look at these two kanji through this lens, you will see that it's actually a very simple mistake, one existing radical is replaced by another existing radical. It is very easy to imagine software that was working exactly like that: interpreting kanji as a combination of radicals rather than individual unrelated symbols
This reminds me of the pregnant man emoji [0] but it turns out they decided to handle that in an unusual way [1] so not really.
[0] https://emojipedia.org/pregnant-man
[1] https://www.reddit.com/r/technology/comments/u7x3l9/comment/...
It occurs to me that we can use 彊 to mean "a completely unknown concept that cannot be named". For example if you ask, "when Cthulhu rises from its slumber, what is thefirst thing it will do? Probably it will 彊.
This sounds like a SCP entry
Perhaps the meaning of this character is sealed in some vault
That's not on my keyboard... Can I just stick with "ineffable"?
Not exactly, it doesn't look like the definition or even connotations of "ineffable" include "unknown".
It does in the sense that ineffable things are unable to be described in language (conceptually: unknown to language).
Anything that is known can be described in language by giving it a name.
Hey, that's today's smbc https://www.smbc-comics.com/comic/r The problem is, there's no good reason to think the concept of 彊 in your mind is at all similar to the concept of 彊 in my brain. After all, we (probably) don't pronounce it the same, right?
More mundanely, you could also use it as a marker character, given that nobody except for sneaky little QA hobbitses will ever use it in their input.
So www.catb.org/~esr/jargon/html/M/mu.html
Well, vast swaths of the Kangxi dictionary (which serves as "sources" for probably most of the CJK characters) are such "ghost" characters as described in the article...
The peculiar properties of CJK characters and the philosophy (apparently the Japanese did not like Unicode's tendencies towards Aristotelian essentialism) under which they were implemented in Unicode probably singlehandedly forced unicode to expand beyond the BMP....
> apparently the Japanese did not like Unicode's tendencies towards Aristotelian essentialism
Do you mean Han Unification? I.e. that conceptually equivalent characters which are written differently in Japan and China received only a single unicode code-point, and are rendered the Chinese way by default on most computers?
Yes. As far as I understood it, the result was an incoherent mess.
I think some characters got different code points, while others were merged. And apparently the Japanese complained bitterly over the ones that were merged. If you had read any articles about that, this is probably what you have in mind right now.
And I'm also a bit tilted by the ones that had different code points, because when processing CJK text now we have to deal with characters that are (in my native Cantonese) essentially the same, looks similar (to my undiscriminating eyes), yet having different code points so that things like text search sometimes don't work.
Of course I'm not "blaming" the Japanese, if anything the simplified vs traditional Chinese thing is much more of a practical problem, and the conflicting code points I deal with on a routine basis are more of a Hong Kong vs Taiwan thing, but I was told that the Unified CJK thing adopted a different "philosophy" from the rest of Unicode (which I think really is some kind of Aristotelian essentialism...) mostly due to vocal objections from the Japanese.
Han-unification was the result of Asian scholars, in particular the Ideaographic Research Group [1]
I have no idea what "Aristotelian essentialism" is supposed to mean, or if you are saying that the unification was that.
> And apparently the Japanese complained bitterly over the ones that were merged.
and
> but I was told that the Unified CJK thing adopted a different "philosophy" from the rest of Unicode [...] mostly due to vocal objections from the Japanese.
Seems to contradict each other.
[1] https://en.wikipedia.org/wiki/Ideographic_Research_Group
I think the Aristotelian (though I might have reached for Platonic) idea is that there is an "A" which is essential, and that you map many various "A"s to that one A - which results in weirdness when you leave the Latin-alphabet based languages.
(Arguably even in English we run into the fact that there is not one "A" - there are many "A"s that sound and act completely different, and we collapsed them into one representation.)
Many times I have asked my Hungarian wife whether a sound is "a" or "a with an accent" (A vs Á), but they are different letters in Hungarian, not just variations of one as they are in Italian. We made a mess with "accent unification".
Which is always fun when you think that clearly nobody thinks of "o or o with a leg" (O vs Q).
That's not how Unicode works, A Α and А are different characters (latin, greek, cyrillic).
Sorry, probably not very standard terminology. I somehow picked it up from the OReilly book: Fonts & Encodings: From Advanced Typography to Unicode and Everything in Between
Page 55:
""" there are characters with no glyphs. glyphs that can correspond to a number of different characters according to context. Glyphs that correspond to multiple characters at the same time (with weightings assigned to each), and even more possibilities.
The problem of glyphs and characters is so complex that it has gone beyond the realm of computer specialists and has come to be of interest even to philosophers. For example, the Japanese philosopher Shigeki Moro, who has worked with ideographic characters in Buddhist documents, goes so far in his article Surface or Essence: Beyond Character Model Set [274] as to say that Unicode's approach is Aristotelian essentialist and to recommend supplanting it by an approach inspired by Jacques Derrida's theory of writing [114, 115]. The reader interested in the philosophical aspects of the issue is invited to consult [165,156], in addition to the works cited above. """
I think "essentialist" is probably a good description of the philosophy of how Unicode defines characters as opposed to fonts and glyphs, so I adopted it.
Are there examples of characters with no glyphs and glyphs with multiple characters? Does it mean there is a motion of a certain word but nobody knows how to write it or there are multiple ways to write it? When going from traditional to simplified Chinese many words with different meanings were merged which is kinda weird. Like sprout 發 and hair 髮 now being a single word 发. Is that what its trying to say?
> I was told that the Unified CJK thing adopted a different "philosophy" from the rest of Unicode
What would that philosophy be about? Sounds apocryphal. Unicode has never done "unification" like that for other languages/scripts?
i/ı/i, ö/ø/ø̈/oͤ: Same same, different codepoints.
Search and sorting is a mess everywhere. Depending on your locale, ö sorts either after o or after z. Sometimes it's semantically and phonetically equivalent to o wrt search but moreoften not. https://en.wikipedia.org/wiki/%C3%96
Well, Unicode didn't unify 門/门 either, but the appearance of the second glyph changes a lot depending on whether you use a Chinese font or a Japanese one. Meanwhile they did unify all the graphical variants of Latin characters with identical semantics, it's just that everyone is so used to the result that you hardly notice not being able to control whether g has a hook or a looped tail, and nobody expects to copy some German blackletter text into a French document and have it show up visually distinct, without explicitly setting the font.
I was involved in this before Unicode be a e Unicode when it started as discussions with Xerox and Apple—Xerox at that time had their XCCS standard. Those discussions directly morphed into Unicode. [[I was even there when we picked the name Unicode for what we had home to produce.]]
Conceptually we long debated unifying everything and in an ideal perfect world we would have done it. The reason was one primary goal for a new standard was to make it easily parseable and having unique rather than repeated codes was key to that. Sadly in the end we did not unify everything only to get buy-in from all major countries to support. That’s even why you see the roman/asciii characters repeated within Unicode itself—like as romaji. This was all well in good until we came to CJK and the number characters with semantical overlap was huge that this was more seriously considered—infact we started investigating this at Xerox before even thinking about Unicode and that work predated and influenced and leveraged the work done later.
Yes, and Cyrillic letters also look quite different in Bulgarian and Russian fonts. So what?
Is that true? Other than things like not seeing a Ы, Э or Ё in Bulgarian, the modern printed fonts look the same to me, though I am not from either country.
Some letters look different: https://en.wikipedia.org/wiki/Bulgarian_alphabet#/media/File...
Same thing, different code points is the opposite of han unification. Han unification is same code point, different thing.
Unicode has done the same unification for cuneiform languages. If you’re dealing with texts in Hittite, Sumerian, or Akkadian, you need to make sure you’re using the right fonts, otherwise the text will look entirely wrong.
Unicode merges some Chinese and Japanese characters into same code points and leave others as they are, on random per-character basis. So Chinese guys gets a ton of Japanese mixed texts and it causes tons of problems.
Japanese users are angry when system chooses Chinese fonts over Japanese, Chinese users apparently suffer with a font mishmash of Chinese and Japanese all the time like a mid-word capitalization. The official sanctioned solution is to just commit a genocide and nuke the offending language out of the system or to attach IVS to every Japanese characters which is like zero padding every single characters by a byte or two.
It's really putting everyone in pain and making absolutely no one happy, which sounds like what a good compromise tends to be, but LLMs seem to be struggling with there being multiple completely separate language families sharing codepoints but not character shapes or syntax. I've seen a smaller LLM model use made up verbs spilling over from the other language. I believe ML guys don't like datasets that do that kind of things. Same apply to image models.
I don't know but last I glanced at it, the Unification was implied to be coming from Chinese pressure than Japanese. Each of Kanji/Hanzi countries already had each own acceptable common use character sets and those could have been just populated into each own spaces. That seemed like the original plan at some point.
There has to be a reason why it happened(as to why Unicode suddenly started insisting it has to fit inside a 2^16 total chars or whatever).
>having different code points so that things like text search sometimes don't work.
Cross language search seems like a hack to me. Searching in Chinese should find Chinese words and searching in Japanese should find Japanese words. Being able to search in Japanese and get Chinese results is not what most users want, unless they don't have a proper keyboard.
Do you feel similarly about french and english?
I think you underestimate the similarities. For instance Japanese names are not translated to Chinese. Theyre just read with Chinese pronunciations. So Chinese will regularly interact with Japanese content (im guess it happens the other way around too, but i dont have the personal experience)
Interesting. I guess there are no well working normalization functions as there are for the different code points that can make umlauts (äöü)?
Meanwhile, Latin a and Cyrillic а, which look identical, have the same origin and are often pronounced identically too, get separate codepoints. CJK unification was not based on any clear preestablished principles but simply an attempt to cling to 16-bit codepoints, Asian languages be damned.
Cling to 16 bit codepoints sounds like a legitimate explanation? Latin and Cyrillic didn't would have saved no relevant amount of space.
It looks like the CJK unified space is over 20,000 characters, so that's a real technical magnitude distinction compared to Latin and Cyrillic. "Asian languages be damned" seems like a bad faith read, compared to "Java and Windows char is 16 bits and that will never change realistically" (and in fact they still haven't, even in 2026 things which rely on UTF16 instead of UCS2 are still commonly bugged unfortunately)
We also have at least 15 copies of Latin alphabet in Unicode, all differently styled/fonted (there is chancery cursive, mathematical cursive, the fraktur version, the full-width version, etc.)
Yes, but most of those are intended to be used exclusively for math, where the different styles do have semantic meaning. Even in introductory textbooks, it's pretty common to see the roman, calligraphic, and double-struck variants of a single letter all used with different meanings.
Plus, these characters were only added 20+ years later, and in the supplemental planes, not the BMP.
Letterlike Symbols and Fullwidth Forms are in BMP (and I generously consider Enclosed Alphanumerics to not be a copy of a Latin alphabet).
Letterlike Symbols is a bit of a weird grab bag of characters, most of which are clearly distinct from their "regular" Latin versions, but I do agree that a few of them are pointless exact copies of regular Latin letters.
The fullwidth/halfwidth stuff is a bit of a mess, but you could argue that these are actually CJK characters that just happen to resemble Latin characters (much like how Greek and Cyrillic both happen to have letters that look a lot like the Latin "A"), since they only exist for compatibility with older CJK encodings. This wouldn't be a very good argument though :)
That’s correct, it was deemed critical to making Unicode 16bits (although there were other good technical reasons but they would apply to unifying other languages as well.)
> The peculiar properties of CJK characters and the philosophy (apparently the Japanese did not like Unicode's tendencies towards Aristotelian essentialism) under which they were implemented in Unicode probably singlehandedly forced unicode to expand beyond the BMP....
Note that Han Unification has a history predating Unicode, and the Unicode work very much followed on from that.
The most notable is CCCII, developed in Taiwan in the early 80s, and then standardised in various places.
That's not to say that it hasn't been controversial (it has!), nor to say that it hasn't caused problems (it has!), but it's also unfair to say that it's just Unicode doing its own special thing.
What is the rationale to the unifying direction of changes in Unicode?
The Han unification probably was about fitting CJK into the 16 bits aka BMP. It was a blunder indeed.
It is a blunder that Japan could not see that they just copied other characters in to JIS and it’s also why we see duplicated Romaji characters within Unicode. The duplication is frankly bad regardless of number of bits need to encode although with 16 it was indeed a requirement.
Before that, there was the case of ÿ and Ÿ in the base IBM character set (see https://en.wikipedia.org/wiki/Ÿ). The former exists in a handful of proper nouns in French and Hungarian (as in about five proper nouns), but only as a y with a diæresis, which would normally be handled with a Unicode modifier symbol. It is also used in a couple of endangered languages in South America, which would not have influenced the character getting in to a PC character set. It is thought that possibly it was supposed to represent the Dutch "ij" character, but apparently that is never written as "ÿ" in print. Ÿ seems to not be used by any language.
> Ÿ seems to not be used by any language.
Surely those aforementioned non-initial cases would sometimes find themselves in a piece of all-uppercase text? You'd have found such things in print media even before the advent of computers.
Not everything is upper/lower convertible like ß for example (as far as I know)
At least until 2017, I imagine there are others though.
> In 2017, the Council for German Orthography officially adopted a capital form ⟨ẞ⟩ as an acceptable variant, ending a long debate. https://en.wikipedia.org/wiki/%C3%9F
Great, so they added Ÿ but not n̈, making it very difficult to write music reviews.
(Not sure how the latter will render in your browser).
Fascinating. But, I guess it's better to have superfluous invalid characters than missing real ones.
The community can use this free character to denote "gay"
Xu Bing has a book that consists entirely of invented characters:
https://en.wikipedia.org/wiki/A_Book_from_the_Sky
(2018)
I've seen YouTube videos on this topic before.
> The original character (𡚴) was not added to JIS or Unicode until much later and doesn't display on most sites for me.
It's from Unicode version 3.1 (published 2001) so this is surprising.
Should probably have "(2008)" in the title
A language where you can make up new characters seems anathema to a universal character encoding.
Saw headline, expected branch prediction vulnerability involving Unicode, was surprised at a completely different topic.
So what is the spectre? I thought this was proposing something new like the manifesto
> The original character (𡚴) was not added to JIS or Unicode until much later and doesn't display on most sites for me
Why didn't they simly replace the original bad one?
> nine hundred pages. Imagine tracking down a single character without a page reference
Not that hard to imagine, OCR existed back then?
> Why didn't they simly replace the original bad one?
Starting with version 2.1 they made it so that code points can't ever be unassigned. People were getting really upset with the removals, since they can result in old data being misinterpreted. (Er, nothing actually was removed in 2.1, but it probably didn't become official policy until 3.0… ?)
Besides, it's useful to have the bad characters in there for people who want to write posts like this one discussing them.
But it's even more useful for people who want to use the actual character to be able to use it! And you can discuss using other means, not like drawings or old standard data disappears Weird absolutism re error preservation instead of striving for correctness
> Not that hard to imagine, OCR existed back then?
How do you train OCR on a character that doesn't exist though? Even these days, OCR often confuses the 52 Latin characters, so I wouldn't expect as good of results with older technology and some 20k CJK characters.
It does exist? It's part of Unicode!
It's listed in the Unicode character database, but I doubt that it was in (m)any fonts 20 years ago, it would have been in zero printed books [0], and nobody knew the meaning of the character, so I'd argue that it's more an artifact of the Unicode compilation process than a "real" character.
[0]: Aside from the "Overview of National Administrative Districts" book mentioned in the article
You don't need (m)any fonts, only need one font used by the book and you can get it the same way it was originally done - gluing some parts of other characters together. Or you could just draw it if the Unicode version is too dissimilar from the book version for a reliable OCR. (though if the character came from this single book, likely the Unicode reference is its replica?)
(Talking about Japanese here)
I don't know if this is how OCR would work here, but they're made of components called radicals. It's kind of like how letters make words, but in a two dimensional layout, and one letter was wrong and a nonexistent word was made.
For example 彁 is made of parts of 引 and 歌 (the first that popped into my mind).
IIRC there's only like 200-something radicals, though some can be hard to spot due to how they overlap or nest inside each other.
I find 鸚哥（いんこ） to be the easiest way to type the right hand side of 彁.
OCR was slow and unreliable and was for a very long time.
"slow" - wasn't like they were pressed for time. It took them almost 20years to even start the investigation! Also not certain you needed that much reliability compared to what was available to match one pic to a quality scan to get a reasonable number of candidates
Is anyone using these characters now for anything? No youth language or online slang using it?
There's some fan fiction that treats the characters as names of ghosts and comes up with back stories and visualizations for them (the article links to a compilation, but it's 403ing for me). Nothing mainstream though.
It's something! I propose that the first 4 of those characters should in English be called Blinky, Pinky, Inky, and Clyde.
See also https://en.wikipedia.org/wiki/A_Book_from_the_Sky (a 1988 art project based on intentionally making up ghost characters by hand and printing a whole book with nonexistent but aesthetically plausible Chinese characters). Amusingly, Wikipedia mentions that over a dozen of them (out of about 4,000) were later found to accidentally exist!
"- the spectre of communism. All the powers of old encoding have entered into a holy alliance to exorcise this spectre..."
Well, Communism made it into Unicode (U+262D). I tried to show it here but it gets stripped.
I have it it in my .XCompose as
Never used, but I laugh every time I see it there.As a slightly related tangent, The compose key mnemonic interface for rarely used characters is pretty great, beats trying to remember alt codes.
Originally added to the upstream compose map in https://gitlab.freedesktop.org/xorg/lib/libx11/-/commit/f052..., and replaced in https://gitlab.freedesktop.org/xorg/lib/libx11/-/merge_reque.... Given the people involved in the initial commit, I suspect (but do not have an absolute memory of) it may have been influenced by my ownership of (insert a hammer and sickle symbol here).net.
Figures, it's a capitalist website after all...
YOU SEEM TO BE ABLE TO ALSO TYPE IN LOWERCASE BUT FOR SOME REASon i can't
never mind i pressed caps lock everything is better now