The internet has a new idea for old text, and it comes with a name that sounds like a joke: UTF-8000. It is a proposal for an unlimited version of UTF-8, the encoding that powers most of the text you read online. The pitch is simple: UTF-8000 would let you write strings of any length, with no upper limit on how many bytes a single character can span.
The proposal is a standalone project, not an official Unicode Consortium document. It offers a reference implementation, and it is being promoted as a fun way to think about how encoding works. The creator, who is clearly familiar with the history of UTF-8, argues that the design already contains the seeds of this kind of expansion. The question is whether the idea is a clever hack or a concept with no practical future.
The Two Special Cases
The core claim is that UTF-8000 introduces no new special cases. That is a significant promise. The proposal says there are only two special cases inherited from UTF-8 itself, and both are minor:
- ASCII, which is treated as-is.
- 2-byte UTF-8, which has 4 mandatory content bits to check against overlong encoding, rather than 5 for all longer lengths.
The proposal is explicit that the two cases are the only exceptions. Everything else follows a clean, predictable pattern.
What the Proposal Actually Does
The main contribution, according to the proposal, is clarifying how to split the highest bits of the first byte of a UTF-8 code unit. The proposal breaks those bits into self-synchronization bits and start bits. It then explains how to stripe the start bits across continuation bytes if needed, to support arbitrarily large code units.
The example in the proposal is a 22-byte code unit. Byte number four is the interesting one: it is a continuation byte, a start byte, the final start byte, and it has content bits, but only some of the mandatory content bits. Those bits are straddled between the final start byte and the first non-start byte. The proposal calls this exciting, which is a fair way to describe a byte that does so many jobs at once.
The Story of UTF-8’s Design
The proposal spends time on history. It points out that the earliest draft of UTF-8 used a six-byte start byte that looked like 111111xx. That was changed a few days later to 1111110x, which ensured the number of content bits was not a special case and left room for expansion.
The proposal argues that this design foresight is why UTF-8 can be extended at all. It calls UTF-8 “the capstone of the Unix Philosophy.” The idea is that the encoding’s architecture was carefully planned from the start, and that planning opened the door for this kind of extension.
The proposal credits Ken Thompson and Rob Pike with the original UTF-8 architecture. It also cites the FSS-UTF draft by Dave Prosser and others as the earlier design UTF-8 was meant to replace. The FSS-UTF proposal used a three-byte structure like 110xxxxx 1xxxxxxx 1xxxxxxx, which could not distinguish between first bytes and continuation bytes without prior context. UTF-8 fixed that by making the two types of bytes disjoint.
The Math Behind the Encoding
The proposal gives the numbers for content bits. For an n-byte code unit, the number of content bits is 5n+1, and the number of mandatory content bits is also 5n+1. The proportion of content bits in a code unit approaches 5/8, or 62.5%, as n grows. For ASCII, the proportion is 7/8, or 87.5%, because it is a one-byte code unit.
The proposal notes that the limit is nonzero and does not depend on n. That is the kind of detail that makes encoders smile.
Why It Might Not Matter
The proposal is a fun academic exercise, but it is not a serious proposal from the Unicode Consortium. It is a standalone project, and it is being treated as one. The fact that it reached a notable score on Hacker News is a sign of curiosity, not a sign of adoption.
The two special cases are small. The design is clean. But the question of whether this is the “canonical, correct way to extend UTF-8” is a matter of taste. The proposal argues that it is, but it is not the only possible extension. There are rejected alternatives, like ASCVI, which is described as a version of UTF-8 with 6-bit ASCII. That version is one of the alternatives the proposal mentions rather than a design it endorses.
The Verdict on UTF-8000
The proposal is clever. It is well-argued. It respects the history of the encoding it is extending. But it is also a toy, not a standard. The fact that it can be proposed so cleanly is a compliment to UTF-8’s design. The fact that it is unlikely to be adopted is a compliment to the people who already settled on UTF-8.
The proposal is worth reading for the history lesson alone. It explains the FSS-UTF draft, the choice of self-synchronization bits, and the care that went into the original design. It is a reminder that the text on your screen is the result of decades of careful planning, and that the people who made it are the same people who made Unix.
The idea of UTF-8000 is a fun one. It is a good standalone project. It is not a proposal that will change anything. But it is a good example of how ideas about text keep moving, even when the standards have already been written.
Source material: “UTF-8000: Unlimited UTF-8,” jb2170.com.
Get the Notebook.
The day's best stories and every fresh verdict, in plain English, in your inbox by seven. One email a day, no more.