tonsky.me

The Absolute Minimum Every Software Developer Must Know About Unicode in 2023 (Still No Excuses!)

2023-10-02T00:00:00Z

+
There Ain’t No Such Thing As Plain Text.
+
It does not make sense to have a string without knowing what encoding it uses. You can no longer stick your head in the sand and pretend that “plain” text is ASCII.
+

A lot has changed in 20 years. In 2003, the main question was: what encoding is this?

In 2023, it’s no longer a question: with a 98% probability, it’s UTF-8. Finally! We can stick our heads in the sand again!

+ +

The question now becomes: how do we use UTF-8 correctly? Let’s see!

What is Unicode?

Unicode is a standard that aims to unify all human languages, both past and present, and make them work with computers.

In practice, Unicode is a table that assigns unique numbers to different characters.

For example:

The Latin letter A is assigned the number 65.
The Arabic Letter Seen س is 1587.
The Katakana Letter Tu ツ is 12484
The Musical Symbol G Clef 𝄞 is 119070.
💩 is 128169.

Unicode refers to these numbers as code points.

Since everybody in the world agrees on which numbers correspond to which characters, and we all agree to use Unicode, we can read each other’s texts.

Unicode == character ⟷ code point.

How big is Unicode?

Currently, the largest defined code point is 0x10FFFF. That gives us a space of about 1.1 million code points.

About 170,000, or 15%, are currently defined. An additional 11% are reserved for private use. The rest, about 800,000 code points, are not allocated at the moment. They could become characters in the future.

Here’s roughly how it looks:

+ +

Large square == plane == 65,536 characters. Small one == 256 characters. The entire ASCII is half of a small red square in the top left corner.

What’s Private Use?

These are code points reserved for app developers and will never be defined by Unicode itself.

For example, there’s no place for the Apple logo in Unicode, so Apple puts it at U+F8FF which is within the Private Use block. In any other font, it’ll render as missing glyph 􀣺, but in fonts that ship with macOS, you’ll see .

The Private Use Area is mostly used by icon fonts:

Isn’t it a beauty? It’s all text!

What does `U+1F4A9` mean?

It’s a convention for how to write code point values. The prefix U+ means, well, Unicode, and 1F4A9 is a code point number in hexadecimal.

Oh, and U+1F4A9 specifically is 💩.

What’s UTF-8 then?

UTF-8 is an encoding. Encoding is how we store code points in memory.

The simplest possible encoding for Unicode is UTF-32. It simply stores code points as 32-bit integers. So U+1F4A9 becomes 00 01 F4 A9, taking up four bytes. Any other code point in UTF-32 will also occupy four bytes. Since the highest defined code point is U+10FFFF, any code point is guaranteed to fit.

UTF-16 and UTF-8 are less straightforward, but the ultimate goal is the same: to take a code point and encode it as bytes.

Encoding is what you’ll actually deal with as a programmer.

How many bytes are in UTF-8?

UTF-8 is a variable-length encoding. A code point might be encoded as a sequence of one to four bytes.

This is how it works:

+ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +

Code point	Byte 1	Byte 2	Byte 3	Byte 4
U+`0000`..`007F`	`0xxxxxxx`
U+`0080`..`07FF`	`110xxxxx`	`10xxxxxx`
U+`0800`..`FFFF`	`1110xxxx`	`10xxxxxx`	`10xxxxxx`
U+`10000`..`10FFFF`	`11110xxx`	`10xxxxxx`	`10xxxxxx`	`10xxxxxx`

If you combine this with the Unicode table, you’ll see that English is encoded with 1 byte, Cyrillic, Latin European languages, Hebrew and Arabic need 2, and Chinese, Japanese, Korean, other Asian languages, and Emoji need 3 or 4.

A few important points here:

First, UTF-8 is byte-compatible with ASCII. The code points 0..127, the former ASCII, are encoded with one byte, and it’s the same exact byte. U+0041 (A, Latin Capital Letter A) is just 41, one byte.

Any pure ASCII text is also a valid UTF-8 text, and any UTF-8 text that only uses codepoints 0..127 can be read as ASCII directly.

Second, UTF-8 is space-efficient for basic Latin. That was one of its main selling points over UTF-16. It might not be fair for texts all over the world, but for technical strings like HTML tags or JSON keys, it makes sense.

On average, UTF-8 tends to be a pretty good deal, even for non-English computers. And for English, there’s no comparison.

Third, UTF-8 has error detection and recovery built-in. The first byte’s prefix always looks different from bytes 2-4. This way you can always tell if you are looking at a complete and valid sequence of UTF-8 bytes or if something is missing (for example, you jumped it the middle of the sequence). Then you can correct that by moving forward or backward until you find the beginning of the correct sequence.

And a couple of important consequences:

You CAN’T determine the length of the string by counting bytes.
You CAN’T randomly jump into the middle of the string and start reading.
You CAN’T get a substring by cutting at arbitrary byte offsets. You might cut off part of the character.

Those who do will eventually meet this bad boy: �

What’s �?

U+FFFD, the Replacement Character, is simply another code point in the Unicode table. Apps and libraries can use it when they detect Unicode errors.

If you cut half of the code point off, there’s not much left to do with the other half, except displaying an error. That’s when � is used.

var bytes = "Аналитика".getBytes("UTF-8");
+var partial = Arrays.copyOfRange(bytes, 0, 11);
+new String(partial, "UTF-8"); // => "Анал�"

Wouldn’t UTF-32 be easier for everything?

NO.

UTF-32 is great for operating on code points. Indeed, if every code point is always 4 bytes, then strlen(s) == sizeof(s) / 4, substring(0, 3) == bytes[0, 12], etc.

The problem is, you don’t want to operate on code points. A code point is not a unit of writing; one code point is not always a single character. What you should be iterating on is called “extended grapheme clusters”, or graphemes for short.

A grapheme is a minimally distinctive unit of writing in the context of a particular writing system. ö is one grapheme. é is one too. And 각. Basically, grapheme is what the user thinks of as a single character.

The problem is, in Unicode, some graphemes are encoded with multiple code points!

+ +

For example, é (a single grapheme) is encoded in Unicode as e (U+0065 Latin Small Letter E) + ´ (U+0301 Combining Acute Accent). Two code points!

It can also be more than two:

☹️ is U+2639 + U+FE0F
👨‍🏭 is U+1F468 + U+200D + U+1F3ED
🚵🏻‍♀️ is U+1F6B5 + U+1F3FB + U+200D + U+2640 + U+FE0F
y̖̠͍̘͇͗̏̽̎͞ is U+0079 + U+0316 + U+0320 + U+034D + U+0318 + U+0347 + U+0357 + U+030F + U+033D + U+030E + U+035E

There’s no limit, as far as I know.

Remember, we are talking about code points here. Even in the widest encoding, UTF-32, 👨‍🏭 will still take three 4-byte units to encode. And it still needs to be treated as a single character.

If the analogy helps, we can think of the Unicode itself (without any encodings) as being variable-length.

An Extended Grapheme Cluster is a sequence of one or more Unicode code points that must be treated as a single, unbreakable character.

Therefore, we get all the problems we have with variable-length encodings, but now on code point level: you can’t take only a part of the sequence, it always should be selected, copied, edited, or deleted as a whole.

Failure to respect grapheme clusters leads to bugs like this:

+ +

or this:

+ + + +

Just to be clear: this is NOT correct behavior

Using UTF-32 instead of UTF-8 will not make your life any easier in regards to extended grapheme clusters. And extended grapheme clusters is what you should care about.

Code points — 🥱. Graphemes — 😍

Is Unicode hard only because of emojis?

Not really. Extended Grapheme Clusters are also used for alive, actively used languages. For example:

ö (German) is a single character, but multiple code points (U+006F U+0308).
ą́ (Lithuanian) is U+00E1 U+0328.
각 (Korean) is U+1100 U+1161 U+11A8.

So no, it’s not just about emojis.

What’s "🤦🏼‍♂️".length?

The question is inspired by this brilliant article.

Different programming languages will happily give you different answers.

Python 3:

>>> len("🤦🏼‍♂️")
+5

JavaScript / Java / C#:

>> "🤦🏼‍♂️".length 
+7

Rust:

println!("{}", "🤦🏼‍♂️".len());
+// => 17

As you can guess, different languages use different internal string representations (UTF-32, UTF-16, UTF-8) and report length in whatever units they store characters in (ints, shorts, bytes).

BUT! If you ask any normal person, one that isn’t burdened with computer internals, they’ll give you a straight answer: 1. The length of 🤦🏼‍♂️ string is 1.

That’s what extended grapheme clusters are all about: what humans perceive as a single character. And in this case, 🤦🏼‍♂️ is undoubtedly a single character.

The fact that 🤦🏼‍♂️ consists of 5 code points (U+1F926 U+1F3FB U+200D U+2642 U+FE0F) is mere implementation detail. It should not be broken apart, it should not be counted as multiple characters, the text cursor should not be positioned inside it, it shouldn’t be partially selected, etc.

For all intents and purposes, this is an atomic unit of text. Internally, it could be encoded whatever, but for user-facing API, it should be treated as a whole.

The only modern language that gets it right is Swift:

print("🤦🏼‍♂️".count)
+// => 1

Basically, there are two layers:

Internal, computer-oriented. How to copy strings, send them over the network, store on disk, etc. This is where you need encodings like UTF-8. Swift uses UTF-8 internally, but it might as well be UTF-16 or UTF-32. What's important is that you only use it to copy strings as a whole and never to analyze their content.
External, human-facing API. Character count in UI. Taking first 10 characters to generate preview. Searching in text. Methods like .count or .substring. Swift gives you a view that pretends the string is a sequence of grapheme clusters. And that view behaves like any human would expect: it gives you 1 for "🤦🏼‍♂️".count.

I hope more languages adopt this design soon.

Question to the reader: what to you think "ẇ͓̞͒͟͡ǫ̠̠̉̏͠͡ͅr̬̺͚̍͛̔͒͢d̠͎̗̳͇͆̋̊͂͐".length should be?

How do I detect extended grapheme clusters then?

Unfortunately, most languages choose the easy way out and let you iterate through strings with 1-2-4-byte chunks, but not with grapheme clusters.

It makes no sense and has no semantics, but since it’s the default, programmers don’t think twice, and we see corrupted strings as the result:

+ +

“I know, I’ll use a library to do strlen()!” — nobody, ever.

But that’s exactly what you should be doing! Use a proper Unicode library! Yes, for basic stuff like strlen or indexOf or substring!

For example:

C/C++/Java: use ICU. It’s a library from Unicode itself that encodes all the rules about text segmentation.
C#: use TextElementEnumerator, which is kept up to date with Unicode as far as I can tell.
Swift: just stdlib. Swift does the right thing by default.
UPD: Erlang/Elixir seem to be doing the right thing, too.
For other languages, there’s probably a library or binding for ICU.
Roll your own. Unicode publishes rules and tables in a machine-readable format, and all the libraries above are based on them.

But whatever you choose, make sure it’s on the recent enough version of Unicode (15.1 at the moment of writing), because the definition of graphemes changes from version to version. For example, Java’s java.text.BreakIterator is a no-go: it’s based on a very old version of Unicode and not updated.

Use a library

IMO, the whole situation is a shame. Unicode should be in the stdlib of every language by default. It’s the lingua franca of the internet! It’s not even new: we’ve been living with Unicode for 20 years now.

Wait, rules are changing?

Yes! Ain’t it cool?

(I know, it ain’t)

Starting roughly in 2014, Unicode has been releasing a major revision of their standard every year. This is where you get your new emojis from — Android and iOS updates in the Fall usually include the newest Unicode standard among other things.

+ +

What’s sad for us is that the rules defining grapheme clusters change every year as well. What is considered a sequence of two or three separate code points today might become a grapheme cluster tomorrow! There’s no way to know! Or prepare!

Even worse, different versions of your own app might be running on different Unicode standards and report different string lengths!

But that’s the reality we live in. You don’t really have a choice here. You can’t ignore Unicode or Unicode updates if you want to stay relevant and provide a decent user experience. So, buckle up, embrace, and update.

Update yearly

Why is "Å" !== "Å" !== "Å"?

+ +

Copy any of these to your JavaScript console:

"Å" === "Å"
+"Å" === "Å"
+"Å" === "Å"

What do you get? False? You should get false, and it’s not a mistake.

Remember earlier when I said that ö is two code points, U+006F U+0308? Basically, Unicode offers more than one way to write characters like ö or Å. You can:

Compose Å from normal Latin A + a combining character,
OR there’s also a pre-composed code point U+00C5 that does that for you.

They will look the same (Å vs Å), they should work the same, and for all intents and purposes, they are considered exactly the same. The only difference is the byte representation.

That’s why we need normalization. There are four forms:

NFD tries to explode everything to the smallest possible pieces, and also sorts pieces in a canonical order if there is more than one.

NFC, on the other hand, tries to combine everything into pre-composed form if one exists.

+ +

For some characters there are also multiple versions of them in Unicode. For example, there’s U+00C5 Latin Capital Letter A with Ring Above, but there’s also U+212B Angstrom Sign which looks the same.

These are also replaced during normalization:

+ +

NFD and NFC are called “canonical normalization”. Another two forms are “compatibility normalization”:

NFKD tries to explode everything and replaces visual variants with default ones.

NFKC tries to combine everything while replacing visual variants with default ones.

+ +

Visual variants are separate Unicode code points that represent the same character but are supposed to render differently. Like, ① or ⁹ or 𝕏. We want to be able to find both "x" and "2" in a string like "𝕏²", don’t we?

All of these have their own code points, but they are also all Xs

Why does the ﬁ ligature even have its own code point? No idea. A lot can happen in a million characters.

Before comparing strings or searching for a substring, normalize!

Unicode is locale-dependent

The Russian name Nikolay is written like this:

+ +

and encoded in Unicode as U+041D 0438 043A 043E 043B 0430 0439.

The Bulgarian name Nikolay is written:

+ +

and encoded in Unicode as U+041D 0438 043A 043E 043B 0430 0439. Exactly the same!

Wait a second! How does the computer know when to render Bulgarian-style glyphs and when to use Russian ones?

Short answer: it doesn’t. Unfortunately, Unicode is not a perfect system, and it has many shortcomings. Among them is assigning the same code point to glyphs that are supposed to look differently, like Cyrillic Lowercase K and Bulgarian Lowercase K (both are U+043A).

From what I understand, Asian people get it much worse: many Chinese, Japanese, and Korean logograms that are written very differently get assigned the same code point:

U+8FD4 in different locales

Unicode motivation is to save code points space (my guess). Information on how to render is supposed to be transferred outside of the string, as locale/language metadata.

Unfortunately, it fails the original goal of Unicode:

+
[...] no escape sequence or control code is required to specify any character in any language.
+

In practice, dependency on locale brings a lot of problems:

Being metadata, locale often gets lost.
People are not limited to a single locale. For example, I can read and write English (USA), English (UK), German, and Russian. Which locale should I set my computer to?
It’s hard to mix and match. Like Russian names in Bulgarian text or vice versa. Why not? It’s the internet, people from all cultures hang out here.
There’s no place to specify the locale. Even making the two screenshots above was non-trivial because in most software, there’s no dropdown or text input to change locale.
When needed, it had to be guessed. For example, Twitter tries to guess the locale from the text of the tweet itself (because where else could it get it from?) and sometimes gets it wrong:

+ +

Why does `String::toLowerCase()` accepts Locale as an argument?

Another unfortunate example of locale dependence is the Unicode handling of dotless i in the Turkish language.

Unlike English, Turks have two I variants: dotted and dotless. Unicode decided to reuse I and i from ASCII and only add two new code points: İ and ı.

Unfortunately, that made toLowerCase/toUpperCase behave differently on the same input:

var en_US = Locale.of("en", "US");
+var tr = Locale.of("tr");
+
+"I".toLowerCase(en_US); // => "i"
+"I".toLowerCase(tr);    // => "ı"
+
+"i".toUpperCase(en_US); // => "I"
+"i".toUpperCase(tr);    // => "İ"

So no, you can’t convert string to lowercase without knowing what language that string is written in.

I live in the US/UK, should I even care?

+ +

quotation marks “ ” ‘ ’,
apostrophe ’,
dashes – —,
different variations of spaces (figure, hair, non-breaking),
bullets • ■ ☞,
currency symbols other than the $ (kind of tells you who invented computers, doesn’t it?): € ¢ £,
mathematical signs—plus + and equals = are part of ASCII, but minus − and multiply × are not ¯\_(ツ)_/¯,
various other signs © ™ ¶ † §.

Hell, you can’t even spell café, piñata, or naïve without Unicode. So yes, we are all in it together, even Americans.

Touché.

What are surrogate pairs?

That goes back to Unicode v1. The first version of Unicode was supposed to be fixed-width. A 16-bit fixed width, to be exact:

Version 1.0 of the Unicode Standard, October 1991

They believed 65,536 characters would be enough for all human languages. They were almost right!

When they realized they needed more code points, UCS-2 (an original version of UTF-16 without surrogates) was already used in many systems. 16 bit, fixed-width, it only gives you 65,536 characters. What can you do?

Unicode decided to allocate some of these 65,536 characters to encode higher code points, essentially converting fixed-width UCS-2 into variable-width UTF-16.

A surrogate pair is two UTF-16 units used to encode a single Unicode code point. For example, D83D DCA9 (two 16-bit units) encodes one code point, U+1F4A9.

The top 6 bits in surrogate pairs are used for the mask, leaving 2×10 free bits to spare:

   High Surrogate          Low Surrogate
+        D800        ++          DC00
+1101 10?? ???? ???? ++ 1101 11?? ???? ????

Technically, both halves of the surrogate pair can be seen as Unicode code points, too. In practice, the whole range from U+D800 to U+DFFF is allocated as “for surrogate pairs only”. Code points from there are not even considered valid in any other encodings.

This space on a very crammed Basic Multilingual Plane will never be used for anything good ever again

Is UTF-16 still alive?

Yes!

The promise of a fixed-width encoding that covers all human languages was so compelling that many systems were eager to adopt it. Among them were Microsoft Windows, Objective-C, Java, JavaScript, .NET, Python 2, QT, SMS, and CD-ROM!

Since then, Python has moved on, CD-ROM has become obsolete, but the rest is stuck with UTF-16 or even UCS-2. So UTF-16 lives there as in-memory representation.

In practical terms today, UTF-16 has roughly the same usability as UTF-8. It’s also variable-length; counting UTF-16 units is as useless as counting bytes or code points, grapheme clusters are still a pain, etc. The only difference is memory requirements.

The only downside of UTF-16 is that everything else is UTF-8, so it requires conversion every time a string is read from the network or from disk.

Also, fun fact: the number of planes Unicode has (17) is defined by how much you can express with surrogate pairs in UTF-16.

Conclusion

To sum it up:

Unicode has won.
UTF-8 is the most popular encoding for data in transfer and at rest.
UTF-16 is still sometimes used as an in-memory representation.
The two most important views for strings are bytes (allocate memory/copy/encode/decode) and extended grapheme clusters (all semantic operations).
Using code points for iterating over a string is wrong. They are not the basic unit of writing. One grapheme could consist of multiple code points.
To detect grapheme boundaries, you need Unicode tables.
Use a Unicode library for everything Unicode, even boring stuff like strlen, indexOf and substring.
Unicode updates every year, and rules sometimes change.
Unicode strings need to be normalized before they can be compared.
Unicode depends on locale for some operations and for rendering.
All this is important even for pure English text.

Overall, yes, Unicode is not perfect, but the fact that

an encoding exists that covers all possible languages at once,
the entire world agrees to use it,
we can completely forget about encodings and conversions and all that stuff

is a miracle. Send this to your fellow programmers so they can learn about it, too.

There’s such a thing as plain text, and it’s encoded with UTF-8.

Thanks Lev Walkin and my patrons for reading early drafts of this article.

+ +

A case for ClojureScript 2.0

2023-06-26T00:00:00Z

I was complaining the other day about the ergonomics of ClojureScript and realized an interesting thing.

Quick context:

Clojure is a modern Lisp.
ClojureScript is a Clojure dialect that compiles to JS.
From its very beginning, it relied on the Google Closure compiler in an elaborate plan to confuse people with Clojure/Closure naming (joking).
It has two important compilation modes: :none and :advanced.
:none is what you are supposed to develop with.
:advanced is what you ship: smaller bundle size, stripped of unused code, better performance, worse stacktraces.

So, I was complaining about compilation times and ergonomics of using :advanced mode and other devs could not understand me. Apparently, they all work in :none and their experience is much better.

This is where an interesting chain of cause and effect starts that leads (in my opinion) to what ultimately should become ClojureScript 2.0.

You see, the very existence of :advanced mode means you can’t really develop in :none.

I know, sounds like clickbait. Let’s unpack.

Advanced makes your code faster

First, :advanced does not just packages your code and trims the bundle size. It also improves its performance and can change its behavior.

Better performance is fine, of course. Who doesn’t love a little bit of extra speed that computer gives you for free, with no work from your side?

(Rich Hickey, for one. He once famously made a case why last should be slow where it could’ve been faster.)

Unless you are doing benchmarking (as I was), so you have to rely on :advanced and have to suffer worse experiences with everything else because of that.

Advanced may break your code

You see, most bundlers (to my knowledge, I might be ignorant here) try to do the best they can while not changing the behavior of your code. For example, if they can prove some code is unused, only then will they remove it. If they can’t, the code stays. Better safe than sorry.

Google Closure is different. Google Closure actively tries to destroy your code. You have to work against it to prove that your code is, in fact, used. Or that it shouldn’t be changed. Or that if you access, for example, className property on a JS object, it should stay named className and not be renamed to some fy or worse. The presumption of innocence does not apply here.

From ShadowCLJS README:

+
Ideally we want to use :closure as our primary JS Provider since that will run the entire application through :advanced giving us the most optimized output. In practice however lots of code available via npm is not compatible with the aggressive optimizations that :advanced compilation does. They either fail to compile at all or expose subtle bugs at runtime that are very hard to identify.
+

You can’t auto-generate externs. Nobody can

DataScript, unfortunately, is one of those libraries that need externs. Not because we do some weird shit there, but because of the nature of the problem.

Basically, queries are data, and data doesn’t get munged by Google Closure. But datoms are classes so their fields are getting munged by default. That’s why we have to write externs to work around that.

It’s not always bad, though. I’ve heard stories that people can develop whole applications without ever experiencing this problem.

I’ve also heard that there’s some “AI magic” (meaning: highly indeterministic heuristics) that is supposed to “automagically” detect cases like that and just “do the right thing™”.

Which is supposed to be good, right?

Well, not exactly. The fact that auto-deduced externs exist means people might forget that externs are sometimes essential for ClojureScript code to work.

For example, ShadowCLJS, one of the most popular ClojureScript dev tools today, ignores hand-written externs by default. Because they are supposed to be “automatically deduced”. And it works. Until it doesn’t. As you can guess, their users then come to me claiming that “DataScript is broken”.

Well, it is. But I didn’t break it. It’s the way we do things is broken.

BTW, did I tell you that upgrading your ClojureScript version might break things in new and exciting ways? Because it updates Google Closure, too, and the extern-deducing algorithm might change unpredictably between versions. And then it’s your problem, because, well, we didn’t really promise you anything, did we?

The ultimate promise of Google Closure compilation is: your code might work. It might not. It also might change between versions. Good luck.

Why is advanced mode needed?

When ClojureScript started, the main premise was that people will build websites with it.

After 10 years, I’d say that ClojureScript is best suited for web apps, not pages. The minimal bundle size, the performance—you won’t really put stuff like that on your landing page.

But a productivity app? Custom editor? Some complex UI? Sure! People don’t really care about bundle size in that case. They are already committed to using it, they have a JS bundle probably cached (unless you release 10 times a day), so it’s much less of a problem.

What I’m saying is: since we are not getting into really super-small, super high-perf, low overhead JS territory, maybe we can relax our constraints a little and choose a less aggressive bundler? The one that maybe produces slightly less optimal code, but code that doesn’t subtly and unexpectedly break?

Is there really a difference between, say, a 500k bundle and a 1M bundle? A practical one? One that users will definitely notice in a meaningful way?

Why not just always use `:none` mode?

It might seem that having more options is always better. Hey, do you want small bundle sizes and good perf? We got you covered. Great dev experience? We’ve got you too!

And that is partially true. For app developers, at least. I think some people just ship :none mode and it works for them. Why wouldn’t it?

For library authors, it’s worse. Because :advanced mode exists, just the fact of its existence, means we have to take it into account. We don’t really get to choose. People use it → we have to support it. In some sense having more options made life harder for us.

You can always look at a choice like that two ways. Tesla can charge your car for free or replace your battery with no waiting time. Free or fast, says Elon Musk. But it’s also a choice between slow or paid. PS5 games have performance mode or quality mode. A good picture or fast gameplay. Or: big latency or worse picture? You don’t just choose good parts here. You also choose bad ones.

How is JVM Clojure doing?

It’s very interesting to look at what JVM Clojure is doing differently. This is how my rant on Twitter started, actually: I was wondering why on JVM, which is designed for statically-typed languages, the Clojure experience is much more dynamic than on JS, where it’s almost comparable with C++ development (long building times, lots of options, bad stacktraces, etc)?

Well, because ClojureScript accidentally complected two things: performance optimizations and minification. I know Clojure devs are trained to be scared of the word “complected”, and its use here is intentional: I am trying to scare you.

Look at Clojure experience. I develop without any notion of jars, classes, paths, etc. There are no compilation options either. It Just Works™. When the time comes, I can compile my Clojure classes (or not) and package everything into a jar, which I then ship.

So there are two modes on JVM Clojure as well: dev mode and prod mode. Yet the dev code behaves exactly how it will in production. There’s no compromise. No choice to make. I can safely work in dev mode until the time comes to ship my code. And I know I don’t even need to check it a second time—it’ll just work. It’s guaranteed to work, even though the storage format (jar) is different.

Why can’t it be that way in ClojureScript? Because it uses Google Closure for both bundle size and performance optimizations.

You see, the ClojureScript compiler outputs less performant code and relies on Google Closure to improve its performance. So even if you personally are ok with larger bundles, it’s still really hard to leave free performance on the table.

What if advanced mode didn’t exist?

So what am I proposing? Basically,

Ditch Google Closure.
Move whatever performance optimizations it does into the ClojureScript compiler. Or accept that it’ll be slightly slower.
Use whatever bundler JS people use. Even if it outputs larger bundles. It’s okay, ClojureScript is already pretty thick anyways. The important part is that it should only make safe transformations and not try to destroy your code.

JS is an ecosystem. A strange one, but a huge one, too. So it was a very strange choice to ignore it completely or make it really hard to use. One of the selling points of JVM Clojure always was: to use whatever Java libraries you need. Using Java from Clojure is easier than from Java (not kidding).

Whereas in ClojureScript it’s more like: don’t use JS libraries. It’s very hard. There are a million “buts”. Are you in node or a browser?

That’s not the spirit, I would say.

And no matter what Rich Hickey's reasoning was, Google Closure is not part of the JS ecosystem. Nobody uses it, except, maybe, for Google.

Getting rid of Google Closure will make interop with JS much simpler (as far as I understand). So we will only potentially lose a little bit in bundle sizes and (maybe) performance? But there are so many low-hanging performance fruits in ClojureScript anyways maybe nobody will notice.

What’s important is what we’ll gain:

The Simplicity.

+ +Ease of Mind.

+ +Happiness.

I fell in love with Clojure because of how simple everything was. To this day I’m still reflecting on how the same things are unnecessarily complicated in other languages.

And I wish the same for ClojureScript users, I want them to feel the same transformative experience.

Where are your patches?

I know getting rid of Google Closure is a huge step. I’m not even sure if that’s possible in the current implementation or the current ecosystem.

That’s why I called this post ClojureScript 2.0. It’s a huge change. Lots of work. But I believe it’s the right path.

I also believe Michiel Borkent is working in the right direction with Cherry 🍒. I don’t know all the details but it looks like how I imagined the Clojure compiler for JavaScript should look like. So maybe help him out?

All in all, the goal of this post was not to diss on ClojureScript. It’s absolutely great that it exists, and its existence has been paying my bills for the last seven(-ish?) years at least. I just was excited that I finally saw how a very early decision (use Google Closure) eventually led to “Clojure feels like the future, ClojureScript feels like developing C++” in some cases. I hope I described that path clearly enough and it’s of interest to you too.

Again, I’m not saying it’s wrong, bad, or anything, or that anybody should’ve predicted it. It took me 10 years to realize what was going on. I only hope it will help someone in the future if any new initiatives get developed.

Peace.

+ +

Humble Chronicles: Managing State with Signals

2023-05-18T00:00:00Z

After the previous post, I figured that the best way to decide on the direction for Humble UI is to make an experiment.

And I did. I implemented a reactive/incremental computation engine (signals) and wrote a simple TodoMVC in it. Following are my thoughts on it.

Signals

The idea behind signals is very simple: you declare some mutable (!) data sources:

(s/defsignal *width
+  16)
+
+(s/defsignal *height
+  9)

And then create a derived (computed) state that depends on those:

(s/defsignal *area
+  (println "Computing area of" @*width "x" @*height)
+  (* @*width @*height))

Now, the first time you dereference *area, it is computed:

@*area => 144
+;; Computing area of 16 x 9

After that, any subsequent read is cached (notice the lack of stdout):

@*area => 144

If any of the sources change, it is marked as dirty (but not immediately recomputed):

(s/reset! *width 20)

But if you try to read *area again, it will recompute and cache its value again:

@*area => 180
+;; Computing area of 20 x 9
+
+@*area => 180
+;; (no println)

You can check out implementation here and some usage examples here. The implementation is a proof-of-concept, so maybe don’t use it in production.

Stable object references

The appeal of signals is that, when data changes, only the necessary minimum of computations happens. This is, of course, cool, but not entirely free — it comes at a cost of some overhead for managing the dependencies.

I was particularly interested in using signals for Humble UI because they provide stable references. Let’s say you have a tabbed interface that has a checkbox that enables a text field:

+ +

Now, our state might look somewhat like this:

(s/defsignal *tab 
+  :first)
+
+(s/defsignal *checked?
+  false)
+
+(s/defsignal *text
+  nil)
+
+(s/defsignal *tab-content
+  (column
+    (ui/checkbox *checked?)
+    (when @*checked?
+      (ui/text-field *text))))
+
+(s/defsignal *app
+  (case @*tab
+    :first  ...
+    :second *tab-content
+    :third  ...))

and the beauty of it is unless the user switches a tab or plays with a checkbox, *tab-content will be cached and NOT recomputed because its dependencies do not change!

And that means that no matter how many times we dereference *tab-content e.g. for rendering or layout, it will always return exactly the same instance of the checkbox and text field. As in, the same object. Same DOM node, if we were in the browser.

Cool? Cool! No diffing needed. No state tracking and positional memoization either. We can put all internal state into objects as fields and not invent any special “more persistent” storage solution at all!

This was my main motivation to look into incremental computations. I don’t really care about optimal performance, because—how much is there to compute in UI anyways?

And also—it’s not obvious to me that if you make every + and concat incremental it’ll be a net win because of overhead. But stable objects in a dynamic enough UI? Lack of diffing and VDOM? This is something I can use.

Props drilling

One of the examples where VDOM model doesn’t shine is props drilling. Imagine an app like this:

(ui/default-theme
+  {:font-ui ...}
+  (ui/column
+    (ui/row
+      (ui/tabs
+        ...
+        (ui/tab
+          (ui/button
+            (ui/label "Hello")))))))

The actual details don’t matter, but the point is: there’s a default theme at the very top of your app and a label somewhere deep down.

If you pass font-ui as an argument to every component, it will create a false dependency for every intermediate container that it passes through. When the time for the update comes, the whole UI will be re-created:

+ +

In a perfect world, though, font-ui change should only affect components that actually use that font. E.g. it shouldn’t affect paddings, backgrounds, or scrolls, but should affect labels and paragraphs.

Well, incremental computation solves this problem beautifully! If you make your default font a signal, then only components that actually read it will subscribe to its changes:

+ +

On the other hand, how often do you change fonts in the entire app? Should it really be optimized? The question of whether this use case is important remains open.

Reactive vs Incremental

Now let’s dig into implementation details a little bit. What we have so far is, I believe, called reactive, but not incremental. To be called incremental we must somehow reuse, not just re-run, previous computations.

A simple example. Imagine we have a list of todos and a function to render them. Then we can define our UI like this:

(s/defsignal *todos
+  ...)
+
+(s/defsignal *column
+  (ui/column
+    (map render-todo @*todos)))

This would work fine the first time, but if we add a new to-do, the whole list will be re-rendered. *todos changes, *column body gets re-executed, render-todo is applied to every todo again by map.

To solve just this problem, we could introduce incremental s/map that only re-computes the mappings that were not computed before:

(def column
+  (ui/column
+    (s/map render-todo *todos)))

Under the hood, s/map caches previous computation and its result and, when re-evaluated, tries to reuse it as much as possible. Meaning, if it already saw the same todo before, it will return a cached version of (render-todo todo) instead of calculating it anew.

Two important things to note here. First, if we care about object identities, we have to use incremental map. Otherwise adding new todo to the end of the list will reset the internal component state of every other one. Not good!

Second, although “incremental map” sounds fancy and smart, under the hood it does the same thing that React does: diffing. It diffs new collection against previous collection and tries to find matches.

It is (probably) a perf win overall, but, more importantly, diff still does happen. That’s the reason why all incremental frameworks have their own versions of for/map:

{#each arr as el}
+  {el}
+{/each}

I can imagine it could be better when a diff happens on the data layer instead of on the final UI layer because generated UI is usually much larger than the source data. Either way, at least you can choose where it happens.

The bad news is, you have to think about it, whereas in React model you usually don’t bother with such minute details at all.

One can imagine that we’ll need incremental versions of filter, concat, reduce etc, and our users will have to learn about them and use them if they want to keep stable identities. And we’ll have to provide enough incremental versions of base core functions to keep everyone happy, and potentially teach them to write their own. Sounds harsh.

Effects

One important feature we’re missing in our incremental framework is effects.

We implement a mixed push/pull model: recalculating values is lazy (not done until explicitly requested), but marking as dirty is eager (immediate dependencies are marked as :dirty and their transitive deps are marked with :check, which means might or might not be dirty):

(s/defsignal *a
+  1)
+
+(s/defsignal *b
+  (+ 10 @*a))
+
+(s/defsignal *c
+  (+ 100 @*b))
+
+@*a ; => 1
+@*b ; => 11
+@*c ; => 111
+
+(:state *b) ; => :clean
+(:value *b) ; => 11
+
+(s/reset! *a 2)
+
+(:state *b) ; => :dirty
+(:value *b) ; => 11
+(:state *c) ; => :check
+(:value *c) ; => 111
+
+@*b ; => 12
+
+(:state *c) ; => :dirty
+(:value *c) ; => 111
+
+@*c ; => 112

Or for us visual thinkers:

+ +

For details, see Reactively algorithm description.

An effect is a signal that watches when it gets marked :check (something down the deps tree has changed) and forces its dependencies to see if any of them are actually :dirty. If any of them are, it evaluates its body:

(s/defsignal *a
+  1)
+
+(s/defsignal *b
+  (mod @*a 3))
+
+(s/effect [*b]
+  (println @*a "mod 3 =" @*b))
+
+(s/reset! *a 2) ; => "2 mod 3 = 2"
+(s/reset! *a 3) ; => "3 mod 3 = 0"
+(s/reset! *a 6) ; => (no stdout: *b didn’t change)

This is exactly what we need to schedule re-renders. We put an effect as a downstream dependency on every signal that was read during the last draw. That means we’ll create an explicit dependency for everything that affected the final picture one way or another.

+ +

Then, when any of the sources change and the redraw effect is actually a downstream dependency on it, we’ll trigger a new redraw.

Disposing signals

What I did have problems with is resource management. First, let’s consider something like this:

(s/defsignal *object
+  "world")
+
+(def label
+  (s/signal (str "Hello, " @*object "!")))

Now imagine we lose a reference to the label. Irresponsible, I know, but things happen, especially in end-user code. The simplest example: we’re in REPL and we re-evaluate (def label ...) again. What will happen?

Well, due to the nature of signals, they actually keep references to both upstream (for re-calculation) and downstream (for invalidation) dependencies. Meaning, the previous version of the signal will still be referenced from *object along with the new one:

+ +

We can introduce dispose method that could be called to unregister itself from upstream, but nobody can guarantee that users will call that in time. It’s so easy to accidentally lose a reference in a garbage-collected language!

And this is what I am struggling with. The signal network has to be dynamic. Meaning, new dependencies will come and go. But de-registering something doesn’t really feel natural in Clojure or even Java code, and there’s no way to enforce that every resource that is no longer needed will be properly disposed of.

A common solution is to make downstream references weak. That means, if we lost all references to the dependant signal (label on the picture below), it will eventually be garbage collected.

+ +

What I don’t like about that solution (that we use anyways in the prototype) is that until GC is called, those unnecessary dependencies still hang around and take resources e.g. during downstream invalidation.

+ +

One idea is to dispose of signals explicitly when their component unmounts. It works for some signals, but not in general. Consider this:

(s/defsignal *text
+  "Hello")
+
+(ui/label *text)

*text signal is created outside of the label and shouldn’t be disposed of by it. At the same time,

(ui/label
+  (s/signal (str @*text ", world!")))

In this case, the signal is created specifically for the label, thus should be disposed of at the same time as the label. But how to express that?

Keep in mind that we probably want both use cases at the same time:

(ui/column
+  (ui/header *text)
+  (ui/label
+    (s/signal (str @*text ", world!")))
+  (ui/label *text))

Eventually, unused signals will be cleaned up by GC, so we can rely on that. I’m just not sure what sorts of problems it might cause in practice.

Disposing components

Same problem I have with signals I also have with components. Because all components are values, nothing stops me from saving them in a var, using them multiple times, etc. Consider this UI:

(s/defsignal *cond
+  true)
+
+(def the-label
+  (ui/label "Hello"))
+
+(def *ui
+  (s/signal
+    (if @*cond
+      the-label
+      (ui/label "Not hello"))))

If we toggle *cond on and off, the-label will appear and disappear from our UI, calling on-mount and on-unmount multiple times. So if we do some resource cleanup in on-unmount, we should somehow restore it in on-mount? Feels strange, but why not?

(core/deftype+ Label [*paint *text ^:mut *line]
+  protocols/ILifecycle
+  (-on-mount-impl [_]
+    (set! *line
+      (s/signal
+        (.shapeLine
+          core/shaper
+          (str (s/maybe-read *text))
+          @*font-ui
+          ShapingOptions/DEFAULT))))
+  
+  (-on-unmount-impl [_]
+    (s/dispose! *line)
+    (set! *line nil)))

This way, if a component needs some heavy resources for rendering (textures, pre-rendered lines, or other native resources) it can clean it up and restore only when it’s actually on the screen. Or rely on GC once again (not recommended).

Mounting components

Another thing that I used to take for granted since the world switched to React: lifecycle callbacks. A lot of stuff comes down to these callbacks. Enabling/disabling signals. Freeing expensive resources held by components. Users’ use cases, like setting a timer or making a fetch request. It is so convenient to be able to tie some expensive resource’s lifetime to the lifetime of a component. We certainly want these!

How does React do it? Well, it takes mount/unmount API away from you and takes control over it, so it can guarantee to call you back at the right time.

The solution I came up with is very simple: the component is mounted if it was drawn during render, and not mounted otherwise. At the very top level, I’m keeping track of everything that was rendered last frame and what is rendered this frame. For new stuff, -on-mount is called, for stuff that’s no longer visible, -on-unmount. The gotcha here is, as I said above, that some components might “come back” after being unmounted. I guess it’s ok?

Gotchas

Working with an incremental framework breaks both imperative and functional intuition. It’s a whole other thing. I made a lot of mistakes and had to think about stuff I usually don’t have to think about. Here are a few gotchas:

Dependency too wide

Imagine we want to render a TODO from very simple EDN data:

+ +

We might write something like this:

(defn render-todo [*todo]
+  (let [*text (s/signal
+                (str (:id @*todo)))]
+    (ui/label *text)))

This render function returns a label object that has a signal as its text. So far so good.

The problem is, we over-depend here: we only use :id from *todo but we depend on the entire thing. A better solution would be:

(defn render-todo [*todo]
+  (let [*id   (s/signal (:id @*todo))
+        *text (s/signal (str *id))]
+    (ui/label *text)))

which seems a bit too tedious to write. It probably doesn’t matter all that much in this particular case, but what if computations are more expensive?

My point is: it’s too easy to make this mistake.

Ambrose Bonnaire-Sergeant has pointed out that Reagent and CljFX solve this by providing an explicit API:

@(subscribe [:items])

Dependency at the wrong time

Imagine you have a UI like this:

+ +

You have a signal that is hooked up to your text field and a button that converts it into a label:

(s/defsignal *text
+  "Your name")
+
+(s/defsignal *list
+  [])
+
+(def app
+  (ui/column
+    (s/mapv ui/label *list)
+
+    (ui/text-field {:placeholder "Type here"}
+      *text)
+
+    (ui/button
+      #(s/swap! *list conj *text)
+      (ui/label "Add"))))

Do you see it? We actually store the original signal in *list instead of making a copy. This way, when we edit text, every element in our list will also be edited!

We might fix it like so:

#(s/swap! *list conj (s/signal @*text))

but it’s no good either.

Yes, we create a new signal, but it depends on the old one :) This is an API problem, and I think maybe I should have separate functions for source signals and derived signals. Right now the proper way to write it would be:

#(let [text @*text]
+   (s/swap! *list conj (s/signal text)))

which is almost identical! but the result is very different.

It reminds me a lot about Clojure laziness puzzles, which is both ok (we all learned to deal with them) and not so much (the best way to deal with laziness is not to use it).

Recomputing too much

There’s another gotcha in the previous example. column takes a collection or a signal that contains a collection, so we have to satisfy that:

(ui/column
+  (s/signal
+    (concat
+      (mapv ui/label @*list)
+      [(ui/text-field ...)
+       (ui/button ...)]))))

But now our s/signal will re-create a text-field and a button each time *list changes. The solution might be:

(let [text-field (ui/text-field ...)
+      button     (ui/button ...)]
+  (ui/column
+    (s/signal
+      (concat
+        (mapv ui/label @*list)
+        [text-field
+         button]))))

which, again, kind of breaks referential transparency. Depending on where we allocate our components, an app behaves differently. Doesn’t matter for the button, as it doesn’t have an internal state, but does matter for the text field.

Alternatively, we might introduce a version of concat that accepts both signals wrapping sequences as well as sequence values. Then argument evaluation will lock the text-field value for us:

(ui/column
+  (s/signal
+    (s/concat
+      (s/mapv ui/label *list)
+      [(ui/text-field ...)
+       (ui/button ...)])))

Ambiguity

It was not always clear to me which parts of the state should be signals and which should be values. Right now, for example, a list of todos is signal containing signals that point to todos:

(defn random-todo []
+  {:id       (rand-int 1000)
+   :checked? (rand-nth [true false])})
+
+(s/defsignal *todos
+  [(s/signal (random-todo))
+   (s/signal (random-todo))
+   (s/signal (random-todo))
+   ...])

This way list of todos could be decoupled from the todos themselves. When we add new todo, we need to change the list and generate a new component. But when an individual todo is e.g. toggled, it’s handled entirely inside and shouldn’t affect the list.

I guess this solution is okay, although double-nested mutable structures do give me pause.

Could the same be done “single atom”-style? Probably, with some sort of keyed map operator and lenses?

(s/defsignal *todos
+  [(random-todo)
+   (random-todo)
+   (random-todo)
+   ...])
+
+(def *todo-0
+  (s/signal
+    {:read  (nth @*todos 0)
+     :write #(s/update *todos assoc 0 %)}))

The same ambiguity problem happens here:

(s/defsignal *text
+  "Hello")
+  
+(ui/label *text)

(s/signal
+  (ui/label @*text))

Should I use a label that contains a signal or a signal that contains a label? Both are viable.

This is not necessarily a problem, just an observation. I guess I prefer Python’s “There should be one—and preferably only one—obvious way to do it” to Perl’s “There’s more than one way to do it”.

Repeating computations

I have a few constants defined in my app, including *scale (UI scale, e.g. 2.0 on Retina) and *padding (in logical pixels, e.g. 10).

But actual rendering requires screen pixels, not UI pixels. For that, I was using the derived signal calculated inside the padding constructor:

(defn padding [*amount]
+  (map->Padding
+    {:amount (s/signal (* @*scale @*amount))}))

The problem? I was using default *padding everywhere:

(padding *padding ...)
+...
+(padding *padding ...)
+...
+(padding *padding ...)
+...

This way I ended up with dozens of equivalent signals (different identities, same value, dependencies, and function) that multiply the same numbers to get the same result.

Is it bad? In this case, probably not. It just doesn’t feel as clean, considering that the rest of the app uses the absolute required minimum of computations and the dependency graph is carefully constructed.

But I don’t see a way to merge identical signals together, either. I guess we’ll have to live with this imperfection.

Pre-compilation

I started this experiment inspired by Svelte, Solid, and Electric Clojure. All of them have compilation steps that I wanted to avoid.

The most non-obvious result I get from this is that it looks like you need pre-compilation for better ergonomics and resource management. Both of these problems go away if we don’t let users interact with our incremental engine directly, but instead, do it for them.

We can replace calls to if/map/concat with their incremental versions transparently, track dependencies reliably, and add dispose calls where needed—basically, all these things you can’t trust humans to get right.

I am also getting reports that Reagent (that has a similar thing, r/track) is hard to use correctly at scale. Can anyone confirm?

Maybe it’s worth running another experiment to see if I can get pre-compilation working and how much it helps.

Results

Some preliminary results from the experiment:

It works

After some massaging, I was able to build incremental TodoMVC that keeps the state of its components that do not directly change.

Here’s a video:

+ + + + +

The magenta outline means that the component was just created and is rendered for the first time.

Notice how when I add new TODO only its row is highlighted. That’s because the rest reuses the same components that were created before.

When you switch between tabs, it causes some of the rows to be filtered out. When you go back to “All”, only the ones that were not visible are recreated.

Also, notice the same effect on tabs: when you switch e.g. from “All” to “Active”, “All” becomes a button but “Active” becomes just a label, so they both have to be recreated. But “Completed” stays a button, so it doesn’t get recreated.

And the last thing: when I toggle TODOs, nothing gets highlighted. This is because I made labels accept signals as text:

(s/defsignal *text
+  "Hello")
+
+(ui/label *text)

So the label could stay the same while the text it displays changes. Not necessary, but feels neat, actually. Another way to do it would’ve been:

(s/signal
+  (ui/label @*text))

Then it would be highlighted on the toggle:

+ + + + +

It feels very satisfying

...knowing no computation is wasted on diffs and only the necessary minimum of UI is recreated.

Props drilling works

I made UI scale, padding, and button fill color signals and when I change them necessary parts of UI are updated:

+ + + + +

This feels very satisfying, too: knowing that you made the dependency very explicit and very precise, not the hacky “let’s just reset everything just in case” way. And it requires no special setup, it “just works”.

No VDOM needed

I don’t have to implement VDOM and diffing! And I don’t need both heavy- and lightweight versions of each component. I don’t need to track the state separately from the components. That’s a huge burden off my shoulders.

We need incremental algorithms

I do need to provide a set of incremental algorithms. Incremental map, incremental filter, concat etc. for macro, too.

Ideally, we want users to be able to write their own.

It breaks intuition

Working with incremental computations could be tricky. Making a mistake is easy, and double-checking yourself is hard, so it’s hard to know if you are doing the right thing.

But it seems that the stakes are not that high: the worst that could happen is you re-create too much and your performance suffers. I’d say it’s a \~similar deal you get with React.

Is there a deeper reason?

There’s probably a good reason React won and FRP/incremental remain marginal technologies that have been tried dozens of times. I understand the appeal, but I also see how it’s not everybody’s cup of tea.

OTOH, Reagent seems to be doing fine in Clojure land, although many people prefer to pair it with re-frame.

Source code

If you are curious, the code is on Github. The run script is at scripts/incremental.sh.

Let me know what you think! And I’m going to try VDOM approach next. And then I guess I’ll have to make a decision matrix.

+ +

Humble Chronicles: State Management

2023-04-29T00:00:00Z

Recently I’ve been trying to improve state management and component API in Humble UI. For that, I’ve tried to read and compile all the possible known approaches and synthesize something from them.

I haven’t decided on anything for Humble UI yet, let’s say I’m in an experimenting phase. But I think my notes could be useful to quickly get a birds-eye overview of the field.

This is a compilation of my research so far.

Object-oriented user interface (OOUI)

Classic UIs like Swing, original native Windows/macOS APIs and the browser’s DOM are all implemented in an OOP manner: components are objects and they have methods: addChild, removeChild, render. That fits so well, actually, that you might think OOP was invented specifically for graphical UIs.

The problem with that approach is code duplication: you have to define both how your initial UI is constructed and how it will change in the future. Also, amount of transitions scales as N² compared to the amount of states, so at some point you either get overwhelmed or miss something.

Nevertheless, OOUIs show great performance, a very straightforward programming model and are widely used everywhere.

+ +

Humble UI — current state

Given the above, it’s only natural that Humble UI started with the OOP paradigm. Yes, we have stateful widgets and component instances, not functions or classes.

Look, mutable fields and inheritance! In Clojure!

(defparent AWrapper [child ^:mut child-rect]
+  protocols/IComponent
+  (-measure [this ctx cs]
+    (when-some [ctx' (protocols/-context this ctx)]
+      (core/measure child ctx' cs))))
+
+(core/deftype+ Clip []
+  :extends core/AWrapper
+  
+  protocols/IComponent  
+  (-draw [_ ctx rect ^Canvas canvas]
+    (canvas/with-canvas canvas
+      (canvas/clip-rect canvas rect)
+      (core/draw child ctx rect canvas))))

These objects can lay out, draw themselves and handle events, but in true Clojure fashion, they can’t be modified.

I mean, theoretically, sure, I could add something like setChildren and the like, but what’s the point if we are not going in that direction anyway?

But wait! — you’d say. I’ve certainly seen Humble UI apps modifying themselves!

+ + + + +

And you’ll be right. There’s a special type of component, dynamic, that doesn’t have a fixed list of children. Instead, it takes a function that generates and caches them based on its inputs. When inputs change, old children are thrown away and new ones are generated.

In this example, when *clicks changes, a new label will be created and the old one will be thrown away.

(def *clicks
+  (atom 0))
+
+(def app
+  (ui/dynamic _ [clicks @*clicks]
+    (ui/label (str "Clicks: " clicks))))

You can get quite far with that approach. However, not far enough. The problem with dynamic is that it throws away the entire subtree, no matter which parts have changed. Consider

(ui/dynamic _ [p @*p]
+  (ui/padding p
+    (ui/rect (paint/fill 0xFFF3F3F3)
+      (ui/label "Label"))))

In this example, only ui/padding component needs to be re-created, but its children would be thrown away and re-created, too. Sometimes it can be fixed, actually, by writing it this way:

(let [body (ui/rect (paint/fill 0xFFF3F3F3)
+             (ui/label "Label"))]
+  (ui/dynamic _ [padding @*padding]
+    (ui/padding padding
+      body)))

Remember — components are values, so the body reference will stay the same and will be captured by dynamic.

This is both good and bad: it works, but it’s kinda backward (you wouldn’t want to write your UI this way).

It also creates very confusing “owning” semantics. Like, who should “unmount” the body in that case? Should it be padding? Or dynamic? Actually, it can be neither of them, because body outlives them both, and they have no way of knowing this.

Why is it a problem? Stateful components. If I throw away and re-create the text field, for example, it’ll lose selection, cursor position, scroll position, etc. Not good.

Funnily enough, current implementation of text field asks you to hold its state because it has nowhere to put it reliably.

But overall, we managed to get quite far with this approach and polish some stateful components, so I don’t consider it a waste.

Declarative UIs

So at some point, programmers decided: we’ve had enough. Enough with OOUI, we don’t want to write

window.add(button);
+window.show();

anymore. We want comfort!

And that’s how declarative UIs were born. The idea is that you describe your UI in some simpler language, bind it to your data, and then the computer goes brrr and displays it somehow.

Well, why not? Sounds good, right?

This is what people have come up with.

Templates

You describe your UI in XML/PHP/HTML/what have you, sprinkle special instructions on top, give it to a black box and it magically works!

Example: Svelte

+
+

The upsides of this approach are that language is probably very simple and very declarative, and also that you can do a lot of optimizations/preprocessing before turning it into UI. Maximum declarativity!

The downsides are, of course, that you have to work in a second, “not real” language, which is usually less powerful, harder to interop with, and less dynamic.

Probably because of the limitations of templates, the MVVM pattern was born: you prepare your data in a real language and get it into a shape that can be consumed by a simple template.

Procedural DSLs

You call builder functions in the right context and your UI framework somehow tracks them and turns them into components. Examples would be Dear Imgui or Jetpack Compose:

ImGui::Text("Hello, world %d", 123);
+if (ImGui::Button("Save"))
+    MySaveFunction();
+ImGui::InputText("string", buf, IM_ARRAYSIZE(buf));
+ImGui::SliderFloat("float", &f, 0.0f, 1.0f);

Notice that you don’t explicitly “add” or “return” components anywhere. Instead, just calling builders is enough. Almost procedural style :)

The upside: the code is very compact.

The downside: you can’t work with components as values. Like, can’t put them in an array and reverse it, or take the first 10, or something like that. Your builders become lambdas, and lambdas are opaque: you can’t do much about them except call. Call sites start to matter, where they normally don’t: what looks like a normal program has a few non-obvious gotchas.

Value-oriented DSLs

In value-oriented DSLs, you return values from your components. Like in React:

export default function Button() {
+  return (
+    
+  );
+}

Notice that you return the button, not call some constructor that adds it to the form. How you get — doesn’t matter. The only thing that matters is the actual value you return. You can do what you want with it: use, ignore, compare, use twice, cache, throw away.

This is also the most natural way to write programs in my opinion: pure functions taking and returning data. It also suits Clojure the best, so we’ll probably want something like that for Humble UI.

Note also that

Adventures in REPL implementation

2023-03-09T00:00:00Z

It’s a strange thing to announce, but I wrote Clojure plugin for Sublime Text. Again.

I mean, the previous version worked fine, but it had a few flaws:

REPL depended on syntax highlighting (yikes!),
the whole implementation was in a single file,
it was hard to add REPLs.

So, let’s do it again, almost from scratch, and right this time!

What is REPL?

In a nutshell, REPL consists of three parts: client, server, and communication protocol between them.

Here’s an architectural diagram for you:

+ +

REPL Client

As you can see from the diagram, REPL clients live in a variety of environments dictated by their host: Java for Idea, Python for Sublime Text, JS for VS Code, etc. In my case, it was Sublime Text, so the environment I was stuck with happened to be Python 3.8.

Of course, writing client-server apps is not hard in any language. Unfortunately for us, REPL client needs to be able to read, speak and actually understand Clojure for a few features:

Problem 1: Automatic namespace switching

When you go to a file and eval something there, you want it to be run in the context of that file’s namespace. But how to figure out which namespace it is, without parsing Clojure source file?

The level of understanding is non-trivial: there could be multiple namespace declarations, not necessarily at the top of the file:

+ +

The declaration itself could be complex, too, containing comments and/or meta tokens before the actual name:

+ +

Problem 2: Form boundaries

I want a shortcut that evals “the topmost form” around my cursor. To do so, I need to know where those boundaries are.

Notice how I don’t explicitly “select” what I want to evaluate. Instead, REPL client finds form boundary for me:

+ + + + +

This is tricky, too. For the very least, you can count parens, but even then you’d have to be aware of strings.

To make things harder, Clojure also has reader tags #inst "2023-02-24", metadata ^bytes b and different weird symbols like @ or #' that are not wrapped in parens but are still considered to be part of the form.

Bonus points for treating technically second-level forms inside (comment) as top-level.

All of this requires a really deep understanding of Clojure syntax.

Problem 3: Indentation and pretty-printing

Clojure Sublimed originally started when I wasn’t happy with what happened when I press “Enter” in Clojure file. Cursor would go to the wrong place, and I (subjectively) was spending too much time correcting it, so I decided to fix that once and for all.

+ + + + +

Indentation is not really a REPL concern, but it’s another part of Clojure Sublimed that requires a model of Clojure code.

Indentation logic re-applied to the whole file is formatting, so I got this one for free (both follow Better Clojure formatting rules).

+ + + + +

Finally, there’s a question of pretty printing, which is basically indentation + deciding where to put line breaks. Normally this would be done on Clojure side, but doing it on a client has clear advantages:

you have to send less data when transferring evaluation results (no need to send spaces at the beginning of the line),
it works for every Clojure REPL the same,
it can adjust for your current editor configuration instead of some arbitrary server-side number like 80 characters.

Wrapping on current window width

Another upside is that I can adjust pretty-printing rules to my liking, of course.

Clojure parser in Python

So, client lives inside your code editor and needs to understand Clojure before it starts communicating with it. Meaning, without Clojure runtime. Meaning, we had to parse Clojure in Python!

This is where things get hard because support for libraries, especially native ones, is not great in Sublime Text. Meaning, pure Python implementation!

I was put off by that task for a long time because it felt enormous. In the first Clojure Sublimed version, I deduced this information from syntax highlighting (Sublime Text already parses your source code for highlighting, and you can kind of access the results of that). But this time I wanted to make things right.

And, in fact, it turned out not to be all that bad! Clojure, like any Lisp, is relatively easy to parse. This is the entire grammar:

+ +

Full source at GitHub.

Some notable details below.

Parser: Cutting Corners

I cut some corners to write less code and get max performance by e.g. not distinguishing between numbers, keywords, symbols — they are all just tokens. It’s probably not hard to add them, but I don’t really need them for what I do, so they are not there.

Parser: Performance

I haven’t spent any serious time trying to optimize the parser. In its current state, it can go through clojure.core (enormous 8000 loc file) in 170ms on my M1 Mac.

Clojure itself does the same in roughly 30-50 ms, though, so there’s definitely potential for improvement.

Parser: Incrementality

Should be possible one day :) So far performance is good enough to re-parse on each “enter” keypress.

Parser: Testing

Parse trees could be quite hard to navigate, and twice as hard to compare.

Parsing also requires a lot of tests to get everything right and to avoid regressions. So having a nice and ergonomic way to write and inspect tests was super important.

I ended up copying test syntax from tree-sitter.

+ +

Test runner just compares the actual result with the expected one string-wise and if they are different, reports an error in a nice to comprehend table:

+ +

Having this early on saved me a ton of time and I am 100% happy I made that investment.

Parser: Error recovery

Parsing valid Clojure code is quite easy. But code in the process of editing is not always correct, even syntactically. That means that our parser has to work around errors somehow!

So I decided to see how people smarter than me do it and found this:

+
In the yacc and bison parser generators, the parser has an ad hoc mechanism to abandon the current statement, discard some parsed phrases and lookahead tokens surrounding the error, and resynchronize the parse at some reliable statement-level delimiter like semicolons or braces.
+

So yeah, I guess no beautiful theory on error recovery.

For our purposes, though, it was quite simple: see something that you don’t understand? That must be an error. Most stuff gets consumed as a symbol or a number, though, so these were rare.

We did accept some invalid programs as valid, but that’s okay for our use case: find expression boundaries, throw it over the fence, and let Clojure work out the rest of the details.

Parser: Accidentally quadratic

One funny thing happened during testing: I noticed that sometimes the parser was becoming ultra-slow on moderately-sized files. Like, seconds instead of milliseconds. That means I had quadratic behavior somewhere.

+ +

And that was indeed the case. The first version of the parser, roughly, was parsing parens/brackets/braces like this:

Seq(Char("["),
+    Repeat(Choice('_gap', '_form')),
+    Char("]"))

This basically means: if you see an opening bracket, consume forms and whitespace inside as long as you can, and in the end, there must be a closing bracket.

Well, what if it’s not there? That means it wasn’t a 'brackets' form in the first place! This is technically correct, but also means we have to mark the opening bracket as an error and then re-parse everything inside it again. That’s your quadratic behavior right here!

A simple change got rid of this problem:

Seq(Char("["),
+    Repeat(Choice('_gap', '_form')),
+    Optional(Char("]")))

Technically, this accepts incorrect programs. In practice, though, it works exactly as we need for indentation: we only care about opening parens up to the point of the cursor and don’t really care what happens after it.

Parser: Conclusion

Writing the parser was very fun! So many little details to figure out and get right, but in the end, when everything snaps into place, it’s so satisfying!

I also now understand why Lisps were so popular back in the day: they are really made for ease of implementation. Hacking together an entire parser from scratch in a weekend — can you imagine it for something like C++ or Python?

Anyways, if you need Clojure parser in Python, take a peek at my implementation — maybe it’ll help you out!

Protocol

Let’s move to the second part of our architecture: the communication channel.

How does a server talk to a client? Die-hard Clojure fans would answer immediately: EDN! But it’s not that simple.

Yes, EDN is the simplest thing for Clojure users. But what about the rest of the world? Don’t forget that on the other side there’s an arbitrary platform and, despite Rich’s best efforts, EDN is not as widespread as we’d like.

Just send... forms?

This is what clojure.core.server/repl does. Basically, it’s the same interactive experience as with command-line REPL, but over a socket:

+ + + + +

Not machine-friendly at all.

Half-EDN

Type in forms, receive EDN-formatted output. clojure.core.server/io-prepl does that:

+ +

Half machine-friendly and you have to be able to parse EDN.

JSON + EDN

Following the half-EDN example, our protocol doesn’t have to be symmetric, either. If we make ease of implementation our first priority, we can go crazy:

Client sends EDN, which is easy to parse in Clojure,
Server sends JSON, which could be parsed with Python stdlib.

Not elegant, but, you know, gets the job done. In both cases, messages are simple enough to be composed with string concatenation, so we only really care about parsing.

The only problem I have with this solution is that it offends my sense of beauty.

Bencode

nREPL also had this problem: a common denominator for multiple clients in all possible languages. Their answer? Bencode.

Bencode is a simple binary encoding developed for BitTorrent. And when I say simple, I mean very simple. Yes, simpler than JSON.

This is the entire Bencode grammar:

number = '0' / '-'? [1-9][0-9]*
+int    = 'i' number 'e'
+list   = 'l' value* 'e'
+dict   = 'd' (value value)* 'e'
+string = length ':' bytes
+length = '0' / [1-9][0-9]*
+value  = int / list / dict / string

And here are some actual messages when communicating with nREPL server:

+ +

Bencode is not supported out of the box by either Python or Clojure, but implementation easily fits in 200 LoC.

Problem? It’s binary. Unfortunately, you can’t run binary protocols on top of Socket Server, only the text ones. So to be able to use bencode you’ll have to start your own server.

Clojure Sublimed uses bencode for connecting to nREPL servers.

MessagePack

MessagePack is beautiful, exactly as I would’ve designed a compact binary serialization format. Everything is length-prefixed, super-simple to implement and you can support only parts that you actually use.

But it’s binary, so can’t be used on top of Socket Server, and I’m not prepared to write my own REPL server yet.

Consider voting for this issue and the situation might change! I believe Clojure deserves binary REPLs as much as text-based ones.

EDN both ways

A lucky coincidence saved me here. Remember the first part where I was writing Clojure parser? Guess what? Since EDN is a subset of Clojure, my parser also can parse EDN well enough to understand REPL server responses!

This is what my upgraded Socket Server REPL looks like on the wire:

+ +

Yes, it looks like nREPL over EDN.

No, it’s not exactly nREPL, it’s subtly different (see the server breakdown below), so there can be more chaos.

Did I invent another wheel? Maybe. But it’s a good wheel and it suits my needs well.

A note on message boundaries

The tricky part of EDN-on-the-wire? How to separate messages.

Clojure has a streaming parser: it consumes data from socket char-by-char and parses it as it goes until it reads a complete form. That’s why, for example, you can’t evaluate something like (+ 1 2 in the REPL, no matter how many times you press Enter.

But my Python parser wasn’t streaming :( You give it a string, it’ll parse it. But it can’t tell you how much of that string to read from a socket. If only TCP was message-oriented — one can dream!

So the solution was... split on newlines :) Lucky for me, the default Clojure printer escapes newlines in strings, so it can’t occur inside the message.

EDN doesn’t exactly forbid newlines, though, so let’s hope they won’t suddenly start to appear one day.

Server

Finally, the third and final part of our architecture: the server. When I only started learning about Clojure and Lisps, I imagined that REPL is literally that:

+ +

Because of that ignorance, it was hard for me to understand why there are different REPL implementations and why you need to “implement” REPL at all.

Let’s go from the simplest case to more complex ones.

Naive REPL

Funny enough, the function I showed you above works:

+ + + + +

It’s very fragile, though: it’ll die on the first exception.

It also doesn’t do many things which you’ll see more sophisticated REPLs provide.

clojure.main/repl

This is the REPL you get when you run clj or clojure command-line utility.

It works essentially the same, but does a little bit of extra work for you:

First and most notably, it prints a command prompt that displays the current namespace:

+ +

Which you can actually customize to your liking:

+ +

Then, it catches and prints exceptions, so your REPL doesn’t die when you make a mistake:

+ +

It also stores the last calculated values in special *1..*3 dynamic vars and the last exception in *e. These variables do not exist outside of REPL:

+ +

Another convenience that default REPL does is requiring some stuff from clojure.repl and clojure.pprint:

+ +

Were you wondering where (doc) in REPL comes from? Now you know.

Finally, it isolates vars like *ns* or *warn-on-reflection* so that when you set! them in your REPL session it doesn’t alter their root bindings.

Quite a bit of nuance, huh? With all that, clojure.main/repl is still considered very basic. There’s more stuff you can do!

clojure.core.server/repl

Basically the same as main/repl, but for the access over the network.

The only difference is output. If your entire program IS the REPL, you don’t have to do anything special with it.

But if you are connecting dynamically to a working program, things get trickier. Where should (println "Hello") print?

If it prints to stdout of the process, you won’t see it in your REPL. It’ll go to wherever the server process redirects its standard output.

So what Server REPL does is it redefines *in*/*out*/*err* to socket streams instead of process’s stdout and sends to you what you print over the network. Everybody gets their own stdout!

+ + + + +

Really tricky stuff to figure out, but essential to understand if you consider yourself an advanced Clojure REPL user.

The rest is the same. Server REPL literally calls into main/repl after rebinding *in*/*out*/*err*.

clojure.core.server/io-prepl

pREPL is Clojure team’s answer to nREPL and critique that Clojure Socket REPL is not machine-friendly. It’s basically server/repl but with EDN-formatted output:

+ +

pREPL consumes raw Clojure forms but outputs EDN-structured data.

In terms of what it does for you, it also formats exceptions (not in a Clojure-aware way, unfortunately) and synchronizes your output so that two threads can’t print simultaneously. But that’s about it.

The main problem with pREPL is that it’s based on EDN and, thus, aimed at Clojure clients first and foremost.

nREPL

nREPL is a third-party server started by Chas Emerick and lately adopted by Bozhidar Batsov. It’s a separate library that you have to add to your project and start the server yourself.

As Rich Hickey put it, ”nREPL is not a REPL, it’s remote evaluation API”. He’s not wrong, but I think that’s exactly what tooling authors need: remote eval API, not interactive console.

First, nREPL is machine-friendly both ways. It receives bencode-d data and sends bencode-d data back.

Second, it walks an extra mile for you:

Its eval optionally accepts file name and position in that file, so that stack traces would contain the correct position.
It limits the size of the output to a user-provided threshold, saving you from printing infinite sequences which are not rare in Clojure.
It provides interruption for already executing evals.
Some of its functions like lookup and load-file solve problems that their Clojure alternatives don’t.
Some are just conveniences like completions.
It’s extensible, allowing you to create your own operations.

All this stuff is very useful and doesn’t come “naturally” with naive REPL implementations.

The downside? You need to add nREPL server dependency to your app. It also has a noticeable startup cost (~500ms on my machine).

Extended nREPL

Since nREPL is extensible, one can extend it to do even more. That’s what the first version of Clojure Sublimed did and still does. Including:

Formating stack traces in a Clojure-aware way and sending them back with errors:

+ +

Parallel evaluation and execution time:

+ + + + +

It worked well for me for 1.5 years, but still, you know, nREPL dependency, startup time, NIH syndrome. I wanted to give REPL a shot on my own.

REPL, upgraded

Even the simplest REPL still has the full power of Clojure in it! We can start with something very basic, like server.repl, send our own server’s code to it first thing after connecting, and then take control over stdin/stdout and start serving our own protocol with our own execution model.

This is called “upgrading” your REPL and that’s how Christophe Grand’s Unrepl works, for example. The beauty of it is zero dependencies: you only need Clojure and nothing more. Everything you need you bring with you.

In our case, it looks like this. First, we send a lot of Clojure code (unformatted, because machine doesn’t care):

+ +

Then, we receive this:

+ +

Which basically means “Yes, I’ve heard you”.

This is all happening inside basic server/repl. It looks messy because it was designed for human consumption (eye-balling), and we don’t even try to interpret it. We just cross our fingers and hope everything we sent works.

At this point, we’re ready to “upgrade” our REPL. This is how we do it:

+ +

(repl) is a function we defined in our initial payload. {"tag" "started"} is the first message of our own protocol. I really, really, really hope here that it will not be messed up by other output (printing in Socket Server is not synchronized, and everyone who worked with Clojure REPL in the terminal knows how often it messes up your output).

After the client sees {"tag" "started"} somewhere in the socket, it considers the upgrade to be finished and now works in our own nREPL-like EDN-based protocol.

Clojure Sublimed REPL Server

Our upgraded Clojure Sublimed REPL does all the same basic stuff that nREPL does. The only practical difference for clients is batch evaluation: send multiple forms together (e.g. when evaluating the whole buffer) and get separate results for each one.

nREPL eval-buffer:

+ +

Clojure Sublimed eval-buffer:

+ +

Under the hood, though, it’s a completely new REPL. It sits on top of Socket Server, yes, but it has its own evaluation model and its own protocol. It’s clean, minimal, fast to load, and works much better with Clojure Sublimed client than nREPL.

I don’t want to release yet separately from Clojure Sublimed (yet?), but, you know, take a peek at the implementation anyway.

Your own REPL!

The original version of Clojure Sublimed (client) was organized quite poorly and adding new REPLs was problematic.

New, refactored Clojure Sublimed was designed to be easy to extend. Out of the box, we ship with these now:

JVM nREPL which installs a few extra middlewares.
(new) Raw nREPL for non-JVM environments (babashka, etc).
Shadow-CLJS nREPL which (now) adapts better to shadow-cljs quirks.
(new) JVM Socket REPL which works on top of bare-bone Clojure Socket Server.

And there could be more! If you are interested, let me know, or, better, jump in with a PR! I promise it should be much easier now. I even wrote docstrings ~~everywhere~~ at some places :)

Conclusion

So, Clojure Sublimed v3 is out there. To sum up the major differences:

REPL doesn’t depend on syntax highlighting,
new JVM Socket REPL,
easier to add new REPLs,
client-side pretty-printer,
faster indenter and formatter.

As always, you can get the new version in Package Control or on Github:

+ +

Let me know what you think! Issues are open :) And happy Clojur-ing!

You were going to ask anyway

Color scheme: Niki Berkeley.

The font on screenshots: Berkeley Mono.

+ +

tonsky.me

The Absolute Minimum Every Software Developer Must Know About Unicode in 2023 (Still No Excuses!)

What is Unicode?

How big is Unicode?

What’s Private Use?

What does U+1F4A9 mean?

What’s UTF-8 then?

How many bytes are in UTF-8?

What’s �?

Wouldn’t UTF-32 be easier for everything?

Is Unicode hard only because of emojis?

What’s "🤦🏼‍♂️".length?

How do I detect extended grapheme clusters then?

Wait, rules are changing?

Why is "Å" !== "Å" !== "Å"?

Unicode is locale-dependent

Why does String::toLowerCase() accepts Locale as an argument?

I live in the US/UK, should I even care?

What are surrogate pairs?

Is UTF-16 still alive?

Conclusion

A case for ClojureScript 2.0

Advanced makes your code faster

Advanced may break your code

You can’t auto-generate externs. Nobody can

Why is advanced mode needed?

Why not just always use :none mode?

How is JVM Clojure doing?

What if advanced mode didn’t exist?

Where are your patches?

Humble Chronicles: Managing State with Signals

Signals

Stable object references

Props drilling

Reactive vs Incremental

Effects

Disposing signals

Disposing components

Mounting components

Gotchas

Dependency too wide

Dependency at the wrong time

Recomputing too much

Ambiguity

Repeating computations

Pre-compilation

Results

It works

It feels very satisfying

Props drilling works

No VDOM needed

We need incremental algorithms

It breaks intuition

Is there a deeper reason?

Source code

Humble Chronicles: State Management

Object-oriented user interface (OOUI)

Humble UI — current state

Declarative UIs

Templates

Procedural DSLs

Value-oriented DSLs

Declarative-OO duality

Summing up

Reconciliation

Full top-down reconciliation

Optimized top-down reconciliation

Surgical point updates

State management

Internal state

External state

All state external

Single atom

State in incremental frameworks

Component DSL

Parent → child dependencies

Child → parent dependencies

Component persistence

Live reload

Conclusion

What does `U+1F4A9` mean?

Why does `String::toLowerCase()` accepts Locale as an argument?

Why not just always use `:none` mode?