A syllable is a movement, and a codec frame is a moment. Playing single frames in a row never sounds like speech however carefully you pick them, because /ba/ is not a b followed by an a — it is the second formant sweeping up out of a closure, and the sweep is the information. So these are whole syllables, synthesised with moving formants, encoded, and kept together as clusters of two or three frames. Hovering a cluster decodes all of it in one call, so the codec carries the movement.
Each cell plays its phone and shows the Mimi semantic token it encodes to.
The other page gives every symbol its own 80 ms frame, and a row of them does not sound like speaking. That is not a shortcoming of the codec — it is what speech is. The identity of a consonant lives in the direction its formants travel as the mouth opens, which is why /b/, /d/ and /g/ are distinguishable at all when their bursts are nearly identical noise. Sampled into independent tiles and reassembled, the trajectory is exactly what is lost.
So each cluster is a whole syllable. A source-filter synthesiser — a buzz at the larynx frequency shaped by three moving resonances — produces 160 to 480 ms of continuous audio, which is encoded and kept together as two, three or four frames. Hover and all of its frames are decoded in a single call, so the codec's own temporal model carries the movement rather than being asked to stitch fragments.
The emoji spell the syllable rather than being assigned by measured category. A consonant always gets the same emoji and a vowel always gets the same emoji, so 👄😮 is "bah" every time and 🐍😁 is "see". That makes a cluster readable, typeable and copy-pasteable, and because the tokeniser matches longest-first, a two- or three-emoji cluster is a single token.
Getting the synthesiser to make actual vowels took three attempts, and the first two failed the same way: the first formant swamped everything, so an /i/ had 85% of its energy below 270 Hz and its 2250 Hz second formant — the thing that makes it an /i/ — was inaudible, while /ba/ and /da/ differed by 18 Hz of spectral centroid. The cause was not the source but the resonator: peak gain in that filter goes as 1/((1−r²)·sinθ), so a 280 Hz resonance had about seven times the gain of a 2250 Hz one. Cancelling both terms put the formant amplitudes back under the synthesiser's control. The phoneme classifier — which knows nothing about how these were made — now agrees that all nine vowels are vowels, and hears three distinct ones among them.
The consonants were wrong, and only one half of it was fixable. Their burst spectrum was hardcoded at 2600 Hz for every stop, so /b/, /d/ and /ɡ/ released identically — onset-frame centroids of 2526, 2542 and 2540, a spread of 16 Hz — and two of them collided on the same semantic token. Only the transition after the burst was place-specific, and the burst is the loudest part. Giving each its own place — bilabial diffuse and low, alveolar sharply high, velar a compact mid peak — takes the spread to 3374 Hz, and the codec keeps all of it.
The other half is the codec's and cannot be synthesised around. A stop's release is a step, and measured before and after a round trip, the energy in the first 10 ms relative to the next 10 goes from 99 to about 0.4 — the step becomes a ramp. At 12.5 Hz a frame is 80 ms, and a burst of 12 ms with a 55 ms transition after it is entirely inside one token. Everything that distinguishes one stop from another is sub-frame, so they arrive softened however sharply they are synthesised.
Singing is length plus vibrato plus a pitch glide, and nothing else. The sung clusters are marked in violet and take a 🎵 prefix, which also keeps a sung /a/ from spelling the same as a spoken one. A rising note takes ↗️ and a falling one ↘️ — without that, a note that glides spelled identically to a flat one and could not be typed at all.