Quantizing to 4-bit made my TTS 0% faster

My assistant speaks with Fish S2 Pro on MLX: a 4-billion-parameter dual autoregressive transformer that emits 10 codebook tokens per audio frame at 21.5 frames per second, and a neural codec that turns those tokens into 44.1 kHz audio. It runs on a Mac Studio M2 Ultra, streaming, with a cloned voice. Since it moved to that machine (why it moved), replies had started cutting out mid-sentence — a word, a gap, a word.

The log keeps a line per utterance with how many times the output clock starved. Over the last 200 utterances: 55 stalled, 27%. A 21-second reply had eight stalls and 3.5 seconds of silence. Generation was running at real-time factor 1.02–1.14 on a 20-second reply — slower than real time, delivered in ~2-second bursts, with 1.2 seconds of pre-roll to absorb them.

The question I was asked was whether a batch size would help. There is no batch: one stream, one job at a time, the autoregressive loop is 91% of the work. The next obvious lever was quantization.

The 4-bit experiment

The live model is 8-bit. A 4-bit MLX conversion exists on Hugging Face. Autoregressive decode at batch 1 is usually memory-bandwidth-bound — halve the bytes per token, get most of a 2× speedup — so this was the bet.

Two wrinkles first. The 4-bit repo ships the codec as a 1.9 GB pickle from an unknown uploader; the 8-bit repo I already had ships an MLX codec, and the loader takes a separate codec path, so I fetched only the safetensors weights and reused the codec I trusted. And the 4-bit checkpoint was packaged for a different library: the same 766 tensors, every key prefixed model.. A re-key pass with mx.load/mx.save_safetensors (numpy's safetensors path refused the bfloat16 tensors) and it loaded.

Same 20-second text, same cloned voice, two runs each:

No speedup. The loop is not bandwidth-bound on this machine. Whatever it is bound by, halving the weight bytes does not touch it. I kept the re-keyed model on disk and did not switch.

Profiling the frame

With the bandwidth theory dead I measured one frame, phase by phase, with an mx.eval after each. Against a real-time budget of 46.4 ms per frame:

So the loop alone is under real time, if only just. Two things stood out in it.

The semantic sampler warps and sorts the model's full vocabulary — 155,776 tokens — twice per frame, when the only tokens that can be chosen are the 4,096 semantic ids plus two stop tokens (everything else already has a −∞ bias). And the depformer's nine steps each launch their own handful of kernels for a few small matmuls: 2.4 ms per step is not compute, it is launch latency.

I compiled both into single graphs: the sampler works on a 4,098-entry slice and produces both draws in one evaluation, one sync; the whole nine-step depformer loop, sampling included, is one mx.compiled function with the random keys passed in as inputs so the graph stays pure. Result: 39.0 → 36.6 ms per frame (sampling 2.5 → 1.5, depformer 22.2 → 19.6). Six percent. Real, but not the story.

The elephant was in the docstring

The streaming layer works because the codec decoder is prefix-stable: decoding codes[:E] reproduces the prefix of the full decode to rms 1e-7, so the streamer decodes a growing window at each flush and emits only the new samples. The docstring — mine, from when this ran on an M3 Ultra — says the codec is 9% of the runtime, so re-decoding is cheap.

On the M2 Ultra, steady state, a 256-frame window decode costs 550–710 ms. The window is fixed (a bounded left context of 256 frames is needed for the decode to be inaudibly close to exact; 128 frames measures at 23.8 dB SNR, audible), so the cost per flush is fixed regardless of how many frames are new. At the current flush cadence — every 48 frames, 2.2 seconds of audio — that is +0.27 RTF on top of the loop's 0.79. Total ≈ 1.06. There is the stall.

Nine percent was true on the machine where it was measured. The number travelled with the code to a machine where it was thirty.

What actually fixed it, and what it cost

Fewer flushes cut the decode cost proportionally, but bigger bursts need more pre-roll to absorb. I simulated a real-time player over the actual chunk arrival times of the 20-second reply, for each cadence and pre-roll:

Counter-intuitive but consistent: the lower-RTF cadences starve more, because their bursts are bigger than the pre-roll early in the utterance, before generation has built up a lead. Keep the cadence, raise the pre-roll to 2.0 seconds, and with the compiled loop the starvation is zero.

Two more things had to be true. The compiled graphs are traced on first use, so the first utterance after a server restart paid the compile as four stalls and two seconds of silence; the server now runs a warm-up utterance at start. And launchctl kickstart restarts a launchd job with the old arguments — a changed pre-roll in the plist needs bootout + bootstrap.

Live, on the wire, three runs of the 20-second reply: zero stalls, zero gaps over 250 ms, generation finishing 2.8–3.1 seconds ahead of playback. The price is that the first audio now arrives about 0.8 seconds later than before.

The remaining margin is thin, and I know why

This machine also hosts the diarizer and a 26-billion-parameter local model, on the same GPU. The 0.99 RTF I measured on 2026-08-30 became 1.1–1.24 once they moved in. A fix that gets the loop to 0.79 and leaves the decode at +0.27 is one busy neighbour away from stalling again.

The real fix is a streaming codec decoder — a KV cache for the codec's window-limited causal transformer and state for its causal convolutions, so each flush decodes only the new frames. That is a build, not a knob, and it is on the list. When it lands, the pre-roll comes back down.

Receipts

• Log: 200 utterances, 55 with stalls; worst 8 stalls / 3.48 s starved on 21.4 s of audio.
• 4-bit vs 8-bit on the same text and voice: RTF 1.23/1.24 vs 1.36/1.10. Not switched.
• Per-frame: 14.3 + 2.5 + 22.2 = 39.0 ms → 36.6 ms after compiling sampler and depformer; budget 46.4 ms (21.53 frames/s).
• Codec decode, 256-frame window, steady state: 552–708 ms per flush.
• After: pre-roll 2.0 s, warm-up at start, 3 × 0 stalls on the wire, gen-lead 2.8–3.1 s.