Six Bits for the Layers That Never Sleep

I released a new quantisation of GLM-5.3-Flash this week: GLM-5.3-Flash-MLX-mixed-4_6bit, an MLX build for Apple Silicon that fills a gap between the two builds people usually choose from. It is ~179.7 GB, about 7% faster at decoding than the popular mixed-4/8 build on the same machine, and scored identically to it on my 64-scenario assistant suite. This post is about the reasoning behind the split, because the reasoning generalises far past this one model.

97% of the parameters barely matter for speed

GLM-5.3-Flash is a mixture-of-experts model, and like most MoE models the overwhelming majority of its weights — about 97% — live in the routed experts. Those experts are the reason the model runs on a desk at all: for any given token, only 8 experts out of hundreds actually fire. The other 96%-something sit in memory doing nothing.

That asymmetry splits the model into two populations with opposite jobs:

• The routed experts decide how big the model is. Nearly all the parameters, mostly dormant.
• Everything else — attention projections, shared experts, embeddings, the language-model head — runs on every single token. These layers decide how many bytes each decode step has to read from memory, and on Apple Silicon memory bandwidth is the binding constraint on speed.

So the precision of the experts sets the download size. The precision of the always-on layers sets how fast the model answers. Any mixed-precision scheme is really just choosing where on that trade-off to sit.

The gap between the two existing builds

The two builds people usually pick between, on this model:

build	routed experts	always-on layers	size
uniform 4-bit	4-bit	4-bit	~178 GB
mixed-4/8	4-bit	8-bit	~182 GB

Notice the strangeness: the 8-bit upgrade costs almost nothing in size (4 GB) because experts stay at 4-bit. And the uniform 6-bit build, at ~256 GB, spends its extra bits mostly on layers that are idle most of the time — which is the worst possible place to spend memory if speed is your goal.

PipeNetwork's published measurements on this model showed uniform 6-bit is statistically indistinguishable from 8-bit quality (ΔNLL +0.0011, with a confidence interval that includes zero). That makes 6-bit the natural floor for the always-on layers. My build takes that conclusion to its endpoint: experts at 4-bit, everything else at 6-bit — hence the name, mixed-4_6bit, after PipeNetwork's mixed-4_8bit convention where the first number is the experts.

Compared to mixed-4/8 it reads roughly 15% fewer bytes per token at essentially the same total size. The 8 bits the 4/8 build spent on always-on layers buy quality those layers don't need, and cost bandwidth every single step.

What I actually verified

Quantising a 314-billion-parameter model by trusting a converter is how you ship a model that is quietly worse and can't tell anyone why. So every quantised tensor was dequantised and compared against an independent dequantisation of the FP8 source: the e4m3 format decoded by hand in NumPy from the raw bit layout, block scales read straight from the safetensors bytes. The reference shares no code with MLX's converter — agreement is not the converter checking itself.

The numbers behaved exactly as two extra bits predict. Relative L2 error of the dequantised weights:

tensor group	this build	error	mixed-4/8	error
routed experts	4-bit	9.28%	4-bit	9.28%
shared experts	6-bit	2.28%	8-bit	0.73%
attention projections	6-bit	2.38%	8-bit	0.74%

The 6-bit layers carry roughly a quarter of 4-bit's error. FP8-sourced and bf16-sourced tensors landed at the same error at the same bit width, which tells me the FP8 dequantisation adds no error of its own.

A few things were deliberately not quantised: the MoE router and correction bias, the multi-head-curvature arrays and decay parameters (float32 — router errors misroute tokens, and block-selection errors in the lightning indexer compound, so those projections got 8-bit). The multi-token-prediction layer was dropped entirely; I don't use speculative decoding on this box.

The measurements

Mac Studio M3 Ultra, 512 GB, both builds served through mlx_vlm.server with a wired memory limit, temperature 0. Decode speed across five prompt types, three runs each, cold prompt cache:

prompt	mixed-4/8	this build
prose	28.2 tok/s	30.2 tok/s
code	28.0	30.1
scheduling	28.3	30.1
creative	28.5	30.3

About 7% faster decode — close to the ~9% the byte-count reduction predicts once fixed per-token overhead is accounted for. Time to first token unchanged at 0.23 s.

Task accuracy on a private 64-scenario assistant suite (calendar, tasks, messages, diarized voice notes; English, Lebanese Arabic, and Latin-script Arabizi), two trials each, graded on end state: 76.6% for both builds, identical. The only difference was one Arabic scenario where the phrasing carried a diacritic my grader didn't accept — a grading artifact, not a model failure.

The honest caveats: outputs are not token-identical to mixed-4/8 even at temperature 0 — different weights resolve near-ties differently — though the differences read as rewording, not different content. And I have not measured perplexity for this build yet; the weight-level and task-level evidence both point to quality matching mixed-4/8, but no NLL number exists.

The lesson that generalises

The mistake most people make with quantised models is treating precision as one number: "should I run the 4-bit or the 6-bit?" But in a mixture-of-experts model, precision is two decisions, and they answer different questions:

• How big is the download, and does it fit in memory? → the experts.
• How fast does it answer once loaded? → the always-on layers.

Spending uniform precision on both means overpaying one of the two bills. The 4_6 split is simply the cheapest point on the curve that doesn't give up measurable quality anywhere I can detect — and the method for finding such points is available for any MoE model: figure out which layers touch every token, protect those, and squeeze the ones that mostly sleep.

If you're on Apple Silicon with the memory to run it, the build is up on Hugging Face, MIT-licensed, with the full recipe in the model card. Base model © Z.ai; the conversion recipe and runtime classes follow PipeNetwork's GLM-5.3-Flash MLX work, and their published quality table is what motivated the split in the first place.