The diarizer wasn't wrong. The clustering was.

My meeting recorder is a phone on the table. It records, ships 30-second chunks to a Mac at home, and when I stop, the whole file goes through a two-step pipeline: pyannote segments the audio into anonymous speakers, then each speaker gets one embedding and is matched against the voices I have enrolled. Whisper large-v3 supplies the words. The note lands in my vault with [Name] on every line. (Teaching it new voices from real recordings, and how that went wrong, is its own post.)

On 2026-09-21 it came back with an 86-minute conversation between two people, and one of them had said everything. Seventy-three minutes under my name, out of seventy-five minutes of speech. The other person, who did most of the talking, had zero lines. A third label, 16 seconds long, had been given to a visitor who was there for half a minute.

The same evening, same enrollments, same machine, a 40-minute recording of the same two people plus two others came back correctly attributed. So something about this file.

The obvious diagnosis, and why I didn't trust it

The obvious story is "the voice model can't tell these two apart — enrol more samples". Both voices are adult male, same accent, same language. The enrollments were also lopsided: one built from 12,434 seconds of old voice notes, the other from 1,133 seconds. Plausible.

But the pipeline had left a receipt. Before running pyannote, it does a cheap "how many voices, and whose" pass: sliding 1.5-second windows, embedded and clustered, each cluster matched against the enrolled voices. That check is only used to decide whether to run the expensive pipeline. On this file it had seen both people clearly: five clusters matching speaker A at cosine distances 0.17–0.27 (about 1,490 windows), one big cluster matching speaker B at 0.16 (1,251 windows). The embedder knew there were two voices. It just had no say in the transcript.

To be sure, I cut ten 20-second slices out of turns labelled B and ran them through the identify endpoint alone. Four came back as A at 0.16–0.32. Three as B at 0.20–0.25. Three were mixed — both voices inside the same 20 seconds, fast back-and-forth. One of the A slices started with the other person's name being addressed. The voice prints were fine.

What actually happened

pyannote's diarization is its own clustering, with its own threshold, run before any enrolled voice is consulted. On this recording it produced three labels: one with 73 minutes of speech, and two tiny ones. Both people were inside the big label. Then the naming step did exactly what it was written to do: embed the whole label, take the mean, match it. A mean over two voices leans toward one of them. The label became B.

Why this file and not the other one the same evening? The differences: 86 minutes instead of 40, a phone in a reverberant hotel lobby instead of a watch on a conference table, and two similar voices interrupting each other every few seconds. pyannote's fixed clustering threshold is calibrated for cleaner conditions; in a reverberant room, two similar voices with short turns look like one to it. The 40-minute meeting had two more speakers with very different voices, which gives the clustering more structure to hold on to.

The small labels were not people either. One held 16 seconds spread across 77 fragments — single words, some Whisper garbage — with the visitor's real half-minute mixed in. Naming that label after the visitor had put 77 scraps of the other two people's speech under her name.

The fix that isn't retraining

If the embedder can separate the two voices at window level, the fix belongs after pyannote, not in the enrollments: look inside each label, turn by turn.

The rule I landed on, in two parts:

Deciding to split needs confidence. Every turn of a label gets its own mean embedding (turns shorter than a 1.5-second window get none). Count, per enrolled voice, the seconds of turns that match it confidently (distance < 0.32). A label is split only if at least two voices each own ≥10 seconds and ≥15% of the label's confidently matched speech. Below that, the label is left exactly as pyannote produced it — a single confident match to a third voice on a 3-second turn is noise, not a fourth speaker.

Once split, no threshold. Every turn of the label goes to the nearer of the two voices, however far both are. They are both known to be in there; the question is only which. Turns within 3 seconds of a turn vote on it at reduced weight, so a one-second fragment follows its neighbours instead of flipping a coin. A dead tie goes to the label's majority voice.

The rest of the pipeline is untouched: labels that were not split are still named from all of their speech, and two labels are never mapped onto one person.

The first version was wrong too

My first implementation split the label but kept a threshold: turns matching a voice below 0.32 were assigned, everything else went to the majority. On the re-run, the label split — and 74% of it still went to B. The reason is in the numbers above: at turn level, in that room, most of A's turns sat at 0.30–0.40 from his own enrollment. Confident enough to prove he was present, not confident enough to claim each turn. With a threshold, "not sure" collapses to "the other one", which is exactly the original bug wearing a new hat.

Removing the threshold after the split decision, and adding the neighbourhood vote for the short turns, gave: A 31 minutes, B 44 minutes. Not the 60/40 I would have guessed from memory, but the transcript reads right. I checked twelve random 15–20 second slices of the corrected labels against independent identification: 12 of 12 agree (A at 0.19–0.39, B at 0.21–0.25).

The visitor's fragment label stayed unnamed. It is 21 seconds of scraps with a real exchange inside; a label is not a person just because a person is in it.

Cost, and what it can't do

Embedding runs at roughly 0.004× real time warm, so the per-turn pass adds nothing you can notice to a job that is already a whole-file Whisper pass plus pyannote. Seven unit tests pin the decision logic on synthetic vectors; the model's actual separating power is a calibration question the tests cannot answer.

It cannot help when two un-enrolled voices are merged — there is nothing to match against. It cannot fix Whisper's word timings. And it does not change pyannote's clustering itself, which is still where the damage is done; it only recovers from it when the enrollments make recovery possible.

The general lesson is older than this bug. When a pipeline has a cheap early signal and an expensive later decision, and they disagree, the disagreement is the diagnosis. The check knew. It just wasn't asked.

Receipts

• Recording: 86 min, 265 word-grouped segments, language en, whole-file pass on a Mac Studio (pyannote 3.1 + Whisper large-v3 on MLX).
• Before: label B 4,377 s, two fragment labels 16 s and 11 s. Speaker check: A ≈1,490 windows (0.17–0.27), B 1,251 windows (0.16).
• Probes: 10 × 20 s slices of "B" → 4 A, 3 B, 3 mixed.
• Refinement v1 (threshold + majority): A 476 s, B 4,105 s. v2 (nearest voice + neighbour vote): A 1,843 s, B 2,662 s. Independent check 12/12.
• The corrected transcript went in as a new dated entry next to the original; the original stays. Notes are immutable once written, and a correction that overwrites its own evidence is not a correction.