I taught my speaker model the wrong voice

Speaker identification in my pipeline works by enrollment. Each known voice is a single L2-normalised vector — the mean of embeddings from a few minutes of that person's speech — stored next to a manifest of what went in. At transcript time, every diarized label is embedded from its own turns and matched against the store: nearest enrolled voice under a cosine distance of 0.45 gets the name, otherwise the label stays SPEAKER_NN.

Enrollments used to be built offline from curated audio (the diarizer's own failure mode is a separate story). The obvious next feature: when a transcript comes back with an unknown label and I can tell who it is, let me type the name and have the pipeline learn that voice on the spot, from that recording. Pool the label's turns, embed, merge into the existing vector (or create one), done. One checkbox on the review screen: also teach this voice, ticked by default.

It worked on the first try. The second day it polluted an enrollment.

What went in

The recording was a room with two people. The diarizer had split them into two labels, but not cleanly — one label held a young speaker's voice plus long stretches of the adult in the same room. I named the label after the young speaker, left the box ticked, saved. The pool was 129 seconds of the adult's speech merged into the young speaker's enrollment at the weight of a full sample.

Nothing failed loudly. The transcript note was written. The manifest recorded a new sample. And from then on, that adult's voice would match the young speaker's enrollment rather than her own — a wrong name on every future recording, and no error anywhere.

I noticed because the next day's recordings started attributing the wrong lines. The store keeps a dated backup before every write, so the restore was one directory copy. The question was how to make the button safe.

Why "just check the audio" is not enough

The first instinct is to look at what you are about to teach and reject it if it is bad. But "bad" here has three different shapes:

1. Too little. A pool of 4 seconds is not a voice, it is a moment. Enrollment maths on one short clip produces a vector that matches nothing reliably — worse than no enrollment, because it will produce confident wrong names.
2. Too mixed. The pool contains two voices. The mean lands between them. This is the case above.
3. Someone else. The pool is clean but it is a voice you already have under another name. Teaching it creates a second, competing print for the same person.

Case 1 is a length check. Cases 2 and 3 are the same check from different angles: does this pool sound like someone I already know, other than the name being taught?

The purity gate

Before teaching, the pool is run through the identify endpoint — the same window-level clustering the pipeline uses to decide whether to bother with diarization at all. It returns the clusters found in the pool and, for each, the nearest enrolled voice and distance.

The gate refuses when:

• the pool is under 5 seconds of speech; or
• the best-matching enrolled voice is not the name being taught and sits closer than 0.35.

That second rule catches both the mixed pool (part of it will match the other person) and the clean-but-already-known pool. The refusal is specific — "this sounds like X (closeness 0.28), not Y" — and it comes back to the review screen before the note is written, because enrollment can fail for reasons that need a decision, and the note is immutable once it exists.

It is a gate, not a wall. A genuinely mislabelled speaker has to be fixable from that screen, so the refusal offers an override: teach it anyway, recorded in the manifest as forced. The default is to refuse; the human can insist. What the human cannot do any more is pollute an enrollment by accident.

What the same check tells you about the store

Running identify on the pools also produced a number I had not measured before: how close the enrolled voices are to each other. The embedder is trained on adult speech. Two young speakers in the store sit roughly twice as close to each other as the closest adult pair; the two thinnest adult enrollments (26 and 9 seconds) are the next-closest pair. Every test on those pairs landed right — but the margin is slim enough that the transcript caveat now says so, per name: "probably", "one of the two", rather than a bare label.

That caveat is not decoration. The note header carries it, the summary generator sees it, and the rule downstream is that nobody gets assigned an action item from a transcript label without a human confirming the voice. A wrong name in a transcript is an annoyance; a wrong name that becomes "you agreed to do X" is a different category of error.

The button, after

The review screen still has one checkbox per unknown label, ticked by default — the friction has to stay near zero or the store never grows. Behind it: length check, identify pass, refusal with the closest match and its distance, override with a recorded reason, backup of the store before every write, and a hourly sync of the store to the second machine that also runs the pipeline.

The enrollment that got polluted was rebuilt from its original samples. The 129 seconds are in a backup directory with a timestamp in its name, which is where they belong.

Receipts

• Enrollment: one mean vector per voice, cosine match threshold 0.45; window embeddings 1.5 s / 0.75 s hop, ~0.004× real time warm.
• Incident: 129 s of the wrong voice merged into an enrollment at full sample weight; restored from the pre-write backup.
• Gate: refuse below 5 s of pooled speech, or when the closest enrolled voice other than the target is under 0.35. Override recorded as forced.
• Two young voices measure about twice as close to each other as the closest adult pair on this embedder.