Local first, big machine as fallback

Two Macs run everything my assistant does. A Mac Studio M3 Ultra hosts the main model — the one my coding sessions and the assistant's own thinking go through — plus the embedding model, a hybrid speech-to-text endpoint and the store of enrolled voices. A Mac Studio M2 Ultra next to it runs the assistant's services: the review app, mail and message mirrors, the phone-call agent, and since a month ago the cloned-voice TTS.

Then I added a workload that never sleeps: continuous audio from a dozen cameras around the house, speech only, diarized, translated, summarised, searchable. Each conversation clip is a whole-file Whisper pass, a pyannote pass, and a local-model summary. On a busy day that is hours of audio, and it was all going to the M3 Ultra — the machine whose GPU I do not want anything else touching while the main model is serving.

The decision

Move the pipeline to the M2 Ultra, entirely: a local copy of the diarizer service (Whisper large-v3 on MLX plus pyannote 3.1), a local 26B model behind a local LiteLLM, and only the embeddings still from the big machine, because the vector index has to be one model. The M3 Ultra stays as a fallback for both the diarizer and the model, switchable per component from a settings page, and a checkbox to turn the fallback off entirely for the days the big machine is busy with something that must not be disturbed.

Two constraints came with the move. The M2 Ultra has 128 GB and already hosts services; the pipeline had to be bounded. And the enrolled voices had to stay a single store — enrol on the big machine only, sync to the small one hourly — because two stores drift.

What "bounded" meant in practice

• A 10-minute cap per clip. A conversation that runs longer is cut into clips; a single 40-minute Whisper job on a shared GPU is a way to starve everything else.
• A footprint guard. MLX's buffer cache grows without limit as vocoder buffers interleave with autoregressive buffers; a process that was 3 GB at start was 42 GB after one long utterance on this machine, and its decode latency had gone from 0.3 to 2.6 seconds. The diarizer service now restarts itself when idle above 12 GB; the TTS server caps its MLX cache at 8 GB. Measured on the TTS: RTF 2.4 → 1.07 from the cap alone.
• A Metal watermark in the environment so the GPU allocator gives memory back sooner.
• One job at a time per service. The diarizer's model lives on one thread; jobs queue. No batching, no concurrency, no surprises.

try_each

The fallback is one function: try the primary, on failure try the secondary, record which one answered. Every clip carries a served record — which machine diarized it, which model summarised it, whether that was the fallback — so a bad week is diagnosable from the data, not from memory.

I tested five scenarios before calling it done, each with clips of 20 seconds or less, because the big machine was serving and I was not going to load it with real audio to prove a point:

1. Both up, primary local → served locally, fallback flag false.
2. Local diarizer stopped → served by the big machine, diarize_fallback: true, clip processed.
3. Local model stopped → local diarizer, remote summary.
4. Fallback disabled, local diarizer stopped → clip marked failed with the reason, retried later, nothing sent to the big machine.
5. Primary switched to the big machine in the settings → served remotely, local left idle.

The self-check that runs every few minutes now has a key for the local diarizer (health on loopback, footprint under 20 GB) and knows to stay quiet when the settings say the local stack is not in use.

The bill, three days later

On 2026-08-30 the TTS on this machine measured RTF 0.99 with the cloned voice — no margin, but real time. Three weeks later, with the diarizer and the 26B model resident on the same GPU, the same measurement was 1.10 to 1.24, and 27% of spoken replies were stalling. Nothing in the TTS had changed. Its neighbours had.

That is the honest cost of local-first on a shared GPU: every service is a tenant, and a tenant that used to have the floor to itself does not. The fix on the TTS side was a compiled inner loop and more pre-roll (the details), and it holds — with a margin I can name and do not love. The structural answer, which I have not built, is scheduling: a whole-file pass on a long clip should yield to a spoken reply, and today nothing tells the GPU that.

Would I do it again

Yes, and earlier. The big machine's GPU is the one resource I cannot buy more of on a whim, and the pipeline that moved was the one workload that never stops. The rules — cap the job, guard the footprint, record who served, test the fallbacks small — are the same ones I would give a client with two servers and one that matters more. The part I would add on day one is the contention accounting: know which tenants share a GPU before one of them starts stuttering.

Receipts

• Two machines: M3 Ultra (main model, embeddings, STT, voice store), M2 Ultra 128 GB (services, TTS, and now the camera pipeline: local Whisper large-v3 + pyannote 3.1, local 26B via LiteLLM).
• Bounds: 10-min clip cap; diarizer restart when idle > 12 GB; TTS MLX cache cap 8 GB (RTF 2.4 → 1.07).
• Five fallback scenarios, ≤ 20 s clips each; every clip records served.
• Contention: TTS RTF 0.99 (2026-08-30, alone) → 1.10–1.24 (2026-09-22, shared), 27% of replies stalling until the loop and pre-roll were fixed.