Building a live meeting mode that runs entirely on the CPU

On-device transcription and on-device meeting notes are both solved problems in 2026. On-device diarization is not — and live/streaming diarization with an unknown speaker count is still a research frontier no open-source tool ships cleanly. The winning move is therefore not to chase real-time diarization but to reuse the offline diarizer you already have, and run it as a post-pass.

Why not the alternatives

The three things to accept

• Speakers are told apart by neural embeddings, not pitch. Pitch/F0 is neither stable within a speaker nor separable between speakers; the field switched to x-vectors / ECAPA-TDNN years ago. (§A)
• Live diarization is a downgrade, not an upgrade. Committing a speaker label in real time is irrevocable while the model keeps re-learning, so any live overlay shows churn. Offline global clustering avoids the entire problem. (§B, §F)
• Unknown speaker count is doable but imperfect. Clustering auto-picks k (eigengap / threshold), but count estimation is wrong in a meaningful minority of sessions and tends to under-count. Let users correct it. (§C)

Neural speaker embeddings plus clustering — not "voice frequency" or pitch.

Why pitch / frequency alone fails

A single acoustic scalar like fundamental frequency (F0) cannot separate speakers, for two independent reasons that are well established in forensic phonetics:

• Large intra-speaker variation. The same person's mean F0 swings widely with emotion and stress — a documented forensic case measured mean F0 of 228 Hz during an incident versus 140 Hz under normal conditions. F0 is "not so effective in the field." (F0 in real-life speaker ID)
• Large inter-speaker overlap. Different speakers occupy overlapping F0 ranges; when target and masker pitch coincide, F0 gives "almost no benefit." (F0 & intelligibility in noise, PMC)

A learned DNN embedding is trained end-to-end to be speaker-discriminative, capturing voice quality, spectral envelope and articulation jointly — placing each speaker at a point on an identity manifold where cosine distance separates them robustly across content and channel. This is the i-vector → x-vector generational shift. The canonical models: the x-vector (Snyder 2018), ECAPA-TDNN (Desplanques 2020, 0.87% EER on VoxCeleb1), and the GE2E d-vector (Wan 2018) behind Resemblyzer.

On-device embedding models — and what sherpa-onnx already ships

The pragmatic choice is a small Apache-2.0 model that sherpa-onnx already pre-converts to ONNX, so there is no custom export and no gated download. 3D-Speaker CAM++ (7.2M params, 0.65% EER) and ERes2Net-base (6.6M) are the best size/accuracy trade-off; sherpa-onnx also ships WeSpeaker and NeMo TitaNet variants.

Takeaway. Your codebase already notes ECAPA is unreliable on sub-second windows. CAM++/ERes2Net are the right on-device choice: Apache-2.0, tiny, high accuracy, already ONNX-converted by sherpa-onnx, no HuggingFace token. Reuse the one you have for the file path.

Offline clustering is accurate and finds the speaker count globally, but produces no live labels. Online systems produce live labels but are either count-capped, GPU-bound, or gated.

Offline (the accurate baseline)

Segment → extract embeddings → cluster the whole recording, so the speaker count is discovered globally. The open standard is pyannote.audio 3.1 (powerset local segmentation + AHC over ECAPA embeddings); VBx (BUT) is the classic VB-HMM baseline. Your sherpa-onnx path is exactly this, offline-only, and it is the accuracy ceiling for a CPU app.

Online / streaming — the landscape

Two families discover new speakers on the fly. Clustering-based online methods (diart, UIS-RNN, online GMM/i-vector) spawn a new cluster when nothing matches → truly unbounded count. End-to-end streaming methods (Sortformer, FS-EEND) are faster and overlap-aware but capped at the training speaker count (≤4) — the open research gap.

The practical hybrid (recommended)

Stream transcription live; run the heavy diarization as a post-pass at meeting end (or in rolling chunks). The OpenWhispr production pattern makes this concrete: live path emits a provisional CAM++ label every ~1 s once ≥1.6 s of speech has accumulated; the end-of-meeting pass runs full offline diarization on the complete WAV for clean global labels, then reconciles. Their churn-control rules are worth copying verbatim:

• User-set names are locked and never overwritten by the batch pass — "the worst UX is silently overwriting a user's correction."
• Running-mean embedding averaging handles voice drift rather than centroid overwrite.
• Retroactive relabel of past unnamed speakers when a name is first assigned; locked mappings untouched.

Memory/compute for 1–2 h is a non-issue: 512-d embeddings ≈ 2 KB/speaker (~20 KB for a 2-h, 10-speaker meeting); the whole ONNX stack is ~45 MB and CPU-only. The trade-off is simply that final labels arrive at stop, not live — acceptable for a notes tool.

Clustering picks k automatically — but expect it to be wrong in a meaningful minority of meetings, and to lean toward under-counting.

How k is chosen

• Maximum eigengap on the spectral affinity matrix — the workhorse since Wang 2018 (LSTM d-vectors). NME-SC (Park 2019) jointly auto-tunes k and the affinity binarization threshold with no dev-set tuning (17% rel. speaker-error improvement on CALLHOME); now in NeMo.
• Cluster-distance threshold — what sherpa-onnx exposes: set num_speakers = -1 and a cluster_threshold and its FastClustering picks the count. (API example)
• HDBSCAN — density-based, no k needed, robust to outliers. (overview) Silhouette/BIC are the classic criteria, largely displaced by eigengap for embeddings.

Real-world accuracy

• Speaker-counting accuracy was 72% with ResNet101+VBx vs only 56.4% with x-vector+AHC — even a strong 2021 system miscounts ~1 session in 4, "mostly due to underestimation." (ECAPA embeddings for diarization)
• Count error scales with difficulty: ~2.43 avg speaker-count error on DIHARD-III "webvideo", ~1.00 on "meeting." (robustness of spectral clustering)
• Oracle (known) count consistently beats estimated count on DER; the estimate degrades all systems. (ibid.)

Design implication. Auto-estimate the count, but treat it as a first guess: surface it, let the user merge/split speakers, and lock corrections. Do not present the machine count as ground truth.

Clustering structurally cannot represent two people talking at once — and real meetings overlap 12–20% of the time.

• Why it breaks clustering. One embedding per segment → one speaker; overlap frames get mis-assigned to a single cluster. Even perfect VAD + perfect clustering still misses the second speaker in overlap regions, a hard DER floor equal to overlap duration. (DIHARD-II is still hard)
• Overlap is substantial in meetings: AMI ~15.9%, AliMeeting ~19%, CALLHOME ~12.6%, vs VoxConverse ~3.5%. Missed speech is the dominant DER component in meetings, driven by unhandled overlap. (Benchmarking Diarization Models 2025)
• EEND / powerset natively handle it. pyannote's powerset encoding (Interspeech 2023) cut overlap misses 13.1% → 9.9% and improved DER ~8–11% relative. The pyannote segmentation model sherpa-onnx re-exports is this powerset model, so you already benefit locally.
• But overlap handling is not a silver bullet: even oracle overlap detection moved DER only 23.8% → 22.2% in one study. (ibid.)

Pragmatic stance for notes: perfect overlap attribution is not required for readable minutes. The powerset segmentation you already ship is enough; accept a small missed-speech floor rather than chasing overlap-aware resegmentation.

Only three options are clean to ship: sherpa-onnx, 3D-Speaker / WeSpeaker, and SpeechBrain — all Apache-2.0, ungated, self-contained, CPU-capable. Everything with a streaming or SoTA-accuracy story drags in a gated or non-commercial or GPU-only constraint.

Verdict. Stay on sherpa-onnx — you already use it, it's the most turnkey Apache-2.0 CPU diarizer, ungated, and handles unknown count. If you ever truly need live diarization, diart is the only mature open streamer, but budget for the gated-model bundling problem.

Word timestamps → speaker turns

• Don't trust faster-whisper's native word timestamps for tight sync. Whisper was never trained to emit them; they're recovered post-hoc via DTW on cross-attention and are documented as inaccurate (word start sometimes later than its end). The field-standard fix is WhisperX's external wav2vec2 forced alignment; CrisperWhisper / whisper-timestamped are alternatives.
• Assignment = max temporal overlap. Each word (with accurate timing) is assigned to the diarization segment it overlaps most — WhisperX's assign_word_speakers() is the reference. Words straddling a boundary go to the larger-overlap speaker. (pyannote+Whisper)
• Real-time alignment is unsolved off-the-shelf — provisional streaming diarization re-clusters, so speaker indices swap between chunks ("A"→"B"), causing label churn. Batch/offline overlay avoids it entirely. This is another argument for the post-pass. (streaming permutation problem)

Endpointing for continuous 1–2 h audio

• Silero VAD — MIT, ONNX, CPU, a 30 ms chunk in <1 ms/thread; it's already the engine inside faster-whisper's vad_filter, which merges speech into ≤30 s Whisper windows (bounding memory) and reduces hallucination/WER on long form. (Silero, faster-whisper)
• Semantic endpointing (only if you ever add conversational turn-taking, not needed for a notes tool): open options are Pipecat smart-turn v3 (open weights, 8 MB, 12 ms CPU, 23 langs) and LiveKit Turn Detector — both layer on top of Silero VAD, not instead of it.

For meeting mode: Silero VAD to chunk the stream + faster-whisper with word_timestamps=True for provisional live text, then re-align words to the offline diarization by max-overlap in the post-pass. If you need karaoke-tight timing in the final export, add wav2vec2 forced alignment (per-language model) — otherwise skip it.

Fully offline minutes are realistic on CPU — provided you chunk by speaker turn and map-reduce, use separate schema-constrained passes, and budget minutes, not seconds.

The pipeline that works

• Chunk on speaker-turn boundaries (never mid-utterance) with 10–20% overlap; map-reduce is mandatory because a 1–2 h meeting is tens of thousands of tokens — MeetingBank council meetings average ~28k tokens. The advertised 128K context of small models is nominal; effective context is far lower.
• Separate passes for summary / decisions / action-items, each emitting JSON keyed to diarized speaker IDs (owner_speaker_id). Force valid JSON with llama.cpp's GBNF grammar-constrained decoding rather than hoping. (pipeline)
• Guard the two known failure modes: high factual-error rate (QMSum: 74% of strong-model summaries had factual errors) and hallucinated attribution. Use grounding constraints ("do not infer"), [unclear] marking, a strict decision definition (chosen + agreed), and speaker-ID consistency checks.

What CPU can realistically run

From the one rigorous CPU-only benchmark (CEUR-WS 2025, llama.cpp Q4_K), generation throughput:

The realistic band is Phi-4-mini, Qwen2.5-3B/7B, Llama-3.2-3B, Gemma-2-2B at Q4_K_M (<1% quality loss, ~75% smaller). CPU prefill of tens of thousands of tokens — not just generation — is the real cost, so end-to-end minutes for a 2 h meeting take minutes, not seconds. Grounding: AMI/ICSI/QMSum/MeetingBank are the corpora; AutoMin/ELITR is the minuting task; ROUGE is standard but a weak proxy, so evaluate qualitatively.

Prior OSS to learn from: ownscribe is closest to your design — local-first CLI, bundled Phi-4-mini GGUF, Markdown + JSON minutes (Summary / Key Points / Action Items / Decisions). Meetily uses Ollama. None ship GBNF-grammar JSON — a differentiator you can claim.

A fully-offline tool can already match cloud on transcription and on local-LLM notes. Cloud still leads on exactly two things — high-quality live diarization, and persistent voiceprint naming — and the second is an unoccupied on-device niche.

• Otter — cloud; its distinctive feature is learn-a-voice-once, auto-name-forever, but that's a stored biometric in the cloud. (docs)
• Granola — bot-free capture but cloud transcription/LLM; on desktop it only splits "Me" vs "Them", no real diarization. (docs)
• MacWhisper — on-device, but diarization is beta (pyannote v4), over-splits, post-hoc only. (review)
• OSS (Meetily, Hyprnote, Vibe) — transcription + local-LLM notes are solid, but live diarization is POC/beta/planned across the board. (Meetily, Hyprnote)
• Apple — best-in-class on-device transcription (WWDC25 SpeechAnalyzer), but ships no speaker labeling at all. (WWDC25 · 277)

Positioning. Your encrypted-corpus voiceprint (ADR-012) plus an offline diarization post-pass hits the two gaps at once: on-device diarization and persistent, private, cross-meeting speaker naming — which no on-device competitor offers.

Capture → stream STT → live provisional labels → stop → offline diarization post-pass → reconcile → notes. Every stage CPU-only and offline; no gated model.

New dependencies: essentially none heavy — you already have sherpa-onnx (diarization), faster-whisper (STT + Silero VAD), and an optional llama.cpp for notes. The work is orchestration and the reconcile/naming UX, not new ML infrastructure.