D27. Word times by forced alignment: wav2vec2 character models and a Viterbi search in TypeScript (12 Oct 2026)

The fine timeline (D28) anchors blocks and cues to words of the narration, and neither Pocket TTS nor a recording says when each word is said. Evaluated on 12 Oct 2026 on 80 lines said by the ten product voices (40 French, 40 English, 415 s), against 448 word boundaries taken from the pauses in the audio’s energy (−45 dB, pauses of 80 ms or more; −40 and −50 dB move the medians by a few milliseconds). The constraints: permissive licences for code and weights (no NC, GPL or AGPL), Node and ideally the product’s onnxruntime-node 1.23.2, French and English then other languages, a reasonable download on first use, about 50 ms on word boundaries.

• The aligner. A wav2vec2 CTC character model per language gives, every 20 ms, the probability of each character; a Viterbi search in TypeScript (@elabora/node align/ctc.ts, torchaudio’s forced_align semantics, checked against a brute-force search) finds where the line’s known text is said (the WhisperX approach). It runs on onnxruntime-node 1.23.2 (no new dependency) in a process of its own that keeps its model and ends after two idle minutes (see « Long recordings » below).
• Long recordings (12 Oct 2026). A recording of 90 s (AAC, 48 kHz) imported into a line stopped the engine and closed the module: ONNX Runtime aborted (SIGTRAP in BFCArena::Extend, the allocator’s request for a buffer it could not get) in the worker thread, which took the whole process with it, and the aligner had windows of 40 s whose peak was 2.4 GB on this machine. Now the audio is read in windows of 10 s whose outer 2 s are given up (the stretches kept are laid end to end: the CTC emissions of a frame depend on a few hundred ms around it), ONNX Runtime’s memory arena and memory pattern are off (they keep every block grown to, rounded up to a power of two, and the first run’s size), and the aligner runs in a process of its own (align/process.ts: the program that runs the engine with ELECTRON_RUN_AS_NODE, as the command line does; ending after two idle minutes frees the model): if ONNX Runtime still aborts, the line fails with E_ALIGN_CRASHED (« Les mots… n’ont pas pu être calés ») and the engine goes on. A recording tried against each line of a screen is read once (the process keeps the emissions of the last audio). Measured on the 90 s recording with the French model, peak memory footprint of the process that runs the model (/usr/bin/time -l, M5 16 GB): 40 s windows with the arena 2.44 GB (RSS 2.1 to 2.5), 20 s windows without 1.50 GB, 10 s windows without 1.12 GB (the model alone is 0.8 GB), the same 7 to 9 s of inference. Seams: the 40 lines of the French corpus (5 voices, 208.7 s, 642 words) laid end to end and aligned as one recording, each word’s start against the start the line alone gives it (the measure above): in windows of 10 s, median 8 ms, 90 % within 18 ms, 99 % within 25 ms, at most 28 ms (205 words within a second of a seam: median 8 ms, at most 28 ms); in one pass of 208 s median 8 ms, at most 42 ms; windows of 20 s, 6 s with 1 s: no worse. A line past 45 s is a warning, W_NARRATION_LINE_LONG: it is hard to follow and to correct; nothing is blocked.
• The models. French: jonatasgrosman/wav2vec2-large-xlsr-53-french (Apache-2.0) at revision 7c79e10, exported to ONNX (opset 17) with its matrix products quantized to int8, 355 MB; the file is XDcobra’s export, pinned at revision feb7081, which gives the same word starts as our own export of the same revision for 98 % of words (the others one frame apart). English: facebook/wav2vec2-base-960h (Apache-2.0) as Xenova’s quantized export (revision a19f851), 95 MB. Downloaded on the first alignment in a language into <local app data>/aligners/<model>/ (ELABORA_ALIGNER_MODELS_DIR; ELABORA_ALIGNER_MODELS_URL for a mirror), pinned and checked by SHA-256, resumed, one download at a time across processes (models/download.ts, shared with the voices). No permissive French model of base size exists (VoxPopuli fine-tunes and Meta’s MMS weights are NC); the English base model aligns French far worse (10 % of starts over 97 ms).
• Measured. Word starts: median 30 ms (French) and 25 ms (English), 9 in 10 within 74 and 69 ms, 78 and 79 % within 50 ms. Starts come about one frame late (biases +28 and +19 ms): every start is moved 20 ms earlier, which gives medians of 22 and 24 ms, 9 in 10 within 61 and 49 ms. Ends come early (−34 and −112 ms): they are moved by that much, up to the next word’s start; anchors use starts. Costs on an Apple M5 under load: 83 ms (French) and 60 ms (English) per second of audio, loaded in 0.5 and 0.1 s, 0.9 and 0.4 GB at the peak; int8 changes no start for 98 % of words and runs 2.3 times faster than fp32. The product’s port gives the prototype’s start for every word of the corpus (588 French, 599 English; the elisions it now splits aside).
• Discarded. Whisper with word timestamps (transformers.js 4.3.1, MIT weights): starts 119 to 158 ms late, 9 in 10 within 210 to 255 ms, 450 ms per second of audio for small, and it brings onnxruntime-node 1.30 (no Intel Mac binary), onnxruntime-web and sharp (libvips, LGPL-3.0), 471 MB of packages. NVIDIA’s NeMo FastConformer CTC through sherpa-onnx (CC-BY 4.0): 80 ms steps, median 95 to 103 ms, 9 in 10 within 315 to 392 ms, and 5 of 40 fast French lines have more pieces of text than steps. Kyutai STT: no ONNX export, 2.4 GB. Qwen3-ForcedAligner-0.6B (Apache-2.0, 11 languages): at least 1 GB, 80 ms steps, a language-model tokenizer; not tried, the one to watch. aeneas (AGPL), Montreal Forced Aligner and Gentle (Kaldi, dictionaries) are out.
• Text. The words are those of narrationWords (one tokenizer for anchors, the editor and the aligner: French elisions are words of their own, aligned run on with the next word). Numbers, times, ordinals, decimals and spoken symbols are spelt out as voices say them (French and English rules: « 9 h 30 », « 2e », « 3,5 % », « 9:30 », « 2nd », « 1,200 »), a symbol said with the word before it; letters fit the model’s vocabulary (case, accents it lacks, ligatures); a word with no letter of the vocabulary, or a number in a language without rules, is absorbed by a wildcard. Spelling out beats the wildcard, which swallows the pause after a number at the end of a line (French end errors, 9 in 10: 104 ms against 428 ms).
• Swallowed words. Each word’s score is the mean probability of its characters where the alignment puts them. A word of two letters or more under 0.25 is one the voice may have swallowed (W_NARRATION_WORD_SWALLOWED, and doubtful in the alignment’s result): on lines with a word cut out of their audio, with the product’s words, it catches 15 of 19 French cut words (the misses are short articles) and 20 of 20 English, and flags 0 of 40 intact French lines and 3 of 40 English ones. A badly pronounced word scores 0.5 to 0.8: the score finds a missing word, not a mispronounced one.
• When. A line a voice makes or a recording imported is timed at once, in the same change as its audio, when the aligner of its language is on the computer (ELABORA_ALIGN=0 turns it off; the tests do); elabora narration align [screen] [--line] [--force] (MCP elabora_align_narration, engine request alignNarration with alignProgress events, stopped by cancelVoice; aligners says what is installed) times the others, downloading the aligner first, through the command set_narration_words, one change a line, refused when the audio changed meanwhile. The sidecar keeps words (with score), aligner and text, what the audio says: when the line’s text changes, its words are followed by a word diff (D28) and editors quote what the voice still says. A translated course aligns its own voice (D28’s markers name the words).
• Transcription (12 Oct 2026, for recordings imported without their text; replaced on 15 Oct 2026 by D46’s transcriber): the same model, decoded greedily (the likeliest character of each frame, no language model), gives a draft with its words timed, sentences cut at pauses, numbers in words; on the corpus 28 % (French) and 20 % (English) of words are wrong (homophones such as « contrôler » for « contrôlez », proper names): a text for the author to check, never taken as it is (transcribeRecording, elabora narration transcribe). A checked text keeps its audio with keep_narration_audio (« La voix dit bien ce texte »), its words following by a word diff. Whisper transcribes far better but was set aside for its runtime (D27 above); it can come later as a transcriber of its own if drafts prove too rough. Revisit if: recorded human voices align worse than these synthetic ones (not measured; the xlsr-53 models were trained on Common Voice’s amateur recordings), the 0.25 threshold or the 20 ms correction do not hold on other texts, another language is needed (one character model and its number rules: check each model’s licence), Qwen3-ForcedAligner or a successor gets a small ONNX export, or the French model’s 0.9 GB matters on testers’ computers.

Revised by D57 (19 Oct 2026): the aligners’ files are downloaded from Elabora’s mirrors of XDcobra’s, jonatasgrosman’s and Xenova’s repositories (elabora/wav2vec2-large-xlsr-53-french-onnx, elabora/wav2vec2-large-xlsr-53-french, elabora/wav2vec2-base-960h-onnx), the same bytes, sizes and SHA-256, so the installed models stay current; the upstreams and revisions named above are those of the mirrors.