The nyra forced aligner

A laugh has to go somewhere

A forced aligner turns the transcript into a chain of phone states and finds the most likely path through the audio that visits them in order. Every frame of audio has to be explained by some state. If the transcript reads so that was fun but the speaker laughed after so and said uh before fun, the laugh and the filler still have to be assigned to something. The aligner could assign them to silence, but in practice we observe that it smears their acoustics into the neighbouring words, and the boundaries of those words end up hundreds of milliseconds off.

Noise breaks the same search in a different place. To see where, it helps to see how boundaries are actually extracted. Each 10 ms frame of audio is turned into a feature vector, MFCCs in MFA's case. Every phone state owns a Gaussian mixture model (GMM) fitted to the feature vectors of the frames it was trained on, and for a new frame each state's GMM returns a likelihood: how probable this frame's features are under this phone. Laying the transcript's states out as rows and the frames as columns gives a grid of these likelihoods. The aligner then searches for the most likely path through the grid that visits the states in order, moving only forward in time, and each step of the path from one state to the next is a boundary. Timestamps are never predicted directly; they fall out of the path.

Everything therefore rests on how decisively the GMMs assign likelihood to the right phone for each frame. That is a property of the features: a GMM can only be decisive if the frames of one phone form a tight cluster that does not overlap the clusters of other phones. Clean MFCCs form such clusters and MFA aligns read speech well. Under noise the MFCC clusters spread and overlap, the GMMs of several states assign similar likelihood to the same frame, the most likely path is no longer the correct one, and boundaries drift. MFA compounds this with its default search beam of 10: it prunes any path that falls more than the beam behind the momentary best, and under noise that is exactly what happens to the correct path. Widening the beam recovers part of the loss (see the results below). Features whose phone clusters stay tight under noise fix it at the source: the GMMs fitted on them keep assigning likelihood decisively, and the path stays put whatever the beam. That is what our front end is built for.

Three changes to a classical aligner

We kept the Kaldi GMM-HMM architecture MFA uses: every unit is a three-state left-to-right HMM, trained monophone-first and then triphone, each stage supervised only by its own alignments. No external aligner, manual segmentation or boundary label is used as a training target at any stage. Once the WavLM features are extracted on a GPU, training takes hours on CPU. What changed is the features, the unit inventory and the data.

1. Self-supervised features instead of MFCCs

A frozen WavLM-Large encoder replaces the MFCC front end. WavLM is pretrained for robustness to acoustic corruption, and its layer-22 hidden states are the ones that correspond most closely to phonemes. We linearly interpolate them from their native 20 ms stride to the 10 ms grid the HMM works on and reduce them to 39 dimensions with PCA, matching MFA's MFCC dimensionality; the bottleneck is also necessary, since flat-start GMM training on the 1024-d features was unstable. After the monophone stage we return to the WavLM features, splice them over ±1 frame, and fit a 39-d linear discriminant (LDA) projection supervised by the monophone model's own per-frame state labels. The triphone model is trained in that space.

2. Explicit units for fillers and vocal sounds

Words are phonemized with espeak-ng onto a canonical inventory of 39 English phones. We extend that inventory with dedicated units for the fillers [UH] and [UM] and twelve non-lexical vocal events: laughter, breath, cough, sigh, and so on. Transcripts follow the verbatim conventions of CrisperWhisper 2.0, so each event appears inline as a bracketed tag, and each tag maps to a one-phone lexicon entry with its own acoustics and duration statistics. Where a standard aligner must ignore a laugh or smear it into the neighbouring words, ours assigns it a unit with matching acoustics. The aligner is built for exactly the transcripts CrisperWhisper 2.0 produces: give it that output and every filler and vocal sound in the audio has a unit to align to.

3. A projection that has already seen noise

WavLM already gives us fairly noise-robust features: swapping MFCCs for WavLM alone roughly halves the error at 0 dB SNR. The LDA projection is where that robustness is refined. LDA picks the directions in feature space along which the phone states are furthest apart relative to how much each state's own frames scatter: it maximises between-class over within-class variance. We fit it on the clean training features together with noise-corrupted copies of the same utterances (MUSAN music at 5–15 dB SNR), both carrying identical frame labels, since additive noise does not move a boundary. Because a clean frame and its noisy copy now belong to the same class, whatever the noise does to the features counts as within-class scatter, and any direction in which noise moves a frame is penalised. What survives are the directions in which phones differ but noise barely registers. In that space the frames of a phone form a tight cluster whether or not the audio is noisy, so the GMMs fitted there keep assigning likelihood decisively under noise. That is what holds the boundaries in place at 0 dB SNR, where the noise is as loud as the speech itself.

Verbatim training data

The event units are supervised only by tag occurrences in the training transcripts, so those transcripts have to be verbatim. We assembled 159 hours from eight sources, roughly half conversational, all transcribed verbatim with fillers and, where annotated, vocal sounds. To widen the event inventory we also insert clips from VocalSound and Nonspeech7k into detected pauses in clean LibriSpeech audio and add the tags at the same transcript positions.

Accurate where speech is messy

We score word boundaries against manual annotations on three held-out sets: TIMIT (read), Buckeye (conversational) and FluencyBank Timestamped (speech from people who stutter). Per word, the error is the mean of its onset and offset error; we report the mean over words (MAE, lower is better). Event tokens are excluded from word metrics for every system, and ASR-based systems are scored only on correctly recognised words, so timing is judged independently of transcription errors.

On read speech we are on par with the best classical configuration: MFA's us_arpamodel leads by 1.9 ms on TIMIT. The difference shows once the speech is no longer read. MFA rises to 25.3 ms on Buckeye and 122 ms on FluencyBank, 6.6× its read-speech error. Our error barely moves between read and conversational speech (20.5 vs. 20.6 ms) and rises far more gently on stuttered speech (76.9 ms, 17% below CrisperWhisper 2.0, the next best). Retraining MFA's unmodified recipe on our data (MFA our_data) helps on Buckeye but not on FluencyBank, so the data alone does not explain the full performance gap.

Fewer catastrophes, not just a better average

Averages hide the errors that break downstream use. For typical words the HMM systems are nearly indistinguishable; MFA is even marginally sharper at the median (13.2 vs. our 14.0 ms). The profiles separate in the tail. 99% of our words land within 97 ms of the reference versus 196 ms for MFA's best configuration; errors beyond 200 ms are 2.7× rarer (0.37% vs. 0.98%) and errors beyond one second 3× rarer. Averaged over each system's worst 5% of words, our error is 122 ms against MFA's 220.

Left of about 30 ms the two HMM systems overlap; from there our curve runs below every other at every severity. The other baselines fail in characteristic ways: NeMo is imprecise everywhere but rarely catastrophic, Qwen3-FA has an MFA-like mid-tail with 3× our rate beyond 200 ms, and CrisperWhisper 2.0's tail sits above ours at every threshold. The result is typical-word precision on par with the best classical system, with far fewer severe errors.

A note on MFA's search beam

MFA ships with a Viterbi beam of 10. On spontaneous speech that default silently prunes the optimal path: widening it to 400 improves us_arpa on Buckeye from 48.0 to 25.3 ms, and under noise most utterances fail to align at all at the default. All MFA numbers in this post therefore use the wide beam. Our aligner is essentially beam-independent (20.3 ms across a sweep from 5 to 2000): its emission likelihoods keep the correct path near the top at every frame, whereas overlapping MFCC densities let it fall behind on disfluent or noisy frames, where it gets pruned.

What the event units buy

The direct test of the unit inventory is the regular words next to a laugh, a filler or a word cutoff. Ours stays within 10 ms of its dataset-wide error next to fillers and cutoffs (26.3 and 28.7 ms) and reaches 52.8 ms next to laughter. Every MFA configuration inflates next to a filler, to 1.6–2.1× its own dataset-wide error, and lands at 120–230 ms next to laughter.

The component ladder separates the causes. Fillers have a spelling, so with verbatim transcripts they can already be represented as words; dedicated filler units add a small gain (47.5 to 41.9 ms in the MFCC arm) and the front end contributes the rest. Laughter has no spelling. There the units halve the error in the WavLM arm (107.8 to 52.4 ms) and cut it 2.7× with MFCCs (215.6 to 79.7 ms).

Boundaries that stay put at 0 dB SNR

We mix Buckeye with held-out MUSAN noise and with Gaussian noise at signal-to-noise ratios of 20, 10, 5 and 0 dB; at 0 dB SNR the added noise is as loud as the speech. Gaussian noise never appears in training (the noise-paired projection only saw MUSAN music), so it measures generalisation to an unseen noise family.

The ablation shows where the robustness comes from. Swapping MFCCs for WavLM roughly halves the error at 0 dB SNR (170 to 74 ms under MUSAN, 273 to 136 under Gaussian noise), and the noise-paired projection improves it again, to 50 and 96 ms. At 0 dB SNR with Gaussian noise, MFA english_mfa misplaces words by 2.0 s on average, us_arpa by 417 ms and WhisperX by 2.7 s; ours by 96 ms.

Use it

The aligner takes audio and a verbatim transcript and returns word- and phoneme-level timings, with fillers and vocal events as segments of their own, and runs at roughly 140× real time on a single NVIDIA L40 GPU. It is designed to run on the verbatim transcripts CrisperWhisper 2.0 produces: transcribe with CrisperWhisper 2.0, align with the nyra forced aligner, and every laugh and uh in the audio has a segment to belong to.

English only for now. Multilingual verbatim transcription withCrisperWhisper 2.0 makes extending the recipe beyond English feasible, and that is the next step.