How precisely do speech systems time words?

Datasets, systems and metrics

We use three corpora with hand-labelled word boundaries: Buckeye (conversational interviews, 80,507 words), FluencyBank (people who stutter, 5,487 words) and TIMIT (read sentences, 14,043 words), the last one also with added noise.

Forced aligners are given the transcript and only place it in time. ASR systems transcribe and time in one go; their output is text-aligned to the reference and only correctly recognised words are timed, so timing is scored separately from transcription. Every table therefore also reports word recall, the share of reference words a system's timing numbers cover.

The main metric is Acc@X, the share of words whose start and end both lie within X of the reference. 25 ms is roughly the length of a phone; 200 ms is “roughly right”. We also report the mean absolute boundary error (MAE, the average of a word's start and end error) and mIoU, the mean intersection over union of predicted and reference word intervals: the time both intervals share, divided by the time either covers. A word placed exactly scores 100%, a word that does not overlap its reference at all scores 0%. Unlike MAE, IoU is relative to the word's length, so a 30 ms error costs more on a short function word than on a long content word.

Speed is how many times faster than real time a system processes one 20-second clip, measured one clip at a time on a single NVIDIA L40 GPU (MFA runs on CPU), or as the full request time for commercial APIs: uploading the audio, queueing, processing and returning the response. These requests were sent one at a time from a cloud VM in Montréal with a fast connection (Ookla speed test against a Montréal server: about 7.8 Gbit/s down, 17 Gbit/s up and 0.7 ms ping). The measured time still depends on the route to each provider and its load at the time, so it reflects the latency a client sees rather than the speed of the model alone.

MFA is listed twice: with its default search beam (10, retry 40) and with a wide beam (400, retry 4000), which drops almost no utterances but is slightly slower.

Accuracy vs. speed

The chart plots mean boundary error on Buckeye against processing speed. The Pareto frontier connects the systems that no other system beats on both axes at once.

Two systems are on the frontier: the nyra forced aligner (20.1 ms at 418× real time) and CTC-Segmentation (81.0 ms at 582×). Among the remaining systems, Fish Audio (35.5 ms), Olewave Olign v1 and MFA with a wide beam (both 35.7 ms), CrisperWhisper 2.0 (38.8 ms) and Qwen3-ForcedAligner (39.4 ms) have the lowest errors, at 21×, 5×, 0.8×, 35× and 359× real time. Widening MFA's beam lowers its error from 48.9 to 35.7 ms, but MFA stays slower than real time either way. How the nyra forced aligner works is described in its release post.

Results per dataset

Both tables are sorted by Acc@50ms. FluencyBank is harder for every system: repetitions, blocks and prolongations make word boundaries less clear, and errors rise accordingly.

Reading the tables

Accuracy should be read together with recall: the ASR systems recognise 74–93% of the words and are timed only on those. The MAE is heavy-tailed, so a few failed utterances can dominate it (MFA with the default beam on FluencyBank: 228 ms mean, 35 ms median). The tighter collars separate systems most clearly: at 200 ms most systems are above 90% on Buckeye, while at 25 ms the range is 1.0% to 63.3%.

Two of the aligners were trained on the evaluation corpora: the MWA Buckeye model on Buckeye recordings and the MWA TIMIT model on TIMIT (marked with * in the tables). Their scores on those sets are in-domain and not directly comparable with the other systems.

Robustness to noise

We add white Gaussian noise to TIMIT at 30 down to 0 dB signal-to-noise ratio; at 0 dB the noise is as loud as the speech. Because systems start missing words under noise, this table uses F1@50ms: a word counts only if it is recognised and placed within 50 ms.

Olewave Olign v1 has the highest clean score (83.8), ahead of the nyra forced aligner (80.2) and MFA (77.7). At 20 dB most systems stay within a few points of their clean score; MFA is the exception, with both beam settings (77.6 to 63.9 and 77.7 to 64.0). Below 10 dB the more accurate systems lose ground quickly, and at 0 dB the highest scores are 46.9 (MMS-FA), 45.7 (nyra forced aligner) and 38.3 (xAI Grok STT). MMS-FA and several commercial APIs hardly change with noise but start from a lower level, which suggests their timing resolution, not recognition, limits the score.

Where the benchmark goes from here

The current version covers English and a single noise type. We plan to extend it to more languages and to more realistic acoustic conditions: babble, music, reverberation and the channel effects of phone and video calls, in addition to white noise. Beyond added noise, we want to include realistic in-the-wild recordings: far-field and handheld microphones, clinical and home environments, and spontaneous speech captured outside the studio, where noise, reverberation and speaking style vary together rather than one at a time.

The larger limitation is the alignment scenario itself. All three corpora contain one speaker at a time, so the benchmark only measures alignment of non-overlapping speech. Real conversations are not like that: speakers interrupt, talk over each other and backchannel while the other is still speaking. Measuring timing in these situations needs reference annotations for overlapping speech, and building them is the next step. They will show how current methods behave when speakers overlap, and they are a prerequisite for developing methods that align several speakers jointly rather than one stream at a time.

A joint model is not the only route. A cascaded system could first separate the recording into one audio stream per speaker and then align each stream on its own. That would let existing single-speaker aligners be reused as they are, and an overlap benchmark would show how far such a pipeline gets compared with aligning all speakers at once.

Adding a system

The corpora are licensed and cannot be redistributed, so the repository contains the scorer, text-free manifests, every system's predictions and the scored results. To add a system, contribute a small adapter that turns one audio clip into word timings; we run it on all three datasets, the noise sweep and the speed test, and add the results.

Commercial APIs change over time; all their results are from October–November 2026.