Evaluate normalized recognition errors and the complete timed interaction

Align a hypothesis with a reference using a declared tokenization, normalization, and edit-scoring protocol, then count substitutions, deletions, and insertions. Case, punctuation, filled pauses, compounds, numerals, and language-specific segmentation can change both alignment and denominator. Apply the intended rules consistently to reference and hypothesis, with documented exceptions. NIST's SCTK documentation provides concrete alignment and word-scoring conventions and configurable alternatives; an experiment should record its tool version and options rather than using the name WER as if every scoring setup were identical.

Reference transcripts can contain annotation disagreement or ambiguity, so curate them under a stable policy and retain adjudication records where needed. Do not tune normalization on the final test set merely to erase a particular model's errors. WER can exceed one hundred percent when insertions are numerous. A reference with zero words has no ordinary per-utterance WER denominator; report its insertions and the dataset's aggregation policy explicitly. A high-quality final transcript also does not reveal how often provisional text changed before finalization or how long the user waited.

A corpus WER pools error counts and reference words. An unweighted average of utterance or cohort WERs gives equal weight to those units instead, which can produce a different answer. Report the choice and inspect language, accent, microphone, noise, speaking rate, utterance length, and interaction style where the data supports those analyses. Small cohorts have uncertainty and may need additional collection rather than a confident ranking. Keep the original test scope visible when an overlap exclusion, failed recording, or unsupported format changes which cases contribute to a score.

Quality beyond word accuracy can include critical entity correctness, task success, clipped speech, synthesis intelligibility, prosody, and interruption success. Automatic proxies should be labeled as proxies and compared against appropriate human judgments or task outcomes. Measure false starts and repeated or missing audio as well as transcript edits. A system can achieve similar pooled WER while systematically losing quiet final words or failing a smaller user cohort. Those effects matter to the intended interaction even if the largest clean-audio subset dominates the aggregate denominator.

Separate audio capture to partial recognition, capture to stable committed text, actual speech end to endpoint decision, and actual speech end to first audible system response. Report first synthesized bytes separately from sound leaving the output device. Use compatible monotonic timing within a clock domain or a justified cross-clock mapping; subtracting unsynchronized host timestamps can create meaningless latency. Decompose one critical path carefully so overlapped work is not counted twice. A real-time factor measures processing duration relative to audio duration and does not bound a particular live response's delay.

Include failures, timeouts, cancellations, underruns, and heavy tails in the evaluation rather than computing percentiles only over easy completed sessions. Report the population and treatment of censored outcomes. The sampling lab isolates an iid binary audit: Reset, explicitly set BOTH p=0.1 and n=20, predict mean-error variance 0.0045, then Play or Step; Reset and re-set BOTH p=0.1 and n=100, predicting 0.0009. Edits retain t; playback highlights exact outcomes. Its CDF approximation bound is not a WER or latency guarantee, and it cannot represent insertion counts, aligned transcripts, correlated speakers, or interaction timing.