How to evaluate speech recognition models

Learn how to evaluate speech-to-text models beyond Word Error Rate. This guide covers Semantic WER, Missed Entity Rate, ground truth correction, and practical benchmarking frameworks for 2026.

Part 1: Understanding core evaluation metrics

Word Error Rate (WER): The foundation

Word Error Rate remains the industry standard, and for good reason. It’s simple, reproducible, and gives you a single number to compare across vendors. But you need to understand what it actually measures. (For a deeper primer on WER as a metric, see Is Word Error Rate Useful?.)

WER calculates the percentage of words that differ between a reference transcript (ground truth) and the AI-generated transcript. It counts three types of errors:

• Insertions (hallucinations): Words the AI added that weren’t in the reference
• Deletions: Words in the reference that the AI missed entirely
• Substitutions: Words where the AI transcribed something different (e.g., “Cadillac” as “cataracts”)

The formula:

WER = (Insertions + Deletions + Substitutions) / Total Words in Reference × 100

A simple example:

• Reference: “The quick brown fox jumps over the lazy dog” (9 words)
• AI output: “A quick brown fox jumped over the lazy dog” (9 words)
• Errors: 2 substitutions (“The” → “A”, “jumps” → “jumped”)
• WER: (2 / 9) × 100 = 22.2%

The catch: Before you calculate WER, both transcripts must be normalized. You need to lowercase everything, strip punctuation, convert numbers to words, and standardize contractions. A transcript that says “don’t” will score differently than “do not” if you skip normalization—even though they’re semantically identical.

AssemblyAI recommends using the open-source Whisper Normalizer for English transcription and the Basic normalizer for other languages.

Semantic WER: Beyond word-for-word accuracy

Here’s where things get interesting. Traditional WER assumes every word matters equally. But in modern Voice AI applications, transcripts often feed directly into LLMs, not human readers. An LLM doesn’t care whether the transcript says “cannot” or “can’t”—it understands the meaning either way. Yet traditional WER penalizes both equally.

Semantic WER addresses this. Instead of comparing words, it uses a reasoning model to evaluate whether the meaning of the transcript is preserved. The process:

1. An LLM (like Claude Sonnet) receives both the AI transcript and the reference transcript
2. The LLM judges whether they convey the same meaning and information
3. The system calculates accuracy based on semantic equivalence, not word-for-word matching

Missed Entity Rate (MER): What really matters in specialized domains

WER treats all words equally, but not all words are equal. In medical transcription, getting “hydrochlorothiazide” wrong is catastrophic. Missing a filler word like “um” is harmless.

Missed Entity Rate (MER) fixes this. It measures accuracy specifically on high-value words like drug names and dosages, proper nouns and company names, medical procedures and diagnoses, numbers, dates, and addresses, and domain-specific terminology.

Instead of averaging errors across an entire transcript, MER focuses on the words that matter most for your application.

LLM-as-a-Judge evaluation

For the highest-confidence evaluation, use a reasoning model (like Claude Sonnet 4.6 with extended thinking) as a judge. This works especially well for comparing transcripts where subjective quality matters.

Part 2: The ground truth problem

Here’s what nobody talks about: the biggest source of evaluation error isn’t the speech-to-text model. It’s the ground truth.

When customers complained that Universal-3 Pro scored worse than our older models, we investigated their ground truth files. What we found was shocking. The human-transcribed references contained systematic errors and inconsistencies that penalized our newer model for being more accurate than the reference. We published the full investigation in Why your WER benchmark might be lying to you—it’s essential reading before you run any benchmark.

Why ground truth quality matters

Imagine a customer transcribes audio and creates a reference file. They have a human transcriber listen and type what they hear. This reference becomes the “truth” you’ll compare models against.

But here’s the problem: human transcribers make mistakes too. They might mishear words and transcribe them consistently wrong, normalize speech in inconsistent ways (“alright” vs “all right”), miss medical abbreviations or specialized terminology, include filler words inconsistently, or disagree on formatting (capitalization, punctuation, numbers).

When your reference contains these errors, a more accurate model can score worse because it transcribes what was actually said, not what a fallible human thought they heard.

How to fix ground truth quality

This is where the Truth File Corrector comes in. Released in the AssemblyAI dashboard, this tool automatically:

1. Detects semantic equivalence issues across your reference file
2. Flags potential transcriber errors using language models
3. Suggests corrections based on domain knowledge
4. Allows you to review and approve changes before re-running evaluation

The result: your benchmark reflects model quality, not transcriber inconsistency.

Part 3: Practical benchmarking frameworks

Step 1: Prepare high-quality reference audio

Select 10–20 audio files that represent your real use cases: a mix of speaker accents and dialects, variety of audio qualities (clean studio, conference calls, noisy environments), real domain-specific terminology, and different speaker counts and overlapping speech.

Step 2: Create ground truth with quality controls

Option 1 (recommended): Use Universal-3 Pro to generate transcripts, then have domain experts review and correct only clear errors. This is faster than human transcription from scratch and you know the baseline quality is high.

Option 2: Hire professional transcribers. Specify that they should transcribe what’s actually said (not “clean up” grammar), and use industry-standard formatting for your domain.

Step 3: Normalize both transcripts

Before calculating metrics, normalize both the reference and model output using the Whisper Normalizer:

from whisper_normalizer.english import EnglishTextNormalizer

normalizer = EnglishTextNormalizer()
reference_normalized = normalizer(reference_transcript)
ai_normalized = normalizer(ai_transcript)

Step 4: Calculate multiple metrics

Don’t stop at WER. Run at least three metrics:

import jiwer

# Traditional WER
wer = jiwer.wer(reference_normalized, ai_normalized)

# Calculate error breakdown
output = jiwer.compute_measures(reference_normalized, ai_normalized)
substitutions = output['substitutions']
deletions = output['deletions']
insertions = output['insertions']

Step 5: Use the Truth File Corrector

Before finalizing your benchmark:

1. Upload your reference file to the AssemblyAI dashboard
2. Run it through the Truth File Corrector
3. Review flagged inconsistencies
4. Approve or modify corrections
5. Re-run your evaluation with the cleaned reference file

Step 6: Run streaming vs. async evaluation separately

If you’re evaluating both streaming and pre-recorded models, benchmark them independently. They’re optimized for different tradeoffs.

Part 4: Comparison table of evaluation metrics

Our recommendation: For most applications, calculate WER + MER on domain-specific terms as your primary metrics. Use Semantic WER if your transcripts feed into LLMs. Add LLM-as-a-Judge evaluation for regulated industries (medical, legal) where subjective quality matters.

Part 5: Real-world evaluation patterns

Enterprise medical transcription

Use MER on clinical terminology as your north star. Primary metric: MER on drug names, procedures, diagnoses. Secondary metrics: traditional WER, Semantic WER. Ground truth: use the Truth File Corrector for consistency. Separate streaming and async benchmarks (medical ambient scribes need streaming; batch documentation needs async).

Voice agent transcription

For voice agents passing transcripts to LLMs, focus on Semantic WER. Primary metric: Semantic WER (what the LLM understands). Secondary metric: deletion rate (hangs are worse than minor substitutions). Benchmark scenario: real customer conversations with overlapping speech. Evaluate Time to First Token for agent responsiveness.

Contact center analytics

Balance accuracy with sentiment detection. Primary metric: WER on critical information (customer names, issues, account numbers). Secondary metrics: sentiment analysis accuracy, emotional tone preservation. Run on actual calls with background noise and accent diversity.

Part 6: The benchmarking SDK and GitHub tools

AssemblyAI provides open-source tools to automate evaluation:

GitHub Repository: AssemblyAI Benchmarking SDK (free, open source)

The SDK includes automated WER calculation with Whisper Normalizer, MER calculation for domain-specific terms, visualization of error breakdowns, batch evaluation across multiple models, and export results to CSV/JSON for analysis.

pip install assemblyai-benchmark-sdk

python -m aai_benchmark \
  --reference truth.txt \
  --hypothesis model_output.txt \
  --language en \
  --metrics wer,mer,semantic_wer

The Truth File Corrector is built into the AssemblyAI dashboard—no setup required. It’s free for all users.

Part 7: Streaming vs. async benchmarking

Pre-recorded (async) evaluation

Async models have time to process fully, so optimize purely for accuracy. Calculate WER on full final transcripts, use normalized ground truth for fair comparison, benchmark against curated datasets, and separate audio scenarios (clean, conference calls, noisy).

Streaming evaluation

Streaming models balance accuracy against latency, so evaluate both dimensions:

• TTFT (Time to First Token): How quickly the model returns the first word
• TTCT (Time to Complete Turn): How long to finalize a complete utterance
• Accuracy on final transcripts: WER on completed turns (not partial hypotheses)
• Deletion rate: Especially important—a missed word causes the conversation to hang

When comparing streaming vs. async, don’t use pure accuracy—evaluate accuracy at the latency level you’ll actually deploy at.

Final words: Evaluation is a continuous process

Evaluation doesn’t end when you choose a model. It’s ongoing. Your use case evolves. Audio patterns change. Ground truth quality matters more than you’d expect. And metrics like Semantic WER and Missed Entity Rate tell a different story than traditional WER.

Quick action steps:

• Run your evaluation on at least 10 representative audio files with multiple models
• Calculate WER and domain-specific metrics (MER for medical, deletion rate for voice agents)
• Use the Truth File Corrector to validate ground truth quality before finalizing results
• Separate streaming and async evaluation—they optimize for different things
• Compare Semantic WER if your transcripts feed into LLMs