AI transcription accuracy: How it works and where it goes wrong

How speech becomes text

1. Speech is picked out. A separate small model finds where in the sound people are talking. Silence and music are left out as far as possible.
2. Sound becomes a picture. The waveform turns into a table of which frequencies dominate at each moment.
3. The model predicts words. Whisper reads the sound in windows of up to 30 seconds and writes the text piece by piece, using the context.
4. The result is cleaned. Our software removes repeats, stock phrases invented in silence and hesitation sounds.

No stage “knows” the meaning. The model picks the most likely continuation given the huge amount of sound and text it was trained on.

Where it goes wrong

The last is the most dangerous, because the result is a tidy, believable sentence. The peer-reviewed Careless Whisper study documented this in English recordings. We cover what it means for newsrooms in AI transcription ethics.

Invented sentences in silence

Whisper was trained on pairs of sound and text gathered from the internet, and a large share of them appear to be subtitles. That seems to be why, on silence, music or applause, it tends to write what appears at the end of subtitles. While building our engine we saw these most often:

• “Thanks for watching.”
• “Please subscribe.”
• “Subtitles by the Amara.org community”
• “[Music]”, “(applause)” and similar sound notes

LiveType filters these, does not send trailing silence to the model and does not re-run unchanged sound. That clears the common patterns, but it cannot catch every invention: We cannot know a new one in advance.

Realtime Whisper latency: Why live text lags

A model cannot safely write a word before hearing what follows. So a faint guess appears first, and it turns solid once later passes confirm the same words:

Three things set the lag: Model size, your graphics card and how the speech flows. The page shows the lag as it happens, suggests a smaller model if your device falls behind and a more accurate one if it has room to spare. You can edit solid text straight away. If you will quote something, wait until it is final.

Speaker diarization: What it means and how it works

Speaker diarization means working out who spoke when. Whisper does not do it: It only writes text. So speaker diarization with Whisper always needs other models:

1. Splitting: While people talk, one model looks at the last few seconds and finds how many different people are speaking.
2. Recognising: A second model makes a voice print of each speaker. Prints are compared, and the same print gets the same number.

It is deliberately cautious. Two people merged under one number cannot be fixed later, while one person split into two is solved by giving both one name. So you may see more numbers than people. Voice prints may count as biometric data, so they never go to the server and stay encrypted on your device. See speaker names.

Why piece length matters in the cloud

In the cloud, sound goes in pieces cut at silences, so no word is split. That is why there is no faint text there: The text arrives as each piece finishes.

Piece length affects the result for some languages. In our own tries on Groq, Turkish text fell apart in pieces of two minutes and longer, so files go in one-minute pieces. That is our own trial, not a research measurement. Cloud transcription and remote devices compares the options.

How to improve Whisper accuracy

• Record cleanly. Moving the microphone closer helps more than any setting.
• Choose the right language. The wrong one ruins the text.
• Use the largest model your device can carry. Choosing a model helps.
• Fix names early. You cannot give a list of names in advance: Whisper copies the style of any text it is given, so we closed that route. You correct them in the text.

When a human must check

If text will be attributed to a person, carries legal weight, will be published or becomes research data, a person must check it. A machine’s text is a draft, and whoever uses it is responsible for it. Do not treat a quote as verified until you have heard it. For a step-by-step check, use the list in how to transcribe a recording.