How accurate is Whisper on Turkish? We measured the models

Results

int8, q4 and fp32 are the same model compressed to different degrees: fp32 is not compressed, int8 brings the file to about a quarter and q4 to about an eighth. A smaller build downloads faster, but its accuracy can change.

* On the conversations without timestamps, on a CPU, outside a browser. WebAssembly in a browser is slower and a graphics card through WebGPU much faster. ** M4 runs the same model in fp16 on a graphics card; fp16 does not run on a CPU, so we tested the closest build. On a graphics card M1 and M2 run this way too (fp16 encoder, q4); the CPU counterparts of those builds scored 50.4% for Base and 38.7% for Small on conversation.

These times are for writing a recording once from start to end. A live engine works through the same audio several times until words settle, so Turbo cannot be expected to keep up with live audio on a CPU, which is why LiveType’s Turbo models (M3 and M4) ask for a graphics card.

How our numbers compare with published ones

OpenAI published word error rates per language in the paper that introduced Whisper. The Turkish rows next to ours:

• No Turkish figure for Turbo. When OpenAI released Turbo it gave no number for Turkish and said that across languages it performs similarly to Large-v2. Our 9.5% on read sentences is close to Large-v2’s official 8.4%.
• An independent measurement: a peer-reviewed study by Polat and colleagues found error rates between 4.3% and 14.2% for Whisper without fine-tuning on five Turkish datasets.
• On read sentences the gap is small: we are 3 points behind the official figure for Small, and Turbo is about 1 point behind Large-v2. The shrunken builds (int8, q4) that fit in a browser, and different ways of normalising text, can explain that. For Tiny and Base the gap is over 10 points: small models suffer more from being shrunk.
• On conversation the gap is large: both official sets are sentences read by one person. On overlapping talk Turbo (14.5% and 17.9%) stayed near Large-v2’s Common Voice result (14.5%), while Small scored about one and a half times its official Common Voice figure (23.7%).

Why the small models fell apart on conversation

Most of Base’s and Small’s errors came from a few long stretches rather than single words. Where several people spoke at once, the model either skipped the stretch or wrote one sound over and over: Small’s int8 build wrote line after line of “hıhıhı” in the podcast. One such stretch adds several points even when the rest is right. With timestamps on, these loops mostly went away: Small’s int8 build fell from 44.1% to 29.5%, and on the same podcast from 90.2% to 45.2%.

Turkish is hard on small models too. Words carry a lot in their suffixes, so one wrong sound makes a whole word count as an error, which is why the app labels M1 (Base) as low accuracy.

The build moved the results in different directions from test to test. On read sentences Base’s int8 build was far better than fp32, and Small’s builds tied. On conversation the int8 builds fell behind with timestamps off and the fp32 builds with timestamps on; on average Base’s two builds are level (62.0% and 63.1%) and Small’s fp32 build is 3 points ahead (33.7% and 36.8%). With four recordings it is hard to say how much of that is chance.

Why one Turbo build scored over 100%

139.6% is not a typo. Extra words count as errors, so the rate can pass 100%. We first took the result for a fault in our test, but the conversations gave 100.9% with timestamps off. The int8 Turbo build wrote “görüp görüp” 199 times in one recording and “biz biz” 368 times in another, and left out half of a third. With timestamps on the loops thinned out, but at 44.6% it still trailed Small. The q4 build of the same model was among the best in both tests. We do not ship the int8 build.

The q4 build had a trap of its own: with timestamps off it skipped nearly all of the 30-second economics clip and wrote a single word, so we checked whether the test was at fault. Our code was not. The clip is exactly 30 seconds, the whole window Whisper hears at once, and in that window the q4 build jumped to a phrase at second 20 and stopped writing. One second shorter, the error rate fell to 10.8%; with timestamps on it was 16.9%. Fed through LiveType’s engine as if live, Turbo q4 wrote this clip with fewer errors than any other model, 6.0%.

Other builds have such traps too: the fp32-encoder build wrote only “Altyazı M.K.” (a subtitle credit) for the clip’s first 15 seconds. That is also why we average the two modes: on the other recordings the same q4 build made a few more errors with timestamps on, ending at 18.5% and 17.2%. Measured one way only, the score would hang on which recording the model happened to stumble over.

The general lesson: “Whisper Turbo is this accurate” means little unless it says which converted build was tested.

Clean clips vs live broadcasts

Short, clean sentences are where Whisper is most comfortable. A real broadcast adds music, noise, people talking over each other and a density of names, and a live engine has to write before the sentence ends. The conversations carry the overlapping talk and the names, so they predict a broadcast better than the read sentences do. Four recordings are still few: use these numbers to compare models, not to predict your own result.

These figures are for the model alone. LiveType does not hand audio to Whisper as it is: speech detection drops silent stretches, the engine has the same stretch written several times and commits only the words that agree, and it guards against invented repeats. With Small’s int8 build (M2 on a CPU) running in that engine, fed the same four recordings as if they were live, the error rate was 33.3%. That is between the two scores it got writing each whole recording at once (29.5% and 44.1%), and the engine writes while the talk is still going on.

AI transcription accuracy explains why and where the model goes wrong.

Which model when

On your own device the audio stays with you (private transcription). Being able to edit while it transcribes also softens a smaller model’s mistakes, since you fix each one as you see it.

The most useful test is your own recording. 30 minutes on the free plan is enough, your first transcript has the steps, and longer work is covered under pricing. When we test new models we will update this page, and the date is at the top.