Using Whisper to transcribe a D&D session, and where it falls down

What Whisper is, briefly

Whisper is OpenAI’s speech recognition model. The code and the model weights are released under the MIT licence, which means you can download it and run it on your own machine for free, indefinitely, with no account and no per-minute charge.

It comes in six sizes. The smallest, tiny, is 39 million parameters and needs about a gigabyte of VRAM; the largest, large, is 1.55 billion parameters and wants around ten. There is also a turbo model at 809 million parameters, roughly eight times faster than large. For a four-hour session of six people talking over each other, the size you pick matters more than any other setting — small will disappoint you and large will take a while on a laptop.

You can also call it as an API instead of running it, which costs money per minute but needs no GPU. Both routes are legitimate; the local route is free and slower, the API route is fast and metered. The current per-minute rate and what it works out to for a four-hour session is in the transcription cost breakdown, where it is kept sourced and dated rather than quoted here where it would quietly rot.

Weakness one: it has never heard of your NPCs

Speech recognition predicts the most probable word given the sound and everything before it. "Thordak Ironforge" is not a probable word sequence in any general training corpus, so you get "Thor Dak iron forge", or "thought Dak", and crucially you get a different wrong answer in different sessions. One NPC ends up under four spellings and your transcripts stop being searchable, which was the entire point of having them.

This is not a bug and no amount of better audio fixes it. It is what happens when you point a general-purpose model at invented proper nouns.

The fix: tell it the names in advance

Whisper accepts a prompt that biases decoding toward vocabulary you expect. OpenAI’s own documentation describes it as improving "recognition of names, acronyms, formatting, or recording-specific vocabulary", and specifically calls out product names, technical terms and acronyms. Your campaign’s cast list is exactly that problem wearing a different hat.

The constraint worth knowing before you build anything around it: on whisper-1 the prompt is capped at 224 tokens. That is a few hundred characters, not a glossary. You cannot paste your whole wiki in; you have to choose which names matter most for this session, which in practice means the recurring cast plus whoever is about to appear.

If you are rolling your own pipeline, that budget is the interesting engineering problem. Feeding it your player characters and the handful of NPCs currently in play will do more for transcript quality than upgrading the model size.

• Prompt with the names you expect this session, not every name you have ever used
• Player characters first — they are said most often and misspelled most visibly
• Recurring NPCs and place names next
• Re-check the budget: 224 tokens on whisper-1 is roughly 850 characters

Weakness two: it does not know who is talking

Whisper transcribes speech. It does not label speakers, so what you get back is a wall of text with no indication of who said which line. For a meeting that is annoying. For a session where the DM voices four NPCs and the players interrupt constantly, it is the difference between a usable record and a transcript nobody reads.

Two workarounds, and they are genuinely different. If you record with a multi-track bot like Craig you already have one audio file per person — transcribe each separately and you have perfect attribution, at the cost of running the job several times and interleaving the results by timestamp. See the Craig guide for how to get those tracks.

Alternatively, OpenAI now offers a separate transcription model that does speaker diarization directly, returning segments with speaker, start and end metadata. If speaker labels are what you need and you are already paying per minute, that is a more direct route than stitching multi-track output together yourself.

The file size limit that catches people out

If you use the API rather than running locally, uploads are capped at 25 MB. A four-hour session is comfortably over that in any decent quality, so you will be splitting the file whether you planned to or not. Compressed formats buy you room; splitting on silence rather than on a fixed clock avoids cutting someone off mid-sentence.

Running Whisper locally has no such limit, which is one of the better arguments for the local route if you have the hardware and the patience.

When rolling your own is the right call

Be honest about this: if you are comfortable at a command line, have a machine that can run a large model, and only want the text, Whisper locally is free and it works. There is no subscription that beats free, and the MIT licence means nobody can take it away from you or change the price.

What you are signing up for is the pipeline around it — splitting long audio, managing the prompt budget per session, handling speaker attribution, and then doing something with the resulting text, because a transcript is not notes. If that sounds like a project you would enjoy, it is a genuinely good one.