OtakuBench
Which local model should translate the visual-novel line on your screen? Ranked on 200 hard Japanese lines (idioms, keigo, negation, dialect, ellipsis). Only models fast enough to read along with are included: under 2 s per line on an RTX 3090.
1Gemma 4 31B IT (Q5_K_M)9.851.21 s/line · fits 24 GB
9.29/10 on 31 full scenes0 of 200 lines scored 5 or lessMost common slips: voice, ambiguity, terms22.9 GB VRAM loadedLines translated well (9 or 10 out of 10), by category. 20 lines each, so treat differences under about 20 points as noise.Idioms & proverbs80%Wordplay & irony95%Dialect & voice100%Politeness (keigo)100%Indirect speech100%Emotional subtext95%Omitted subjects100%Culture & institutions100%Literary style95%Onomatopoeia95%
2Gemma 4 31B IT9.821.09 s/line · fits 24 GB
9.23/10 on 31 full scenes1 of 200 lines scored 5 or lessMost common slips: voice, implication, ambiguity20.0 GB VRAM loadedLines translated well (9 or 10 out of 10), by category. 20 lines each, so treat differences under about 20 points as noise.Idioms & proverbs85%Wordplay & irony100%Dialect & voice95%Politeness (keigo)100%Indirect speech100%Emotional subtext95%Omitted subjects100%Culture & institutions90%Literary style95%Onomatopoeia100%
3Gemma 4 26B-A4B IT9.730.28 s/line · fits 24 GB
9.55/10 on 31 full scenes1 of 200 lines scored 5 or lessMost common slips: voice, who did what, idioms16.6 GB VRAM loadedLines translated well (9 or 10 out of 10), by category. 20 lines each, so treat differences under about 20 points as noise.Idioms & proverbs75%Wordplay & irony75%Dialect & voice95%Politeness (keigo)100%Indirect speech100%Emotional subtext100%Omitted subjects95%Culture & institutions95%Literary style100%Onomatopoeia90%
4Gemini 3.5 Flash-Lite (cloud)*9.72
9.48/10 on 31 full scenes0 of 200 lines scored 5 or lessMost common slips: additions, ambiguity, idioms* Gemini: full scenes use only the first sentence of the system prompt; with the full prompt it returns some scenes untranslated.Lines translated well (9 or 10 out of 10), by category. 20 lines each, so treat differences under about 20 points as noise.Idioms & proverbs90%Wordplay & irony70%Dialect & voice100%Politeness (keigo)90%Indirect speech100%Emotional subtext100%Omitted subjects100%Culture & institutions95%Literary style90%Onomatopoeia95%
5Gemma 4 12B IT9.650.42 s/line · fits 12 GB
9.45/10 on 31 full scenes2 of 200 lines scored 5 or lessMost common slips: idioms, voice, omissions7.8 GB VRAM loadedLines translated well (9 or 10 out of 10), by category. 20 lines each, so treat differences under about 20 points as noise.Idioms & proverbs90%Wordplay & irony75%Dialect & voice90%Politeness (keigo)100%Indirect speech100%Emotional subtext100%Omitted subjects95%Culture & institutions80%Literary style90%Onomatopoeia95%
6Qwen3.8-27B (UD-Q6_K_L)9.621.33 s/line · fits 24 GB
9.19/10 on 31 full scenes2 of 200 lines scored 5 or lessMost common slips: voice, idioms, omissions22.0 GB VRAM loadedLines translated well (9 or 10 out of 10), by category. 20 lines each, so treat differences under about 20 points as noise.Idioms & proverbs65%Wordplay & irony80%Dialect & voice85%Politeness (keigo)95%Indirect speech100%Emotional subtext85%Omitted subjects95%Culture & institutions75%Literary style95%Onomatopoeia100%
7Qwen3.8-27B9.571.17 s/line · fits 24 GB
9.29/10 on 31 full scenes3 of 200 lines scored 5 or lessMost common slips: voice, idioms, implication15.1 GB VRAM loadedLines translated well (9 or 10 out of 10), by category. 20 lines each, so treat differences under about 20 points as noise.Idioms & proverbs60%Wordplay & irony80%Dialect & voice85%Politeness (keigo)95%Indirect speech100%Emotional subtext85%Omitted subjects95%Culture & institutions75%Literary style95%Onomatopoeia100%
8Hy-MT2-30B-A3B9.520.17 s/line · fits 24 GB
9.29/10 on 31 full scenes2 of 200 lines scored 5 or lessMost common slips: voice, ambiguity, idioms17.4 GB VRAM loadedLines translated well (9 or 10 out of 10), by category. 20 lines each, so treat differences under about 20 points as noise.Idioms & proverbs85%Wordplay & irony65%Dialect & voice80%Politeness (keigo)95%Indirect speech100%Emotional subtext85%Omitted subjects95%Culture & institutions95%Literary style90%Onomatopoeia90%
9Ornith-1.5-35B-A3B9.400.41 s/line · fits 24 GB
9.29/10 on 31 full scenes6 of 200 lines scored 5 or lessMost common slips: idioms, additions, voice20.2 GB VRAM loadedLines translated well (9 or 10 out of 10), by category. 20 lines each, so treat differences under about 20 points as noise.Idioms & proverbs65%Wordplay & irony65%Dialect & voice85%Politeness (keigo)100%Indirect speech85%Emotional subtext95%Omitted subjects75%Culture & institutions85%Literary style90%Onomatopoeia100%
10Qwen3.6-35B-A3B9.370.42 s/line · fits 24 GB
9.35/10 on 31 full scenes6 of 200 lines scored 5 or lessMost common slips: voice, idioms, omissions20.2 GB VRAM loadedLines translated well (9 or 10 out of 10), by category. 20 lines each, so treat differences under about 20 points as noise.Idioms & proverbs60%Wordplay & irony70%Dialect & voice80%Politeness (keigo)95%Indirect speech95%Emotional subtext90%Omitted subjects90%Culture & institutions80%Literary style80%Onomatopoeia95%
11TranslateGemma 27B IT9.330.57 s/line · fits 24 GB
9.39/10 on 31 full scenes3 of 200 lines scored 5 or lessMost common slips: voice, idioms, omissions16.7 GB VRAM loadedLines translated well (9 or 10 out of 10), by category. 20 lines each, so treat differences under about 20 points as noise.Idioms & proverbs65%Wordplay & irony65%Dialect & voice75%Politeness (keigo)90%Indirect speech85%Emotional subtext85%Omitted subjects80%Culture & institutions85%Literary style85%Onomatopoeia90%
12Gemma 4 E4B IT9.300.17 s/line · fits 8 GB
9.13/10 on 31 full scenes7 of 200 lines scored 5 or lessMost common slips: idioms, voice, who did what3.5 GB VRAM loadedLines translated well (9 or 10 out of 10), by category. 20 lines each, so treat differences under about 20 points as noise.Idioms & proverbs60%Wordplay & irony70%Dialect & voice75%Politeness (keigo)90%Indirect speech90%Emotional subtext85%Omitted subjects85%Culture & institutions80%Literary style90%Onomatopoeia90%
13Hy-MT2-7B9.210.16 s/line · fits 8 GB
8.74/10 on 31 full scenes5 of 200 lines scored 5 or lessMost common slips: idioms, voice, terms5.0 GB VRAM loadedLines translated well (9 or 10 out of 10), by category. 20 lines each, so treat differences under about 20 points as noise.Idioms & proverbs55%Wordplay & irony70%Dialect & voice85%Politeness (keigo)85%Indirect speech95%Emotional subtext85%Omitted subjects85%Culture & institutions70%Literary style85%Onomatopoeia95%
14TranslateGemma 12B IT9.200.30 s/line · fits 12 GB
8.87/10 on 31 full scenes6 of 200 lines scored 5 or lessMost common slips: voice, idioms, additions7.8 GB VRAM loadedLines translated well (9 or 10 out of 10), by category. 20 lines each, so treat differences under about 20 points as noise.Idioms & proverbs55%Wordplay & irony60%Dialect & voice60%Politeness (keigo)100%Indirect speech80%Emotional subtext75%Omitted subjects80%Culture & institutions70%Literary style95%Onomatopoeia85%
15Sugoi-14B-Ultra9.050.30 s/line · fits 12 GB
8.90/10 on 31 full scenes9 of 200 lines scored 5 or lessMost common slips: voice, idioms, additions8.8 GB VRAM loadedLines translated well (9 or 10 out of 10), by category. 20 lines each, so treat differences under about 20 points as noise.Idioms & proverbs55%Wordplay & irony65%Dialect & voice65%Politeness (keigo)90%Indirect speech85%Emotional subtext80%Omitted subjects75%Culture & institutions85%Literary style70%Onomatopoeia85%
16Gemma 4 E2B IT8.700.11 s/line · fits 8 GB
8.77/10 on 31 full scenes18 of 200 lines scored 5 or lessMost common slips: idioms, voice, modality1.9 GB VRAM loadedLines translated well (9 or 10 out of 10), by category. 20 lines each, so treat differences under about 20 points as noise.Idioms & proverbs25%Wordplay & irony70%Dialect & voice55%Politeness (keigo)90%Indirect speech80%Emotional subtext75%Omitted subjects60%Culture & institutions60%Literary style80%Onomatopoeia65%
17LFM2.5-24B-A2B (empty-think)8.060.16 s/line · fits 16 GB
7.45/10 on 31 full scenes28 of 200 lines scored 5 or lessMost common slips: idioms, voice, who did what14.7 GB VRAM loadedLines translated well (9 or 10 out of 10), by category. 20 lines each, so treat differences under about 20 points as noise.Idioms & proverbs15%Wordplay & irony45%Dialect & voice55%Politeness (keigo)60%Indirect speech60%Emotional subtext60%Omitted subjects70%Culture & institutions35%Literary style70%Onomatopoeia75%
18Flint 40B-A2.5B (empty-think)7.880.27 s/line · fits 24 GB
7.61/10 on 31 full scenes37 of 200 lines scored 5 or lessMost common slips: idioms, who did what, voice21.1 GB VRAM loadedLines translated well (9 or 10 out of 10), by category. 20 lines each, so treat differences under about 20 points as noise.Idioms & proverbs20%Wordplay & irony35%Dialect & voice55%Politeness (keigo)60%Indirect speech55%Emotional subtext55%Omitted subjects70%Culture & institutions40%Literary style55%Onomatopoeia60%
·Hy-MT2-1.8Bbeing judged: 0/200 lines
Score out of 10, judged by GPT-6.1 Sol. Click a model for its full-scene score and typical mistakes. New models are added as the daily judging run finishes them.
What's being tested?
Each model runs the way a screen translator would: a short instruction, a few lines of dialogue history, and the line on screen. That means a 2k context window, a 256-token output cap, and thinking switched off. Nobody waits 40 seconds for a line.
The 200 lines are written to catch the mistakes that ruin a scene: 油を売る read literally as "selling oil", 猫をかぶる as "pretends to be a cat", or a negation or a who-did-what reversed. Separately, the 31 full scenes test whether names, speakers and tone hold up across a conversation.
How is it scored?
The judge gives each translation 0–4 for meaning, 0–4 for intent and voice, and 0–2 for natural English. It treats the reference as one valid reading, not the only one, so creative equivalents aren't penalised. It doesn't know which model it's judging. A line the model failed to produce scores 0.
One judge and one run per model: differences of a few tenths between neighbours are within noise. More in Method.
Why speed first?
A translation that arrives after you've clicked to the next line is useless. Before any quality testing, every model must finish a line in under 2 s (median) and 3 s (slowest 10%), measured with 6 lines of history. On a 3090 almost everything passes, even dense 27–31B models at about 1.2 s. On smaller hardware the trade-off matters more. Latency has every model's speed and memory use.
Which models didn't make the cut?
Of the 50 candidates, 44 produced translations. Out for screen use:
- Thinking models that won't stop thinking. The three LLM-jp-4.1 Thinking models, and Qwen3-Swallow-8B-RL, which reasons in its visible reply.
- GPT-OSS-Swallow-20B: 2.9 s per line even at low reasoning effort.
- Nemotron 3 Nano 30B-A3B: too big for 24 GB at Q4, so it's deferred to a larger GPU.
Some others needed a workaround to run at all: gpt-oss-20b at low effort, LFM2.5-24B with an empty think block. These are labelled wherever they appear.
Can I see the test data?
The 50 development lines and 6 scenes are public. The 200 evaluation lines and their reference translations are encrypted in the repository, so the test doesn't leak into future models' training data. See Data policy.