Benchmark

Does the glot pipeline actually ship fewer broken strings than asking a model to "translate this JSON"? glot-bench measures it on a public evaluation set, with the same models on both sides, and a judge that is not glot's own code.

The dataset

bench/data/messages.jsonl: 500 English UI messages written for the benchmark (CC0), modeled on patterns from open-source app catalogs (issue trackers, chat, billing, file sharing, CI dashboards). Each line has an id, a category, the source, a UI description, a ui_role, the glossary terms it uses and, for buttons and tabs, a pixel budget (fit).

The glossary is bench/data/glossary.toml. The set is generated by bench/data/gen.py, which deduplicates it. The 40 ambiguous strings keep repeated texts on purpose, since only the context differs.

Target locales: fr, de, es, pl, ru, ar, ja, zh. They cover 1 to 6 CLDR plural categories, a right-to-left script and two CJK languages.

The two conditions

• raw: the prompt a developer would write. JSON in, JSON out. It asks the model to keep placeholders, ICU syntax and tags, and passes the glossary and each string's UI description. There is no validation and no retry. Everything the model returns is written.
• glot: the full glot translate pipeline with the same model. That means the system prompt with the target's CLDR plural categories and the glossary, the pixel budget per string, validation, two targeted retries, and escalation to a stronger model (claude-haiku-5-5 → claude-sonnet-5-5, gpt-5-mini → gpt-5). A message that still fails is rejected: it is left untranslated instead of written broken.

What counts as broken

A message is broken when it was written to the catalog and fails one of these checks:

1. glot's validator (glot_core::validate). It checks ICU syntax, lost or invented placeholders, missing CLDR plural categories, translated keywords, broken tags, the glossary and do-not-translate terms (inflected forms accepted), and the pixel width measured with the same font metrics as glot check.
2. An independent judge (bench/judge/icu-judge.mjs). It parses with FormatJS's @formatjs/icu-messageformat-parser (the parser behind intl-messageformat and react-intl) and takes the required plural categories from the JS engine's own CLDR data (Intl.PluralRules). It flags syntax errors, placeholder or tag drift, missing plural categories and translated select keys. Because of it, glot is not the only one grading glot.

The report shows both the combined rate and the independent-judge-only rate. It also reports rejected messages, cost, tokens (including prompt-cache reads), calls and latency per call for every (model, locale, condition).

The benchmark does not score meaning or fluency (there are no human reference translations). It measures what breaks a build or a screen.

Reproduce

cd bench/judge && npm install && cd ../..                      # independent judge
export ANTHROPIC_API_KEY=… OPENAI_API_KEY=…
cargo run -p glot-bench -- run --sample 40 --budget-usd 3 --dry-run   # estimate only
cargo run -p glot-bench -- run --models claude-haiku-5-5,claude-sonnet-5-5,gpt-5-mini \
  --locales fr,de,es,pl,ru,ar,ja,zh --sample 40 --budget-usd 3
cargo run -p glot-bench -- report --docs web/content/docs/benchmark.md

• Estimate first. run prints the expected and worst-case cost of every cell before calling anything. It refuses to start if the expected total exceeds --budget-usd, and it stops calling the API once real spend reaches that cap.
• Resumable. Every paid answer is cached in bench/cache/, keyed by the exact request. Finished cells are skipped. Re-running the same command after an interruption costs nothing for the work already done.
• Offline check. --mock serves every model with the mock-noisy provider. It breaks a seeded share of answers the way models do: a lost or invented placeholder, a dropped plural branch, a translated keyword, a broken tag or brace (GLOT_MOCK_SEED, GLOT_MOCK_NOISE, default 15 %). Use it to verify the metrics end to end without a key.

Results land in bench/results/<date>/: report.md, report.json, chart.svg and the raw outputs per cell.

Results

Run bench/results/2026-10-18-real.

Broken = written to the catalog and failing glot's validator (ICU syntax, placeholders, CLDR plural categories, keywords, tags, glossary, pixel width) or the independent FormatJS + Intl.PluralRules judge. Rejected = refused by glot after retries/escalation and left untranslated instead of shipped broken (raw prompts never reject).

Summary (all locales)

Per locale

Why messages broke