Methodology
What a leaderboard row actually means, and the scoring rules behind every WER, RTFx, and parity verdict. Short by design — enough to trust the numbers and to reproduce them.
A row is a tuple, not a model
A leaderboard row is never “Whisper large-v3.” It is the full identity tuple that produced one scored run. Two of its axes — backend and hardware — move both speed and quality dramatically on local inference, so the site keeps every axis legible and comparable rather than collapsing it into a single number.
highlighted axes move speed and quality — each is a column and a filter, never a footnote.
The tuple is hashed into a stable identity_key that de-duplicates rows, groups derived leaderboards, and permalinks each result detail page.
Quality, throughput, and latency are separate axes
There is no single composite “AI score.” Quality, throughput, and latency are reported independently, so a fast model with poor accuracy or slow responses cannot quietly win.
Quality uses the task headline such as WER, chrF++, or DCF. Throughput uses rtfx_native, rtfx_wall, or tokens/s. RTFx is never latency, even when it is above 1× realtime.
Latency uses producer-observed named events: ASR first-partial/finalization, text-generation TTFT/ITL/completion, and VAD first-speech/speech-boundary/finalization. Throughput and latency remain hardware- and profile-scoped.
WER normalization
Before scoring, both hypothesis and reference are put through one pinned, versioned pipeline so a WER means the same thing on every row:
- unicode_form — text is normalized to NFC so canonically-equivalent Unicode compares equal.
- normalizer — a pinned [email protected] normalizer (lower-casing, punctuation and whitespace handling). The id@version travels with every result, so a normalizer change is a visible, versioned event — never a silent shift in the numbers.
- remove_diacritics: false for Russian — diacritics are kept, so ё/й are preserved and not folded onto е/и. Removing them would understate WER on Russian.
wer_norm is computed on this normalized text; wer_ortho is the cased, un-normalized orthographic WER kept alongside it for reference. The reference set is content-hashed (references · sha256) so the exact targets are pinned too.
Micro-aggregation
Per-language metrics use micro-aggregation (aggregation: "micro"): errors and reference tokens are pooled across all utterances, then the rate is taken once — total edits ÷ total reference length. This weights every word equally, so long utterances count for more than short ones, rather than averaging per-utterance rates (macro), where one short sentence could swing the score.
Runs with errors are flagged as lower-confidence via n_error / n_missing counts and never silently averaged in. The row is built to grow into median ± CI (n=142 devices) once crowd aggregation lands.
Parity tolerances
Parity answers one question: did the port change the transcripts? Hold the model equal, vary the backend or chip, and check that the output stays equivalent within explicit tolerances. The verdict is PASS only when every per-language delta stays inside its bound; otherwise FAIL.
A report states what was held equal (shared) versus what changed (differs), shows each quality_delta against its tolerance, and reports identical_hypothesis_rate as a gauge. Mode quality is hardware-independent (transcripts only); mode full adds one-axis speed deltas. This is the check for validating a CUDA→MLX or PyTorch→Core ML port.
The schemas are the contract
Results and parity reports are validated against exact 0.6.0 JSON Schemas and the pinned Registry. ASR, text-generation, and VAD streaming latency use producer-observed events with explicit population counts.
Not affiliated with Apple or the model vendors. Cloud baselines, when they land, are labeled separately and never mixed into a hardware leaderboard.