omni-benchOMNI
…

Documentation

Run the benchmark with an agent, understand the supported workflows, and publish scored results.

omni-bench is a framework, not a model runner: hosts own inference, while the framework owns datasets, identity, measurement, scoring, parity, and publication. The public contract is 0.6.0.

Agent integration

Use omni-bench with coding agents

The repository ships one maintained skill covering every benchmark family and the complete artifact pipeline. Install it into the project where your agent will implement or test a producer.

Prepare → run → score → diff

Canonical manifests, artifacts, results, identity, validation, and producer/scorer boundaries.

Quick PR qualification

Run exact experimental quick Tasks with omni-bench qualify. Public runs produce PR-ready Markdown and a 1200×630 PNG card; they never mix with full leaderboard populations.

ASR · batch and streaming

Transcriber and StreamingTranscriber adapters, paced audio chunks, WER/CER, latency, and RTFx.

Text generation

Decode throughput, TTFT/ITL, shared-prefix cache scenarios, and host-reported backend timing.

Structured tool calls

Strict single-turn JSON call generation on pinned BFCL v4 simple_python prompts. This static task does not claim native tool APIs, execution, agent behavior, or official BFCL leaderboard equivalence.

Translation · QA · IFEval

FLORES+ chrF++, final-answer exact/accuracy, prompt-strict instruction following, and dataset gates.

Voice activity detection

ActivityDetector and StreamingActivityDetector adapters, AVA-Speech preparation, DCF/miss/false alarm, RTFx throughput, decision latency, and stream integrity.

Python · Swift · parity

JCS identity equality, implementation comparisons, signed deltas, timeline parity, fixtures, and conformance checks.

Install project skill
mkdir -p .agents/skills/omni-bench
curl -fsSL https://raw.githubusercontent.com/beshkenadze/omni-bench/main/.agents/skills/omni-bench/SKILL.md \
  -o .agents/skills/omni-bench/SKILL.md

Project-local installation keeps the instructions versioned with your work. Agents that support the .agents/skills convention discover it automatically; otherwise point the agent at SKILL.md explicitly.

Open the complete omni-bench skill on GitHub →
Completeness gate

A realtime benchmark needs more than a score

Every family is reviewed across six independent dimensions. RTFx and tokens/s are throughput, never latency; batch-only is an adapter limitation, not proof that latency is irrelevant.

Quality

Correctness against the task reference or protocol.

Throughput

Work completed per unit time: RTFx, tokens/s, and cache speedup.

Latency

Delay between named events with a declared clock and aggregation.

Streaming integrity

Pacing, chunking, endpointing, cache continuity, and backpressure.

Resources

Memory and other declared resource high-water marks.

Parity

Evidence that implementations preserve comparable output.

VAD 0.6.0: decision latency is explicit

VAD reports DCF, miss, false alarm, F1, RTFx, first-speech and pooled speech-boundary decision latency, finalization, pacing drift, consumer lag, and chunk completion. RTFx remains throughput, never latency.

On-device execution

Control a compiled iPhone host from the CLI

App Bench pairs an iOS 17 execution agent with a local Mac/Linux/WSL controller. The phone runs only compile-time registered text-generation or ASR providers while foreground and Armed. Use the normal project skill and exact Registry quick Tasks; device_control_version: 1.0.0 is transport, while the benchmark contract remains 0.6.0.

Agent-friendly CLI after pairing
omni-bench device serve --advertise-host 192.168.1.10 --json
omni-bench device list --json
omni-bench device qualify ios.DEVICE \
  --candidate mlx.model \
  --task qa.gsm8k.test.quick.v1 \
  --measurement-profile text_generation.streaming_single.v1 \
  --run-profile '{"delivery":"streaming","chunk_ms":20,"warmup_samples":0,"concurrency":1,"family_parameters":{…}}' \
  --json

The platform API key stays on the controller. Inputs are downloaded and SHA-256 verified before inference; phone clocks and RSS own the measurements. Raw Evidence is scored by the trusted Python service into a private Result visible in My Runs, then publication is explicit. Controller or network time is never reported as model latency.

Publishing

Upload scored results

Score locally, then publish the resulting result.json with a personal API key. The steps below cover the request and error contract.

Step 1

Get an API key

API keys live on your profile. Create one, copy it once (it is shown a single time), and keep it secret — it is your upload credential. The same profile includes My Runs: a searchable, filterable inventory where you can remove runs you own.

Open your profile →
Step 2

POST your result.json

Send the bare result document (the exact shape the scorer emits, validated .strict()) to POST /api/v1/results with your key in the x-api-key header. The body must carry schema_version — only 0.6.0 is accepted today.

curl
curl -X POST https://omni-bench.example/api/v1/results \
  -H "x-api-key: $OMNI_BENCH_API_KEY" \
  -H "content-type: application/json" \
  --data-binary @result.json

The request body is validated against the versioned Result JSON Schema (the contract the scorer and the API share). Its TypeScript type and Zod validator are generated from that schema, so the API can't drift from the data.

Result contract & JSON Schema (methodology) →

Uploads are PUBLIC by default. To stage a result as private, add the header x-omni-visibility: private.

Step 3

Read the response

201 Created

On success the API returns the new result id and its identity key (the content-addressed key that groups re-scores of the same tuple).

{
  "resultId": "res_01J…",
  "identityKey": "sha256:8bbc0ff…"
}

Errors

  • 401Missing or invalid API key.
  • 409The run_id is already owned by another principal (or an identity race — retry).
  • 413Body over the 4 MB cap — a real result is a few KB, so this means an abusive payload.
  • 422Invalid JSON, an unsupported schema_version, or a document that fails schema validation (the response lists the Zod issues).
Derived evidence

Upload an Analysis Report

After every source result.json has been uploaded, send the exact core-produced analysis-report.json to POST /api/v1/analysis. The API resolves the Analysis Profile and parameter schema from the pinned Registry, then binds every ordered input digest to a readable Result. It does not recompute the observations.

curl
curl -X POST https://omni-bench.example/api/v1/analysis \
  -H "x-api-key: $OMNI_BENCH_API_KEY" \
  -H "content-type: application/json" \
  --data-binary @analysis-report.json

Public reports require public input Results. Add x-omni-visibility: private to stage an analysis over private evidence, then publish the Results before calling POST /api/v1/analysis/:id/publish. The response links to its detail page; GET /api/v1/analysis/:id exports the exact immutable JSON.