Permanent benchmark record
Medi Bench
A versioned framework for comparing grounded workflow quality, accepted-workflow cost, and response speed as models change. Rules are fixed within each benchmark version.
- Displayed evidence
- V1 archived
- Latest candidate
- V3 pilot
- Integrity
- Ed25519 verified
PILOT / NOT FOR PUBLIC CLAIMS. The July 2026 V3 methodology is recorded, but no V3 model metrics are published. Its 260 cases, 3 repeats, and 2,340 observations are development/pilot process counts only, not comparative performance claims. The verified May 2026 V1 snapshot remains the last-known-good public evidence.
Last-known-good evidence
Medi Bench V1
The archived chart did not publish corpus size, repeats, confidence intervals, secondary rates, or prompt-level material.
V1 reported grounded-answer rate, cost per accepted answer, and p95 latency. It is archived directional evidence with no published corpus size, repeats, or confidence intervals, and is not directly comparable with the planned V3 weighted methodology.
| Model | Grounded answer rate | Cost per accepted answer (USD) | p95 latency | Confidence | Run |
|---|---|---|---|---|---|
| 99% | $0.0040 | 9.7s | The archived aggregate fixture did not publish confidence intervals. | Not published · Not published prompts · No repeats published | |
| 100% | $0.0008 | 8.1s | The archived aggregate fixture did not publish confidence intervals. | Not published · Not published prompts · No repeats published | |
| 98% | $0.0035 | 10.6s | The archived aggregate fixture did not publish confidence intervals. | Not published · Not published prompts · No repeats published | |
| 95% | $0.0005 | 14.3s | The archived aggregate fixture did not publish confidence intervals. | Not published · Not published prompts · No repeats published |
Methodology
Fixed rules before results.
The contract fixes track definitions, missing-value behaviour, public-data boundaries, and publication stops before any candidate can appear on the chart.
Scoring
- V1 reported grounded-answer rate, cost per accepted answer, and p95 latency.
- The archive uses only the exact rounded values visible in the May 2026 chart; missing information remains null.
Public-data boundary
- Public artifacts are aggregate-only. Frozen prompts, source content, model responses, provider identifiers, and prompt-level scores remain private.
- Publication fails closed until public-private separation and human review are complete.
Model eligibility
Planned is not published.
V3 records the approved comparison matrix. Public eligibility remains pending until access, pricing, completeness, and review are represented in a signed aggregate artifact.
| Model | Tier | Public status |
|---|---|---|
| GPT-5.4 mini | Tier 1 candidate | Pending / not published |
| GPT-5.6 Luna | Tier 1 candidate | Pending / not published |
| GPT-5 mini | Tier 1 candidate | Pending / not published |
| GPT-4.1 mini | Tier 1 candidate | Pending / not published |
| GPT-5.6 Terra | Tier 2 comparator | Pending / not published |
| GPT-5.6 Sol | Tier 2 comparator | Pending / not published |
| GPT-5.4 | Tier 2 comparator | Pending / not published |
| GPT-5.5 | Tier 2 comparator | Pending / not published |
Tracks
One model. Two questions.
01
Oracle-context
Measures model behaviour with the approved evidence context supplied directly.
- The oracle-context track isolates model behaviour by supplying the approved evidence context directly.
- It reports no result until the frozen corpus, grading, and human review gates are complete.
02
End-to-end
Measures the complete Medi retrieval, policy, grounding, citation, and answer path, including failures and latency.
- The end-to-end track includes retrieval, policy, model response, citation selection, failures, cost, and latency.
- Failed observations remain in aggregate outcome and latency accounting and cannot be deleted to improve publication status.
Confidence intervals
Uncertainty belongs beside the score.
- V3 specifies prompt-clustered bootstrap confidence intervals so repeats do not count as independent prompts.
- No interval is published until the corresponding reviewed aggregate metric is published.
The archived V1 fixture did not publish intervals or repeat counts. The interface says so directly rather than implying precision that the source aggregate does not support.
Limitations
- PILOT / NOT FOR PUBLIC CLAIMS: V3 has no reviewed public model result artifact in this snapshot.
- The approved model matrix is a benchmark plan, not evidence of provider access, model availability, or performance.
- Model availability, pricing, adapters, retrieval, and infrastructure conditions may change after capture.
- Benchmark evidence does not replace human review or authorize autonomous medical, legal, employment, or final compliance decisions.
Historical versions
History only moves forward.
2026-07-09
Medi Bench V3
PILOT / NOT FOR PUBLIC CLAIMS. Methodology recorded; model results not published.
2026-05-27
Medi Bench V1
Verified archived aggregate and current last-known-good public evidence.
Change log
What changed.
Recorded the two-track V3 methodology, publication stop rules, and approved model matrix without publishing model metrics.
Retained the archived aggregate V1 snapshot as the last known good public evidence.
Machine-readable snapshots
Each response includes the parsed snapshot and its Ed25519 integrity metadata. The index is append-only and hash-chained.