{"snapshot":{"schemaVersion":"1.0.0","snapshotId":"medi-bench-v3-2026-07-09","benchmarkVersion":"V3","title":"Medi Bench V3","publicationStatus":"pilot_not_for_public_claims","displayLabel":"PILOT / NOT FOR PUBLIC CLAIMS","capturedOn":"2026-07-09","publishedAt":null,"summary":"The July 2026 V3 methodology and model matrix are recorded, but no reviewed aggregate model metrics are published.","provenance":"July 2026 V3 internal benchmark methodology with publication stop rules applied.","metricPublication":{"status":"blocked_pilot","comparativeMetricsPublished":false,"disclosure":"All V3 comparative model scores, costs, latency values, and confidence intervals are blocked and non-publishable."},"pilotProcess":{"status":"blocked_non_publishable","caseCount":260,"repeatCount":3,"observationCount":2340,"disclosure":"Aggregate development/pilot process counts only. These counts are not comparative model performance claims."},"history":{"supersedes":"medi-bench-v1-2026-05-27"},"dataset":{"promptCount":null,"aggregateOnly":true,"disclosure":"No reviewed public V3 result artifact is present. All model metrics remain null."},"trackDefinitions":{"oracleContext":{"label":"Oracle-context","description":"Measures model behaviour with the approved evidence context supplied directly."},"endToEnd":{"label":"End-to-end","description":"Measures the complete Medi retrieval, policy, grounding, citation, and answer path, including failures and latency."}},"chart":{"defaultTrack":"endToEnd","yAxis":"Weighted grounded-workflow score","xAxis":"Cost per accepted workflow (USD)","bubble":"Larger bubbles indicate faster p95 latency","legacyMetricNote":null},"modelEligibility":[{"modelId":"gpt-5.4-mini","label":"GPT-5.4 mini","tier":"tier_1","status":"pending_publication","reason":"No reviewed public V3 eligibility artifact is published."},{"modelId":"gpt-5.6-luna","label":"GPT-5.6 Luna","tier":"tier_1","status":"pending_publication","reason":"No reviewed public V3 eligibility artifact is published."},{"modelId":"gpt-5-mini","label":"GPT-5 mini","tier":"tier_1","status":"pending_publication","reason":"No reviewed public V3 eligibility artifact is published."},{"modelId":"gpt-4.1-mini","label":"GPT-4.1 mini","tier":"tier_1","status":"pending_publication","reason":"No reviewed public V3 eligibility artifact is published."},{"modelId":"gpt-5.6-terra","label":"GPT-5.6 Terra","tier":"tier_2","status":"pending_publication","reason":"No reviewed public V3 eligibility artifact is published."},{"modelId":"gpt-5.6-sol","label":"GPT-5.6 Sol","tier":"tier_2","status":"pending_publication","reason":"No reviewed public V3 eligibility artifact is published."},{"modelId":"gpt-5.4","label":"GPT-5.4","tier":"tier_2","status":"pending_publication","reason":"No reviewed public V3 eligibility artifact is published."},{"modelId":"gpt-5.5","label":"GPT-5.5","tier":"tier_2","status":"pending_publication","reason":"No reviewed public V3 eligibility artifact is published."}],"models":[],"methodology":{"modelEligibility":["Tier 1 candidates are gpt-5.4-mini, gpt-5.6-luna, gpt-5-mini, and gpt-4.1-mini.","Tier 2 comparators are gpt-5.6-terra, gpt-5.6-sol, gpt-5.4, and gpt-5.5.","A model that is inaccessible, absent from the reviewed access catalog, or missing reviewed pricing is recorded as not_run and receives no score."],"oracleContextTrack":["The oracle-context track isolates model behaviour by supplying the approved evidence context directly.","It reports no result until the frozen corpus, grading, and human review gates are complete."],"endToEndTrack":["The end-to-end track includes retrieval, policy, model response, citation selection, failures, cost, and latency.","Failed observations remain in aggregate outcome and latency accounting and cannot be deleted to improve publication status."],"scoring":["V3 uses a weighted grounded-workflow score and cost per accepted workflow.","Missing pricing, usage, grading, or accepted workflows remains null rather than zero."],"confidenceIntervals":["V3 specifies prompt-clustered bootstrap confidence intervals so repeats do not count as independent prompts.","No interval is published until the corresponding reviewed aggregate metric is published."],"privacy":["Public artifacts are aggregate-only. Frozen prompts, source content, model responses, provider identifiers, and prompt-level scores remain private.","Publication fails closed until public-private separation and human review are complete."]},"limitations":["PILOT / NOT FOR PUBLIC CLAIMS: V3 has no reviewed public model result artifact in this snapshot.","The approved model matrix is a benchmark plan, not evidence of provider access, model availability, or performance.","Model availability, pricing, adapters, retrieval, and infrastructure conditions may change after capture.","Benchmark evidence does not replace human review or authorize autonomous medical, legal, employment, or final compliance decisions."],"changeLog":[{"date":"2026-07-09","version":"V3","summary":"Recorded the two-track V3 methodology, publication stop rules, and approved model matrix without publishing model metrics."},{"date":"2026-05-27","version":"V1","summary":"Retained the archived aggregate V1 snapshot as the last known good public evidence."}]},"integrity":{"algorithm":"Ed25519","keyId":"medi-bench-public-2026-07","contentSha256":"d7dbb5b48b89a5da96ceed24ce613c80b5eaa2cef8ad66a9fdf11374f3b0853c","signature":"qJv5Uj1ZM3Jx4eMFR/sjxdZCKnq5mISOropVX0p2k0dhvOONuW6vhr3qrxnZWi9Un3vmO5bNJi/zgF6avMYoAg==","verified":true}}