Run serve · mode serve · 200 decisions · tiers haiku → opus
Summary
OKSkipped-case error vs references is within tolerance 5%: the upper bound 4.8% is at or below it. From the audit alone (weighted disagreement 2.5%, interval 0.07%-12.7%) the status would be INCONCLUSIVE; that is what a run without references reports.
The fast tier (haiku) answered 90% of 200 tasks without escalating. On those, it disagrees with opus on an estimated 2.5% (95% CI 0.1–12.7%, 51 audits, n_eff 42). Against reference answers, the skipped cases are wrong on 1.7% (95% CI 0.6–4.8%, 181 graded). That is about 3.0 wrong answers among the 181 skipped cases (1.0–8.6).
Answered by fast tier
90.5%
181 of 200 tasks
Escalation rate
9.5%
95% CI 6.2–14.4%
Served accuracy
98.5%
95% CI 95.7–99.5%, 200 graded
Cost per task
$0.0016
Savings vs always opus
86.5%
95% CI 85.6–87.3%; always opus: $0.012/task
Audit overhead
192.2%
$0.60 audit spend
Expected wrong skipped cases
3.0
95% CI 1.0–8.6
Audit by confidence bin
Skipped cases by served confidence (95% CI in brackets)
Confidence bin
Skipped
Audited
Disagreement (CI)
Error vs references (CI)
[0.00, 0.50)
0
0
—
—
[0.50, 0.70)
0
0
—
—
[0.70, 0.80)
0
0
—
—
[0.80, 0.90)
44
15
13.3% (1.7–40.5%)
4.5% (1.3–15.1%)
[0.90, 0.95)
30
16
0% (0.0–20.6%)
3.3% (0.6–16.7%)
[0.95, 1.00]
107
20
0% (0.0–16.8%)
0% (0.0–3.5%)
Whiskers are 95% CIs; n is the number of audits per bin.
Tier usage and costs
Decisions served per tier
Tier
Role
Served
Share
haiku
fast
181
90.5%
opus
final, reference
19
9.5%
Audit coverage of skipped cases
Quantity
Count
Skipped
181
Audited
51
Not selected for audit
130
Audit pending
0
Audit errors
0
Undecided (judge could not decide)
0
No audit record
0
Graded against references
181
Unweighted disagreement (biased)
3.9% (1.1–13.2%)
Audit tier (opus) error vs references
0% (0.0–8.4%)
Costs (USD)
Quantity
Value
Serving cost (total)
$0.31
Audit cost (total)
$0.60
Cost per task
$0.0016
Est. cost per task, always opus
$0.012 ($0.011–$0.012)
Estimated savings
86.5% (85.6–87.3%)
Audit overhead (audit / serving)
192.2%
Decisions with unknown serving cost
0
Decisions with unknown audit cost
0
Threshold sweep
Offline threshold sweep over 200 eval-mode tasks (15 threshold settings); accuracy is measured against reference answers.
cheapest thresholds with accuracy >= 0.9900 (best single tier 'opus' at 1.0000 minus 0.01) on the selection split; validated on 60 held-out decisions (seed 0); held-out accuracy falls below the target; treat with caution
Baselines on all 200 usable tasks
Router
Accuracy (CI)
Cost / task
Escalation
Tier share
always haiku
95.0% (91.0–97.3%)
$0.00047
0%
haiku 100%, opus 0%
always opus
100% (98.1–100.0%)
$0.012
0%
haiku 0%, opus 100%
oracle (lower bound)
100% (98.1–100.0%)
$0.0010
5.0%
haiku 95%, opus 5%
recommended (all data)
99.0% (96.4–99.7%)
$0.0024
16.5%
haiku 84%, opus 16%
The oracle serves from the cheapest correct tier without paying for the tiers it skips; no real router can reach it.
Pareto frontier
Thresholds
Accuracy (CI)
Cost / task
Escalation
Skipped error (CI)
haiku ≥ 0
95.0% (91.0–97.3%)
$0.00047
0%
5.0% (2.7–9.0%)
haiku ≥ 0.55
95.5% (91.7–97.6%)
$0.00057
1.0%
4.5% (2.4–8.4%)
haiku ≥ 0.6
96.5% (93.0–98.3%)
$0.00080
3.0%
3.6% (1.8–7.3%)
haiku ≥ 0.8
98.5% (95.7–99.5%)
$0.0016
9.5%
1.7% (0.6–4.8%)
haiku ≥ 0.85
99.0% (96.4–99.7%)
$0.0024
16.5%
1.2% (0.3–4.3%)
recommended
haiku ≥ 0.9
99.5% (97.2–99.9%)
$0.0042
31.5%
0.73% (0.1–4.0%)
haiku ≥ 0.95
100% (98.1–100.0%)
$0.0061
46.5%
0% (0.0–3.5%)
Calibration
Confidence calibration per non-final tier
Tier
Scored
ECE
Brier
AUROC
AURC
Missing scores
haiku
200
0.063
0.041
0.910
0.008
0
ECE and Brier: lower is better. AUROC: how well the score separates right from wrong answers (0.5 = chance). AURC: area under the risk-coverage curve, lower is better.
Reliability diagram for haiku (200 scored tasks).Risk-coverage curve for haiku.
Caveats
est_savings treats serving cost per task as exact; its interval reflects only the uncertainty in the always-opus cost estimate, and excludes audit cost.
Methodology
Disagreement is the share of skipped cases (answered by a non-final tier without escalating) whose answer differs from the final (audit) tier's answer to the same task, as decided by the configured comparator or judge. In serve mode only a random sample of skipped cases is shadow-audited; each audited case is weighted by 1/π, the inverse of its known inclusion probability (a Hajek inverse-probability-weighted estimator), so the estimate describes all skipped cases rather than only the audited ones. In eval mode every tier answers every task, so every skipped case is observed (π = 1). The audit tier is a proxy, not ground truth: disagreement counts differences from its answers, not verified errors, unless reference answers exist, in which case the error rate against them is reported as well. Audits that are pending, failed or undecided are treated as nonresponse: within each inclusion-probability stratum, completed audits are up-weighted by the inverse of the stratum's response rate, which assumes audits are missing at random within their stratum; strata without any completed audit cannot be represented and are excluded from the estimate (a caveat says so). Intervals are Wilson score intervals for unweighted proportions and Korn-Graubard intervals (Clopper-Pearson at the effective sample size n_eff) for weighted ones; when a stratum holding at least 10% of the skipped cases has fewer than 10 completed audits, the status is never reported as ok. The status uses a fixed-sample interval: checking it repeatedly as audits accumulate and stopping at the first ok inflates the chance of a false ok, so fix the number of audits in advance or use a stricter confidence level for repeated checks. The savings interval reflects only the uncertainty in the cost of always using the final tier. Sweep accuracies are Wilson intervals; recommended thresholds are chosen on a selection split and re-evaluated on a held-out split.