shadowgate report: sonnet-serve

Run sonnet-serve · mode serve · 1,680 decisions · tiers haiku → sonnet

Summary

BREACHSkipped-case error vs references exceeds tolerance 5%: the lower bound 7.9% is above it. From the audit alone (weighted disagreement 4.1%, interval 2.3%-6.6%) the status would be INCONCLUSIVE; that is what a run without references reports.

The fast tier (haiku) answered 80% of 1,680 tasks without escalating. On those, it disagrees with sonnet on an estimated 4.1% (95% CI 2.3–6.6%, 471 audits, n_eff 386). Against reference answers, the skipped cases are wrong on 9.3% (95% CI 7.9–11.0%, 1,344 graded). That is about 125 wrong answers among the 1,344 skipped cases (106–147).

Answered by fast tier
80.0%
1,344 of 1,680 tasks
Escalation rate
20.0%
95% CI 18.2–22.0%
Served accuracy
86.2%
95% CI 84.5–87.8%, 1,680 graded
Cost per task
$0.0027
Savings vs always sonnet
69.4%
95% CI 68.8–70.0%; always sonnet: $0.0089/task
Audit overhead
90.6%
$4.12 audit spend
Expected wrong skipped cases
125
95% CI 106–147

Audit by confidence bin

Skipped cases by served confidence (95% CI in brackets)
Confidence binSkippedAuditedDisagreement (CI)Error vs references (CI)
[0.00, 0.50)00——
[0.50, 0.70)00——
[0.70, 0.80)00——
[0.80, 0.90)33717211.6% (7.2–17.4%)21.4% (17.3–26.1%)
[0.90, 0.95)2831382.9% (0.8–7.3%)9.5% (6.6–13.5%)
[0.95, 1.00]7241611.2% (0.2–4.4%)3.6% (2.5–5.2%)

Highlighted bins have a disagreement interval entirely above the tolerance.

Skipped-case disagreement by confidence binBars show the disagreement per bin with 95% CI whiskers; the dashed line is the tolerance.0%5%10%15%20%[0.00, 0.50) n=0n=0[0.00, 0.50)[0.00, 0.50)[0.50, 0.70) n=0n=0[0.50, 0.70)[0.50, 0.70)[0.70, 0.80) n=0n=0[0.70, 0.80)[0.70, 0.80)[0.80, 0.90) 12% [7.2%, 17%] n=172n=172[0.80, 0.90)[0.80, 0.90)[0.90, 0.95) 2.9% [0.8%, 7.3%] n=138n=138[0.90, 0.95)[0.90, 0.95)[0.95, 1.00] 1.2% [0.15%, 4.4%] n=161n=161[0.95, 1.00][0.95, 1.00]tolerance 5%served confidencedisagreement
Skipped-case disagreement by confidence binBars show the disagreement per bin with 95% CI whiskers; the dashed line is the tolerance.0%5%10%15%20%0.00 n=0n=00.000.000.50 n=0n=00.500.500.70 n=0n=00.700.700.80 12% [7.2%, 17%] n=172n=1720.800.800.90 2.9% [0.8%, 7.3%] n=138n=1380.900.900.95 1.2% [0.15%, 4.4%] n=161n=1610.950.95tolerance 5%served confidence (bin start)disagreement
Whiskers are 95% CIs; n is the number of audits per bin.

Tier usage and costs

Decisions served per tier
TierRoleServedShare
haikufast1,34480.0%
sonnetfinal, reference33620.0%
Audit coverage of skipped cases
QuantityCount
Skipped1,344
Audited471
Not selected for audit873
Audit pending0
Audit errors0
Undecided (judge could not decide)0
No audit record0
Graded against references1,344
Unweighted disagreement (biased)5.5% (3.8–8.0%)
Audit tier (sonnet) error vs references6.5% (4.2–9.4%)
Costs (USD)
QuantityValue
Serving cost (total)$4.55
Audit cost (total)$4.12
Cost per task$0.0027
Est. cost per task, always sonnet$0.0089 ($0.0087–$0.0090)
Estimated savings69.4% (68.8–70.0%)
Audit overhead (audit / serving)90.6%
Decisions with unknown serving cost0
Decisions with unknown audit cost0

Threshold sweep

Offline threshold sweep over 1,680 eval-mode tasks (32 threshold settings); accuracy is measured against reference answers.

Accuracy vs cost per task32 simulated threshold settings (gray), the Pareto frontier (line), single-tier and oracle baselines, and the recommended point (star).82%84%86%88%90%92%$0.000$0.002$0.004$0.006$0.008$0.010cost per task (USD)accuracyoperating pointsPareto frontieralways haikualways sonnetoracle (lower bound)recommendedoracle (lower bound)always sonnetrecommendedalways haikuoperating pointsoperating pointsPareto frontierPareto frontieralways haikualways haikualways sonnetalways sonnetoracle (lower bound)oracle (lower bound)recommendedrecommended
Accuracy vs cost per task32 simulated threshold settings (gray), the Pareto frontier (line), single-tier and oracle baselines, and the recommended point (star).82%84%86%88%90%92%$0.000$0.005$0.010cost per task (USD)accuracyoperating pointsPareto frontieralways haikualways sonnetoracle (lower bound)recommendedoracle (lower bound)always sonnetrecommendedalways haikuoperating pointsoperating pointsPareto frontierPareto frontieralways haikualways haikualways sonnetalways sonnetoracle (lower bound)oracle (lower bound)recommendedrecommended
Gray: every simulated threshold setting. Line: Pareto frontier. Star: recommended thresholds.

Recommendation

Objective
max-savings
Thresholds
haiku ≥ 0.88
Selection accuracy (1,176 tasks)
87.3% (95% CI 85.3–89.1%)
Selection cost per task
$0.0048
Selection escalation rate
41.5%
Held-out accuracy (504 tasks)
88.7% (95% CI 85.6–91.2%)
Held-out cost per task
$0.0041
Held-out escalation rate
34.7%
Accuracy vs always sonnet
-1.6 pp (95% CI -3.0 to -0.4 pp)

cheapest thresholds with accuracy >= 0.8641 (best single tier 'sonnet' at 0.8741 minus 0.01) on the selection split; validated on 504 held-out decisions (seed 0)

Baselines on all 1,680 usable tasks
RouterAccuracy (CI)Cost / taskEscalationTier share
always haiku83.0% (81.2–84.8%)$0.000580%haiku 100%, sonnet 0%
always sonnet88.3% (86.6–89.7%)$0.00880%haiku 0%, sonnet 100%
oracle (lower bound)90.2% (88.7–91.6%)$0.00137.2%haiku 93%, sonnet 7%
recommended (all data)87.7% (86.1–89.2%)$0.004539.5%haiku 61%, sonnet 39%

The oracle serves from the cheapest correct tier without paying for the tiers it skips; no real router can reach it.

Pareto frontier (showing 16 of 21 points)
ThresholdsAccuracy (CI)Cost / taskEscalationSkipped error (CI)
haiku ≥ 083.0% (81.2–84.8%)$0.000580%17.0% (15.2–18.8%)
haiku ≥ 0.183.1% (81.2–84.8%)$0.000600.12%16.9% (15.1–18.7%)
haiku ≥ 0.2583.2% (81.4–84.9%)$0.000640.42%16.6% (14.9–18.5%)
haiku ≥ 0.383.3% (81.5–85.0%)$0.000670.60%16.5% (14.8–18.3%)
haiku ≥ 0.483.8% (82.0–85.5%)$0.000872.2%15.6% (13.9–17.4%)
haiku ≥ 0.584.0% (82.2–85.7%)$0.00113.8%15.0% (13.4–16.9%)
haiku ≥ 0.684.5% (82.7–86.1%)$0.00146.2%13.9% (12.3–15.7%)
haiku ≥ 0.6585.4% (83.6–87.0%)$0.002315.2%10.7% (9.2–12.5%)
haiku ≥ 0.785.5% (83.8–87.1%)$0.002315.5%10.6% (9.1–12.3%)
haiku ≥ 0.8286.8% (85.1–88.3%)$0.003427.1%8.0% (6.6–9.7%)
haiku ≥ 0.8586.8% (85.1–88.4%)$0.003427.5%7.9% (6.5–9.5%)
haiku ≥ 0.8887.7% (86.1–89.2%)$0.004539.5%5.3% (4.1–6.9%)recommended
haiku ≥ 0.9287.9% (86.2–89.3%)$0.005449.6%4.4% (3.2–6.0%)
haiku ≥ 0.9387.9% (86.3–89.4%)$0.005550.8%4.1% (3.0–5.7%)
haiku ≥ 0.9688.2% (86.6–89.7%)$0.006967.7%2.8% (1.7–4.5%)
haiku ≥ 0.9888.3% (86.6–89.7%)$0.008689.3%0.56% (0.1–3.1%)

Calibration

Confidence calibration per non-final tier
TierScoredECEBrierAUROCAURCMissing scores
haiku1,6800.0240.1100.8190.0620

ECE and Brier: lower is better. AUROC: how well the score separates right from wrong answers (0.5 = chance). AURC: area under the risk-coverage curve, lower is better.

Reliability: haikuAccuracy per confidence bin with Wilson intervals; points on the diagonal are perfectly calibrated.0%20%40%60%80%100%0%50%100%mean confidenceaccuracyperfect calibrationobserved
Reliability diagram for haiku (1,680 scored tasks).
Risk-coverage: haikuError rate among the skipped cases as the threshold is lowered and the tier answers more cases alone.0%5%10%15%20%0%50%100%coverage (share answered by tier)risk (error among skipped)tolerance 5%risk
Risk-coverage curve for haiku.

Caveats

Methodology

Disagreement is the share of skipped cases (answered by a non-final tier without escalating) whose answer differs from the final (audit) tier's answer to the same task, as decided by the configured comparator or judge. In serve mode only a random sample of skipped cases is shadow-audited; each audited case is weighted by 1/π, the inverse of its known inclusion probability (a Hajek inverse-probability-weighted estimator), so the estimate describes all skipped cases rather than only the audited ones. In eval mode every tier answers every task, so every skipped case is observed (π = 1). The audit tier is a proxy, not ground truth: disagreement counts differences from its answers, not verified errors, unless reference answers exist, in which case the error rate against them is reported as well. Audits that are pending, failed or undecided are treated as nonresponse: within each inclusion-probability stratum, completed audits are up-weighted by the inverse of the stratum's response rate, which assumes audits are missing at random within their stratum; strata without any completed audit cannot be represented and are excluded from the estimate (a caveat says so). Intervals are Wilson score intervals for unweighted proportions and Korn-Graubard intervals (Clopper-Pearson at the effective sample size n_eff) for weighted ones; when a stratum holding at least 10% of the skipped cases has fewer than 10 completed audits, the status is never reported as ok. The status uses a fixed-sample interval: checking it repeatedly as audits accumulate and stopping at the first ok inflates the chance of a false ok, so fix the number of audits in advance or use a stricter confidence level for repeated checks. The savings interval reflects only the uncertainty in the cost of always using the final tier. Sweep accuracies are Wilson intervals; recommended thresholds are chosen on a selection split and re-evaluated on a held-out split.