# shadowgate report: demo-n400-s0-serve

Run demo-n400-s0-serve · mode serve · 400 decisions · tiers fast → slow

## Summary

**Status: BREACH** — Skipped-case error vs references exceeds tolerance 5%: the lower bound 8.9% is above it. From the audit alone (weighted disagreement 10.8%, interval 5.1%-19.4%) the status would be BREACH; that is what a run without references reports.

The fast tier (fast) answered **65%** of 400 tasks without escalating. On those, it disagrees with slow on an estimated **10.8%** (95% CI 5.1–19.4%, 89 audits, n\_eff 84). Against reference answers, the skipped cases are wrong on **12.3%** (95% CI 8.9–16.9%, 260 graded). That is about 32.0 wrong answers among the 260 skipped cases (23.0–43.8).

- **Answered by fast tier:** 65.0% (260 of 400 tasks)
- **Escalation rate:** 35.0% (95% CI 30.5–39.8%)
- **Served accuracy:** 92.0% (95% CI 88.9–94.3%, 400 graded)
- **Cost per task:** $0.00038
- **Savings vs always slow:** 54.8% (95% CI 53.6–55.9%; always slow: $0.00084/task)
- **Audit overhead:** 44.6% ($0.068 audit spend)
- **Expected wrong skipped cases:** 32.0 (95% CI 23.0–43.8)

## Audit by confidence bin

**Skipped cases by served confidence (95% CI in brackets)**

| Confidence bin | Skipped | Audited | Disagreement (CI) | Error vs references (CI) |
| --- | ---: | ---: | ---: | ---: |
| \[0.00, 0.50) | 0 | 0 | — | — |
| **\[0.50, 0.70)** | **13** | **7** | **100% (59.0–100.0%)** | **100% (77.2–100.0%)** |
| **\[0.70, 0.80)** | **15** | **9** | **77.8% (40.0–97.2%)** | **86.7% (62.1–96.3%)** |
| **\[0.80, 0.90)** | **6** | **4** | **75.0% (19.4–99.4%)** | **66.7% (30.0–90.3%)** |
| \[0.90, 0.95) | 3 | 0 | — | 33.3% (6.1–79.2%) |
| \[0.95, 1.00\] | 223 | 69 | 0% (0.0–5.2%) | 0.45% (0.1–2.5%) |

_Highlighted bins have a disagreement interval entirely above the tolerance._

_Charts are included in the HTML report._

## Tier usage and costs

**Decisions served per tier**

| Tier | Role | Served | Share |
| --- | --- | ---: | ---: |
| fast | fast | 260 | 65.0% |
| slow | final, reference | 140 | 35.0% |

**Audit coverage of skipped cases**

| Quantity | Count |
| --- | ---: |
| Skipped | 260 |
| Audited | 89 |
| Not selected for audit | 171 |
| Audit pending | 0 |
| Audit errors | 0 |
| Undecided (judge could not decide) | 0 |
| No audit record | 0 |
| Graded against references | 260 |
| Unweighted disagreement (biased) | 19.1% (12.3–28.5%) |
| Audit tier (slow) error vs references | 0% (0.0–4.3%) |

**Costs (USD)**

| Quantity | Value |
| --- | ---: |
| Serving cost (total) | $0.15 |
| Audit cost (total) | $0.068 |
| Cost per task | $0.00038 |
| Est. cost per task, always slow | $0.00084 ($0.00082–$0.00086) |
| Estimated savings | 54.8% (53.6–55.9%) |
| Audit overhead (audit / serving) | 44.6% |
| Decisions with unknown serving cost | 0 |
| Decisions with unknown audit cost | 0 |

## Threshold sweep

Offline threshold sweep over 400 eval-mode tasks (52 threshold settings); accuracy is measured against reference answers.

_Charts are included in the HTML report._

### Recommendation

- **Objective:** max-savings
- **Thresholds:** fast ≥ 0.89
- **Selection accuracy (280 tasks):** 99.3% (95% CI 97.4–99.8%)
- **Selection cost per task:** $0.00043
- **Selection escalation rate:** 42.1%
- **Held-out accuracy (120 tasks):** 99.2% (95% CI 95.4–99.9%)
- **Held-out cost per task:** $0.00049
- **Held-out escalation rate:** 45.8%
- **Accuracy vs always slow:** -0.8 pp (95% CI -2.5 to +0.0 pp)

_cheapest thresholds with accuracy >= 0.9900 (best single tier 'slow' at 1.0000 minus 0.01) on the selection split; validated on 120 held-out decisions (seed 0)_

**Baselines on all 400 usable tasks**

| Router | Accuracy (CI) | Cost / task | Escalation | Tier share |
| --- | ---: | ---: | ---: | --- |
| always fast | 57.0% (52.1–61.8%) | $0.000027 | 0% | fast 100%, slow 0% |
| always slow | 100% (99.0–100.0%) | $0.00084 | 0% | fast 0%, slow 100% |
| oracle (lower bound) | 100% (99.0–100.0%) | $0.00043 | 43.0% | fast 57%, slow 43% |
| **recommended (all data)** | **99.2% (97.8–99.7%)** | **$0.00045** | **43.2%** | **fast 57%, slow 43%** |

_The oracle serves from the cheapest correct tier without paying for the tiers it skips; no real router can reach it._

**Pareto frontier (showing 15 of 43 points)**

| Thresholds | Accuracy (CI) | Cost / task | Escalation | Skipped error (CI) |   |
| --- | ---: | ---: | ---: | ---: | --- |
| fast ≥ 0 | 57.0% (52.1–61.8%) | $0.000027 | 0% | 43.0% (38.2–47.9%) |  |
| fast ≥ 0.33 | 61.5% (56.6–66.1%) | $0.000074 | 4.5% | 40.3% (35.5–45.3%) |  |
| fast ≥ 0.36 | 69.2% (64.6–73.6%) | $0.00016 | 12.2% | 35.0% (30.2–40.2%) |  |
| fast ≥ 0.39 | 79.0% (74.7–82.7%) | $0.00026 | 22.0% | 26.9% (22.3–32.1%) |  |
| fast ≥ 0.42 | 84.0% (80.1–87.3%) | $0.00031 | 27.0% | 21.9% (17.6–27.0%) |  |
| fast ≥ 0.45 | 87.5% (83.9–90.4%) | $0.00034 | 30.5% | 18.0% (13.9–22.9%) |  |
| fast ≥ 0.48 | 89.5% (86.1–92.1%) | $0.00036 | 32.5% | 15.6% (11.7–20.4%) |  |
| fast ≥ 0.52 | 91.2% (88.1–93.6%) | $0.00037 | 34.2% | 13.3% (9.7–17.9%) |  |
| fast ≥ 0.56 | 92.2% (89.2–94.5%) | $0.00038 | 35.2% | 12.0% (8.6–16.5%) |  |
| fast ≥ 0.61 | 94.0% (91.2–95.9%) | $0.00040 | 37.0% | 9.5% (6.5–13.8%) |  |
| fast ≥ 0.71 | 95.5% (93.0–97.1%) | $0.00041 | 38.5% | 7.3% (4.7–11.3%) |  |
| fast ≥ 0.74 | 97.2% (95.1–98.5%) | $0.00042 | 40.2% | 4.6% (2.6–8.1%) |  |
| fast ≥ 0.77 | 98.2% (96.4–99.1%) | $0.00043 | 41.2% | 3.0% (1.5–6.0%) |  |
| **fast ≥ 0.89** | **99.2% (97.8–99.7%)** | **$0.00045** | **43.2%** | **1.3% (0.5–3.8%)** | **recommended** |
| fast ≥ 0.99 | 100% (99.0–100.0%) | $0.00047 | 46.0% | 0% (0.0–1.7%) |  |

### Calibration

**Confidence calibration per non-final tier**

| Tier | Scored | ECE | Brier | AUROC | AURC | Missing scores |
| --- | ---: | ---: | ---: | ---: | ---: | ---: |
| fast | 400 | 0.185 | 0.093 | 0.999 | 0.114 | 0 |

_ECE and Brier: lower is better. AUROC: how well the score separates right from wrong answers (0.5 = chance). AURC: area under the risk-coverage curve, lower is better._

## Caveats

- est\_savings treats serving cost per task as exact; its interval reflects only the uncertainty in the always-slow cost estimate, and excludes audit cost.

### Methodology

Disagreement is the share of skipped cases (answered by a non-final tier without escalating) whose answer differs from the final (audit) tier's answer to the same task, as decided by the configured comparator or judge. In serve mode only a random sample of skipped cases is shadow-audited; each audited case is weighted by 1/π, the inverse of its known inclusion probability (a Hajek inverse-probability-weighted estimator), so the estimate describes all skipped cases rather than only the audited ones. In eval mode every tier answers every task, so every skipped case is observed (π = 1). The audit tier is a proxy, not ground truth: disagreement counts differences from its answers, not verified errors, unless reference answers exist, in which case the error rate against them is reported as well. Audits that are pending, failed or undecided are treated as nonresponse: within each inclusion-probability stratum, completed audits are up-weighted by the inverse of the stratum's response rate, which assumes audits are missing at random within their stratum; strata without any completed audit cannot be represented and are excluded from the estimate (a caveat says so). Intervals are Wilson score intervals for unweighted proportions and Korn-Graubard intervals (Clopper-Pearson at the effective sample size n\_eff) for weighted ones; when a stratum holding at least 10% of the skipped cases has fewer than 10 completed audits, the status is never reported as ok. The status uses a fixed-sample interval: checking it repeatedly as audits accumulate and stopping at the first ok inflates the chance of a false ok, so fix the number of audits in advance or use a stricter confidence level for repeated checks. The savings interval reflects only the uncertainty in the cost of always using the final tier. Sweep accuracies are Wilson intervals; recommended thresholds are chosen on a selection split and re-evaluated on a held-out split.

---

_Generated by shadowgate 0.1.0 at 2026-10-15 13:09:19 UTC · run demo-n400-s0-serve · mode serve_