How the numbers were made, and what they cannot tell you

Context

In 2021 the team worked on confidential lab-result spreadsheets and HPLC control exports supplied by the industry partner (CSL). The revival had to make the methods explorable in a public web app without exposing anything that came from the partner: values, identifiers, process-stream names, sample-naming rules or charts drawn from the real data. The 2021 coursework and its outputs stay in a private repository, and data/ and plots/ are ignored everywhere.

Recorded on 9 October 2026 for a decision taken when the site was first rebuilt, so the 2026 upgrade (evaluation studies, decision records, optional AI) inherits it explicitly.

Decision

• Every sheet, chart and cluster comes from web/src/lib/synthetic, built from one integer seed (default 30034, the notebooks' random_state) with a small mulberry32 generator.
• Column names mimic generic chromatography exports; every value, stream name, identifier format, sequence name and instrument name is invented.
• Faults are planted on purpose (out-of-control runs, a slow drift, degraded samples, blank and text cells) and listed, so the site can say what is real signal and what is noise.
• The original 2021 code is run on the same synthetic inputs (scripts/parity_reference.py) to prove the TypeScript ports behave like it.
• The optional AI feature (DR-004) only ever sees numbers derived from this synthetic data.

Options considered

1. Anonymise the real data (rename streams, rescale values, jitter dates). Rejected: re-identification risk, and the partner never agreed to any release.
2. Use a public HPLC data set. Rejected: none had both messy lab-result sheets and matching control-assay exports, so the join between parsing and charting (Sample Name to Injection Name) could not be shown.
3. Seeded synthetic generator with planted faults (chosen).
4. Static screenshots of the 2021 outputs. Rejected: they would show partner data, and nothing would be explorable.

Why

The generator keeps the structure that makes the methods interesting (free-text identifiers, several process streams, bracketing control injections, runs that carry one stream's samples) without any of the content. Planted faults also give something the real project never had: ground truth, so a method can be scored on whether it finds what was planted.

What happened

• The parity suite runs the 2021 Python on the synthetic inputs and the ports match: all 43 assay, control and measurement charts reproduce the notebook's order, seven limit lines and printed out-of-control list.
• Over 40 synthetic worlds the parser handles 99.74% of rows (95% bootstrap CI 99.69% to 99.78%; range 99.33% to 100%). That is close to the 2021 figure of about 99.7% by construction: the generator's junk-row rates were chosen to give it. It shows the parser's rules work on the formats I invented, not that they would work on new real formats. This is the weakest number on the site and I say so wherever it appears.
• The planted faults were made findable (control shifts of 5 to 6 sigma, degraded samples 8 to 11 spreads away). Recovery rates on /evaluation are therefore upper bounds for the methods, not estimates of how often a real lab would catch a real fault.
• Nothing from the partner is in this site's code, data or content: every value shown is generated from the seed.

What I'd change

• Calibrate the generator's spreads and fault sizes against published HPLC system-suitability ranges instead of my own guesses, and add autocorrelation within runs, which real controls often show.
• Generate a second, harder world (smaller faults, more realistic typing errors) and report the parse rate and fault recovery on both, so the easy case is not the only one.

Context

The 2021 notebook flagged only points beyond 3 sigma, with sigma the sample standard deviation of every point on the chart. The revival added Western Electric and Nelson run rules as checkboxes, which made it easy to switch on all eight. Nothing on the site said what that costs: each rule adds false alarms, and a QC chart that cries wolf trains people to ignore it. The team had also spotted a run of six rising amino acid results by eye, which is exactly the trend rule (N3), so there was a case for adding it.

Decision

• Default view on /control-charts: the 2021 rule (3 sigma only), so the page still shows what the team did.
• Presets: "2021 notebook", "3-sigma + 8 on one side" (recommended addition), "Western Electric" (rules 1 to 4) and "Western Electric + Nelson" (all eight). Each preset shows its simulated false-alarm probability for a 150-point in-control chart.
• Not recommended: the trend rule on its own, and the four Nelson rules beyond Western Electric.
• Limits: the 2021 all-points limits stay the default view, but the page links to the masking study, and the Phase I switch (limits from the first 40 points) is the recommended way to judge later results.

Options considered

1. 3 sigma only (the 2021 notebook). Fewest false alarms, slow on small sustained shifts.
2. 3 sigma + 8 on one side (chosen as the addition).
3. 3 sigma + trend of 6 (what the team saw by eye).
4. Western Electric rules 1 to 4.
5. All eight Western Electric and Nelson rules.
6. A CUSUM or EWMA chart instead of run rules. Better suited to small shifts, but a different chart from the one the team built.

Why

From /evaluation (4,000 simulated streams per scenario, every rule set on the same streams, 95% intervals):

• "8 on one side" gives most of the detection gain for sustained shifts (about three times faster at 1 sigma) for 2.4 times the false-alarm rate.
• The trend rule costs nearly as many false alarms and buys almost nothing, even against the drifts it is meant for: at 0.05 sigma per point its ratio to the 3-sigma rule is 0.97 (0.97 to 0.98). Noise breaks long monotone runs before they reach six points.
• The four extra Nelson rules detect nothing faster than Western Electric 1 to 4 here (9.4 against 9.5 points at 1 sigma) and cut the in-control ARL from 91.3 to 66.0.
• A preset that shows its cost lets a visitor make the trade-off knowingly instead of ticking every box.

What happened

• The simulation engine reproduces exact values: the 3-sigma rule's in-control ARL interval covers the theoretical 370.4, and "8 on one side" alone gives 255 (2⁸ − 1) in the tests. The published 3-sigma estimate, 361.9, sits near the bottom of its interval; it is a Monte Carlo estimate, not a disagreement with theory.
• The bigger problem turned out to be the limits, not the rules. With the 2021 all-points limits, a sustained 3-sigma shift in the last third of a 150-point chart is caught in only 2.5% (2.0% to 3.0%) of charts, because the shift inflates the sample SD and drags the centre line. Limits from the first 40 points catch it in 100% (99.9% to 100%). At 1.5 sigma the paired gain is 75.8 percentage points (74.5 to 77.0).
• Phase I limits have their own cost: with 40 points the limits are noisy, and 27.2% (25.8% to 28.6%) of in-control charts flag one of the first 100 points, against 21.7% (20.5% to 23.0%) for all-points limits. With 20 Phase I points, 41.1% (39.8% to 42.5%) of analysts get a chart that false-alarms at least twice as often as the textbook chart.
• Across the synthetic worlds, the 3-sigma rule flags 2% of clean runs and Western Electric 1 to 4 flags 18%, because each control has up to 11 measurements charted at once.
• Weak spot: all of this assumes independent, normal results with no autocorrelation. Real HPLC controls can violate all three.

What I'd change

• Add an EWMA chart for small sustained shifts, compared on the same simulated streams, instead of stacking more run rules on a Shewhart chart.
• Control the family-wise false-alarm rate across the measurements of one control (for example a Bonferroni-style widening, or a multivariate T² chart) and simulate it.
• Simulate the run rules with estimated limits too; the Phase I study currently covers the 3-sigma rule only.

Context

The 2021 team compared PCA, t-SNE and UMAP projections with K-Means and DBSCAN, scored by silhouette, and concluded modestly that without labels clusters can only help surface unusual runs. The revival turned that into "review prompts": DBSCAN noise, tiny clusters and points far from their K-Means centre. Two questions were open. How repeatable is a list of review prompts? And does DBSCAN point at the same runs as the control charts, so that the two could be combined into one "suspicious run" score?

Decision

• /odd-runs keeps the wording "prompts for a person to check, not verdicts" and links to the stability results.
• No clustering output is turned into a label, a disposition or an automatic action anywhere on the site, and the optional AI feature is not given clustering output to judge.
• DBSCAN outliers and run-rule signals stay separate views; the site does not merge them.
• A high silhouette is not presented as evidence that a pairing is good for finding odd runs.

Options considered

1. Clusters as labels (for example "normal" and "abnormal" groups). Rejected: no labels exist to validate them, and the 2021 team rejected it too.
2. A combined anomaly score from DBSCAN noise and run-rule signals. Rejected after the agreement study below.
3. Review prompts with stability shown (chosen).
4. Drop clustering from the site. Rejected: it was a third of the 2021 brief, and showing its limits honestly is useful.

Why

From /evaluation (the whole /odd-runs procedure re-run at the reference seed and 20 other algorithm seeds, and on 50 bootstrap resamples; 95% bootstrap intervals for the means):

• Algorithm seed alone changes the answer for some pairings. PCA + K-Means on the amino acid assay chose k = 15 at the reference seed and 4, 10, 11, 12 or 13 at the other 20 seeds. Agreement with the reference has mean ARI 0.59 (0.51 to 0.67), but the mean hides two outcomes: ARI 0.28 at the 5 seeds that chose k = 4 and about 0.70 at the rest.
• Resampling the injections moves the prompts even where the partition is stable. Over 50 bootstrap resamples, PCA + DBSCAN on Protein A has ARI 0.42 (0.31 to 0.55) because the chosen eps jumps between about 0.4 and 1.5; on cation exchange the partition holds (ARI 0.92, 0.89 to 0.95) but the overlap of flagged injections is 0.68 (0.65 to 0.71).
• The most repeatable pairings are the least useful for review: UMAP + K-Means and t-SNE + K-Means reproduce their partitions across seeds (mean ARI 0.96 to 1.00) and flag no injection at any seed, because they fold the odd samples into tidy clusters with a high silhouette.
• DBSCAN and the run rules do not agree: Cohen's kappa between the 3-sigma rule and DBSCAN, over 4,800 runs in 20 synthetic worlds, is −0.01 (−0.03 to 0.01). The rules find runs with planted control faults (69%, 67% to 71%); DBSCAN finds runs with planted degraded samples (82%, 77% to 87%). A combined score would average two detectors that look at different injections.

What happened

• The stability and agreement tables are on /evaluation and are re-checked by the test suite against the published artefact.
• The team's own pairing for the amino acid assay, t-SNE + DBSCAN, came out well: ARI 0.97 (0.94 to 1.00) across seeds with prompt overlap 0.92 (0.85 to 0.97). For cation exchange the same pairing keeps its partition (ARI 0.99) but flags nothing at the reference seed and two to four injections at 10 of the 20 other seeds, so its prompt overlap is 0.50 (0.30 to 0.70).
• A first version of this study used 7 other seeds. Its percentile intervals were too narrow for so few replicates (the amino acid PCA + K-Means figure read 0.52, 0.34 to 0.64), so the published study uses 20.
• DBSCAN also flags 8% (6% to 9%) of runs with no planted problem: a missing value filled with 0 looks as odd as a degraded sample.
• Limitation: data resampling was only run for the PCA pipelines (t-SNE and UMAP refits cost seconds each), so their stability under new data is not measured.

What I'd change

• Report a per-injection "flag frequency" across seeds and resamples (how often each injection is prompted) instead of one prompt list, so a reviewer sees which flags are robust.
• Replace the "where the curve settles" heuristic with a stability-based choice of k or eps (pick the setting whose partition is most repeatable), and compare.
• Run the bootstrap for t-SNE and UMAP in a Web Worker pool, offline, to close the gap above.

Context

Control charts are easy to draw and easy to misread: a first-time reader does not know what "8 on one side" means, or that a chart with 150 points will often show a false alarm. A short plain-English reading of the chart on screen helps. The project has no budget for AI and no server: the site is static on Vercel's free tier. Any AI use also had to be transparent about what is sent, where it goes and what a person decides, informed by the Australian Government's policy for the responsible use of AI in government, the transparency principles of the EU AI Act and the NIST AI Risk Management Framework. Being informed by them is not a claim of compliance with any of them.

Decision

• The feature is optional. Every page works without it, and nothing on the site depends on its output.
• Bring your own key: Anthropic (default, Claude Haiku 4.5, with Claude Sonnet 5.5 as an option) or OpenAI (model id editable). The key is kept in sessionStorage by default, in localStorage only if the visitor ticks "remember on this device", and can be forgotten with one button. It is sent only in the request header to the provider, never to this site, and never logged.
• Input: only the numeric summary of the chart on screen (what is charted, limits and how they were set, active rules and their signals, injected disturbances, run statistics, and the published false-alarm figures for the active rule set). No raw series, nothing about the visitor.
• Output: structured JSON validated with zod, shown under an "AI-generated" label with the model, latency and token usage. A grounding check lists numbers in the reply that are not in the summary.
• Human in the loop: the visitor accepts, edits or rejects each explanation. Every call, including failures, is written to an audit log in the visitor's IndexedDB (input without the key, output, provider, model, latency, token usage, decision), viewable and exportable as JSON or CSV at /ai-log.
• The system prompt forbids release or disposition language: a signal is a prompt to investigate, never proof of a cause.

Options considered

1. No AI. Simplest, but the reading of a chart stays locked behind SPC jargon.
2. A site-funded proxy with my key. Rejected: no budget, and a public endpoint would need abuse controls.
3. Server-side calls with the visitor's key. Rejected: the key would pass through this site's server, which is exactly what visitors should not have to trust.
4. Bring your own key, browser-direct (chosen).
5. A model running in the browser (WebGPU). Rejected for now: large downloads, uneven device support and weaker explanations.

Why

Browser-direct calls keep the key between the visitor and their provider. Sending only the numeric summary makes the input small (about 3,400 characters with the system prompt, roughly a thousand tokens), auditable and free of anything personal. Structured output plus the grounding check turns "the model should only use these numbers" from a hope into something visible on screen and in the log.

What happened

• The client, both provider adapters, error mapping, key storage, redaction, the audit log (memory and IndexedDB) and the grounding check are unit-tested with the network mocked; no test calls a real provider.
• The Anthropic request sends anthropic-dangerous-direct-browser-access: true, structured output through output_config.format, low effort only on Sonnet (Haiku 4.5 rejects the effort setting), and the server-side refusal fallback only on Sonnet 5.5.
• What I have not measured: the quality of the explanations. With bring-your-own-key there is no budget to run an evaluation set, so there is no accuracy number to report. The grounding check catches invented numbers but not a wrong reading built from correct numbers.
• The audit log lives in one browser. Clearing site data deletes it, and nothing is kept server-side by design.

What I'd change

• Build a small evaluation set (charts with known correct readings, scored by a rubric), run it once with a key, and publish the results with intervals on /evaluation, with the AI explanation compared against a template-based explanation generated from the same summary.
• Add a check that the explanation names the right rule for each listed signal, not only the right numbers.