Per-label thresholds + a RoBERTa soft-vote lifted recall to 0.92 (~1.7× production), and a GDPR coverage pass pulled its weakest label off the floor (0.64 → 0.95). The honest cost: ~4% clean false positives. Then the same classifier met documents — where longer text broke the thresholds — and a length-aware fix brought the file path to 4% FP at 100% recall. Both channels are now live.
Detection happens in two layers. Explicit identifiers — SSNs, payment cards (Luhn), and the healthcare-specific NPI (a checksum-validated National Provider Identifier that generic DLP misses) — are caught by a deterministic checksum layer: exact, instant, ~100% precision by construction, no model involved. A number that fails its checksum is dropped, not flagged. Everything below measures the harder job — the semantic layer that judges paraphrased regulated content carrying no identifier at all ("the individual we discussed has been on that medication for weeks"). That's where thresholds, hypotheses, and the soft-vote earn their keep, and where every F1 in this report lives.
Production applies one global 0.95 threshold to every label. Giving each label its own threshold is the biggest single lever (0.570 → 0.802); averaging two NLI models adds the rest (→ 0.836). All on the sharpened-hypothesis config.
Numbers shown are after both hypothesis rewrites (below). In-sample and tuned on the same set — an optimistic ceiling, not a shippable figure; the held-out FP test is the honest check.
Each label tuned to its own optimum. After the rewrites + a GDPR coverage pass, HIPAA reaches 0.92 and GDPR climbs from 0.41 to 0.73 — both at high, safe thresholds. PII is the next candidate for the same treatment.
A first diagnostic pass on 30 clean prompts flagged 37–63% of harmless traffic at the recommended thresholds. Rewriting the GDPR hypothesis, then the HIPAA one, walked the rate down step by step — with F1 rising, not falling, because the sharper wording lifts true positives and drops clean text at once. The 0% here is on 30 prompts; the 199-prompt validation below revises it to the honest ~3%.
Every first-pass false positive was GDPR ("personal data of an EU resident" entailed on any sentence); the small residual was HIPAA ("meditation" / "machine learning" reading as medical). Rewriting each hypothesis to demand a concrete context — an EU person's data request, a specific patient's care — drops both to zero. PII and Sensitive never misfired.
A fresh 199-prompt corpus (74 clean + 125 regulated), disjoint from tuning; thresholds tuned on train, scored on test, over 5 seeded splits. Recall = of the regulated content, how much was caught. Precision = of what was flagged, how much was truly regulated. F1 is their harmonic mean — the single number earlier reports quoted, which sits between the two.
Soft-vote = the deployed fixed thresholds on the full 199-prompt corpus (the operating point). A held-out train/test run with F1-tuned thresholds brackets it more conservatively at recall 0.84 — the deployed config deliberately favors recall.
GDPR was the floor at 0.64 — the jurisdiction + rights-request hypotheses missed sub-types (right-to-be-forgotten, breach notice, lawful basis, DPIA, objection). Two coverage hypotheses + a threshold drop (0.92 → 0.80, safe: clean FP stays 0% to 0.75) lift it to 0.95. The cost: GDPR now cross-labels onto clinical content (file GDPR precision 0.44 → 0.36) — cosmetic, since HIPAA outranks it and drives routing.
The honest read: the deployed soft-vote catches 92% of regulated content (recall 0.915) against production's 53% at 76% precision and ~4% clean false positives. F1 (0.811) sits between precision and recall, so quoting it as recall understates the catch rate. Every label now clears 0.82+ recall — GDPR, once the floor at 0.64, was lifted to 0.95. A false positive only means "redacted / sent on-prem," never blocked — so for a governance layer where a miss is the costly error, trading a little precision for that recall is usually right.
A file is just another content channel: uploads run the identical 4-stage pipeline through /v1/files/govern. But the per-label thresholds were tuned on one-sentence prompts — and a document is not a sentence.
A standing monitor probes known-clean prompts and known-clean documents every 15 minutes. On the same live config, short prompts stayed clean while 3 of 5 clean documents were wrongly redacted — longer text gives the zero-shot NLI far more surface to weakly entail a low-threshold label.
The three misfires split into two different causes — the monitor labels each by which layer fired:
nli — low thresholds (HIPAA 0.49, PII 0.43) over-firing on long text. Presidio — the NER tagging "Banana Bread" as a PERSON, and on_any_entity turning that into PII: a pre-existing bug, unrelated to the soft-vote.
A low label's threshold ramps up toward a precise 0.90 document floor as word count grows 18 → 45. Short prompts (≤18 words) keep their base thresholds, so the chat recall win is untouched; a 37-word memo now needs HIPAA ≥ 0.75, not 0.49. The deterministic stub still backstops keyword/entity content in long docs, so little real recall is lost.
on_any_entity no longer fires PII on a bare PERSON match — it needs a corroborating identifier (email, phone, SSN, address, DOB, MRN…). A recipe title or a byline stops being "PII"; a name with contact details still is.
Measured on 24 clean + 14 regulated documents through the live post-fix config, alongside the 199-prompt held-out chat result. Both channels now run the soft-vote in production.
Files hit perfect recall (1.00) — every regulated document caught — at 0.73 precision. Read that precision carefully: it's mostly GDPR and PII getting over-applied to documents that are already regulated (per-label precision GDPR 0.36, PII 0.56), not clean docs being flagged — the clean-doc FP-rate is just 4%. Those extra labels don't change routing (the doc goes on-prem regardless), but they do stamp labels that aren't strictly true. Regulated documents — clinical notes, EU erasure requests, PII forms, confidential memos — were caught every time: real sensitive documents almost always carry the keywords/entities the stub catches too. The one clean-doc FP (a 35-word "coffee chat" note → PII) is the same low-PII-threshold eagerness seen on chat.
The deployed soft-vote reaches recall 0.915 (production 0.53) at precision 0.757, F1 0.811 — it catches ~72% more regulated content. It held up under held-out train/test validation (which brackets it at 0.84, F1-tuned), so the tuning wasn't overfit. Hypothesis engineering was the lever throughout: sharpening HIPAA/GDPR moved true positives clear of clean text, and a GDPR coverage pass lifted its recall 0.64 → 0.95. On files, recall is a clean 1.00.
The gain isn't free: ~4% of clean chat and ~4% of clean documents get redacted or re-routed when they shouldn't (never blocked), and lowering the GDPR threshold added cross-labeling on clinical content. Both false-positive rates were surfaced by real measurement, not hope — the diagnostic "0%" that first looked too good was a small-sample mirage the bigger sets corrected. A standing FP monitor now watches both channels, so drift shows up as data rather than a surprise.
The soft-vote + per-label + length-aware thresholds + lone-PERSON guard are deployed on the gateway for chat and files. Remaining work: