Foundational Research: Trustworthy Scientific AI
The privacy fix that quietly breaks fairness
Privacy and fairness are audited on separate benches, by separate teams, with separate sign-offs. But they interact. The standard privacy defence, differential privacy, degrades the accuracy of the smallest subgroups the fastest. So a clinical model can pass a privacy audit and a fairness audit run independently, while the very act of making it private is what broke equity for the patients with the least data. Here is that trade, measured on one model and one report card.
Disparate impact of DP
2.6×
minority accuracy loss vs majority
Privacy actually bought
~0
membership-inference advantage
On one model
both planes
real public clinical data
The Challenge
Machine learning on sensitive data needs guarantees against leakage, and clinical AI needs to be equitable across the groups it serves. These are usually treated as two separate assurance problems. But differential privacy, the rigorous answer to the first, is known to concentrate its accuracy cost on under-represented subgroups (the disparate impact of differential privacy, Bagdasaryan et al., 2019). For a mental-health or readmission model, the members most re-identifiable from a training set are often the smallest, highest-stigma clinical subgroups, exactly the ones the privacy fix hurts most. Assuring privacy and fairness apart is structurally blind to this.
One report card
On the real, public UCI Diabetes readmission dataset (predict 30-day readmission; race as the protected attribute), we freeze one logistic model and run both assurance planes on it as we tighten differential privacy: a loss-based membership-inference attack per subgroup, and per-subgroup accuracy. Because differential privacy is randomised, every number is averaged over 40 noise draws.
| Privacy (epsilon) | Membership-inference advantage | Accuracy drop, majority | Accuracy drop, minority |
|---|---|---|---|
| inf (non-private) | 0.004 | +0.000 | +0.000 |
| 4.0 | 0.004 | +0.002 | +0.004 |
| 1.0 | 0.006 | +0.005 | +0.011 |
| 0.5 | 0.006 | +0.009 | +0.028 |
| 0.25 | 0.013 | +0.025 | +0.065 |
At the tightest privacy (epsilon = 0.25) the accuracy cost is not shared evenly. Ordered by subgroup size, the drop is 2.5 points for Caucasians (n=6,150), 4.4 for African Americans (n=1,520), and 8.5 for Hispanics (n=153): a 2.6x disparate impact on the minority groups, growing monotonically as privacy tightens. The smaller the group, the more the privacy fix costs it.
Why one card, not two audits
The membership-inference advantage the differential privacy is meant to reduce is near zero on this regularised model, so the privacy it actually buys here is marginal. Read the situation three ways. A privacy-only audit signs off: "differential privacy applied at epsilon 0.25", a win. A fairness-only audit at a fixed epsilon sees a subgroup gap, but not that the privacy knob caused it. Only the coupled report card shows the trade for what it is: almost all equity cost, for almost no privacy gain, borne by the patients with the least data. The contribution is not a new attack or a new fairness metric, it is putting both planes on one model, across a privacy sweep, so the interaction cannot hide.
"You cannot sign off privacy and fairness on separate forms. The tool that makes a clinical model private is the same tool that widens its accuracy gap for the smallest groups. Put both on one report card, or you will approve a trade you never saw."
Fabio Rovai, Tesseract Academy
An independent, self-initiated study on open data and open tools. The model is a regularised linear classifier, which by nature leaks little to membership inference; higher-capacity or overfit models leak substantially more, and the same card applies to them with a larger privacy side to the trade. Differential privacy is applied by output perturbation, a rigorous closed-form mechanism and the convex analogue of DP-SGD. The dataset is real but specific (US hospital readmission, race); MIMIC-IV, credentialed under a data-use agreement, is the clinical scale-up, including a mental-health arm. Numbers are reported as measured, not inflated, and the disparate-impact trend is monotone across the whole privacy sweep.
Run the report card
Both planes, the full DP sweep, on real data, in Colab or from the repo.
