8
AI
Hosted LLM API vs self-hosted open-weight model for regulated health data
Model quality ↔ Data control
Context
Summarise clinical notes for 40 hospitals without breaking data-residency rules.
We summarise clinical notes for about 40 hospitals. Everything is PHI and contractually has to stay in the EU.
Two camps on the team:
- Hosted frontier model through a cloud provider's EU region with zero data retention, a BAA/DPA and private networking.
- Self-host an open-weight model (70B class, quantised) on our own GPU nodes in the same VPC as the data.
Quality on our eval set: the hosted model scores 91%, the best open-weight model 84% after prompt tuning. The gap matters, but so does being able to say the data never leaves our infrastructure.
How have others weighed this, especially with auditors in the room?
Constraints
- Data
- PHI, EU residency
- Volume
- 2M notes/month
- GPU budget
- $15K/month
- Audit
- ISO 27001 + DPIA
Hosted API (zero retention)vsSelf-hosted open-weight model
Community verdict
Hosted API (zero retention) 60%Self-hosted open-weight model 40%
10 engineers · 6 opinions
With these constraints, what would you choose?
One choice per engineer. You can change it any time.
Hosted API (zero retention)
Self-hosted open-weight model
Data control →
Trade-offs
Dimensions
Hosted API (zero retention)Self-hosted open-weight modelout of 5
- Model quality
- 5Hosted API (zero retention) scores 5 of 53Self-hosted open-weight model scores 3 of 5
- Data control
- 3Hosted API (zero retention) scores 3 of 55Self-hosted open-weight model scores 5 of 5
- Ops burden
Community discussion
6 comments
Have you made this decision in production? Share your reasoning.
Sign in to commentOur auditors cared about four things: where data is processed, whether it is retained, who can access it, and whether it trains the model. A hosted EU endpoint with zero retention, private connectivity and contractual no-training answered all four.
"Self-hosted" is not automatically safer. You now own patching, access control and logging of a GPU fleet.
Agree. We wrote the DPIA for both options and the hosted one was shorter, because the provider had already done half the work.
Procurement is the hidden constraint. Several hospital groups we sell to simply will not sign if a third-party AI provider appears in the sub-processor list. That made the decision for us, regardless of quality.
Seven points of quality is a lot for summarisation, but check where the gap is. Ours was almost entirely medication dosage formatting, which a constrained output schema plus a validation pass fixed. After that the gap was two points.
And at 2M notes/month, a reserved GPU fleet came out cheaper than per-token pricing.
Did that cost comparison include the on-call engineer for the GPU fleet? That is usually where self-hosting maths breaks.
Half an engineer, amortised. Still cheaper at our volume, but not at a quarter of it.