For most of the past two years, the practical choice for European IT teams deploying language models was simple: call an API. OpenAI, Anthropic, Google Gemini, and their peers offered capabilities that no self-hosted alternative could match, and the compliance exposure felt manageable. That calculus is shifting. Open-weights models from Mistral, Meta, and others have closed the capability gap enough that the question is no longer whether self-hosted inference is viable, but whether proprietary APIs are still worth the trade-offs.
The trade-off is not just technical. Every inference request that carries personal data to a non-EU API endpoint is a cross-border data transfer under GDPR. Every model integrated into a product serving EU users sits inside the scope of the EU AI Act, which has been in force since August 2024 and is rolling out obligations in phases through 2027. Understanding exactly what each model type demands of you, operationally and legally, is now a prerequisite for any serious AI deployment strategy.
What open-weights actually means
The term open source is used loosely in AI, and the looseness matters. A model can publish weights while restricting commercial use, prohibiting fine-tuning, or capping deployment scale. For the EU AI Act’s purposes, the exemptions under Article 53(2) apply only to models released under a license that explicitly permits free access, use, modification, and distribution without payment or other restrictive conditions. If the license charges for commercial use above a user threshold, the exemption does not apply.
Models that do qualify for the exemption are released from certain provider obligations: specifically, the obligation to draw up and maintain the full technical documentation covering training and testing processes. But two obligations survive regardless of license: copyright compliance and the publication of a training data summary using the AI Office template. These are not waived by open-source status.
There is also a hard ceiling on the exemption. Any model trained with more than 10²⁵ FLOPs is classified as a systemic-risk GPAI model under Article 51. For those models, the Article 53(2) open-source exemption disappears entirely. The provider faces the full Article 53 obligations plus the additional Chapter V requirements: adversarial testing, incident reporting to the EU AI Office, cybersecurity measures, and annual energy reporting. The threshold was calibrated around GPT-4-class compute at the time of drafting; frontier open-weights models are approaching it.
The deployer obligation that the license cannot touch
Here is the point most teams miss. The AI Act’s open-source exemptions cover the model provider’s obligations, not yours. When you download Llama or Mistral and integrate it into a product that serves EU users, you are a deployer and potentially a provider under the Act. The model’s license is irrelevant to your regulatory status.
Deploying an open-source model in a high-risk context makes you the provider of a high-risk AI system. All eleven high-risk obligations apply in full. The upstream license changes nothing.
High-risk contexts include HR screening, credit scoring, medical diagnostics, and law enforcement support. If your use case touches any of those, the compliance burden is identical whether you use GPT-4 via API or Mistral running on your own hardware. The architecture choice affects data residency and operational control, not the regulatory classification of the system you are building.
GDPR and the inference transfer question
GDPR does not prescribe where data must be stored, but it restricts transfers of personal data to countries without adequate protection. An inference request that includes personal data, a customer name, a medical record fragment, a support ticket, is a transfer if it is processed outside the EU. Even metadata and telemetry can constitute personal data under GDPR’s broad definition.
Running inference on-premises eliminates the transfer question entirely. When the model runs inside your network, client data never crosses a border. GDPR Article 46 simply does not apply because there is no transfer to govern. This is not a compliance workaround; it is the architectural outcome the regulation was designed to incentivise.
The structural risk with proprietary APIs goes further than data residency. GDPR Article 48 prohibits handing personal data to non-EU authorities without an international agreement. The EDPB finalised its guidelines on Article 48 in June 2025, clarifying the conditions under which organisations can lawfully respond to third-country authority requests. Any inference pipeline routed through a US-headquartered provider sits inside this structural tension, regardless of the contractual safeguards in place.
Where proprietary APIs still win
Intellectual honesty requires acknowledging where the proprietary model argument holds. Frontier capability at the absolute edge, multimodal reasoning, long-context tasks requiring hundreds of thousands of tokens, and tasks that demand the most recent training data, still favour the largest proprietary models. The operational burden of running GPU inference at scale is real: hardware procurement, driver management, model serving infrastructure, and capacity planning are not trivial.
For use cases where the data is genuinely non-personal, the volume is low, and the capability requirement sits at the frontier, a proprietary API with appropriate DPA terms may be the pragmatic choice. The EU-US Data Privacy Framework provides a legal mechanism for transfers to certified US providers, though its long-term stability remains a live question for any team planning infrastructure beyond a two-year horizon.
The on-premises inference architecture
For teams that have decided self-hosted inference is the right call, the architecture is more approachable than it was eighteen months ago. Modern serving frameworks handle model loading, batching, quantisation, and request routing. The remaining questions are hardware sizing, model selection, and where inference sits relative to the rest of your data pipeline.
GPU inference and training are a core part of what we operate at DIVERSITY, running on dedicated hardware across our EU datacenters. Teams that do not want to own the GPU layer but need inference to stay inside European jurisdiction can run workloads on our infrastructure, keeping the data residency guarantee without taking on bare-metal operations themselves.
Model selection is now genuinely competitive. For most enterprise inference tasks, a well-quantised 70-billion-parameter open-weights model running on four or eight A100s will match proprietary API quality for structured extraction, classification, summarisation, and retrieval-augmented generation. The gap narrows further when the task is domain-specific and the model can be fine-tuned on internal data, something proprietary APIs make difficult or impossible.
Making the build-vs-buy call
The decision framework is not complicated, but it requires honest answers to three questions. First, does your inference workload process personal data? If yes, on-premises or EU-sovereign infrastructure is the lowest-risk path. Second, does your deployment context qualify as high-risk under the AI Act? If yes, the compliance obligations are the same regardless of model type, so the decision reduces to capability and operational fit. Third, do you need frontier-edge capability that open-weights models cannot yet match? If yes, a proprietary API may be justified, with appropriate contractual and architectural controls.
For the majority of enterprise inference workloads, open-weights models running on EU infrastructure now satisfy all three criteria better than a proprietary API does. The capability gap has closed enough that the sovereignty trade-off is no longer a reasonable price to pay for convenience. The teams that recognise this earliest will be the ones with the most operational flexibility when the AI Act’s enforcement phase begins in August 2026.
