Confidential Computing for LLM Inference: How TEEs and Encryption-in-Use Protect AI Models and Data

When you ask an LLM a question about your medical records, financial history, or proprietary business data, where does that information go? If you’re using a standard cloud-based AI service, the answer is: it’s decrypted, processed in plain text, and potentially exposed to anyone with access to the server-even if that’s just a system administrator or a compromised script. This isn’t theoretical. It’s happening right now in hundreds of enterprises trying to use powerful AI models without realizing they’re handing over their most sensitive data in plain sight.

Why LLM Inference Is a Security Nightmare

Large Language Models aren’t just big software. They’re expensive, proprietary assets. Training a 70B-parameter model can cost over $50 million. On the other side, users are feeding in personally identifiable information, trade secrets, legal documents, and health records. The traditional model-send data to the cloud, let the model run on untrusted hardware, get the answer back-is broken for anything that matters.

This is the AI privacy paradox: you need powerful AI to make sense of sensitive data, but using that AI means exposing the data. For years, encryption was the go-to answer-but encryption only protects data at rest or in transit. Once the data lands on the server, it has to be decrypted to be processed. That’s the gap. Confidential computing closes it.

What Confidential Computing Actually Does

Confidential computing isn’t magic. It’s hardware. Specifically, it’s Trusted Execution Environments (TEEs)-secure, isolated areas inside a processor where code and data are protected even from the operating system, hypervisor, or cloud provider. Think of it like a vault inside the CPU. Even if someone owns the server, they can’t peek inside the vault.

In LLM inference, this means:

• Your prompt is encrypted before it leaves your device
• It travels through the cloud network still encrypted
• Only inside the TEE does it get decrypted
• The LLM runs its calculations on that data-still encrypted in memory
• The response is re-encrypted before it leaves the secure environment

The model weights-the actual brain of the AI-are never exposed outside the TEE. Neither are your inputs. This is called encryption-in-use. It’s the first time in computing history that data can be processed without ever being visible in plain text.

How TEEs Work in Practice

There are three main hardware platforms powering this today:

• Intel TDX (Trust Domain Extensions): Used in modern Xeon processors. Provides memory encryption and remote attestation.
• AMD SEV-SNP (Secure Encrypted Virtualization - Secure Nested Paging): AMD’s answer to Intel TDX. Stronger against certain memory attacks.
• NVIDIA GPU TEEs: Built into H100, H200, and now Blackwell B200 GPUs. This is the game-changer.

NVIDIA’s approach is critical because LLM inference is GPU-heavy. Traditional CPU-based TEEs like Intel SGX slowed down LLMs by 15-25%. NVIDIA’s solution? Run the entire inference inside the GPU’s encrypted memory. Performance loss? Just 1-5%. That’s near-native speed with enterprise-grade security.

The process isn’t simple. Before the model even loads, the system checks:

1. Is this a real, unmodified TEE? (via cryptographic attestation)
2. Does the cloud provider have permission to load this model? (mutual attestation)
3. Is the model’s decryption key securely delivered only to this specific enclave?

If any step fails, the model stays encrypted. No access. No execution. This is how companies like Leidos and major insurers are now running AI on patient records and claims data without violating HIPAA or GDPR.

Cloud Providers Are Betting Big

Every major cloud provider has a confidential computing offering-but they’re not equal.

Comparison of Confidential Computing Platforms for LLM Inference

Platform	Hardware Support	Performance Overhead	Model Loading	Kubernetes Native
NVIDIA H100/H200/B200	GPU TEEs	1-5%	Best (mutual attestation)	No
Azure Confidential Computing	Intel TDX, AMD SEV-SNP	5-10%	Good (application-level encryption)	Yes
AWS Nitro Enclaves	Intel TDX	10-15%	Poor (no GPU TEE)	Partial
Red Hat OpenShift + CVMs	Intel TDX, AMD SEV-SNP	5-12%	Good (with Tinfoil Security)	Yes

NVIDIA wins on speed. Azure wins on integration. AWS is lagging because Nitro Enclaves can’t use GPUs securely yet. Red Hat’s OpenShift integration is the most developer-friendly for enterprises already using Kubernetes.

Real-World Impact: Who’s Using This?

In healthcare, a European insurer replaced its manual claims review system with an AI assistant running in Azure Confidential VMs. Patient data stayed encrypted end-to-end. Accuracy stayed at 92%. Breach risk dropped to near zero.

A Fortune 500 bank moved its credit risk model from on-premises servers to AWS Nitro Enclaves. The model weights were never exposed to the cloud provider. The result? Compliance with GDPR and Basel III requirements without sacrificing model performance.

But it’s not all smooth sailing. One financial firm tried loading a 30B-parameter model into an Intel SGX enclave. It took 47 minutes. Their real-time system couldn’t wait. They had to redesign their pipeline to use smaller models or pre-process inputs into encrypted embeddings.

What You Need to Get Started

If you’re considering confidential computing for LLM inference, here’s what you actually need:

• Hardware: Servers with Intel TDX, AMD SEV-SNP, or NVIDIA H100+ GPUs. No point in trying this on older hardware.
• Cloud or on-prem: Most companies start in the cloud. But if you need data residency (like banks or governments), you’ll need on-prem TEE-capable servers.
• Secure model loading: This is the hardest part. You need mutual attestation-where the enclave proves it’s real, and your model proves it’s authorized to run there.
• Integration: Your MLOps pipeline must now handle encrypted prompts, attestation checks, and encrypted outputs. Tools like Red Hat’s sandboxed containers or Azure’s Attested Oblivious HTTP help.

The learning curve is steep. Security engineers need to understand hardware attestation, Kubernetes extensions, and GPU memory management. Most teams take 8-12 weeks to get their first confidential LLM running.

The Bigger Picture: Why This Is the Future

The market for confidential AI computing is exploding. It was worth $1.7 billion in Q3 2025. By 2027, it’ll be over $8 billion. Why? Because regulations won’t wait.

GDPR, HIPAA, CCPA, and new EU AI Act rules require that sensitive data be protected during processing-not just stored or transferred. Confidential computing is the only technology that meets that requirement.

By 2027, Gartner predicts 90% of healthcare and financial services firms will use it. Right now, only 15% do. That gap is closing fast.

And it’s not just about compliance. It’s about trust. Customers won’t give their data to AI systems they can’t verify are secure. Investors won’t fund AI startups that can’t prove their models aren’t leaking IP. Confidential computing turns security from a cost into a competitive advantage.

What’s Next?

NVIDIA’s Blackwell B200 GPUs, launched in December 2025, now support 200B+ parameter models with under 3% performance loss. Microsoft just added GPU TEE support to Azure Confidential VMs. The Confidential Computing Consortium is building standard APIs for secure LLM serving.

The next step? Making this easy. Right now, you need a team of security engineers to set it up. In two years, it’ll be a toggle in your cloud console.

Until then, if you’re handling sensitive data with LLMs, you’re playing Russian roulette with your privacy and your IP. Confidential computing isn’t optional anymore. It’s the baseline.