Trust before secrets: confidential AI inference with CPU TEEs and GPU Confidential Computing

Here is one of the most important ideas in confidential computing, in one sentence:

Never give your API key to a machine until the machine has proven it is trustworthy.

Most of this blog is about making inference fast on GPUs I rent. This post is about the other half — making it private on hardware I don’t own. Because when you rent a GPU, you are handing your model, your prompts, and your credentials to someone else’s computer, and by default that someone can read all of it. Confidential computing is the hardware mechanism that lets you flip the order: prove the machine is a real, untampered, encrypted environment first — then, and only then, release the secrets.

Table of contents

The threat: on a rented GPU, the provider sees everything

Rent a GPU server the normal way and the trust model is upside down. You send your secrets up front to a VM you don’t control:

  You ──API key / credentials──►  Cloud VM ──►  GPU

The provider owns the host OS, the hypervisor, and the storage. So from outside your container — as root on the host — they can read essentially anything:

docker inspect              # your env, mounts, config
/proc/<pid>/environ         # every environment variable (your API keys)
/proc/<pid>/mem             # the live memory of your process
overlay2/ , disk image      # your files at rest
swap                        # whatever spilled out of RAM

Running your container as root doesn’t help — the threat is below you, in the layers the provider controls. Your API keys, your model weights, your prompts: all readable. That’s the problem confidential computing exists to solve.

TEE: hardware-encrypted VM memory

A Trusted Execution Environment (TEE) moves the encryption boundary into the CPU. You boot a Confidential VM (CVM) on Intel TDX or AMD SEV-SNP, and the CPU encrypts that VM’s memory in hardware with a key the hypervisor never sees:

                 CPU
        ┌──────────────────┐
        │  Intel TDX /     │
        │  AMD SEV-SNP     │   ← memory encrypted by hardware
        └────────┬─────────┘
                 ▼
          Confidential VM       ← hypervisor cannot read this memory

Now the host and the hypervisor are outside the trust boundary. They can schedule the VM, but they cannot read its RAM. That closes the /proc/<pid>/mem and swap-inspection attacks.

But it raises a new question, and it’s the whole game: how do you know the provider actually booted a real CVM, instead of a normal VM that just says it’s confidential? You can’t trust the provider’s word — they’re the party you’re defending against. You need the hardware to prove it. That’s attestation.

Attestation: making the machine prove itself

Attestation is a cryptographic handshake where the CPU itself vouches for what’s running.

1 — Boot is measured. As the CVM boots, the CPU hashes each stage into tamper-proof measurement registers — the firmware, the kernel, the initrd, the root filesystem, the boot configuration. Those hashes become the VM’s identity. Change any byte of the image and the measurement changes.

2 — The VM signs a quote. You send a fresh random nonce and ask it to prove itself. The CVM returns a quote:

quote = {
  measurements,        # firmware / kernel / image hashes
  nonce,               # your freshness challenge
  signature            # signed by a key fused into the CPU
}

The signing key is burned into the silicon by the CPU vendor. The provider cannot forge this signature — they don’t have the key, and no software they run can extract it.

3 — You verify. You check three things against the vendor’s attestation service (Intel’s PCCS / Trust Authority, AMD’s KDS):

   You ──quote──►  Verify:
                     1. signature chains to a genuine Intel/AMD CPU
                     2. measurements == your expected "golden" values
                     3. nonce matches  (it's fresh, not a replay)
                   ✓ valid

Pass all three and you now know, on hardware evidence rather than trust: it’s a genuine TDX/SEV-SNP TEE, running the exact kernel and image you expect, unmodified, right now.

GPU attestation: extending trust to the accelerator

The CPU is trusted — but AI runs on the GPU, and a CPU TEE says nothing about it. So the GPU has to attest too. NVIDIA Confidential Computing (on Hopper and Blackwell — the RTX PRO 6000 / B200 class) gives the GPU its own signed attestation report:

  CVM  ──challenge──►  GPU (CC mode)
       ◄──report────   { genuine GPU, CC mode ON,
                         firmware authentic, untampered }

You verify that report through NVIDIA’s remote attestation service the same way you verified the CPU quote — it chains to NVIDIA’s device certificates. It proves the GPU is genuine silicon, that Confidential Computing mode is actually enabled, and that the firmware hasn’t been swapped.

In CC mode the PCIe link between the CPU TEE and the GPU is encrypted (the CPU encrypts data into bounce buffers; the GPU decrypts into a protected region of VRAM), so the model and data are never in plaintext on the bus.

Why GPU CC is necessary, not just the CPU TEE

This is the part that matters most for AI, and it’s easy to miss. Suppose you have a perfect CPU TEE and encrypted storage, but an ordinary GPU:

  encrypted disk → TDX memory → load model → ──PCIe──► GPU VRAM (plaintext!)

The moment the weights cross PCIe into VRAM, they leave the CPU’s encrypted boundary. Everything I spend my time on in inference lives there in the clear — model weights, the user’s prompt, the activations, the KV cache — and anyone with sufficient control of the GPU environment could read it straight out of VRAM. The CPU TEE protected the front door and left the vault open.

GPU CC extends the encrypted boundary all the way to VRAM:

  encrypted disk → TDX memory → encrypted PCIe → GPU CC (protected VRAM)

Now the weights, prompts, activations, and KV cache stay confidential end to end — through the CPU, across the bus, and on the accelerator where the actual compute happens.

Storage: sealing secrets at rest

Memory is protected at runtime, but what about the disk? You never want plaintext secrets sitting in storage, because storage is one of the layers the provider owns. Two mechanisms:

• Sealing. Instead of secrets.dat, you store secrets.dat.enc, encrypted under a sealing key bound to the attested environment (the SGX/TEE sealing key, derived from the CPU and the measured identity). Only the same verified TEE can derive that key and decrypt. Steal the disk and you get ciphertext.

  disk: secrets.enc ──►  TDX VM ──(sealing key)──►  secrets   (only inside the attested TEE)

• dm-verity gives the root filesystem a read-only, hash-tree-verified integrity guarantee: the root hash is part of the measured boot, so a tampered image either fails to mount or changes the attestation measurement (and your verify step rejects it). Pair it with full-disk encryption (LUKS) for confidentiality of anything written.

Injecting secrets into a running CVM — “trust before secrets”

Now the payoff, and the step everything else was building toward. You release the private info — API keys, SSH keys, database credentials, signing keys — only after both attestations have passed:

   Verify CPU quote   ✓
   Verify GPU + CC     ✓
        │
        ▼  (only now)
   inject API keys / SSH keys / credentials  ──►  running CVM (protected memory)

The secrets are delivered over a channel bound to the attestation (e.g. a TLS session keyed to the verified quote, or a secrets manager that releases only on a valid, fresh attestation) straight into the CVM’s encrypted memory. They exist only in protected RAM; if they must persist, you seal them to encrypted storage; and you can wipe them from memory after use to shrink the exposure window.

Contrast that with the common anti-pattern, which quietly defeats the whole stack:

   host: .env / credentials  ──bind-mount (plaintext)──►  container

If the secrets are bind-mounted from the host in plaintext, the host — the party you’re defending against — sees them before the container ever starts. Having TDX and sealing and dm-verity available doesn’t help if the secrets are still handed over before attestation. The mechanism only matters if you actually gate the release on it.

The complete pipeline

Put it together and a confidential AI inference stack is a single ordered chain:

   Encrypted storage
          │
          ▼
   Attested CPU TEE  (Intel TDX / AMD SEV-SNP)
          │
     release secrets  (API keys, credentials)
          │
          ▼
   GPU Confidential Computing  (encrypted PCIe + protected VRAM)
          │
          ▼
   Confidential AI inference  (weights / prompts / KV cache stay private)

The principle, restated as four steps:

1. Attest the CPU TEE — prove it’s genuine TDX/SEV-SNP running the exact image you expect.
2. Attest the GPU — prove it’s a genuine accelerator with CC mode actually on.
3. Release secrets only after both succeed — never before.
4. Keep secrets in protected memory, and seal them to encrypted storage only if they must persist.

Why I care

I spend most of my time making inference fast — kernels, KV cache, rooflines, multi-GPU serving on Hopper and Blackwell. Confidential computing is the complement: making that same inference private on hardware I rent and don’t control. At the edge I get privacy for free — on a Jetson in your house, nothing leaves the building. In the cloud you can’t rely on physics, so you rely on the CPU and the GPU proving themselves before they ever see a secret.

The honest caveat: a TEE and attestation don’t automatically secure your application — you can still leak a key with a sloppy app on top of perfect hardware. What they give you is the foundation: a way to make “release the secret” conditional on “the environment is verified.” Get that ordering right — trust before secrets — and renting a GPU stops meaning trusting whoever owns it. That’s the difference between a normal GPU rental and a confidential one, and it’s increasingly where serious AI inference is heading.