The confidential API: sending a prompt to a cloud GPU the provider can't read

In the last post I covered the infrastructure side of confidential AI — attesting a CPU TEE and a GPU, then releasing secrets only to a verified environment. This post is the runtime side, and it answers the question that confidential AI platforms actually exist to solve:

How can my application send a prompt to an untrusted cloud GPU without the provider ever seeing it?

The answer is three things working together: attestation + secure key exchange + an encrypted channel — with one detail that makes or breaks the whole thing, which I’ll get to at the end.

Table of contents

HTTPS is not enough

Almost every LLM API today works like this:

  Client app ──HTTPS──►  Cloud server ──►  GPU
                              │
                         prompt in plaintext

HTTPS protects your prompt while it travels the Internet. But TLS terminates at the provider’s edge — a load balancer, a reverse proxy, the server process. From that point on, the prompt is plaintext on a machine you don’t control, and the provider can:

• read it in the request handler,
• log it (intentionally or by default),
• dump the process RAM,
• inspect it in GPU memory.

Encryption in transit solves the wrong half of the problem. The threat for confidential AI isn’t a network eavesdropper — it’s the endpoint itself. So the prompt has to stay encrypted past the network, all the way into a place the provider can’t look: an attested TEE.

The confidential API: encrypt for the TEE, not the server

The fix is to treat the cloud server as a hostile relay and encrypt the prompt for the TEE specifically — with a key the provider can never hold. Here’s the whole exchange.

Step 1 — verify attestation

Before sending anything, the client asks the environment to prove itself and verifies the evidence (this is the previous post in one step):

  Client ──request quote──►  Confidential VM
         ◄──CPU quote + GPU report + nonce──
   verify:  TDX/SEV-SNP ✓   GPU CC mode ✓   measurements ✓   expected image ✓

If any check fails, the client stops here and sends nothing.

Step 2 — a session key only the TEE can hold

Now both sides run an ECDH (Elliptic-Curve Diffie–Hellman) exchange and independently derive a shared secret, which becomes an AES-256 session key:

  Client  ◄──ECDH public keys──►  TEE   ⟹   shared secret  ⟹  AES-256 session key

A passive provider can’t derive this key — it never crosses the wire. But that alone isn’t enough, and here’s the subtle part that everything hinges on: the TEE generates its ECDH keypair inside the confidential VM and embeds its public key in the signed attestation quote (in the quote’s report-data field). So when the client verified the quote in Step 1, it also verified that this exact public key belongs to the attested TEE.

That binding is what stops a man-in-the-middle. Without it, the host could terminate your session with its own key, decrypt, peek, and re-encrypt to the TEE — a valid HTTPS cert says nothing about who’s behind it. With the key bound into the quote, the client knows it’s doing key exchange with the verified TEE and nothing else. (This is the RA-TLS pattern — remote-attestation-bound TLS.)

Steps 3–7 — encrypt, decrypt only inside, re-encrypt

The rest follows. Instead of posting plaintext:

POST /v1/chat   { "prompt": "our Q3 financial report ..." }

the client posts ciphertext:

POST /v1/chat   { "ciphertext": "A83D19...", "nonce": "..." }

and the flow inside the boundary is:

  ciphertext ──►  TEE: AES-decrypt (only the TEE has the key) ──►  prompt
                        │
                        ▼  encrypted PCIe
                  GPU CC: protected VRAM  ──►  inference
                        │
                        ▼
                  TEE: AES-encrypt the output  ──►  ciphertext  ──►  client decrypts

The plaintext prompt exists in exactly two places — the client, and the protected memory of the attested TEE — and nowhere in between. It’s decrypted only inside the TEE (which has the session key the host doesn’t), kept confidential across the bus and in VRAM by GPU Confidential Computing, and the response is re-encrypted before it ever leaves the trusted environment.

The complete flow

  (1) Client ──────────────►  request quote
  (2) TEE    ──────────────►  CPU quote + GPU report  (+ TEE's ECDH pubkey, signed)
  (3) Client ──────────────►  verify attestation                          ✓
  (4) ECDH ── derive ──────►  shared AES-256 key  (bound to the quote)     ✓
  (5) Client ── encrypt ───►  encrypted prompt
                              ▼
  (6) TEE  ── decrypt ─────►  prompt → GPU CC → inference → encrypt result
                              ▼
  (7) Client ── decrypt ───►  plaintext response

Where API keys fit — and the secret-manager pattern

Your own application secrets — keys for an upstream model API, a payments API, a cloud provider — fit the same rule from the trust-before-secrets post: release them only after attestation succeeds, into the TEE’s protected memory.

  verify attestation ✓  ──►  release API key  ──►  TEE memory

But there’s a stronger version. The most secure architecture is one where the client never transmits its long-term key at all. Instead, a secret manager (Vault / KMS) holds the key and releases it only to a TEE whose attestation it has verified:

  Client ──verify TEE──►  Secret Manager (Vault / KMS)
                                │  release only on valid attestation
                                ▼
                          attested TEE

Now the long-lived credential lives in the vault, the client only ever proves “this TEE is genuine,” and the key is handed straight to attested, protected memory — never to the client’s request path, never to the provider’s plaintext.

The detail that makes it real

It’s tempting to read all this as “use TLS to a confidential server.” It isn’t. A conventional HTTPS endpoint proves you’re talking to a server with a valid certificate — it says nothing about whether that server is a genuine, untampered TEE, or whether the prompt survives past the TLS terminator in plaintext.

The thing that makes a confidential API fundamentally different is binding the encrypted session to the attestation: the key you encrypt for is cryptographically tied to the specific TEE you just verified. That single binding turns “encrypt for some server” into “encrypt for this proven, measured, CC-enabled environment, and no one else.”

Put end to end, that’s real confidentiality: the prompt is encrypted in transit, decrypted only inside an attested TEE, processed on a GPU with Confidential Computing on, and re-encrypted before the response leaves. Most of this blog is about making that inference fast; this is the half that makes it private on a machine you rent and will never trust — which, as the roadmap keeps insisting, is increasingly where serious AI inference has to run.