Open-weight models allow clients to run LLMs locally, thus
keeping their data private from untrusted providers. However,
running large models requires massive hardware and storage
resources (especially challenging on resource-constrained de-
vices like smartphones), limiting local execution to smaller
models. This leaves clients with a frustrating compromise:
settle for a less-capable model that can be run locally, or sac-
rifice privacy by sending queries to an external server.

We present an efficient protocol that allows a client to privately
and verifiably outsource LLM inference of an open-weight
model to a pair of malicious (but non-colluding) servers. Pri-
vacy implies that neither server learns anything about the
client’s queries. At the same time, the client can verify the
claimed result using information posted by the model owner
along with the model weights.

Compared to prior state-of-the-art for private LLM inference
(SIGMA, PETS’ 24)—which does not provide verifiability—
our protocol is ≈11–14× faster while imposing no overhead at
the servers (beyond the cost of inference in the original model).
Our protocol also scales to larger models not supported by
prior work: for example, with our protocol a client can run
the Llama 2-70B model using just 179 MB of local storage
(instead of the 140 GB required to run the model locally).