Inference where the server never sees your data

Here's the deal: on August 14, Google published HEIR. The name stands for Homomorphic Encryption Intermediate Representation. What it actually is, is a compiler. What it does is take an AI model that was trained to work on ordinary plaintext data and rewrite it into a version that runs on encrypted inputs.

That matters because of a gap that has always existed in cloud AI. Your data is encrypted in transit. It's encrypted at rest. But at the moment of computation, it has to be plaintext — the raw values land in server memory, even if only for milliseconds. Fully homomorphic encryption (FHE) is the technique that closes that last hole, and HEIR is an attempt to make it usable by people who don't hold a cryptography PhD.

Google's framing in the post: "Today we're excited to showcase HEIR, the latest powerful tool added to our Private Computing Toolkit. HEIR is an open source compiler that unlocks cryptographically-secure private AI inference." The author is Jeremy Kun, a staff software engineer at Google.

Now let some air out of this before we go further. You cannot run a chatbot through it. What Google actually compiled and demonstrated were small, structurally simple models: a recommendation model, a credit card fraud detector, a network intrusion detector, and a hotword spotter. Large language model inference remains far outside what FHE can carry. This story is about where exactly that boundary sits, and why it has started to move.

Why FHE spent two decades being "perfect in theory, impossible in practice"

The idea itself is old. A homomorphic scheme lets you add and multiply ciphertexts such that decrypting the result gives you what you'd have gotten by adding and multiplying the plaintexts. The proof that a fully homomorphic scheme — arbitrary computation, unlimited depth — was even possible arrived in 2009. Ever since, FHE has been a permanent fixture on the "technology that will change everything, eventually" list.

The blocker was speed. A ciphertext is thousands of times larger than the plaintext it hides, and a single operation costs hundreds to tens of thousands of times what the plaintext version costs. Worse, every multiplication grows the noise embedded in the ciphertext, and once that noise crosses a threshold, decryption simply fails. Resetting it requires bootstrapping, which is the single most expensive operation in the entire field.

Put numbers on it and the problem becomes obvious. A computation that takes a microsecond in plaintext becomes a millisecond encrypted — fine, ship it. A computation that takes a second in plaintext becomes something on the order of fifteen minutes. That multiplier is why FHE stayed inside papers for the better part of twenty years. Neither banks nor hospitals allocate budget for a system that is perfectly private and ten thousand times slower.

The second blocker was people. Actually deploying FHE meant choosing a scheme (BGV? BFV? CKKS? CGGI?), setting ring dimensions and modulus chains, managing a noise budget across the whole program, deciding how to pack values into ciphertext slots, and placing relinearization and modulus-switching operations by hand. That isn't merely difficult — it's graduate-level cryptography. Libraries existed. The real bottleneck was that the population capable of using them correctly numbered in the hundreds worldwide.

Which is exactly why a compiler is the right shape for this problem. Every item on that list is a whole-program optimization question. Those are miserable for humans and native to compilers. HEIR aims squarely at that gap.

Google has been building toward this for five years

This didn't appear out of nowhere. Google open-sourced a C++ FHE transpiler back in 2020 — feed it C++, get back a circuit that runs over ciphertexts. At the time it was close to a first of its kind.

In August 2023 Google extended it with a TensorFlow-to-FHE path. The headline benchmark then: compiling a TensorFlow Lite model to FHE and producing a private inference in 16 seconds for a three-layer neural network. Three layers, sixteen seconds — that was the flagship result Google was proud to publish three years ago. Keep it in mind as a yardstick.

Today, the original fully-homomorphic-encryption repository describes itself this way: "What started with a C++ transpiler 5 years ago, morphed into two new Open Source libraries." Those two are HEIR and Jaxite. Jaxite is an FHE backend written in JAX that targets TPUs and GPUs; HEIR is the compiler layer that sits above it. The original transpiler source has been pushed to an archived release tag. Google closed the first generation and bet on the second.

It's worth separating HEIR from Google's other privacy work. Google already ships Android's Private Compute Core, federated learning, and differential privacy libraries. All of those are variations on "send less data, or blur what you do send." FHE runs the opposite direction — send all the data, but make it unreadable to the machine processing it. Because the directions are opposite, these approaches stack rather than compete.

Read the repository honestly, though. github.com/google/heir is Apache-2.0 licensed, sits at 820 stars, and carries the line "This is not an officially supported Google product." The Getting Started page says the nightly binaries are "intended for testing compiler passes and not for production use." Google is not presenting this as a finished product, and Google is the loudest voice saying so.

What HEIR actually is under the hood

The core design choice is MLIR — Google's own compiler infrastructure, built around stacking layers of intermediate representation (called dialects) and lowering high-level code through them progressively. HEIR adds a tower of homomorphic-encryption-specific dialects on top of it.

The paper lays out the layers. At the top sits the scheme-agnostic secret dialect, where you haven't yet committed to any encryption scheme — values are simply marked secret, which lets ordinary MLIR optimizations run first. Below that come mgmt (relinearization, modulus reduction, bootstrapping) and tensor_ext (rotations and ciphertext packing). Then scheme-specific dialects like bgv, bfv, ckks, and cggi. Then backend code-generation dialects — openfhe, lattigo, tfhe_rust, jaxite. And at the bottom, low-level math: polynomial, mod_arith, rns.

Layer Dialects What it handles
Scheme-agnostic secret computation secret Marks values as encrypted; reuses existing MLIR optimizations
Scheme-agnostic HE computation mgmt, tensor_ext, comb Noise management, bootstrapping insertion, packing layout, boolean circuits
Scheme-specific bgv, bfv, ckks, cggi, lwe The actual operations of the chosen scheme
Backend code generation openfhe, lattigo, tfhe_rust, jaxite Emits runnable C++, Go, Rust, Python
Low-level math polynomial, mod_arith, rns, random Ring arithmetic, modular integers, residue number systems

The scheme-and-backend support matrix is in the repository: BGV, BFV, and CKKS run on OpenFHE and Lattigo; CGGI runs on tfhe-rs and Jaxite. On hardware, the paper documents integrations with Belfort Labs FPGAs, Google TPU v6e/Trillium, Intel HERACLES, Niobium BASALISC, and the Optalysys optical accelerator.

The automation is the real payload. HEIR ships 88 of its own optimization passes plus 237 inherited from MLIR. Among them: optimal relinearization placement solved as a mixed-integer linear program, automated bootstrapping insertion for CKKS, cost-model-driven ciphertext packing layout, and polynomial approximation of non-polynomial functions like ReLU via the Carathéodory-Fejér method with Paterson-Stockmeyer evaluation. These are things people used to do by hand with a stack of papers open.

Parameter selection is automated too. The compiler analyzes the full computation graph, finds the maximal noise at each RNS level, picks tight prime moduli that modulus switching can reset noise against, and consults 128-bit security parameter tables to fix the ring dimension. A library API can't do this, because it only ever sees one operation at a time and has to leave generous safety margins. A compiler sees the whole program and can tighten.

The front door got wider as well. HEIR includes a Python frontend that takes Python bytecode — annotated with types marking which values are secret — and converts it into the secret dialect. In the docs it looks like adding a decorator, and calling the function will "encrypt the inputs, run the function, and return the decrypted result." Installation is pip install "heir_py[python,openfhe]". There's also StableHLO support for importing trained ML models.

So what works today, and what doesn't

The four demos Google published make the boundary unusually legible.

First, a Deep Learning Recommendation Model, described as joint work with Belfort Labs, LG, and New York University, which "unlocks serving private content recommendations." The underlying HE-LRM paper carries the actual numbers: 24 seconds on a single-threaded CPU for a UCI health-prediction dataset, and 228 to 489 seconds for Criteo click prediction. The authors also report a 56× speedup over prior state of the art from their embedding compression technique, and note that "GPU and ASIC FHE acceleration can reduce end-to-end latencies to seconds and even sub-seconds."

Second, credit card fraud detection, compiled with Niobium and hardshell.ai. Third, the Kitsune intrusion detection system, compiled with Niobium — this one, in Google's words, "allows a service provider to detect anomalies without revealing the contents of network packets." Fourth, a hotword detector built with Belfort Labs, which "could allow an audio-triggered AI agent to recognize hotwords while protecting the privacy of the audio recordings."

Notice what all four share. Every one is small, shallow, and structurally simple. And every one produces a short output: one recommendation score, one fraud verdict, one anomaly flag, one wake-word decision. LLM inference is the opposite shape — dozens of attention layers, then hundreds of sequential token generations, with FHE overhead multiplying through the whole thing. It isn't close.

So the practical map divides like this. Workloads with a latency budget measured in seconds or longer, running small models over extremely sensitive data, can go FHE today. Medical prediction, financial fraud scoring, biometric matching, personalization that doesn't want to see your history. Workloads that need conversational latency, run large models, or care about throughput are not there yet.

Google didn't fudge this. The post states plainly that homomorphic encryption carries "a nontrivial cost overhead," and then reframes the trade-off: it used to be capability versus privacy, and now it's cost versus privacy. That isn't a marketing line — it's a meaningful shift. Once an impossible problem becomes an expensive problem, hardware and time start doing the work.

Read the benchmark conditions closely, too. The latency figures Google presented are for a single-threaded CPU. That's the most conservative measurement setting available, which makes it both honest and a signal that headroom exists. GPU bootstrapping results in the literature already run orders of magnitude faster than single-threaded CPU, and the dedicated accelerators HEIR has wired up are aimed at exactly that gap.

Apple and NVIDIA took entirely different roads

Google isn't alone in wanting AI that runs on data it can't read. But there are three distinct philosophies here, and they differ at the root.

Apple's Private Cloud Compute is a hardware trust model. Announced June 10, 2024, it runs on custom Apple silicon servers carrying the same Secure Enclave and Secure Boot technology as iPhones, and promises stateless processing where user data exists only to serve one request, no access for anyone including Apple, and verifiable transparency via a public log researchers can inspect. It's a serious design with one structural condition: as Apple's own documentation acknowledges, data must be accessible in plaintext during processing. You are trusting Apple's hardware, and Apple.

NVIDIA's Confidential Computing is the trusted execution environment approach. Hopper was the first accelerator architecture with confidential computing built in, and general access on H100 landed in April 2024. It pairs with CPU TEEs — AMD SEV-SNP and Intel TDX — to isolate the whole workload, and uses remote attestation to verify GPU identity and firmware authenticity. The performance cost is comparatively small: NVIDIA cites roughly 2-5% throughput overhead for most LLM inference, while an independent IEEE ICDCS 2025 measurement study found throughput in non-confidential mode exceeding confidential mode by 45-70% under some conditions. Both agree the bottleneck is PCIe-path encryption, not the computation itself.

FHE removes the trust premise entirely. Apple's model requires trusting Apple silicon. NVIDIA's requires trusting GPU firmware and hypervisor isolation — and TEEs have a documented history of falling to side-channel attacks. FHE takes a different route: a malicious server, compromised hardware, or a full memory dump all yield the same thing, which is ciphertext. The security rests on the computational hardness of lattice problems rather than on any physical component. As a bonus, lattice-based schemes belong to the family considered resistant to quantum attack.

Approach Flagship What you must trust Performance cost LLM inference today?
Hardware enclave Apple PCC Apple silicon, transparency log Low Yes
GPU TEE NVIDIA H100 GPU firmware, CPU TEE, attestation 2-5% (vendor) / 45-70% gap in some independent tests Yes
Fully homomorphic encryption Google HEIR Nothing (lattice hardness) 100x to 10,000x+ No

The trade-off reads cleanly off that table. The TEE camp can run big models right now, provided you're willing to trust somebody. FHE requires trusting nobody, and currently runs only small models. These are not mutually exclusive, either — real deployments increasingly run hybrid architectures, with the sensitive core in FHE and the latency-critical parts in a TEE or in plaintext.

The regulatory demand is already here

Talk long enough about the technology and someone asks who's going to pay for a hundredfold overhead. A large part of the answer is regulation.

The EU AI Act's high-risk obligations under Regulation (EU) 2024/1689 apply from August 2, 2026 — conformity assessments, technical documentation, CE marking, EU database registration. Look at what falls into the high-risk bucket: credit scoring, employment, education, biometrics, critical infrastructure, law enforcement, migration. Plus product-embedded systems covered by EU product safety law, such as medical devices. Nearly every item on that list is a domain where the data cannot leave the building.

The interaction with GDPR matters too. The AI system itself is governed by the AI Act; the personal data flowing through it is governed by GDPR. Organizations deploying high-risk systems frequently end up owing both a GDPR data protection impact assessment and an AI Act fundamental rights impact assessment. Against that double burden, a technical proof that the server structurally cannot read the input simplifies a lot of arguments.

Healthcare and finance feel this most directly. A hospital that wants to use an external AI diagnostic service inherits regulatory and litigation exposure the moment patient data leaves its perimeter. A bank that wants fraud detection in the cloud can't get audit sign-off to put its transaction ledger on a vendor's servers. Until now the menu for these organizations read: build it on-premises, or don't. It is not a coincidence that Google's demo set includes card fraud detection and health prediction.

Regulation doesn't convert to revenue automatically, though. Adopting FHE remains a cost decision. A hundredfold overhead is a hundredfold server bill, and that gets weighed against regulatory risk rather than automatically beating it. The first places FHE lands will be where the alternative isn't a cheaper system but no system at all — B2B contracts serving customers who will never hand over their data under any terms.

Someone is also trying to standardize this

There's one more detail in the paper worth pulling out. The Fully Homomorphic Encryption Technical Consortium on Hardware (FHETCH) has identified HEIR as "an excellent candidate" for standardization, and the compiler working group at the homomorphic encryption standardization body now focuses on HEIR.

That matters because fragmentation has been the chronic disease of this ecosystem. Every scheme has its own library, every library its own API, every accelerator its own interface. In that state you can't even compare optimization techniques meaningfully. When the HEIR paper says the authors ported "a large fraction of the HE literature to HEIR," the point is precisely that comparison becomes possible on a common substrate.

The paper is also explicit about what came before. Google's own earlier transpiler and Porcupine "take hours to compile programs of modest size" because they exhaustively search the optimization space, while HECO requires fully unrolling loops. HEIR's claim is that layered abstraction plus composable passes sidesteps both limits. Four peer-reviewed publications have been built on HEIR so far.

If the compiler becomes the standard, hardware follows. Accelerator companies need a fixed compiler interface before they can decide what silicon to build, and software needs that interface before it can target the silicon automatically. The fact that HEIR already has Belfort, Niobium, Cornami, and Optalysys attached is the tell. The analogy to what CUDA did for GPUs isn't perfect, but it isn't wrong either.

So what actually changes

If you work in security or privacy engineering, there's a new item in the toolbox. Conversations that used to end at "this data can never leave our environment" can now sometimes continue to "how much latency can you tolerate." Scope it realistically, though: today's candidates are small models with second-scale latency budgets. If you want a pilot this quarter, start with something batchable — fraud scoring, anomaly detection.

If you're an ML engineer, the barrier genuinely dropped. You can start from a Python decorator and import trained models via StableHLO. What hasn't gone away is the model surgery. Non-polynomial functions like ReLU get replaced by polynomial approximations; higher approximation degree means more multiplicative depth; more depth means bootstrapping; bootstrapping means cost. You also have to measure the accuracy you lose to the approximation.

If you run enterprise IT or compliance, treat this as a 2027-2028 architecture option rather than a 2026 line item. The useful work right now is inventory: list the workloads your organization is not doing because the data can't move. That list is your FHE candidate pool. Anything caught by the EU AI Act's high-risk classification moves up the priority order.

From an investment view, there are two threads. One is the FHE accelerator companies — Belfort, Niobium, Cornami, Optalysys appearing in a Google announcement is not decorative, because a standardized compiler is what turns their addressable market into something real. The other thread runs the other way: if TEEs get fast enough and trusted enough, the market FHE is aiming at narrows. This race is early and the outcome isn't callable yet.

If you're an ordinary user, nothing changes for you this year. This is infrastructure, and infrastructure takes time to surface as product. But the direction is worth knowing. When you use cloud AI today, what you're actually doing is trusting that a company won't look at your data. A mature FHE stack replaces that trust with mathematics. Belief and proof are different objects, and that difference could eventually become a product-selection criterion.

If you're a cryptography researcher or student, the on-ramp just opened. You can implement a new optimization as a single MLIR pass and benchmark it against a large portion of the existing literature on identical terms. Georgia Tech, Carnegie Mellon, UC Santa Barbara, Illinois Institute of Technology, Purdue, the University of Edinburgh, and Tsinghua are already listed as collaborators, and the project runs monthly meetings and office hours.

🥄 Three Things You're Probably Wondering

— Does this mean my chatbot conversations get encrypted? No, not remotely. What Google compiled were small models — recommenders, fraud detectors — and the published latencies run from tens to hundreds of seconds on a single-threaded CPU. LLM inference is orders of magnitude more computation, with FHE overhead multiplying on top of that. When it becomes feasible is too early to say flatly; it depends on how far accelerators and algorithmic improvements compound.

— Doesn't that make Apple's or NVIDIA's approach better right now? If you need to run a large model today, yes. TEE overhead starts in the single-digit percent range and it's already in production. The catch is that you're trusting a hardware vendor and its firmware. FHE drops that requirement and pays for it in scale. Which one is "better" depends on who you're able to trust and how long you can wait, and the likeliest outcome is that neither replaces the other.

— Why would Google give this away? Google hasn't disclosed its reasoning, so this is inference from structure. FHE is worthless deployed alone — the party handing over data and the party computing on it have to share a scheme, parameters, and toolchain. And making it fast requires dedicated hardware, which hardware companies won't build without a stable interface to target. Releasing under Apache-2.0 and handing it to a standards body is the fastest way to create that shared substrate. Who ends up selling cloud capacity on top of it is a question you can answer yourself.

References

Numbers are as of announcement and may change.