Encrypted BERT-base inference drops the format conversions: MOAI amortizes 2.36 minutes per input, 52.8% faster than THOR

MOAI removes the need to switch data packing formats between layers of encrypted Transformer inference, keeping a fixed column-packing and diagonal-packing handoff through the whole evaluation flow. THOR previously had to convert formats between layers to complete ciphertext-ciphertext multiplication, and also performed an encrypted matrix transpose separately. Across all matrix multiplications of BERT-base, MOAI needs 9,648 homomorphic rotations, against 22,224 for THOR and 16,740 for Powerformer.

Softmax and LayerNorm are rewritten as algorithms that need no homomorphic rotations, while the model itself is left unchanged and needs no retraining. NEXUS and THOR both relied on a set of homomorphic rotations to sum data packed in a ciphertext when computing these two functions, and rotation is among the slowest CKKS operations. Substituting BERT-base parameters, the rotation-free Softmax and LayerNorm together save 2,448 homomorphic rotations, where NEXUS and THOR each need 2,448 for the same item.

On a single A100 GPU, MOAI's end-to-end evaluation of BERT-base takes 52.8% less time than THOR, amortized to 2.36 minutes per input. THOR had reported 10 minutes per input for BERT-base inference on 128 tokens. The two are compared on the same hardware, ring dimension and slot count, and the layer-wise table shows total time falling from 602.26 seconds to 283.95 seconds.

The same packing and evaluation flow also applies to Powerformer, whose functions were already made homomorphic-friendly, and extends to decoder-only models such as LLaMA-3-8B. Powerformer gained speed by replacing Softmax and LayerNorm with power and linear functions and retraining, a route that does not transfer easily to other Transformer architectures. Under Powerformer's parameter settings, MOAI cuts total evaluation time from 344.52 seconds to 152.4 seconds, a 55.7% reduction.

The demonstrated scope is end-to-end encrypted inference of BERT-base with up to 128 input tokens, with amortized time of 574.6 seconds on CPU and 141.3 seconds on a single H200 GPU; the comparison with THOR and Powerformer is run on a single A100 GPU. LLaMA-3-8B results are given as an aggregation of microbenchmarks using 8-token input.

Encrypted evaluation error grows over the first 6 to 7 layers and then stabilizes, and GLUE scores under encrypted input are close to but not identical with plaintext, for example 65.5 versus 65.7 on RTE. LLaMA-3-8B totals 224.53 seconds with 8-token input and 26.94 seconds at short input length, and these figures come from aggregated microbenchmarks rather than a full end-to-end run.