Widening CPU vectors hits the L1 capacity wall: turning private L2 into a software-managed scratchpad gives 1.44x GEMM speedup at 2048 bits

CPU vector width can keep growing, and the private L2 becomes directly addressable by software as a scratchpad rather than only a hardware-managed cache. L1 data cache had been confined to 32 to 64 KB, while the private L2 holds 256 KB to 2 MB but stayed inaccessible to software because of hardware-managed coherence, virtual memory, and missing ISA support. On a gem5 Armv8-A out-of-order core model, FP16 SVE micro-kernels derived from ACL across seven GEMM shapes give a 1.44x geometric-mean speedup and 40% memory-subsystem energy reduction at VL=16, versus 1.12x and 26% at VL=4.

The scratchpad is not a separate SRAM; existing L2 ways switch into SPM mode at runtime, and each cache line becomes a slot addressed by way and set index. Earlier software-managed memories either paid for dedicated physical storage or kept accesses on the tag-checked associative path, whereas CacheFlex reclaims existing L2 capacity and takes pure SPM accesses off tag lookup and translation entirely. In a post-layout commercial 12 nm FinFET implementation, the SPM load/store queue with 32 load and 48 store entries occupies 7,084 square micrometers and L2 SPM control logic 1,293, raising L2 area by 0.31% and total overhead to 2.02%.

A pair of bridge instructions moves data across domains, and ordering between regular memory and the SPM is maintained by two separate load/store queues. Earlier cache-to-scratchpad designs mostly relied on user-level DMA or software-side conflict resolution, while CacheFlex has hardware handle cross-domain RAW, WAW, and WAR hazards under speculative out-of-order execution. spm_cp copies regular-memory data into an SPM slot non-destructively and without allocating, leaving the source line in place; spm_wb writes back through the normal coherence path, and both are inserted into the regular and SPM queues with commit not waiting for installation.

In full-layer inference, the scratchpad stacks with FlashAttention by pinning selected K/V tiles in the SPM for reuse across a group of query blocks. With FlashAttention alone, tiles can still be silently evicted by the hardware replacement policy; with CacheFlex alone, unfused attention is bottlenecked by a score matrix far larger than both L2 and SPM. On LLaMA-3.2-1B and BERT-Base prefill layers at VL=16, the combination gives 1.64x and 1.78x layer speedup over baseline with 38% and 45% memory-subsystem energy reduction; the FlashAttention kernel-level gain reaches 18.3% once the working set approaches or exceeds L2 capacity.

The evaluation is scoped to single-core, mechanism-level analysis, isolating CacheFlex's first-order effects on private-hierarchy data placement and movement without conflating them with shared-resource contention. Workloads cover GEMM and attention kernels that dominate the compute phase of AI inference prefill, plus full prefill layers from LLaMA-3.2-1B and BERT-Base. CacheFlex is described as complementary to hardware prefetching, with benefits growing as vector width and peak compute throughput increase.

The evaluation is single-core and does not cover behavior under multi-core shared-resource contention. Full-layer results come from gem5 modeling plus post-layout 12 nm area and power data, not from silicon measurements. Without FlashAttention, CacheFlex provides negligible speedup on attention itself because the score matrix dwarfs both L2 and SPM. When the working set fits comfortably in L2, the FlashAttention kernel-level gain is at most 3.2%, and the placement policy retains conventional cache mode. The design principles are described as ISA-agnostic, but the concrete implementation and evaluation are based on Armv8-A SVE.