# Edge inference

For scientific workflows that must process sensor data within hard deadlines, UniPipe runs inference serving and online fine-tuning of a deep-neural-network surrogate in a single CUDA context on one edge GPU, handling inference requests and labeled samples in one forward pass and backpropagating only through the labeled subset. On a real ptychographic reconstruction workflow with PtychoNN it lowers inference and reconstruction MSE over a frozen pretrained baseline, reduces the deadline-miss rate over a two-context concurrent baseline, and nearly halves the 95th-percentile latency spread relative to the mean.

For scientific workflows that must process sensor data within hard deadlines, UniPipe runs inference serving and online fine-tuning of a deep-neural-network surrogate in a single CUDA context on one edge GPU, handling inference requests and labeled samples in one forward pass and backpropagating only through the labeled subset. On a real ptychographic reconstruction workflow with PtychoNN it lowers inference and reconstruction MSE over a frozen pretrained baseline, reduces the deadline-miss rate over a two-context concurrent baseline, and nearly halves the 95th-percentile latency spread relative to the mean.

FlashML released FreeToken, an edge-native Mixture-of-Experts serving engine aimed at running frontier open-weight MoE models above 290B parameters locally on personal and consumer hardware. It treats GPUs, CPUs, host memory and interconnects as one elastic inference platform, organizing compute and data movement through bandwidth-adaptive CPU–GPU co-execution, full-layer double-buffered prefill streaming and a global LRU expert cache. The engine also adds semantic anchor checkpoints to skip recomputation after agentic context edits and reallocates VRAM at runtime between expert caches and KV memory, covering models such as DeepSeek-V4-Flash, Qwen3.6-35B-A3B and GLM-5.2 across MXFP4, NVFP4, FP8 and BF16 formats.

FlashML released FreeToken, an edge-native Mixture-of-Experts serving engine aimed at running frontier open-weight MoE models above 290B parameters locally on personal and consumer hardware. It treats GPUs, CPUs, host memory and interconnects as one elastic inference platform, organizing compute and data movement through bandwidth-adaptive CPU–GPU co-execution, full-layer double-buffered prefill streaming and a global LRU expert cache. The engine also adds semantic anchor checkpoints to skip recomputation after agentic context edits and reallocates VRAM at runtime between expert caches and KV memory, covering models such as DeepSeek-V4-Flash, Qwen3.6-35B-A3B and GLM-5.2 across MXFP4, NVFP4, FP8 and BF16 formats.