UniPipe puts serving and fine-tuning in one CUDA context: on a ptychography stream its miss rate tracks an inference-only deployment
Synopsis
For scientific workflows that must process sensor data within hard deadlines, UniPipe runs inference serving and online fine-tuning of a deep-neural-network surrogate in a single CUDA context on one edge GPU, handling inference requests and labeled samples in one forward pass and backpropagating only through the labeled subset. On a real ptychographic reconstruction workflow with PtychoNN it lowers inference and reconstruction MSE over a frozen pretrained baseline, reduces the deadline-miss rate over a two-context concurrent baseline, and nearly halves the 95th-percentile latency spread relative to the mean.
Interpretation
A surrogate on an edge GPU can now keep a hard per-unit deadline while continually fine-tuning, instead of trading accuracy against latency stability. Earlier deployments either froze the pretrained surrogate, letting the error grow over regions underrepresented at training time, or fine-tuned concurrently in a separate CUDA context, where the two contexts contend for streaming multiprocessors, memory bandwidth and the kernel-launch queue. On a real ptychographic reconstruction workflow, two-context concurrency raised both the mean and the fluctuation of per-sample inference latency, and the miss rate widened monotonically with arrival rate; UniPipe's miss rate stays close to the inference-only pretrained baseline across the sweep.
Inference and fine-tuning share one forward pass, with labeled samples handled separately only in the backward pass. The naive approach runs inference batches and training batches separately, repeating forward computation, and small batches leave the GPU underutilized. Each batch mixes unlabeled inference requests with labeled samples whose ground truth comes from the classical solver; one forward pass covers the whole batch, the output is sliced so the unlabeled part is returned as inference replies and the labeled part enters the loss and backward pass; on PtychoNN the forward and backward latency curves are nearly flat below the saturation knee, and a polynomial-time dynamic-programming scheduler sets batch composition from them.
Perspective
The demonstrated scope is a single real ptychographic reconstruction workflow on one edge GPU, with PtychoNN as the surrogate, sensor rates swept from 1,000 to 5,000 diffraction patterns per second, and classical-solver throughput fixed at 16 samples per second. The authors plan to extend the scheduler to adaptive selection of labeled training data and to evaluate other surrogate architectures such as transformers and workflows beyond ptychography.
The system targets a single-device edge node and does not address multi-GPU scaling; the scheduler assumes the forward and backward latency curves are profiled once before deployment and stay stable; the forward pass must be mode-invariant, so architectures with mode-dependent layers such as BatchNorm or Dropout need per-mode profiling or separate sub-batches; drift in classical-solver throughput requires re-running the scheduler. The authors describe the generality under these structural requirements as empirical.