TensorRT stops betting on one build route: across ten workloads, the best tuned route ran 1.14x to 1.63x faster than default

TensorRT performance optimization no longer relies on a single universal scheduling strategy; it evaluates alternative compilation routes per model. When no build route is specified, TensorRT uses the default build route, in which every knob keeps its default value; previously optimization depended on that one route suiting the model at hand. Across ten evaluated workloads, with each default build route normalized to 1.0x, the best eligible route found during tuning delivered performance improvements ranging from 1.14x to 1.63x.

Tuning writes compiler scheduling choices as explicit build routes so the space can be enumerated and compared. A build route is a concrete combination of compiler knob values, where a knob is a configuration setting that changes some aspect of the compilation pipeline, such as a fusion decision or code generation path; TensorRT 11.1 provides more than 200 knobs. The sample trtexec implementation expands a regex-like route expression into a list of concrete routes, builds and measures accuracy and performance for each, excludes routes that crash or fail accuracy, and selects the fastest measured eligible engine.

Accuracy and performance are split into two roles in the optimization rather than merged into one objective. Accuracy determines which candidate solutions are acceptable, and performance ranks only the acceptable candidates, with the developer defining the accuracy boundary appropriate to the application. In a thought experiment with four binary knobs and 16 candidate solutions, tightening the accuracy tolerance from infinity to 0 contracts the eligible set from 16 down to 1, selecting the fastest measured eligible candidate at each tolerance; these counts are illustrative and actual transition points depend on the model, reference data and accuracy metric.

The demonstrated scope is end-to-end measurement of the Global Performance Tuner in TensorRT 11.1 on ten evaluated workloads, plus the tuning loop organized by the trtexec sample implementation; the same implementation in Python is planned for release in Torch-TensorRT 2.15.

The best build route holds only for the model, input shapes, target GPU, software stack and TensorRT and tuner versions used during the search, and a route that improves one deployment may have no effect or cause a regression in another; the counts of 16 candidates contracting with accuracy tolerance are illustrative, and actual transition points depend on the model, reference data and selected accuracy metric.