A glance is enough to queue the next actions: symbolic graphs carry kitchen priors, and a 3M-parameter model leads in short-observation tests
Synopsis
SymAnt merges a scene graph and a knowledge graph into one joint graph and uses the symbolic context found by graph search as the starting point for iterative decoding, so it can predict long action sequences from very short observation windows. In the short-observation settings of Breakfast and 50 Salads, its mean accuracy over classes exceeds the neuro-symbolic baseline NeSCA and several large-model baselines; on EPIC-KITCHENS, the 3M-parameter model performs comparably to fine-tuned models with billions of parameters. Because predictions are anchored in an editable graph, users can add or remove nodes at inference time to change predictions without retraining.
Interpretation
Action anticipation no longer rests on visual history alone: a scene graph and a knowledge graph are first combined into an explicit symbolic context, and the future action sequence is generated from it. Earlier purely neural methods needed longer observation windows and large training sets, while the neuro-symbolic method NeSCA used only a static knowledge graph. In the short-observation settings of Breakfast and 50 Salads, SymAnt's mean accuracy over classes exceeds NeSCA, FUTR, and reasoning-model baselines such as GPT-o1 and DeepSeek-r1.
The method links objects detected in each frame into a scene graph with spatial relations, accumulates it over time into a temporal scene graph, and connects it to a prebuilt knowledge graph through shared objects. From the shared nodes, a joint-graph search alternates a propagation network, an importance network, and a context network to decide which neighbors to expand, and the resulting embedding initializes the iterative decoder. The knowledge graph is built once from the training split and never updated at inference; on Assembly101, a non-cooking assembly dataset, the same graph-construction procedure is applied directly.
Predictions can be changed by hand: activating or deactivating an action node in the graph changes the action sequence the model outputs. Purely neural models offer no comparable handle tied to a specific concept. On 50 Salads, for Cut Tomato, Cut Lettuce, and Serve Salads, deactivating the target node substantially lowers accuracy, while forcing its activation raises accuracy to 100%.
Perspective
The demonstrated scope is offline evaluation on five action-anticipation datasets: third-person Breakfast and 50 Salads, egocentric EPIC-KITCHENS and EGTEA Gaze+, and the assembly dataset Assembly101, plus a qualitative example in a small real kitchen setup.
The method relies on an off-the-shelf object detector, so initial detection failures propagate to later predictions; the knowledge graph can be replaced by a general-purpose source but remains part of the pipeline. The knowledge-editing experiments show that deactivating an action node sharply reduces but does not fully eliminate the corresponding prediction, indicating that visual features partly bypass the symbolic pathway.