Model-Based Online Decision Making via Generative Trajectory Planning

Abstract

Online decision-making faces a tension between efficient inference and long-horizon reasoning. Value-based methods rely on short-horizon bootstrapping that limits the propagation of long-term information, while trajectory optimization methods plan explicitly but must solve a high-dimensional optimization problem at every decision step. We propose Generative Trajectory Planning (GTP), a model-based framework that addresses this tension by learning a generative model as a proposal distribution over action sequences. Instead of optimizing sequences from scratch, GTP samples candidate sequences from a diffusion prior and refines them using learned dynamics, reward, and value models, enabling efficient trajectory-level reasoning. Across continuous control and long-horizon manipulation tasks, GTP matches or exceeds state-of-the-art model-free, diffusion-policy, and model-based baselines, and remains stable in regimes where strong baselines degrade.

Generative Trajectory Planning

GTP uses the generative model as a structured proposal distribution for planning. An ensemble dynamics model supplies sequence-level uncertainty for exploration, while conservative value updates limit the impact of model bias during learning.

Benchmark Results

Locomotion

Manipulation

Ablations

We ablate GTP's planning, candidate augmentation and dynamics ensemble.

Planning Horizon

GTP trains and plans effectively with longer-horizon models: short horizons lack enough lookahead for reliable FetchSlide behavior, while extended horizons let the planner evaluate delayed consequences and select more stable action sequences.

Exploration

Where does each method's end-effector go on PushT-Vision? SAC and DPMD rarely reach the T-block, and TD-MPC2 covers the space around it broadly but without focus. GTP concentrates its visits on the block's contour and contact regions, the states that matter for pushing it into place.

Each animation shows 2,000 evaluation rollouts from that checkpoint in the T-block's frame: the shading accumulates end-effector visits over the episode, and the dots follow 150 sampled rollouts. The value under each video and the plot give the share of steps the end-effector spends within two pusher radii of the block.

BibTeX

@inproceedings{balim2026modelbasedonline,
  title     = {Model-Based Online Decision Making via Generative Trajectory Planning},
  author    = {Balim, Haldun and Du, Yilun and Li, Na},
  booktitle = {Advances in Neural Information Processing Systems},
  year      = {2026}
}