RAD-LAD Rule and Language Grounded Autonomous Driving in Real-Time

Anurag Ghosh1  Srinivasa Narasimhan1  Manmohan Chandraker2,3  Francesco Pittaluga2

1Carnegie Mellon University   2NEC Labs America   3University of California, San Diego

Paper Video

Closed-loop driving demonstrations across diverse long-tailed scenarios from the nuPlan benchmark. Our planner is interruptible and generates coherent textual reasoning alongside motion plans in real-time.

Overview

Abstract

We present LAD, a real-time language-action planner with an interruptible architecture that produces a motion plan in a single forward pass (~20 Hz) or generates textual reasoning alongside a motion plan (~10 Hz). LAD is fast enough for real-time closed-loop deployment, achieving ~3× lower latency than prior driving language models while setting a new learning-based state of the art on nuPlan Test14-Hard and InterPlan. We also introduce RAD, a rule-based planner designed to address structural limitations of PDM-Closed. RAD achieves state-of-the-art performance among rule-based planners on nuPlan Test14-Hard and InterPlan. Finally, we show that combining RAD and LAD enables hybrid planning that captures the strengths of both approaches. This hybrid system demonstrates that rules and learning provide complementary capabilities: rules support reliable maneuvering, while language enables adaptive and explainable decision-making.

Key Insights

01

Two Complementary Challenges

Planning requires both geometric feasibility and semantic reasoning and no single approach covers both.

02

RAD Addresses Geometry

Expands the search space via dynamic topology and data-driven maneuver priors.

03

LAD Addresses Semantics

Enables language-grounded reasoning over ambiguous traffic interactions in real-time.

04

RAD-LAD: Best Together

The hybrid system integrates structured planning with foundation models for complementary strengths.

Method

Classical Planning

Rule-Based Autonomous Driving (RAD)

A flexible rule-based planner that extends PDM-Closed with:

  • Dynamic Topological Replanning — full graph search at every timestep, ensuring proposals always extend from the current ego pose
  • Lane-Change Capability — augments the road topology with adjacent-lane centerlines, enabling lane changes and even opposite-lane maneuvers
  • Goal-Directed Optimization — Euclidean distance-to-goal cost encourages decisive progress
  • Trajectory Vocabulary — data-driven maneuver priors (swerves, bypasses) from clustered training trajectories
  • Context-Aware Rule Relaxation — softly downweights penalties in deadlock situations
RAD dynamic topological replanning vs PDM-Closed static proposals
PDM-Closed's static proposal paths can get blocked. RAD topologically replans at every timestep and augments paths with adjacent-lane centerlines.
RAD goal-directed optimization and trajectory vocabulary augmentation
RAD combines goal-directed optimization with trajectory proposal augmentation to enable lane changes and other maneuvers.
Learned Planning

Language-Based Autonomous Driving (LAD)

An anytime multimodal language model planner that produces a valid motion plan in a single forward pass:

  • Scene Encoding — PlanTF encodes map elements and agents; MLP adapters project into the LLM's token space
  • Planning Decoder — classification over a discrete trajectory vocabulary, not autoregressive waypoint generation
  • Language Supervision — DrivingQA + PlanningQA provide complementary inductive bias for trajectory prediction
  • Interruptible Inference — valid plan always available from the plan token; reasoning tokens generated opportunistically
  • Multimodal Training — two-stage curriculum (alignment → LoRA finetuning) preserving language abilities
LAD architecture diagram
LAD architecture: We encode map and agent features, which MLP adapters project into the LLM's token space as pseudo-tokens. A designated plan token produces trajectory logits via classification over a discrete vocabulary, while reasoning tokens are generated autoregressively dependent on budget — enabling interruptible, anytime inference.
Hybrid Planning

RAD-LAD: Best of Both Worlds

The hybrid planner combines strict rule-following with language-based reasoning. LAD's trajectory refinement head produces per-waypoint offsets for better rules alignment. The refined trajectory is added as a proposal to RAD's cost-based scorer, which selects the final plan from an expanded set of candidates. This enables interpretable planning via LAD and safe planning via RAD.

Results

Performance on Long-Tailed Situations Rules and learning provide complementary capabilities: RAD alone rivals hybrid methods, LAD sets a new learned SOTA, and their combination improves upon both while being competitive on long-tail scenarios.
Type Planner Test14-Hard InterPlan
Expert Log Replay 85.96 —
Rule IDM 62.26 31
Rule PDM-Closed 75.19 42
Rule RAD (Ours) 80.53 72
Learned PlanTF 61.61 32
Learned PLUTO 59.74 —
Learned DiffusionPlanner 69.22 25
Learned FlowPlanner 70.42 —
Learned LAD (Ours) 70.77 40
Hybrid PLUTO 76.88 49
Hybrid STR2-CKS-800m 78.58 45
Hybrid DiffusionPlanner 82.00 —
Hybrid FlowPlanner 80.25 —
Hybrid RAD-LAD (Ours) 81.36 74
Language supervision improves planning performance Language supervision acts as an inductive bias that improves planning. Autoregressive textual reasoning provides a complementary signal to planning.
Component Test14-Hard (R)
PlanTF 61.61
+ 128 Objects 59.73
+ Static Objects 68.49
+ Plan Token 68.90
+ LLM / DrivingQA 69.75
+ PlanningQA (LAD) 70.77
LAD is real-time with textual reasoning Prior work widely assumed language-based planners are too slow for closed-loop deployment, we dispel that notion.
Model Reasoning Latency (ms) Hardware
DriveVLM Yes 410 Orin X-2
DriveGPT4-V2-8B No 2500 —
DriveGPT4-V2-1.5B No 345 —
DriveGPT4-V2-0.5B No 124 —
PlanTF No 12 A6000
DiffusionPlanner No 50 A6000
FlowPlanner No 83 A6000
LAD No 43 A6000
LAD (10 tokens max) Yes 102 A6000
LAD (40 tokens max) Yes 222 A6000

BibTeX

@article{ghosh2025radlad,
  title     = {RAD-LAD: Rule and Language Grounded Autonomous Driving in Real-Time},
  author    = {Ghosh, Anurag and Narasimhan, Srinivasa and Chandraker, Manmohan and Pittaluga, Francesco},
  journal   = {arXiv preprint arXiv:2603.28522},
  year      = {2025}
}
Coming Soon!