Philadelphia

Monocular in-the-wild video; bird's-eye view overlaid on OpenStreetMap.

Recovered 4D point cloud from ego-camera view.

Non-reactive simulation replay from chase-camera view.

Dash2Sim: Closed-Loop Driving Simulation from in-the-wild Dashcam Videos

Anurag Ghosh¹, Francesco Pittaluga², Khiem Vuong¹, Angela Chen¹, Juan Alvarez-Padilla³, Manmohan Chandraker^2,4, Srinivasa Narasimhan¹

¹Carnegie Mellon University ²NEC Labs America ³MIT ⁴UC San Diego

arXiv Code Data BibTeX

We turn in-the-wild monocular videos into 4D driving logs and open the door for closed-loop planner evaluation on realistic long-tailed scenes, without instrumented fleets or handcrafting scenarios in synthetic game engines.

In-the-wild dashcam video in Philadelphia.

Abstract

Self-driving simulations typically rely on data collected in a small number of cities or on hand-authored synthetic scenarios. Dashcam videos cover a far broader range of locations and situations, including rare or long-tailed scenarios. They are considered less usable for simulation because it is difficult to recover accurate 4D scenes from monocular in-the-wild videos. Work zones are one such class of long-tailed situations that dashcams capture. We present Dash2Sim, a framework that turns in-the-wild monocular dashcam videos into metric, geo-referenced 4D driving logs compatible with existing simulators, and verifies each one against an independently maintained map without annotations. We apply Dash2Sim to a large video corpus to create the ROADWork4D benchmark dataset, which spans 4,244 scenes with 2.7M 3D objects across 17 cities. On a verified subset ROADWork4D-CL (2,201 scenes), we study privileged closed-loop planners and find that work zone scenarios are difficult: while rule-based and hybrid planners generalize better than learning-based ones, all fall short, failing to make the lane changes that temporary work zone channels require. Beyond planning, dense depth recovered by Dash2Sim improves novel-view synthesis quality by up to 19% on perceptual metrics, suggesting its potential to provide rich conditioning for closed-loop sensor simulation from monocular videos.

Motivation

Work Zones are a persistent hurdle.

A few work zone failures by commercial ADS and ADAS vehicles, drawn from publicly available information. Contrary to prevailing expectations, work zones remain a persistent hurdle for reliable self-driving. Failure modes include colliding with emergency vehicles, driving into wet concrete, not stopping for human flaggers, getting boxed in by construction equipment, and merging into oncoming traffic.

Significance: Work zones present unique, persistent and safety-critical challenges for autonomous vehicles. Existing open-source fleet-derived datasets have very limited coverage of work zones.

Rare objects & entities: cones, barrels, arrow boards, and a wide vocabulary of temporary signage. Hover to zoom.

Geometry → Semantics → Simulation.

Framework

The Dash2Sim Framework

Dash2Sim produces a metric, geo-referenced 4D driving log from in-the-wild monocular dashcam video in three stages, shown left to right. Geometry recovers a metric ego trajectory and dense per-image depth. Semantics detects, tracks, and lifts agents and long-tailed objects to 3D using promptable foundation models. For Simulation, we retrieve the OpenStreetMap tile and place the ego, agents, and objects.

Contribution: Dash2Sim enables the creation of metric, geo-referenced 4D driving logs from in-the-wild monocular dashcam videos, complementing data derived from fleets to better cover long-tailed scenarios. We shall cover key design decisions next.

Columbus

Recovered point clouds across cities. In these visualizations, we projected all points (including dynamic objects) onto the same coordinates.

How to recover AV-aligned 4D scenes?

Metric Reconstruction via Geo-anchored SfM

Monocular 4D scene recovery is difficult, brittle, and error-prone. For accurate localization, we identify that recovering a global metric reference does not need additional sensors: we retrieve nearby co-located street view images, and run SfM over the monocular dashcam images and co-located images together and use their known GPS to transfer scale and geo-reference the trajectory.

Implication: Less need for fleets to get metric, geo-referenced 4D driving logs. But how do we verify if our logs are accurate at scale?

Median translation error < 10 cm · scale within 0.3% on nuScenes-mini.

Red: coarse GPS tracks. Blue: metric 6-DoF poses recovered by Dash2Sim. Hover to zoom.

Three paradigms for data verification.

How to Ascertain Data Quality at Scale?

Verifying 4D Driving Logs at Scale

Different data collection paradigms afford different verification strategies. The in-the-wild nature of our data source leads us to a novel strategy:

Instrumented collection verifies via cross-modal sensor agreement.
Interactive collection verifies via task success in the environment.
In-the-wild observations (ours) can be verified via cross-source physical consistency in simulation.

In our source videos the driver completed each route without any incidents. Replaying the recorded ego trajectory and rolling out with non-reactive agents therefore verifies 4D log recovery. Any penalty is more likely to be an error in our log or the map. Because this score-based verifier needs no ground truth, it scales to all scenarios across 17 cities.

However, the reverse isn't true: a low score doesn't necessarily mean the log is wrong. The verifier is conservative: a correct log can still score low, say when the simulator penalizes the hard braking that is routine in dense traffic.

Takeaway: Less need for human effort to verify the quality of 4D log recovery.

2,201 verified scenes · nuPlan Driving Score > 90

Dataset & Benchmark

ROADWork4D

A 4D Dataset for Studying Long-Tailed Autonomous Driving

We apply Dash2Sim to in-the-wild ROADWork videos. We recover 4D driving logs of work zone scenes, a class of rare but safety-critical, complex, and dynamic long-tailed situations across 17 US cities.

Scenes: 4,244
US cities: 17
Driving distance: 724.5km
Driving time: 35.2Hours
Frames @ 5 Hz: 633,404
Per-frame 3D boxes: 2.74Million
Instance tracks: 142,863
SfM 3D points: 276.3Million

Verified closed-loop subset — ROADWork4D-CL: 2,201 scenes · 1.50 Million boxes.

Looking to work on 2D perception? Use ROADWork instead. We shall cover downstream applications of ROADWork4D next.

Chicago

PlanTF (17 cities)

Diffusion Planner

PDM-Closed

PLUTO Hybrid

Log replay (top); four planners on the same work zone (below). Note that driving in the correct work zone channels (i.e. in the lane or drivable area not blocked by temporary traffic control objects) is a challenge.

Closed-loop planning

Privileged Closed-Loop Planning with Dashcam Videos

We can study long-tailed planning using the ROADWork4D dataset. Rule-based and hybrid planners generalize to navigating work zones better than learning-based planners. Yet, navigating work zones is hard: all fall short of performing channel-sensitive lane changes that work zones demand and trade off progress for safety. Fortunately, navigating work zones is a learnable task, though: training on more cities steadily improves a learning-based planner.

Planners exhibit different performance characteristics while navigating work zones. Hover to zoom.

Planner	Sens.	Insens.
PlanTF	17.7	22.9
Diffusion	24.0	30.9
Pluto	32.7	43.0
PDM-Closed	53.0	57.4
Pluto-H	57.5	68.8

Channel-sensitive scores for all the planners are much lower than channel-insensitive ones.

Training on even one city improves performance, with data from more cities leading to better results. Hover to zoom.

Novel-view synthesis comparison: OmniRe baseline versus OmniRe with Dash2Sim depth, showing sharper signs and cones on held-out views in Boston and Houston — Dense depth from Dash2Sim improves novel-view synthesis, especially for small long-tailed objects.

Please Consider Citing Dash2Sim

arXiv Code Data

@article{ghosh2026dash2sim,
  title   = {Dash2Sim: Closed-Loop Driving Simulation from in-the-wild Dashcam Videos},
  author  = {Ghosh, Anurag and Pittaluga, Francesco and Vuong, Khiem and
             Chen, Angela and Alvarez-Padilla, Juan and Chandraker, Manmohan
             and Narasimhan, Srinivasa},
  journal = {arXiv preprint arXiv:2606.07366},
  year    = {2026}
}

This website is licensed under a Creative Commons Attribution-ShareAlike 4.0 International License. You are free to borrow its source code; we just ask that you link back to this page.