Autonomous driving has evolved from modular pipelines to end-to-end (E2E) systems that map raw sensors and navigation signals directly to actions. Vision–Language–Action models (VLAs) push this further by replacing discrete commands with free-form natural language. This interface is more intuitive for the driver, but it opens a new vulnerability surface.
In traditional modular stacks, a rule-based Behavior Planner reconciled high-level intent with local traffic conditions before any control action was issued. E2E systems collapse this reasoning into network parameters, and VLAs further widen the input space to open-ended natural language, whose combinatorial coverage cannot be guaranteed during training. The same surface-form instruction can be safe on an open highway and dangerous at a pedestrian crossing.
We are the first to systematically study navigation–scene conflict in the E2E VLA setting. We build a three-axis benchmark suite probing (i) safe navigation following, (ii) navigation–scene conflict awareness, and (iii) robustness to adversarial instructions, and we propose SafeDriveVLA, which decouples conflict reasoning from action generation through an interpretable conflict prediction made before the action span, together with navigation-conditioned world model dreaming that exposes whether the instructed maneuver is feasible before the policy commits.
Our benchmark suite isolates three orthogonal capabilities of a language-conditioned driver. Together, they distinguish a model that ignores instructions, a model that follows them blindly, and a model that genuinely reasons about whether they should be executed in context.
For each meta-command, a pool of paraphrased natural-language instructions is issued on routes synthesized from the CARLA OpenDRIVE topology, with all hazards removed and all lights forced green, isolating whether the model can actually condition on the navigation signal at all.
Manually authored unsafe instructions are triggered at the moment a safety-critical event occurs (Bench2Drive, 150 routes), and ground-truth navigation commands are swapped for conflicting ones at intersections and lane-changes (NavSim, 300 scenarios). The danger is contextual, since the same surface form is benign elsewhere.
Adversarial “jailbreak”-style instructions that are unsafe to execute under any circumstance. Tests whether the VLA can refuse direct harmful commands rather than treating language input as authoritative.
Prior VLAs entangle navigation–scene conflict awareness with action generation, causing the model to blindly predict actions without first reasoning about safety. SafeDriveVLA inserts an interpretable conflict prediction emitted before the action span, attributing every expert frame to one of three driving modes:
Strict Cautious Fallback
Strict: execute the navigation signal immediately. Cautious: follow the signal while prioritizing safety. Fallback: override the signal because the safe action space is empty. By predicting the mode first, the model separates the question of what should be done from how to do it, enabling separate diagnosis of conflict awareness and trajectory quality.
Predicting the mode tells the model that a conflict may exist; a navigation-conditioned world model lets it see that conflict before committing. We pretrain a latent world model on driving video (a frozen V-JEPA 2 encoder paired with an action-conditioned predictor) that imagines how the scene will evolve over the next 2.5 seconds. Unlike prior world models that roll the scene forward under the ground-truth expert action, SafeDriveVLA dreams forward under a per–meta-command action anchor that strictly executes the current navigation signal. When the instructed maneuver is feasible the dreamed latents stay benign; when an adjacent vehicle occupies the target lane or a hazard blocks the path, they encode the conflict and steer the policy to defer or override the maneuver even when it is explicitly instructed. These dreamed latents are handed to the VLA backbone alongside the observations and navigation signal, grounding the predicted mode in an imagined rollout rather than surface form alone.
Expert trajectories are not necessarily optimal; behaviour cloning confuses the model when similar navigation signals and scenes yield different actions. We attribute every frame to strict, cautious, or fallback, giving the model an explicit handle on navigation–scene conflict before the policy commits to an action.
A latent world model is trained to predict future states from the current state and a candidate action, supplying the dreaming substrate over which alternative trajectories can be rolled out and scored for safety.
Conditioned on the navigation signal, the model dreams forward in the latent world to test whether the implied maneuver is compatible with the current scene, then commits to the action consistent with the predicted driving mode.
In CARLA-F and B2D-C, drag the handle to compare SimLingo (left) with SafeDriveVLA (right), and use the thumbnail strip to switch between scenes. Each caption lists the instruction issued to the model.
We report closed-loop results following the order in the paper: overall driving quality on standard Bench2Drive, navigation-following capability on CARLA-F, navigation–scene conflict awareness on B2D-C and NavSim-C, and robustness to adversarial instructions on B2D-Adv. A strong language-conditioned driver should reach state-of-the-art driving quality and instruction following, keep infractions low when the navigation signal conflicts with the scene, and refuse instructions that are unsafe under any circumstance.
Standard Bench2Drive closed-loop benchmark. Nav. = navigation signal (CMD command, WP waypoint, Lan language). * trained with ×0.2 data scale. Higher is better for all metrics. SafeDriveVLA reaches state-of-the-art DS and SR.
| Method | Nav. | Expert | DS ↑ | SR (%) ↑ | Efficiency ↑ | Comfort. ↑ |
|---|---|---|---|---|---|---|
| Prior methods | ||||||
| UniAD | CMD | Think2Drive | 45.81 | 16.36 | 129.21 | 43.58 |
| VAD | CMD | Think2Drive | 42.35 | 15.00 | 157.94 | 46.01 |
| RecogDrive | CMD | Think2Drive | 71.36 | 45.45 | 138.18 | 17.45 |
| ORION | CMD | Think2Drive | 77.74 | 54.62 | 151.48 | 17.38 |
| MindDrive | CMD | Think2Drive | 78.04 | 55.09 | – | – |
| AutoVLA | Lan | PDM-Lite | 78.84 | 57.73 | 146.93 | 39.33 |
| DriveMoE | WP | Think2Drive | 74.22 | 48.64 | 175.96 | 15.31 |
| SimLingo* | WP | PDM-Lite | 81.48 | 53.66 | 246.01 | 42.33 |
| SafeDriveVLA* | WP | PDM-Lite | 83.64 | 59.82 | 260.67 | 52.48 |
CARLA-F navigation-following benchmark. Per-maneuver Navigation Compliance Rate (NCR, %); Speed is Speed Error (lower is better) and Avg. is mean NCR. * trained with ×0.2 data scale. From language alone, SafeDriveVLA leads all language-conditioned models and recovers lane-change NCR from 0%.
| Model | Nav. | Speed ↓ | Turn left | Turn right | Go straight | Left lane | Right lane | Lane follow | Avg. ↑ |
|---|---|---|---|---|---|---|---|---|---|
| Command-conditioned | |||||||||
| ORION | CMD | – | 100.0 | 100.0 | 94.1 | 60.6 | 58.6 | 86.2 | 87.8 |
| MindDrive | CMD | – | 100.0 | 92.2 | 94.1 | 33.3 | 37.9 | 70.7 | 78.8 |
| AutoMoT | CMD | – | 98.2 | 100.0 | 97.6 | 21.2 | 27.6 | 89.7 | 82.0 |
| Language-conditioned | |||||||||
| SimLingo | Lan | 1.44 | 83.6 | 80.4 | 45.9 | 0.0 | 0.0 | 63.8 | 52.4 |
| SimLingo-IF | Lan | 1.31 | 89.1 | 82.3 | 49.4 | 0.0 | 0.0 | 69.0 | 55.6 |
| SimLingo-Safe | Lan | 1.67 | 92.7 | 84.3 | 50.0 | 0.0 | 0.0 | 62.1 | 55.6 |
| SafeDriveVLA* | Lan | 4.27 | 87.8 | 98.0 | 89.0 | 66.7 | 72.4 | 82.5 | 82.7 |
Bench2Drive navigation–scene conflict benchmark (B2D-C). Free-form language instruction is the sole navigation signal; we compare against SimLingo variants, the only baseline supporting language-conditioned driving. Higher DS / SR are better; lower infractions are better. * trained with ×0.2 data scale.
| Method | DS ↑ | SR (%) ↑ | Collision ↓ | Traffic Viol. ↓ | Out of Route ↓ |
|---|---|---|---|---|---|
| Baseline VLA | |||||
| SimLingo | 72.8 | 36.7 | 66 | 60 | 18 |
| SimLingo-IF | 56.3 | 12.7 | 166 | 57 | 32 |
| SimLingo-Safe | 72.8 | 38.0 | 60 | 61 | 11 |
| SafeDriveVLA* | 67.3 | 35.8 | 45 | 9 | 1 |
NavSim-C navigation–scene conflict benchmark. On the real-world NavSim split, existing VLAs degrade substantially when the injected command conflicts with the scene. Higher is better for all metrics; PDMS is the composite score.
| Method | NC ↑ | DAC ↑ | TTC ↑ | C. ↑ | DDC ↑ | EP ↑ | PDMS ↑ |
|---|---|---|---|---|---|---|---|
| AutoVLA | 93.6 | 79.0 | 88.0 | 100.0 | 86.0 | 66.6 | 70.7 |
| ReCogDrive | 95.0 | 91.3 | 92.6 | 100.0 | 94.7 | 76.7 | 81.7 |
| CuriousVLA | 95.3 | 93.0 | 94.7 | 98.3 | 97.2 | 85.4 | 84.4 |
| DriveVLA-W0 | 96.3 | 87.0 | 91.7 | 100.0 | 97.0 | 71.0 | 76.9 |
B2D-Adv adversarial instruction benchmark. Driving Score on benign Bench2Drive routes (B2D) vs. adversarial instructions that explicitly try to override the safety prior (B2D-Adv); higher is better. SafeDriveVLA surpasses SimLingo by ≈70% under attack, and the ablation shows the mode token and world model each help.
| Method | B2D DS ↑ | B2D-Adv DS ↑ |
|---|---|---|
| SimLingo | 71.6 | 43.8 |
| SafeDriveVLA ablation | ||
| baseline (no mode, no WM) | 76.3 | 71.4 |
| w/ mode | 81.6 | 77.5 |
| SafeDriveVLA* (w/ mode + WM) | 82.8 | 78.4 |