LLM-Guided Motion Planning for Autonomous Vehicles: A Safer Approach

TU Delft researchers have developed a system that uses GPT-4o-mini to translate natural-language passenger requests into adjustments of a safety-aware motion-planning controller for autonomous vehicles. Instead of letting the LLM directly control the car, the system tunes the parameters of a traditional model predictive-path integral controller within safe bounds, and requires passenger confirmation before making changes. This approach offers a practical pattern for integrating LLMs into safety-critical systems without sacrificing formal safety guarantees.

How the system works: LLM tunes controller criteria within safety bounds

The system relies on a model predictive-path integral controller previously developed by the researchers. This controller identifies multiple trajectories to a goal and scores them based on speed, steering angle, and collision probability, then selects an optimal path as a combination of high-scoring trajectories. GPT-4o-mini is given the passenger's natural-language request and a description of the driving scenario. Instead of directly adjusting controller settings, the LLM rates the relative importance of the scoring criteria, shifting them up or down around a safe baseline set by the researchers. For example, a request like "I am running late, go fast" increases the weight on speed, while "I feel dizzy" increases smoothness. Before any changes are applied, the system presents a natural-language description of the planned adjustments and asks the passenger to confirm. This human-in-the-loop step catches misinterpretations and allows further refinement.

Testing in the nuPlan simulation environment with eight prompts showed that the system adjusted speed and smoothness in line with user intent. Requests for a more comfortable ride increased smoothness, while urgent prompts led to higher speeds.

Why this matters for builders: a pattern for safe LLM integration

The key architectural decision is separating the LLM from the main controller. Even if the model hallucinates or misinterprets a prompt, it cannot produce dangerous behavior because the controller's safety constraints remain in place. This is a useful pattern for any safety-critical system where you want to offer natural-language customization without giving up deterministic safety guarantees. Nicolas Baumann of ETH Zurich, who published similar work on an LLM-tuned racing car controller, notes that "you get the possibility of language interaction, but you can guarantee that it is going to be within the constraints of this classical controller, so you can bake in safety." However, setting these constraints requires considerable engineering work