Talk to the Car: Autonomous Vehicles Get Personal

This article is part of our exclusive IEEE Journal Watch series in partnership with IEEE Xplore.

The idea of letting a machine do the driving for you may put a lot of people off autonomous vehicles. But research could make it possible to backseat-drive an autonomous vehicle just as you might with a human driver.

Self-driving cars carefully balance a host of parameters to ensure a smooth ride, including things like speed, acceleration, and the smoothness of turns. But human driving preferences can often vary depending on how much of a rush they’re in, whether they’re feeling carsick, or how busy the traffic is.

These cars have a software component called the motion planner, which is responsible for choosing a safe and efficient path through traffic. The motion planner is normally tuned by engineers before the vehicles hit the road so that there’s little scope for passengers to adjust a vehicle’s driving style on the fly. But now researchers at the Delft University of Technology (TU Delft) in the Netherlands have developed a system that uses a large language model (LLM) to translate natural-language user requests such as “I am running late, go fast” into adjustments to a self-driving control system. The researchers posted their preprint on arXiv and are presenting the work at the IEEE Intelligent Transportation Systems Conference in September.

LLMs Personalize Autonomous Driving

The system doesn’t give users direct control over the vehicle’s driving decisions; it simply tunes the parameters of a safety-aware motion-planning algorithm, which helps to keep the vehicle’s behavior within safe bounds. And the system keeps the human in the loop by describing how it’s going to alter its behavior in nontechnical language, and by asking the passenger to confirm before making changes. When the system was tested in simulation, the researchers found it adjusted the speed and smoothness of driving in line with natural-language instructions.

“The motion-planning problem is not only about reaching a place while avoiding collisions, it’s also how you do it,” says lead author Diego Martinez-Baselga, a postdoctoral researcher at TU Delft. “The motivation here is trying to make the way the autonomous car drives adaptable by end users easily, just by talking to the car.”

Previous research has investigated the potential of using LLMs and video-language models (VLMs) to direct decision-making for self-driving vehicles, but the researchers deliberately targeted driving style instead. Using LLMs and VLMs to directly control vehicles faces several challenges, says Martinez-Baselga. These include relatively slow response times, which can make these models unsuitable for the fast-paced decision-making required in driving, and the fact that they can’t provide concrete performance guarantees in the way a deterministic motion planner can.

Instead, the researchers used an LLM’s language and reasoning capabilities to translate fuzzy human preferences…