Teaching a Car to Drive the Way People Learn

NTU researchers have built a self-driving system that keeps learning from the human who corrects it, improving on the road instead of being retrained from scratch for every new place.

Prof Lyu Chen watches over Haohan Yang at the driving simulator, stepping in to correct the car as it drives.

A person learns to drive by doing it badly first. They stall, they brake too late, they take a corner too wide, and an instructor in the passenger seat puts them right. Over months, the corrections grow rarer, and one day they drive alone.

A self-driving car does not learn that way. It is trained heavily before it is allowed out, and once it is on the road, what it knows is largely fixed. When it meets something its training did not cover, it cannot simply learn the lesson and move on.

Prof Lyu Chen and his team at NTU have built a car that can. It keeps learning after it is deployed, from the moments a human driver takes over to correct it, the way an apprentice learns from a master. The work entitled “Human-guided continual learning for multifaceted improvement of self-driving vehicles” was recently published in Nature Communications.

The car does the driving. A person watches over it and steps in when it gets something wrong. The car takes that correction, learns from it, and handles the same situation better the next time. The more it learns, the less the human is needed.

“An apprentice does not relearn everything each time a new situation appears,” Prof Lyu says. “They build on what they already know, receive corrections when necessary, and gradually become more independent. That is very close to the learning process we are trying to reproduce in the vehicle.”

The car that travels badly

A car that drives well in one city can drive poorly in another, and its sensors are not to blame. The rules of the road are local.

Overtaking is on the right in one country and the left in another. When to yield, how much room to leave before changing lanes, how slowly to edge past a cyclist, all of it shifts with local law and local habit. A car can finish a manoeuvre without a scratch and still drive like a tourist.

Four everyday situations where driving is about more than avoiding a collision: yielding when turning, not blocking the lane being entered, giving way to pedestrians and cyclists, and overtaking on the side the local law requires.

Moving a self-driving system to a new place, Prof Lyu says, “is not simply changing the map.” Different roads, different regulations, different weather, different driving manners. Traditionally, adapting to all of it has meant gathering fresh data and retraining much of the system from scratch.

That cost is easy to underestimate. As driverless cars reach more cities around the world, the price of teaching each one a new place becomes one of the real barriers to getting them there at all.

A correction is not a failure

When a human takes over from a self-driving car, most systems treat it as a failure. The takeover is logged as a safety event and the car carries on unchanged, ready to make the same mistake again.

Prof Lyu’s team saw something else in that moment. A takeover is also a demonstration: the human is showing the car exactly what it should have done.

“If a human has already corrected the vehicle once, why should the vehicle make essentially the same mistake again?” Prof Lyu says. The whole method grows from that question, turning the takeovers that already happen into lessons the car keeps.

Prof Lyu points out a situation on the simulated street as Haohan Yang drives.

Learning without forgetting

Teaching the car something new is the easy half. The hard half is teaching it without washing away what it already knew.

Machine-learning systems are prone to this. Learn a new skill, and the new training can quietly overwrite the old, a problem known as catastrophic forgetting. For a car, it would be intolerable: getting better at a rare junction must not make it worse at everyday driving.

So the team built the car to hold two things at once, enough flexibility to absorb a new correction, and enough memory to protect the skills already there.

When the car’s own judgement and the human’s disagree, it does not copy the human blindly. It weighs the correction against what it already does well, and leans toward the human without unsettling the rest.

The proof on the road

Simulations are one thing. The sterner test was a real car on a real track, driving lap after lap while a human watched over it and stepped in where it erred.

Lap by lap, the corrections took. The car’s ethical risk, a measure that weighs how likely a collision is against how much harm it would cause, fell to about a tenth of where it started after only a few rounds of guidance.

The same test track before and after learning. Red marks stretches where the car’s driving carried high ethical risk, blue where it carried little. Across the starting policy and two rounds of human guidance, the track turns from mostly red to mostly blue, with average ethical risk falling from about 0.35 to 0.03.

The part that surprised them

The most striking result came in the rare cases, the odd and awkward situations a car meets least often and handles worst.

After the car learned from corrections in a handful of them, it improved in other rare cases it had never been shown.

“That suggested it was not simply memorising individual examples,” Prof Lyu says, “but beginning to capture something more general about good driving behaviour.”

Different situations, he says, come down to the same task: reach the destination safely, treat other road users properly, follow the rules. A lesson learned in one place carries to the next.

The team tests the system on a driving simulator, running the car through varied street scenes before trials on the road.

What it does not solve

Prof Lyu is careful about what the work does not do.

The car learns from people, and people are not always good drivers. “A human driver is not automatically correct simply because they are human,” he says.

A good correction is one where the car was wrong and the human was plainly safer. Telling the good corrections from the bad, every time, is still an open problem.

Then there are the rare emergencies no rule anticipates. Sometimes the safest thing is to break a rule, to swerve across a line and avoid a car coming the wrong way.

The system is built to drive well within the rules, not to make that leap. That will take a higher kind of reasoning than driving skill alone, and it is still ahead of the field.

Where it is going

What interests Prof Lyu most is a car that never stops learning.

Most self-driving cars stop learning the day they leave the lab. What they know is set, and it stays set. He wants to undo that, and build cars that keep learning from every new place and every correction across their working life, growing safer the more they drive.

“Future autonomous systems should not be viewed as finished products whose intelligence is frozen when they leave the laboratory,” he says.

The idea reaches past cars. A robot in a warehouse, a helper in the home, anything that works beside people could learn this same way. A system that keeps learning from the people around it, he believes, is what will let it earn a place out in the world.

About the research

The research was published in Nature Communications under the title “Human-guided continual learning for multifaceted improvement of self-driving vehicles.” It was supported by the Agency for Science, Technology and Research (A*STAR) and the Ministry of Education, Singapore, with collaborators at Chongqing University, which supported the test vehicle and field site. https://doi.org/10.1038/s41467-026-77370-x

About the researchers

Associate Professor Lyu Chen led the research. He is Director of the Automated Driving and Human-Machine System (AutoMan) Research Lab at NTU’s School of Mechanical and Aerospace Engineering, which he founded in 2018. He earned his PhD from Tsinghua University, with a joint PhD at the University of California, Berkeley, and was a research fellow at Cranfield University before joining NTU. His research focuses on autonomous driving, human-machine collaboration, and intelligent vehicles.
Dr Haohan Yang is a co-first author of the study and a postdoctoral fellow in the AutoMan Research Lab at NTU. He developed the method, designed the experiments, and led the writing of the paper. His research centres on human-guided and continual learning for autonomous driving.
Dr Yanxin Zhou is a co-first author of the study and a postdoctoral fellow in the AutoMan Research Lab at NTU. He led the real-world deployment of the system on the test vehicle and the field experiments. His research centres on learning-based decision-making for autonomous driving.

 

By Karen Chai, NTU MAE Communications