Crisscrossing cyclists weaving through streets are a major challenge for self-driving cars. Assistant professor of robotics Holger Caesar is teaching autonomous cars to ‘see’ cyclists and predict their behaviour. To do this, he needs a lot of data, which he collects the ‘Dutch way’.
Assistant professor Holger Caesar might be the only person on campus who lights up when the steep bridge between the centre of Delft and the university campus opens. Dozens of students gather at the barrier as a meter-long ship slowly passes by. Cyclists wait impatiently for their turn to cross. As soon as the road is clear, chaos ensues. Students scramble to get back on their bikes all at once, pressing their pedals to avoid rolling backward as they all cross the bridge.
Why is Caesar fascinated by this chaotic scene? For him, this traffic situation is a rich source of data on cyclists’ behaviour. Who chooses which route to weave through slower cyclists on the bridge? ‘Until now, very little data has been available on cyclists,’ he says. He is using the cyclists’ data to teach self-driving cars how to deal with cyclists in traffic.
He brings the datasets he develops to the attention of other researchers. ‘There’s a lot of interest in this data. With a good public dataset, we can explore many different research questions,’ he says. His efforts are already making a significant impact. One of his earlier datasets, nuScenes, maps two entire cities in detail. This dataset is widely used by researchers to test new software for self-driving cars.
Teaching cars to see in the rain
The fact that self-driving cars cannot yet properly anticipate cyclists is partly because the cars were developed in America and Asia, says Caesar. ‘Research into self-driving cars started in a desert in Arizona. That’s not representative of many places in the world,’ he says. There are no cyclists or traffic lights there, and the sun is always shining.
Observing cyclists is also a challenge. ‘Cyclists are largely transparent. Think of the space between the spokes,’ he explains. And their behaviour is difficult to predict, says Caesar: ‘Cars follow lanes, cyclists move much more freely.’ The movements they make are also different. ‘They lean back and forth. For example, when a cyclist makes a turn, he first moves in the opposite direction.’ This also needs to be taken into account for safe interaction in traffic.
Cyclists are largely transparent. Think of the space between the spokes
Holger Caesar
In order to teach cars to see cyclists, Caesar first had to think about what kind of ‘eyes’ the car should have. It was important that the vehicle could withstand our wet autumn months. He found the solution in a radar under the bonnet. ‘Unlike cameras, radar can see through the bonnet and through the rain.’ In a dedicated testing area next to the lab, he carried out several test runs. He placed pedestrian ‘Hans’ there, a soft doll that moves back and forth on wheels. Using the radar, the car was able to drive itself around the car park in the rain without endangering Hans.
Research, ‘Dutch style’
Now that the car could see, he needed to teach it what a cyclist looks like and how to behave in traffic. To do this, he needed good examples. But how do you get them? There are few cameras on the roads that focus on cyclists. And a car with sensors to detect cyclists cannot just drive onto the cycle path. Caesar found the solution in – you guessed it – a bicycle.
In the robotics lab, Caesar walks over to the test bike, waving as he approaches. On a computer screen next to the bike, his waving is instantly displayed as a colourful collection of points, showing the laser and radar data in three dimensions. With two lasers and protruding antennas on the back and a third laser on the front, the bike does not go unnoticed in traffic. ‘It’s fun to ride the bike across campus, people look at you and ask what you’re doing,’ he says.
From laser image to dataset
After pedalling around Delft on the test bike for hours, you still do not have a usable dataset. To make the data usable, it needs to be ‘labelled’. Labelling involves linking everything visible in the images to a description of what it is – for example, ‘tree’, ‘cyclist’ and ‘traffic light’. This allows the car to recognise a cyclist when it sees one. Right now, this is often done manually, which is a lot of work. A similar approach is used in the ‘I am not a robot’ pictures in CAPTCHAs, where you are asked to identify squares containing scooters or traffic lights to prove that you are not a robot.
You could develop an application that alerts car drivers when a cyclist makes an unexpected move
Holger Caesar
Caesar wants to automate the labelling process. To do this, he and his team are developing computer algorithms, which he shows on his computer screen. ‘You can see here that everything the algorithm recognises is given a colour,’ he explains. For example, the trees are green and the buildings are orange. The algorithm also predicts the direction in which a road user will move. The most likely location is shown as a green spot, while something less likely is coloured orange.
From Delft to Rotterdam
Caesar envisions many applications for the cyclist dataset. ‘You could develop an application that alerts car drivers when a cyclist makes an unexpected move,’ he says. This could increase road safety for cyclists. He also sees opportunities for collaboration with his colleagues. ‘There are researchers here who are developing bikes that can’t fall over, and they’re very interested in the datasets we’re developing,’ he adds.
After graduate students mapped the centre of Delft and explored the campus on the bike, Caesar now has bigger plans as a cycling enthusiast. Previously, he cycled from Vietnam to Thailand and from Germany to Italy in his free time. He now wants to cover similar distances with the sensor-equipped bike. ‘I want to ride the test bike from Delft to Rotterdam first, and later also to Germany and Belgium,’ he says with a big smile. This will allow him to expand his dataset even further, unlocking new research opportunities for his own team and many other researchers. 'Someday other researchers and entrepreneurs will use these seeds to create numerous useful inventions that will make cycling safer for everyone,’ he concludes.
