Context
As self-driving cars (SDCs) are placed in the real world, they need to understand specific real-world environments. But what exactly do they need to understand — and what obstacles will they encounter?
SDCs have been known to fail to recognise their surroundings in ways that baffle most humans: misidentifying kangaroos, traffic signs, and people with darker skin tones. In some cases these failures are hard to foresee. In many cases, however, helping data science teams better understand the environments in which their cars will actually drive can solve problems early — before resources are spent or damage is done.
Exploring six cities across the E.U. — Berlin, Vilnius, Riga, Zagreb, Ljubljana, and Lisbon — we mapped a wide range of unique obstacles that self-driving cars may face in each. Below, we describe two of our findings in detail.
Findings: Lisbon
Calçada portuguesa and pedestrian motion trajectories
Calçada portuguesa — Portuguese pavement — is a distinctive form of mosaic stonework found throughout Portugal. Beautiful and characteristic of Lisbon, it also creates significant difficulties for those walking on it. And because it affects how people walk, it directly impacts the pedestrian behaviour that self-driving cars must predict.
Three compounding issues define the problem. First, the stones become extremely slippery when wet — polished by years of foot traffic, they form ideal conditions for slipping in rain. Second, the pavement is difficult to maintain: the specialty craftsmen trained to lay it are increasingly rare, meaning holes and loose stones are common and widespread. Third, Lisbon is a hilly city with many narrow sidewalks, making all of the above more hazardous.
These conditions produce three distinct pedestrian behaviours that SDCs must be prepared to handle: tripping, sliding and falling; walking on the road instead of the sidewalk (especially during rain); and kicking loose stones, which can become unpredictable projectiles.
Importantly, the impact of calçada portuguesa is easy to miss during development and testing. Holes are not everywhere, rain is seasonal, and narrow sidewalks are not universal. Direct observation of the resulting behaviours is unlikely unless one actively looks for them — and pedestrian accidents in these contexts are severely under-reported.
Recommendations for Lisbon
- Train models to distinguish between calçada portuguesa and other pavement types. Simply recognising it could already signal SDCs to drive more cautiously.
- During rainy weather and near narrow sidewalks, expect pedestrians to walk on the road and take extra precautions for detecting them and calculating their motion trajectories.
- Prepare SDCs for a higher likelihood of pedestrians tripping, slipping, and falling — and consider detecting pavement holes as a proxy for high-risk zones.
Findings: Zagreb
Advertisements on trams, buses and walls
Zagreb’s 19 tram lines are a central feature of the city’s transport network. Most trams display advertisements across their sides — ranging from simple typography to full-bleed imagery of people and scenes. Similar advertising is found on walls close to roads throughout the city centre.
The model we tested (MSeg1080_RVC segmentation) repeatedly failed to understand that the content of these advertisements was not real. A tram carrying an advertisement was partially recognised as “train” at the front, while the advertisement-covered body was labelled “fence”. Billboards displaying images of cars were recognised as actual cars. People depicted in advertisements were labelled as real people.
The model performs significantly better when the surroundings of an advertisement are visible — allowing context to clarify that it is a billboard rather than its content. In reality, however, trams with full-side advertisements narrowly pass other vehicles, and billboards are often placed on walls directly next to roads, meaning the surrounding context is rarely available to the camera.
The model often fails to distinguish between reality and fiction — recognising advertised cars as real cars, and people in advertisements as real pedestrians.
Recommendations for Zagreb
- Train the model to recognise visual advertisement styles — large slogans, high-gloss finishes, and the characteristic aesthetic of advertising imagery can serve as contextual clues that content is fictional.
- Leverage motion: as an SDC approaches a billboard or advertising tram, earlier frames with visible surroundings can inform later, closer frames where context disappears. Combining temporal awareness with style recognition is the most robust approach.