3D modelling of cities has come a long way. You have probably noticed the change in, for example, Google Earth: zooming in on an urban scene often allows you to ‘fly’ between the buildings. It looks impressive, but something vital is missing. If you want a computer to perform calculations and make predictions, it needs to know the properties of each part of the model. Weixiao Gao spent his PhD working on this problem, resulting in some of the most advanced labelling algorithms in academia.
Why are labels so important in 3D city models? Weixiao explains: “With labels, a computer is better at performing urban spatial analysis. Like detecting the construction and demolition of buildings, or the adding or removing of trees. It also enables much more accurate navigation: for people, but also for robots and drones! Computers can calculate volumes of structures, measure window area, simulate microclimates… all vital for environmental impact. And during conflicts or disasters, these programmes can automatically assess the damage, speeding up reparation efforts.”
The allure of 3D virtual worlds
Weixiao grew up in a small town in Northern China. In the evenings after primary school, he became a big fan of 3D video games like Grand Theft Auto: “That is probably where my interest in virtual worlds started.” Like many gamers, he initially wanted to study computer science. However, he ultimately pursued geoinformation science, a field focused on creating and utilizing maps derived from the real world. “I kept thinking: where is the third dimension? After all, we live in 3D!” So when the opportunity arose, he moved to Delft to work on 3D city models.
Airplanes above Helsinki
The type of 3D model Weixiao works on is called a ‘textured mesh’. The data are sourced from images, which can be taken by satellites, airplanes, or even mobile phones. “In my case, data is derived from airplane images of Helsinki, which is one of the few places with a public database.” The computer identifies some key points in the input images and calculates their 3D coordinates. It then creates a ‘field of triangles’, the mesh, in the shape of the urban scene. The so-called texture consists of cropped images from the original source data which are used to fill in these triangles.
Deep Learning: a false friend?
At the start of his PhD, Weixiao tried manually labelling his 3D models. “It took me 300 hours to label 1 km2. The next step was obvious: efficiency.” That is when Deep Learning took the world by storm. To Weixiao, it seemed the perfect solution. But as he tried and tried again, he realised that full automation was nowhere near feasible. “A.I. is never 100% correct. In fact, current systems for automatic labelling only manage around from 50% to 70% accuracy, which means endless manual corrections.” Instead, Weixiao incorporated A.I. in labelling tools, which massively sped up not only the labelling process but also data generation and error correction.
Milestones in the field
After a long and rocky road, Weixiao is proud of the fruits of his labour. His programme allows detailed labelling of an urban textured mesh: “Not just objects, like ground and building, but also parts, like windows, doors, and road markings. It is one of the first research outputs in the world to allow for this level of detail.” Not only that, Weixiao also created a separate algorithm able to convert the textured mesh into 3D polygon models which are watertight: essential for calculating properties like volume. “As far as I know, this is the first data-driven algorithm which can automatically reconstruct detailed and labelled 3D city models. It is fast, accurate, and allows for 3D printing!”
Applications and collaborations
As you might expect from his enthusiasm, Weixiao plans to stay in academia: “I want to continue pushing this field in the right direction. And I want to explore possible applications for these advancements: virtual worldbuilding, advanced urban planning, calculating impacts, maybe even simulating disasters like floods or storms or earthquakes.” Which leads to his key message: “The field of Urban Data Science is too isolated! We could contribute to almost any research group in BK. Imagine what, for example, Landscape Architecture could do with advanced 3D models.” To the BK community: are you ready to see your work in a virtual world?
Published: October 2024
More information
- Weixiao Gao will defend his PhD thesis on Monday 4 November at 12:30.
- Read Weixiao’s full thesis, titled ‘Semantic understanding of urban scenes from textured meshes’.
- All images in this article are made by and owned by Weixiao Gao.
