Chatting with family on the other side of the world via a hologram or selecting your own camera angles for your favourite club’s football match. TU Delft computer scientist Pablo Cesar is researching the best ways of transmitting and receiving information to make these kinds of applications possible. He was recently awarded the Netherlands Prize for ICT Research 2020.
At first glance, the room where Pablo Cesar works looks like a totally normal office complex, with carpeting, system ceilings and a pinboard with posters about academic events. Behind the doors, you expect to see desks with computers on which mathematicians are doing calculations. But when Cesar opens the door, you walk into a cosy traditional Dutch living room, one your grandmother would feel at home in. There is a photo of the king next to a cuckoo clock, and a Persian rug on the floor.
But this is not where the scientists drink coffee or relax watching television. ‘This is one of our research rooms,’ says Cesar. He points towards an antique lamp. ‘It's fitted with a microphone, so we can record conversations. There are cameras on the walls. Next door is a computer room where we decide what people can see and observe how they watch TV.’
Prestigious prize
The lab disguised as grandmother’s living room provides an indication of how Cesar and his colleagues approach their work. They like to conduct research in as realistic a setting as possible. They use this method to look at a wide range of subjects. For example, Cesar is investigating how we watch TV, he aims to improve teleconferencing and wants to enable hospitals and ill people to communicate with each other.
The computer scientist is an associate professor in TU Delft's Multimedia Computing group and the national research institute for mathematics and computer science Centrum Wiskunde & Informatica (CWI) in Amsterdam. For his contribution to science, he was recently awarded the Netherlands Prize for ICT Research 2020. In its report, the judging panel highlighted the fact that Cesar's work focuses on fundamental problems in human/technology interaction and how these can be modelled. They also praised Cesar for conducting his experiments in a realistic context, rather than a lab setting. ‘We prefer to conduct our research in the wild,’ says Cesar. ‘We’re developing technology for people. That means that we have to get them to test what we come up with in order to improve our ideas.’
We prefer to conduct our research in the wild.
A good example of his approach is the British FA Cup final. Cesar was asked by the BBC and British Telecom to design a second screen for people watching the final. He came up with a menu that allows you to control all kinds of things, including the camera angle, extra action replays or the colours around the screen. ‘It’s normally the programme director that determines what you see at home. This gives the viewer more freedom. The director is still in control, as is appropriate in view of his or her experience. But he or she gives the viewer multiple options to choose from.’
This made it possible to show different things on several screens in a pub full of Chelsea supporters. One TV showed images of the match whereas another just had a close-up of the coach. The sound came directly from the Chelsea supporters’ stand, so their chanting could be heard in the pub. The result was much more tailored to the specific target group in the pub.
It’s normally the programme director that determines what you see at home. This gives the viewer more freedom.
This approach may give viewers more options, but what difference does the science make here? ‘Our primary interest was in the how you transmit all that information at a superfast speed and show it on the screens. Pictures of the match already need to be transmitted to the TV in the pub or at home at amazing speed. In this final, we had to send a lot more information, such as several camera angles and different sound.’
That involves transmitting an awful lot of information simultaneously, because you also want to avoid any delay in the match you’re watching. Cesar is developing ways of transmitting the information as smartly as possible. To achieve this, he uses algorithms that select the relevant information in a smart way. This data only is all put in small packets in a process known as compression – almost like creating a zip file on your computer. The files are pressed together, reducing the space they take up and enabling them to be transmitted faster and more easily. The files are then unpacked at the end user where they are converted to make images or sound. Algorithms also play an important role here. Cesar is investigating how all of this can be done as effectively and quickly as possible. In the process, he always has the viewer’s quality of experience in mind. What compression results in the best image quality, how much image delay is permitted without undermining the connection and what bandwidth is required?
Communicating with family
Cesar is not only working on watching TV and second screens. He’s also researching communication between people. He walks into a room full of boxes. They have been painted pink on one side. There are cameras around them and a researcher is walking round with his arms spread out. Cesar points to a computer screen showing an image of the walking man.
Imagine you want to discuss something important with a family member on the other side of the world. It’s now already possible to telephone them or use Skype. But using the technology that Cesar is now working on, you can create a hologram of someone that makes it look as if he or she is standing in front of you. He is working on this in this room.
‘We create a 3D image of someone. The result is a large file that we compress and decompress when it reaches the receiver. It’s similar to what we did for the FA Cup. But what’s the most practical approach? If I want to talk to a family member in Spain, where I come from, they don’t need to see my back. That means that we don’t have to send all of the images. These are very large files, so we’re investigating how we can make them smaller by removing useless noise.’
The image quality required also plays a role. Displaying someone’s face in high definition and the rest of the body less clearly can already make a big difference. Cesar and his colleagues are testing people’s responses to this. ‘We’re always trying to work out which files are important to send and receive and which are not.’
This up-and-coming technology is not only useful for phoning your family. It can also be used for teleconferences with businesses, or in hospitals. ‘For example, if you need a knee operation, you can have the necessary appointments for tests in the hospital. Any other appointments can be done by a tele-connection, with the doctor receiving a very high definition image of the knee. This will also help the advance of healthcare in the future.’
Disco of the future
But it's not only healthcare – even a disco can benefit from Cesar's research. Several years ago, Cesar was asked to work on a project at the Amsterdam Dance Event (ADE). One of the sponsors asked him to make the club of the future possible. Cesar therefore joined forces with clothing designer Borre Akkersdijk to come up with armbands packed with sensors that checked people’s temperature, location and movements. This information was then transmitted wirelessly to minicomputers on the wall. Some 450 people attended. ‘That’s an amazing number for a scientific study,’ says Cesar. ‘We used machines with artificial intelligence that had to understand where people were and whether they were dancing. We then wanted to share that information with the people at the event. For that, we created a second room that was the opposite of the dance floor.’
During the Amsterdam Dance Event (ADE), Pablo Cesar researched the future of club culture together with textile innovator BYBORRE. Cesar developed a wristband, equipped with a sensor. The sensor makes it possible to analyse where people are, dance, walk and even the way people move.
We used machines with artificial intelligence that had to understand where people were and whether they were dancing.
When the feet really hit the floor, it was very relaxed in the second room. If everyone on the dance floor was acting like a wallflower, the second room was incredibly hectic. ‘By doing this, we learnt how to process a lot of information rapidly and also how to display it immediately to visitors. The result was not only a great disco, but something that we, as scientists, could also learn from.’ In scientific research into what is known as affective computing, there is a lot of focus on such issues as the recognition of emotions by software and investigating audience engagement.
In these kinds of projects, the researchers make a lot of things themselves. This not only includes the software, such as algorithms to do the calculations, but also the sensors built into the armband. Cesar reveals a room full of equipment where there is also a workbench on which all of the hardware is made or adapted.
This is not pure chance. Cesar and his colleagues are interested in the entire chain across which information is transmitted and received. It’s an approach he believes in. ‘We’re living in an age when many scientists are focusing on a very small thing or subject. We look at things from a broader perspective and always focus specifically on how information reaches people. This has been relatively ignored in the scientific world, but our work at TU Delft and CWI is fortunately now changing that. Winning the ICT prize in 2020 will certainly help. I hope that the attention it attracts will focus especially on the development of computer science aimed at end users and that we will be able to continue to investigate that even more effectively.’
Text: Robert Visscher | Portrait photo: Mark Prins
