In order to bring machine learning a step forward it is important to fully understand the social components of human interaction. “For instance, a waiter knows that he needs to walk around a highly engaged group of friends in a bar. A robot waiter, however, will opt for the shortest route available and is most likely to disturb a fun night out,” Laura explains. “In order to help machines better understand complex human behaviour, we need to feed them with examples of real human behaviour gathered from humans interacting ‘in-the-wild’.”
Non-verbal cues
“Humans tend to be very multimodal. We speak but we also use nonverbal communication in different channels. Each of these modalities may not be enough on its own, or they may complement each other. The development of systems with true social intelligence – that can take into account non-verbal cues – still presents us with several complex challenges”, Laura explains. “Humans can understand when another human is upset – these are intrinsic components that we as humans already know – but it is difficult to determine how exactly we reach that conclusion. Social interactions are made up of different elements occurring in parallel, which makes it difficult to analyse their components individually. And most human social interactions occur during free-standing conversations ‘in-the-wild’.”
In-the-wild
Laura analysed humans while speed dating and socialising at events using multimodal streams. “The research analysed humans in their natural settings, in real-life situations. We were able to study the subjects without disturbing them by using wearable devices to register proximity and movement. Information from the sensors was used to try to detect the type of personality that people have, who was speaking and who was quiet, who was gesturing and who wasn’t.”
High-quality dataset
As part of her research, a totally new dataset was created to study social interactions. “In our field people tend to take for granted that datasets exist. But if your particular problem does not fit that dataset then there is not much you can do.” This new dataset enabled researchers to study human behaviour during natural conversations without disturbing the subjects. “It was hard work with many challenges that needed to be overcome, but we ended up with a high-quality dataset that is now publicly available at https://tinyurl.com/matchnmingle.”
