ERC for AI that can judge human social intentions in complex situations

On January 31 it was announced that Hayley Hung received an ERC Consolidator grant for her new project NEON (Nonverbal Multimodal Social Intention Modelling), worth nearly 2 million euros. With it she and her team aim to develop an AI (Artificial Intelligence) system that can judge people's intentions in social interactions. While a proof of concept at first, Hung and her team hope that their work contributes to the ability of AI to understand the complexity of human socio-cognitive processes by observing their behaviour.

With NEON, Hayley Hung and her team aim to tackle a complex problem in the field of artificial intelligence: understanding human intentions. Traditionally, those AI systems are trained to judge situations using clear, mutually exclusive labels. The assumption is that in each situation, there is 'one real truth' that can be derived using those definitive labels. However, when it comes to understanding human intentions, this is hardly ever the case. In reality, different people judge other people's intentions differently. Hung aims to combine these different perspectives into one single system. In this, it also immediately shows its value: by combining and interpreting these different human perspectives, we can learn to understand each other better.

Take the following simple example: you have footage of person A reaching for a glass of water, but at the very last moment the glass is snatched away by person B. 'Traditional' AI would say that it was not the intention of person A to grab the glass. Because their end result was not achieved, it could not have been the intention of person A to grab the glass. It takes human empathy to correctly describe the situation, not only correctly estimating the intentions of person A, but also their intentions were not realised because of a competing goal of person B.

"What makes us humans so socially capable, are our unique and individual empathetic abilities. All of our personal experiences bring some unique insight to the table: like a lens that is particularly adept at creating a crystal-clear image, but only at a specific focal length." The AI that Hung and her team will develop will thus be of a layered construction, judging a social scene simultaneously using different perspectives derived from different traits. All those layers combined, will help AI give a more complete and nuanced judgement, instead of naively looking for 'one real truth'.

To construct the different 'trait-based personalisation layers', Hung and her team will actively collaborate with scientists from different fields, such as Conversational Analysis and Organisational Psychology.

The potential applications of this technology are vast, but one of the more immediate applications is to use the AI in coaching meetings or round table discussions. "We think that one the reasons women are less vocal in meetings, might be because they are not able to speak up. You might see, for example, that a female member of a discussion changes her posture in preparation to speak, but at the very last moments decides not to because another member of the speaks louder and takes the turn. We hope that our AI can coach in those cases." In the case of the example: next meeting the members of the discussion could make a more conscious effort enabling the woman from the example to speak.

Hung and her team follow in a tradition at TU Delft to not only make our Digital Society better, but also more responsible – and in a sense even, more humane. Hayley Hung: "I hope that technologies like NEON will help us understand each other better."