1) Large non-verbal Language Models (LnvLM) for human-agent interaction and cooperation, at AIR-Lab, Tilburg University,
(with Giacomo Spigler).
When we think of communication, our first thought is language. Following this intuition, a vast amount of research in robotics (in particular, human-robot interaction (HRI)) has been dedicated to modeling spoken language. Nonetheless, a significant portion of information exchanged in human-human communication is non-verbal. Despite its importance, current non-verbal behavior by robots in HRI settings is either manually pre-programmed, or controlled by humans via teleoperation.
The objective of the project is to build on recent advances in language modeling and artificial intelligence to create large language models for non-verbal communication, including gestures, eye gaze, facial expressions, proxemics, etc. The developed models will be deployed on various robot platforms to test ecological validity of the proposed approach. Part of the validation will involve experiments with Reinforcement Learning from Human Feedback (RLHF), relying on the trained embeddings of non-verbal communication to both specify tasks and to extract reward signals.
During my postdoc period, I contributed to the following projects:
1) Multimodal Interaction and Communication, at the cluster of Excellence, Science of Intelligence
"The overall goal of this project is to create a robot that can represent and integrate information from different sources and modalities for successful, task-oriented interactions with other agents. To fully understand the mechanisms of social interaction and communication in humans and to replicate this complex human skill in technological artifacts, we must provide effective means of knowledge transfer between agents. The first step of this project is therefore to describe core components and determinants of communicative behavior including joint attention, partner co-representation, information processing from different modalities and the role of motivation and personal relevance (Kaplan, and Hafner, 2006; Kuhlen & Abdel Rahman, 2017; Kuhlen et al., 2017). We will compare these functions in human-human, human-robot, and robot-robot interactions to identify commonalities and differences. This comparison will also consider the role of different presumed partner attributes (e.g., a robot described as “social” or “intelligent”). We will conduct behavioral, electrophysiological, and fMRI experiments to describe the microstructure of communicative behavior."
The second step of the project is to create predictive models for multimodal communication that can account for these psychological findings in humans. Both the prerequisites and factors acting as priors will be identified, and suitable computational models will be developed that can represent multimodal sensory features in an abstract but biologically inspired way (suitable for extracting principles of intelligence; Schillaci et al., 2013). In perspective, the third step of this project is to use these models to generate novel predictions of social behavior in humans.
Throughout the project we will focus on the processing of complex multimodal information, a central characteristic of social interactions, that have nevertheless thus far been investigated mostly within modalities. We assume that multimodal information, e.g. from auditory (speech) and visual (face, eye gaze) or tactile (touch) information, will augment the partner co-representation and will therefore improve communicative behavior.
2) From understanding learners’ adaptive motivation and emotion to designing social learning companions, at the cluster of Excellence, Science of Intelligence
"In this threefold project, we will firstly examine how novel user modelling approaches and feedback strategies in Intelligent Tutoring Systems (ITS) incorporating virtual agents can enhance positive emotions and motivation (self-regulation, goal orientations) and reduce negative emotions in social learning situations and how these approaches can be used to prevent inequalities in education.
We will also be exploring the (moderating and mediating) processes that underlie the relations between pedagogical agents’ ‘behaviors’ and learners’ performance by investigating psychological factors that strengthen or reduce the effects of ITS on learners’ motivation and emotions.
As a third and final objective, we intend to create a robotic learning companion that keeps an updated model/simulation of the learner and their current knowledge, motivational, and emotional state, acting accordingly."
During my PhD period, I contributed to the following projects:
1) The Neurorobotics Platform of Human Brain Project, at the BioRobotics Institute of Scuola Superiore Sant'Anna
"Neurorobotics is about building and simulating robot bodies with an embedded brain and embedded control systems that mimick the structure and function of the nervous system. This leads to an unprecedented opportunity to perform cognitive experiments in silico that bridge the gap between the real and the virtual world.
With the Neurorobotics Platform, researchers from neuroscience, robotics and machine learning can collaboratively design and run virtual closed-loop experiments while relying on a tight integration with other EBRAINS tools and methods."