Gestures provide instant answers
Hand movements and facial expressions are a crucial component of communication
Judith Holler works at the Max Planck Institute for Psycholinguistics and Radboud University, both located in the Dutch city of Nijmegen. She researches the use of gestures and their significance for communication. In a new study, she has investigated the role of gestures in answering questions.
Ms Holler, I invited you for a video interview because I prefer to see my conversation partner, rather than only being able to hear them. Why is that?
Quite simply, it comes down to the fact that our language developed through direct communication, face-to-face. In the past, telephones and the like simply didn’t exist. As a result, even today we use our face, eyes, hands, indeed our whole body to communicate. Gestures and facial expression are unbelievably important.
So how much information do gestures contain?
That remains unclear. We already know that hand gestures, particularly those we produce while speaking, convey a large part of the message. It’s hard to measure exactly how much information this represents. Various studies suggest that gestures often convey up to 50% of a statement, if not more. Of course, this depends heavily on the conversation topic: when we speak about spatial matters, gestures play a particularly important role.
Many people also employ body language when speaking on the telephone. They know that their conversation partner can’t see them – and yet, they wave their arms, nod their head or roll their eyes. What effect does this have?
Firstly, gestures appear to have evolved into a fixed component in our communication. Then again, hand gestures also help the speaker in speech production. Very consciously, try to suppress all gestures and body movements while you’re speaking – it’s actually very difficult. One study has even shown that we speak less fluently when we are prevented from gesticulating, and that our language then becomes less vivid.
So, is it also more difficult to understand language without gestures?
That isn’t totally clear. We know that our brain integrates information from languages and gestures when listening. This occurs in a fast process, at least in an experimental context, where we have to limit ourselves to understanding individual words or very short sentences. And, in natural conversation, everything becomes even more complex. It could be that it requires a great deal of effort from our brains to integrate and analyze the different visual and acoustic signals in a short time.
In your current study, you focused on this very question: What do gestures achieve in conversation?
In part, at least. We analyzed conversations between friends. More specifically, we’re interested in the instances where one person asks something and another person answers. The interval between question and answer is extremely short, no more than 200 milliseconds. In comparison: If you place a picture of a dog in front of a person and ask them what they can see, it takes them around 600 milliseconds before they say the word “dog”. Other scientists have shown this in studies.
So we’re faster when conversing?
Yes, significantly faster, despite the fact our brains have much more to do at the same time! They have to listen, understand and plan an answer. That means that much of this appears to happen in parallel, and initial studies at our Institute suggest that that is indeed the case: when I’m listening, I’m also already planning my answer.
Do gestures also play a role in this?
That was a mystery for us, and is why we have investigated the conversations in such detail. We filmed the conversations from different perspectives using six video cameras so that we could accurately analyze the gestures. We wanted to uncover whether and how the temporal coordination between question and answer interacts. Certain structures in spoken language convey to my counterpart that I’m going to finish speaking soon, such as intonation. That fact has long been known. Yet gestures also appear to play a role. The answers to questions that are contained in head or hand gestures were obvious more quickly. Often, there was barely a pause between question and answer.
Are there certain gestures where this effect was particularly pronounced?
Yes, we were able to demonstrate this for hand gestures with return phases. These are gestures where I make a movement with my hand and then return my hand to its resting position. For instance, when I gesticulate while talking and then stop doing so but continue to speak, this can signal that I am approaching the end of my sentence. This can help my conversation partner to prepare their contribution while I’m still speaking.
Are other gestures also effective?
Other hand and head gestures appear to help us to reply faster. The underlying mechanism for this is difficult to pinpoint at present. Perhaps these gestures convey additional information that makes it easier for us to understand the question. However, to date we have only completed the observational study. Our current target is to test these results in experiments.
How might that work?
We produced video recordings of the conversations. We want to select a few questions and play them to test subjects to measure the cognitive processing of different questions.
Your trial participants were all native English speakers. They are hardly known for their lively use of gestures. The first to come to mind – though a cliché – would be Italians. Could your results be transferred for other languages and cultural groups?
Naturally, different cultures often have an entire repertoire of gestures of their own. The extent to which cultures differ, above all in relation to gestures and temporal coordination in conversation, is something we don’t yet know. We’re extremely interested in such cultural comparisons, and we want to examine that more closely in future. For example, we are already starting to work with recordings of Dutch and Italian conversations. However, it’s very time-consuming, and we’re only in the early stages.
Do children also have to learn gestures when acquiring language?
Children start to gesticulate very early on. They often use gestures even before they start to speak. Having said this, we still know relatively little about when and how children understand gestures. Can a seven-year-old child already combine the meaning of hand gestures and language? That’s another topic that we’re researching at our Institute, and we’re excited to see the results.
Interview: Claudia Doyle