Mind, hands, face and body

Mehr zum Buch

Communication is inherently multimodal, involving not just words but also gestures, facial expressions, gaze, and body movements. These modalities interact in intricate ways, prompting the question: can we analyze them individually? This text proposes that communication scholars can create a "musical score" for this symphony by assigning specific meanings to each signal and developing lexicons for various communicative modalities. While linguists have long compiled dictionaries for verbal languages, there is a call to establish new dictionaries for gestures, gaze, and touch. The first section (Mind) introduces a cognitive model of communication based on goals and beliefs. The following sections (Hands and Face) provide in-depth analyses of gestural and facial communication, highlighting both universal and cultural aspects. They explore the distinction between codified gestures and spontaneous ones, offering insights on creating a dictionary for touch and interpreting eyebrow movements. The final section (Body) presents an annotation scheme for transcribing and analyzing signals across modalities, demonstrating its utility in empirical research and simulations with Embodied Conversational Agents. This tool is applied to various multimodal discourses, revealing how the body can convey indirect messages, deception, irony, and more. Prof. Dr. Isabella Poggi, an expert in psychology and communication, contributes to this field th