A Study of the Interaction Between Hand Gestures and Referring Expressions in Object-referring Practice

The interaction between modes is an important research topic of multimodality studies.This study intends to systematically investigate the patterns of interaction between hand gestures and referring expressions utilized in object-referring practice.As the first step of investigation, four units for segmenting the multimodal data of referring practice are proposed, namely, referential context, referential activity, referential task and referential means.Then, with instances from the self-built corpus, this study analyzes the patterns of interaction between the two modes in terms of form interaction and role interaction.The form interaction between the two modes include four patterns:(1)the interaction between one hand gesture and one referring expression;(2) the interaction between one hand gesture and two or more referring expressions;(3)the interaction between two or more hand gestures and one referring expression; and (4) the interaction between two or more hand gestures and two or more referring expressions.The role interaction between the two modes include three patterns:(1) hand gesture in primary position while referring expression in secondary position; (2) hand gesture in secondary position while referring expression in primary position; (3) hand gesture and referring expression in equal positions.