Michael Johnston
Multimodal systems support more natural and effective human-computer interaction by providing multiple different input and output modes such as speech, pen, hand gesture, gaze, and dynamic graphical displays. Their critical advantage is enabling user input and system output to be expressed in the mode or modes to which they are best suited, given the task at hand, user preferences, and the physical and social environment of the interaction.
This tutorial provides an overview of key issues in the design, implementation, and evaluation of multimodal user interfaces and examines their current and potential applications in mobile computing, information access, interactive entertainment, and the arts. The underlying processing architecture of a sample multimodal system will be presented along with detailed examples showing how representations and techniques from natural language and speech processing can be extended and applied to the parsing, integration, and understanding of multimodal inputs and the generation and presentation of coordinated multimodal output. The tutorial will be introductory in nature and no special knowledge or background is required.
Materials
