ColloquiaFrancis Quek on Thursday, May 2nd

[Margery Ishmael]

Thursday, May 2, 2002
3:00 pm
Ryerson 276 (annex)


FRANCIS QUEK, Associate Professor
Vision Interfaces & System Laboratory
Computer Science & Engineering Department
Wright State University

"Gesture & Discourse: The Signal and Sense of Computational Multimodal 
Language Analysis"

Human intelligence is embodied.  It follows that human language and 
expression are embodied as well.  Discourse is a dynamic process of 
converting thoughts into speech, gesture, and gaze activity.  Grounded on 
the psycholinguistic foundations of the production of such multimodal 
"conversational-acts", we address the interpretation of gesture and speech 
in the context of discourse management.

Our analysis is based on a psycholinguistic device known as a "Catchment". 
This unifying concept between gesture and speech posits that the recurrence 
and flow of the cognitive units underlying discourse results in 
comprehensible structure in the speech content and gestural features.  The 
understanding of how this structure may be extracted from video and audio 
signal, and the determination of the kinds of computable cues that support 
such analysis are the first steps toward the bridging of the signal-sense 
gap in multi-modal interaction.  We set forth the broad picture of our 
interdisciplinary effort, and proceed to bring the concept into focus with 
concrete examples of cues that are computationally accessible from 
video/audio analysis, and demonstrate their efficacy in discourse structure 
recovery.

Our approach involves experiments designed to discover and quantify cues in 
the various modalities and their relation to discourse management, and the 
development of computational algorithms to detect and recognize such 
cues.  We investigate the cues afforded by each mode of interaction (in 
isolation and in tandem); study the spatial and temporal relationships 
among these cues and associate them with topical units in discourse; and 
present a multimedia database system that integrates these elements into a 
coherent whole.  Cues studied thus far include effort, holds and 
handedness, hand symmetries, gaze shifts, spatial anchoring and deictic 
origos, cross-modal temporal integration, "hold tension releases", and 
periodic hand oscillations.

Finally, time permitting, we shall provide further motivation for our 
approach by showing how such discourse analysis may be applied to study the 
communicative deficits in Parkinson's Disease.
http://vislab.cs.wright.edu/~quek/

Host: Stuart Kurtz

*The talk will be followed by refreshments in Ryerson 255*
If you wish to meet the speaker, please send e-mail to marge at cs.uchicago.edu

Persons who need assistance should call 773.834.8977

=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=
Margery Ishmael
Secretary to the Chairman, Department of Computer Science
The University of Chicago
tel. 773.834.8977  fax. 773.702.8487
=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=