ERCIM News No.46, July 2001 [contents]

Multimodal Interfaces for Multimedia Applications: Consequences on the Interface Design

by Laurent Romary and Nadia Bellalem

Under the auspices of the European project MIAMM, two teams of INRIA collaborate to define new types of multimodal interactions involving strong language recognition and understanding capacities.

Considering the ever growing size of information available on line, coupled with the wider variety of potential users which are likely to demand some kind of access to it without necessarily being computer experts, it is mandatory to provide access means which are basically based upon human spontaneous communication abilities. Such interfaces must obviously incorporate dialogue component based on speech and gesture, without losing fundamental qualities like simplicity, efficiency, speed and robustness. Conversely, the system must be able to answer either in natural language or by means of an adequate presentation of available information in a graphical way, probably coupled with force feedback devices to get a better grasp on the actual organisation of the information which is being presented.

The aim is thus to design a human-computer flexible interface which is able to adapt to the communicative capabilities of the user by interpreting the user’s requests in the application context. The development of such interfaces may have strong consequences on more fundamental research on man-machine communication:

• it implies having robust language processing modules, from speech recognition to deep language understanding, comprising the interpretation of multimodal referring expressions (eg demonstrative and gesture) and the contextual understanding of the action to be taken or answer to be given
• it forces the design of new models for the analysis and understanding of gestural trajectories, there again implying the contextual interpretation of possible designations
• these two aspects have to be combined in generic communication models where the complementarity of the two modes (language and gesture) is understood at a semantic level
• dialogue management is even more crucial in a multimodal environment since it contributes to the building of the context (cf. Eg elliptical phenomena) but also ensures a good continuity of the dialogue by ensuring that the right communication act is understood and produced
• finally, one has to design an optimal graphical representation of the task where objects are clearly identified and organized in such a way that the lay-out of the scene as perceived by the user will be predictable.

In the context of a multimodal interface comprising a haptic device, it is essential to analyse how such a presentation can integrate the third dimension (depth). It should be both more natural from a perceptual point of view, and provide situations where direct action on the scene is preferable to communication with the virtual assistant (situations of emergency, re-organization of the scene etc.). This opens a wide range of possible research that has been barely tackled so far.

These various aspects have been for long part of the core activities of the ‘Langue et Dialogue’ team at Loria, mainly in the context of multimodal reference interaction with ‘classical’ designation device, but our strong involment in the future MIAMM European project (Multidimensional Information Access using Multiple Modalities). This project aims at providing a multimodal interface based on a haptic device to provide access to multimedia data. In collaboration with the Parole team at Loria, we will develop the following aspects:

• robust linguistic processing module putting into competition both stochastic models and symbolic ones (based in particular on the TAG (Tree Adjoining Grammar) formalism)
• provide a unified semantic of 3-D gestures as provided by the haptic device, which integrates the relation to perceptual grouping factors
• multimodal dialogue management where both contextual interpretation and interlocution will be managed in a uniform framework to ensure the close interaction between these two functions
• software integration, based on the definition of an multimodal exchange format MMIL (Multi-modal Interface Language), expressed in XML, which will gather up the various results yielded by each module in the system.

The MIAMM project will start in September 2001 and also involves DFKI, Sarrebruck; TNO, Netherlands Organisation for Applied Scientific Research, Delft; Sony Int. Europe GmbH, Stuttgart and Canon Research Center Europe Ltd., Guilford.

Link:
http://www.loria.fr/equipes/led/

Please contact:
Nadia Bellalem — LORIA
Tel: +33 3 83 50 20 33
E-mail: nbell@loria.fr