ERCIM News No.35 - October 1998
ERCIM News No.35 - October 1998
CharToon - A JAVA-based Animation System
by Paul ten Hagen
The 3-year project FASE (Facial Analysis and Synthesis of Expressions), started in spring 1997, has entered its second phase. The aim is to develop a system which can recognize facial movements in front of a camera and use the data thus produced to animate models of the human face. Basic editing tools have been implemented in the CharToon system, consisting of a collection of JAVA programs enabling the interactive construction of parametrized 2D drawings and a set of time curves to animate the drawings. The project is a joint effort of Delft University of Technology (TUD) and CWI and is sponsored by the Dutch Technology Foundation STW.
Previous work, including CWI’s FERSA project in the early 1990?s on automatic real-time lip-synchronous speech animation, had shown that two technologies play a pivotal role to realize such a system: Facial Analysis (FA) and Facial Synthesis (FS). An important aspect of FASE is the integration of both technologies, in order to achieve life-likeness of the estimated facial information and motion. The FA module, covered by TUD, receives as input video images of a face (the ‘user’) and possibly additional information (eg, text and audio sources), on the basis of which the parameters describing the facial movements and expressions are estimated. The FS module, developed at CWI, contains a generic facial model and an animation model. It uses the estimated facial parameters as an input to deform and move the facial model and thus ‘mimic’ the looks and movements of the user. The results are used as a feedback input for FA. In addition, a third module, ‘Model Editing and Parameter Control Interface’ not only facilitates the development and set up of the (face) models and the generation of new animation models, but also provides easy control over the whole process of changing facial parameters – a highly desirable feature, in view of the close interplay between analysis and synthesis. The integration of FA and FS is also done at CWI.
The FS module allows two kinds of facial models:
- 3D physically based ones, aiming at realism
- 2D drawings, aiming at cartoon-like effects, stylized avatar faces or possibly also animation of 2D photographs of a real face.
For the 2D models we developed a Face Editor to design the drawing to be animated, an Animation Editor to specify the time-behaviour of the drawing’s animation parameters, leading to a movie script, and a Face Player to animate the drawing using this script. These three tools are integrated in the CharToon system. The implementation in JAVA makes the system suitable for Web applications.
The hierachical drawing program FaceEditor consists of basic building blocks like polygons and ellipses, objects composed of animated polygons and other shape-changing objects, and ready-made complex objects like an eye with pupil and eyelids.With AnimEditor values of animation parameters for computer facial (or other) models can be graphically specified and modified. The editor operates on a window resembling a musical score, with a ‘staff’? for each parameter and dots on the staff indicating the parameter?s behaviour in time. There is a facility to switch on and off audio channels. AnimEditor is also able to perform constraint management. Finally, in combination with the tool PERSONA, AnimEditor is also used to build realistic 3D facial models on the basis of the input facial parameters. Movies with sound, as constructed by AnimEditor and FaceEditor, can be played by FacePlayer. For small drawings, one can (just) achieve a rate of 20 frames/sec on an otherwise empty Sun Sparc Ultra. Different parts of a drawing can be controlled in parallel.
Current research at CWI concentrates on:
- gathering performer data to distillate generic and person-dependent characteristics of certain facial expressions
- fine-tuning the muscle model, add features to and improve performance of the physically-based model
- developing an interactive conformation tool
- extending the animation editing tool with mechanisms to allow the definition and dynamic use of animations as building blocks, synchronisation and other constraints
- trying out the existing tools in different applications.
The area of possible applications is very wide. The television (production and post-production) and telecommunication (eg, low bandwidth communication such as in videoconferencing and videophoning) industries are interested in realistic 3D rendering of facial expressions (efficient image coding). 2D caricature rendering is of interest for the film industry (animation) and in man-machine interaction (social user interfaces: avatars). Other applications include: entertainment (games, interactive television, synthesized actors), lip synchronisation (text or voice based, possibly also adjusted for the deaf), and facial surgical planning and simulation. Several potential industrial users are represented in the user group accompanying the FASE project (a standard procedure in STW projects): NOB (major Dutch television production company), KPN Research (communication systems), Institute for Phonetical Science (University of Amsterdam), Institute for the Deaf (Sint-Michielsgestel), and General Design (a CWI spin-off company, eg, for the design of Web sites). The CharToon system will be available for researchers free of charge by the end of the year. Information on the Web: http://www.cwi.nl/FASE
Please contact:
Paul ten Hagen - CWI
Tel: +31 20 592 4133
E-mail: Paul.ten.Hagen@cwi.nl