Compositing Computer and Video Image Sequences
by Marie-Odile Berger and Jean-Claude Paul
Augmented reality systems aim at enhancing the users's vision with computer generated imagery. For numerous applications, it is necessary to merge a video signal from a real camera with computer generated objects. Such techniques are of great interest to assess the potential impact of new constructions on local environments (for urban planning decisions, for instance). To make such systems and their applications effective, the computer generated objects (also called virtual objects) must be blended convincingly with the real images. In this context, the ISA project at INRIA Lorraine develops methods and tools devoted to the automatic composition of video and computer images.One major challenge in image composition is to correctly register the real data and the virtual data in order to ensure the geometric coherence of the composed scene along a complete video sequence. This can be done using two basic approaches. The first solution is to use position sensors (for example Polhemus sensors); but instrumenting the real world is not always possible, especially for vast or outdoor environments. The other solution is to use some objects in the scene, the model of which is known, to perform registration. As we often work on architectural outdoor applications, instrumentation is impossible. Hence we are working on temporal model based registration, since the 3D structure of some objects (buildings) in the scene is often known.
Figure 1: The model of the bridge.
Figure 2: The reprojection of the model of the bridge using the computed pose.
Ensuring temporal registration is not sufficient to perform realistic composition. Other significant visual clues to the human perceptual system must be considered: for instance, proper occlusion resolution between real and virtual objects is highly desirable in composition systems. Other photometric interactions between real and virtual objects (continuity of lighting, shadowing) should also be considered.
Unlike numerous existing composition systems, which often require strong and tedious interactions with the user, the aim of the ISA project is to develop tools and algorithms allowing composition to be performed as automatically as possible.
We have first designed a robust algorithm for temporal model based registration. This system detects and tracks various features corresponding to the model, that will be used to compute the camera position and to update the set of visible features. The heart of our system is the pose computation method, which handles various features (points, lines, curves) in a very robust statistical way. This method is then able to give a correct estimate even if tracking errors occur. The reliability of our method has been proven in the project of the illumination of the Paris Bridges: the aim was to test new lighting systems for some bridges around the "Ile de la cite". In a sequence shot at dusk time, we have replaced the bridge with the lighting simulation of the bridge: Figure 1 shows the model of the bridge and Figure 2 exhibits the reprojection of the model using the computed pose. The interested reader may see further results on our web site: http://www.loria.fr/isa.
It must be noticed that our algorithm does not work in real time. This was not a priority for us, because of the cost of the rendering process. However, the tracking and the pose computation can be processed in parallel. Thus, real time applications could be carried out, provided that the computation of the virtual objects is not too expensive.
We are currently working on the problem of occlusions. Theoretically, resolving occlusions could be achieved by inferring a dense map from two consecutive images, which allows the depths of the virtual and the real object to be compared. Unfortunately, despite new advances in 3D reconstruction, the depth map lacks accuracy and cannot be used as is. We therefore investigate a contour based approach without reconstruction. For more information on the ISA project, see: http://www.loria.fr/isa/
Please contact:
Marie-Odile Berger and Jean-Claude Paul - INRIA Lorraine
Tel: +33 3 8359 2070, +33 3 8359 2077
E-mail: berger@loria.fr, paul@loria.fr
