ERCIM News No.32 - January 1998

Object-based Manipulation Tools for Postprocessing Video Applications

by Henri Nicolas

In the framework of the ESPRIT project NEMESIS (NEw MultimEdia Services using analysIs Synthesis), partly carried out by the Temis research group at IRISA/INRIA, Rennes, automatic or semi-automatic tools are being developed which enable object-based manipulations of the original image sequences for post-processing video manipulations.

When a video sequence is acquired by a moving camera, it contains generally both moving objects and a static background. By computing an object-based segmentation of the scene for each image, the background can be separated from the rest of the scene in order to create the mosaic image. This global information can therefore be used for video editing (for example suppression of an object), or to virtually change the camera point of view, ie, the camera location and its focal. This leads to the creation of video sequences which have never been acquired by a camera, but which are obtained using real data provided by the real camera shot. This kind of automatic tools is of great interest for the post-production world in which, with the existing techniques, long and tedious manual efforts are necessary to achieve such manipulations.

In practice, once the images have been acquired by the camera, a human operator defines the manipulations which must be done on the objects existing in the original data. Furthermore, in order to obtain very high qualities on the computed image sequences, human intervention may be required to correctly set-up the algorithms according to the image characteristics. Nevertheless, the use of the proposed approach in a purely automatic way is possible (see the illustrations below) but lower image quality can be expected.

Different techniques and algorithms have been developed in order to design such manipulation tools. In practice, an artificial image sequence is created using the four following steps:

Precise camera estimation is required in order to create the mosaic image. This is obtained by using a three parameters camera motion model, ie two translational and one zoom motion parameters. For each image, these parameters are estimated using a temporal linking technique combined with a comparison with the currently obtained mosaic image.

The creation of the mosaic image requires an object-based segmentation for each original image in order to differentiate the background from the other objects. For the first image, a human operator has to define the objects which must not be included in the mosaic (this may include static objects which cannot be automatically separated from the background by motion estimation). For the other images, an object-based motion estimation technique combines with the current mosaic image permits to select the information which can be used to update the current mosaic.

The mosaic is defined as the global view of the scene background resulting from an image sequence obtained by a moving camera. For each image, this mosaic image is created or updated using the current initial image and taking account the camera motion estimation and the object-based segmentation previously estimated.

A human operator specifies the treatment which must be done on the images (for example, positioning of a given object in the center of the images). The artificial sequence is therefore computed using the original images, the segmentation and the mosaic images.

First versions of these tools have already been developed which modify artificially the camera point of view or eliminate a given object from the scene using camera motion estimation, object-based segmentation and mosaic images. Nevertheless, new techniques continue to be developed in order to improve the quality of the results. Finally, a major perspective of this work is related to the mixing of synthetic objects together with real data.

Please contact:

Henri Nicolas - IRISA/INRIA
Tel: +33 2 99 84 74 25
E-mail: hnicolas@irisa.fr

return to the contents page