Personalized and Adaptive Multimedia Retrieval

by Joemon M. Jose and Jana Urban

Data accumulation has become an integral part of our life, be it text (eg email) or multimedia data (eg photographs). This is mainly due to the proliferation of computers, networking devices and consumer devices like digital cameras and camcorders. However, our creation prowess is not matched by any comparable search facilities. As a result, finding relevant information from such archives becomes a cumbersome process. Even after a decade of research in content-based image retrieval (CBIR), users must plough manually through their media archives in order to find what they need.

There are a number of reasons for this scenario. The major one is that the state-of-the-art in image processing does not allow us to identify and extract any meaningful segments from images. Images are still represented using low-level features. However, the images’ low-level feature representation does not reflect the high-level concepts the user has in mind (semantic gap) and – partly due to this –users can experience serious difficulties in effectively formulating and communicating their information need (query formulation problem).

Any reasonable solution to CBIR should address the issue of semantic gap, and providing help with query formulation is one way to do this. In our approach to image retrieval, we employ an adaptive scheme for image retrieval.

Adaptive Models for Image Retrieval
Difficulties with query formulation are addressed using an adaptive query learning scheme and an innovative search interface. We have developed an interface (see Figure 1) in which a user starts browsing with one example image. Subsequently, a new set of similar images is presented to the user.

As a next step, the user – through selecting one of the returned images – updates the query, which now consists of the original image and the selected image from the returned set of images. After a couple of iterations the query is based on a set of images. In this scheme, the retrieval process is iterative: updating the system’s knowledge of the user’s information need based on the user’s implicit feedback. In the underlying interaction model, the user builds up a browsing tree of interesting images by choosing one image from a recommended set to be appended to the browsing path in each iteration. The system’s recommendations are then based on a query constructed from the current path of images.

In this approach, the emphasis is placed on the user’s activity and the context, rather than any predefined internal representation of the data. A path represents a user’s motion through information, and taken as a whole is used to build up a representation of the instantaneous information need.

In a nutshell, our method supports both browse-based and query-based approaches. It supports a query-less interface, in which the user’s selection of an image is interpreted as evidence of its relevance to her/his current information need. It therefore allows direct searching without the need for formally describing the information need. For the query, each image in the path is considered relevant, but the degree of relevance is dependent on the age: it decreases over time when new images are appended. In this way, the retrieval model is a special kind of relevance feedback model, in which a query is implicitly refined by the user’s selection of images for feedback. It recognizes and addresses the dynamic nature of information needs, and has the advantage of allowing for an intuitive and user-centred search process.

A user study involving design students demonstrated the effectiveness of this approach. The evaluation showed that people preferred the search process in the ostensive browsing scheme, felt more comfortable during the interaction, and found the system more satisfactory to use compared to a traditional CBIR interface.

Image Retrieval by Group Organization
Another approach to providing effective image retrieval is to develop mechanisms to support the handling of the retrieved set of images. This will allow users to handle retrieved sets of images more easily by organizing them into relevant groups. The system can then capture the context of the user, and will thus be able to recommend images more tuned to their needs.

We introduced a system called EGO (Effective Group Organization), which facilitates retrieval in context (see Figure 2). EGO is a system for the management of image collections, supporting the user through the process of personalization and adaptation. In this system, we stress the need for a search system that provides flexible and extensible interfaces. The idea is to help the user in query formulation: by facilitating qualitatively high interaction, the system learns the needs of its users and can provide searches accordingly.

Our approach encourages the user to group and organize their search results and thus provide more finely grained feedback for the system. It combines the search and management process, which helps the user to conceptualize their search tasks and to overcome the query formulation problem. The system assists the user by recommending relevant images for selected groups. The user can therefore concentrate on solving specific tasks rather than having to think about how to create a good query in accordance with the retrieval mechanism.

Please contact:
Joemon M Jose, University of Glasgow, UK
E-mail: jjdcs.gla.ac.uk