Retrieval of Images based on Visual Content: A Biologically Inspired Multi-Agent Architecture

by Socrates Dimitriadis, Kostas Marias and Stelios Orphanoudakis

Understanding the mechanisms of visual perception is important in a number of application areas, including content-based image retrieval (CBIR). An interdisciplinary approach to CBIR, in the framework of cognitive science, is expected to yield significant results where traditional approaches have failed to do so. In the Computational Vision and Robotics Laboratory (CVRL) of the Institute of Computer Science - FORTH, an ongoing research effort aims to develop and implement an experimental platform for the investigation of CBIR, based on a biologically inspired multi-agent architecture.

In recent years, interdisciplinary research has produced significant results in a number of areas in which traditional approaches appeared to have reached their limit. Based on the joint effort of biological neuroscientists, computational neuroscientists, psychologists, and computer scientists, the interdisciplinary field of cognitive science has emerged as the field of research in which brain function is investigated with respect to behaviour and mental activity. Visual perception is a major brain activity and has attracted the attention of many cognitive scientists. Understanding the mechanisms of visual perception is important in a number of application areas, including content-based image retrieval (CBIR).

In the Computational Vision and Robotics Laboratory (CVRL) of ICS- FORTH, an ongoing research effort is aims to develop and implement a biologically inspired image retrieval methodology. Due to the fact that images represent a particularly large volume of information, the efficient and possibly intelligent browsing of images based on visual content is becoming increasingly important in application fields which make use of large image databases, such as diagnostic medical imaging, remote sensing, entertainment, etc. Surfing on the Web may also be facilitated and made more intelligent if visual browsing is exploited.

Humans undoubtedly possess the ability to process visual information efficiently and to identify images as being similar based on their visual content. However, computational approaches currently fall short of matching this ability. The reasons for this are many and cannot be explored in this short article. It is our research objective at ICS-FORTH to develop and implement CBIR mechanisms that are perceptually relevant and based on a biologically inspired system architecture. We must therefore take into account what is currently known about how humans process visual stimuli and subsequently represent and store visual information, so that it may later be retrieved efficiently from visual memory.

It is known that the human cerebral cortex consists of many cortical areas, each one specialised and able to execute a specific task. The visual cortex consists of a set of specialised modules that are responsible for the processing of a specific visual feature and can execute their respective tasks independently and in parallel. An image similarity decision is not based on any one of these processes acting in isolation, but is rather the result of the integration of their individual contributions. This biological paradigm resembles the computational approach adopted in multi-agent systems. Exploiting parallelism in complex dynamic environments with a high level of uncertainty is a key ingredient of multi-agent computational systems that model biological processes and behaviours.

Architecture of the content-based image retrieval system.

A biologically inspired multi-agent architecture has been developed at ICS-FORTH, and provides a platform both for the experimental investigation of a variety of problems arising in CBIR and for the assessment of the perceptual relevance of similarity retrieval schemes involving the cooperation of selected agents. As shown in the figure , this architecture consists of four main components: a set of software agents, an image database, a graphical user interface, and a voting system that supports a set of alternative voting schemes. Images can be imported into the database, viewed individually or in galleries, and retrieved based on their visual content. Images that are similar to a query image may be retrieved from a specific set of candidate images (gallery) or from all images contained in the database. The query image and the set of candidate images are both selected by the user. Each agent works independently and computes the similarity between the query image and each candidate image, based on its specialised similarity criterion and matching algorithm. Integration of partial results obtained by different agents is achieved through the voting system, using a voting scheme selected by the user. The system loads and handles dynamically the set of available agents and voting methods. Furthermore, the reasoning on which the final ranking of candidate image similarity is based is exported, thus allowing the user to interpret the response to a specific query and to fine-tune relevant parameters in order to improve the response to subsequent queries. This reasoning consists of the contribution of each agent to the final ranking of each candidate image, according to the voting scheme selected by the user.

The CBIR system is fully scalable and can easily be extended with additional agents or voting schemes in order to take into account the specific requirements of different experiments and the class of images under consideration. It has been developed in JavaTM and makes use of the Java Advanced Imaging package.

Please contact:
Stelios Orphanoudakis, ICS-FORTH
Tel: +30 2810 391605
E-mail: orphanou@ics.forth.gr