INEX: Evaluating XML Retrieval Effectiveness
by Mounia Lalmas
The Initiative for the Evaluation of XML Retrieval (INEX) is an international campaign involving more than fifty organizations worldwide. It provides a means of evaluating retrieval systems that provide access to XML content.
Documents today contain a mixture of textual, multimedia, and metadata information. One way to format this mixed content is according to the eXtensible Mark-up Language (XML). In contrast to HTML, which is mainly layout-oriented, XML follows the fundamental concept of separating the logical structure of a document from its layout. XML thus offers the opportunity to exploit the logical structure of documents to allow more precise searching. Providing effective access to XML-based content has become a key research issue.
Effective access to XML repositories is the core of XML retrieval research. XML retrieval systems aim to exploit the logical structure of documents to retrieve, in response to a user’s query, document components (ie XML elements) rather than whole documents. Implementing this more focused form of retrieval means that an XML retrieval system must not only find relevant information, but also determine the appropriate level of element granularity to return to the user. Evaluating the effectiveness of these systems therefore requires test-beds utilizing criteria that take into account the imposed structural aspects.
In 2002, INEX started to address these issues. An infrastructure was established, and a large XML test collection and appropriate scoring methods were provided for the evaluation of content-oriented XML retrieval systems.
Evaluation is carried out using test collections assembled specifically for evaluating particular retrieval tasks. A test collection consists of a document collection, a set of user requests (ie topics) and relevance assessments. The characteristics of traditional test collections have been adjusted to appropriately evaluate XML retrieval effectiveness: the document collection comprises documents marked up in XML, the topics specify requests relating to both content and structure, and the relevance assessments are made at element level. In addition, relevance is measured such that it appropriately quantifies the system’s ability to return the correct granularity of XML elements.
Ad Hoc Retrieval
The main retrieval task to be performed in INEX is ad hoc retrieval. This can be described as a simulation of how a library might be used, and involves the searching of a static set of documents using a set of topics. In INEX, the library consists of XML documents, the queries may contain both content and structural conditions, and in response to a query, arbitrary XML elements may be retrieved.
In 2005, we identified two ad hoc retrieval sub-tasks that depend on how structural constraints are expressed. In the content-only sub-task, it is left to the retrieval system to identify the most appropriate XML elements to return to the user. Three different strategies have been defined, depending on the preferred output format of an XML retrieval system. In a focused strategy, we assume that a user would prefer the single element most relevant to the topic; in a thorough strategy, we assume that a user would prefer all highly relevant elements; and in a fetch and browse strategy, we assume that a user is interested in highly relevant elements contained within highly relevant documents. An extension of this sub-task is when a user adds structural constraints to the topic to narrow down the number of returned elements.
In the content-and-structure sub-task, the structural constraints are explicitly stated in the topics and can refer to where to look for the relevant elements (ie support elements) and what types of elements to retrieve (ie target elements). Structural constraints can be interpreted as either strict or vague, and these interpretations can be applied to both support and target elements, giving a total of four strategies for this sub-task.
Tracks
INEX has separate tracks looking at specific issues in XML retrieval. The interactive track aims to investigate the behaviour of users when interacting with XML documents, and to develop effective user-based approaches to XML retrieval. The heterogeneous track is concerned with cases where an XML collection comprises documents from different sources and based on multiple document structures. The multimedia track aims to provide an evaluation platform for XML retrieval systems that include text, images, speech, and video. The document-mining track aims at defining techniques for effectively mining information from XML documents, with a focus on classification and clustering.
INEX is led by Norbert Fuhr from the University of Duisburg-Essen, Germany, and Mounia Lalmas from Queen Mary University of London, United Kingdom, and is partly funded by the DELOS Network of Excellence on Digital Libraries administrated by ERCIM.
Links:
INEX: http://inex.is.informatik.uni-duisburg.de/
DELOS: http://www.delos.info/
Please contact:
Mounia Lalmas
Queen Mary University of London, UK
E-mail: mounia
dcs.qmul.ac.uk
