 |
ERCIM News No.47, October 2001 [contents]
|
CLEF2001: Workshop of the Cross-Language Evaluation Forumby Carol Peters The results of the second evaluation campaign of the Cross-Language Evaluation Forum were presented at a Workshop held on 3-4 September, immediately before ECDL2001 in Darmstadt, Germany. Cross-Language Evaluation Forum provides an infrastructure for the testing and tuning of mono- and cross-language information retrieval systems working with European languages. The main focus of the activity is to stimulate the development of multilingual information retrieval systems, ie systems capable of matching a query in one language against document collections in many languages. As a result of the success of the CLEF2000 campaign, it was decided to extend the test suite provided for this year’s participants. In particular, the multilingual corpus was increased to nearly one million documents in six languages instead of four languages: Dutch and Spanish were added to the English, French, German and Italian collections. The main task of CLEF2001 thus required participants to search a document collection in five languages, extracting their queries from a topic set offered in twelve different languages (nine European and three Asian languages). The same topic set could be used in the bilingual retrieval task to search either a Dutch or an English target collection. Tasks testing monolingual, domain-specific, and interactive text retrieval systems were also offered. Thirty four groups participated in one or more of the tracks – fourteen more than in 2000. The results of the CLEF2001 experiments were presented in a two-day Workshop held in Darmstadt at the beginning of September. The Workshop was attended by nearly fifty researchers and system developers from both academia and industry. In the opening session, one of the organizers, Martin Braschler of Eurospider Information Technology, Zürich, gave an overview of the campaign and summarised the main trends that emerged this year. All the traditional approaches to the cross-language task had been employed but many innovational strategies were also tested. It was notable that many groups had built on results reported in CLEF 2000. The rest of the first day was dedicated to descriptions of the experiments and discussions on a number of the key issues affecting cross-language systems. These included the problem of merging the results of a search over a number of document bases in different languages in a meaningful way, and a comparison between methods used for indexing texts in multiple languages. On the second day, ideas for new activities were presented. Julio Gonzalo (UNED, Madrid) and Douglas Oard (University of Maryland) first reported the results of the experimental interactive track that was focussed on the end user and the document selection task. The aim was to investigate how the results of a cross-language system can best be presented to users who have no knowledge of the target language(s) in order to facilitate their selection of those documents that do contain relevant information and merit further examination. Marcello Federico of itc-IRST, Trento, then illustrated a proposal for the design of a track for a preliminary testing of systems for cross-language spoken document retrieval, thus moving CLEF towards an investigation of multilingual multimedia systems. In the following session, Donna Harman from the US National Institute for Standards and Technology (NIST) and Noriko Kando (National Institute of Informatics, Japan) reported on the activities of the TREC and NTCIR evaluation campaigns for cross-language systems (for Arabic and Asian languages, respectively) . In their next campaign, NTCIR intends to adopt the CLEF model for evaluating multingual retrieval systems operating on Chinese, Japanese, Korean and English. The session concluded with an invited talk by Ellen Voorhees (NIST) on the philosophy of IR evaluation in which she outlined the methodology that has been studied for TREC and adopted by CLEF. In the afternoon, the criteria adopted by CLEF to create the topic sets and produce the relevance assessments were described by Christa Womser-Hacker (Univ. Hildesheim, Germany) and Djoerd Hiemstra (Univ. Twente, Netherlands). In the final session of the Workshop, the preliminary programme for CLEF 2002 was outlined. While the multilingual task will remain unaltered, the bilingual task has been considerably strengthened. Groups will be able to test their systems choosing from target document collections in at least five European languages and perhaps more. Investigations are currently underway as to the feasibility of adding both Finnish and Swedish. Monolingual system testing will again be offered and the interactive track will be extended to become an official CLEF task in 2002. Copies of the Workshop presentations are available on-line on the CLEF Web site. The proceedings will be published for the second year running by Springer in their Lecture Notes for Computer Science series. CLEF is a collaboration between the US National Institute for Standards and Technology and the European Commission. CLEF2000 and 2001 were supported by the DELOS Network of Excellence for Digital Libraries. CLEF 2002 will be funded independently under the 5th Framework IST programme of the Commission. Link: Please contact: |
iei.pi.cnr.it