Fourth DELOS Workshop on Evaluation of Digital Libraries:
Testbeds, Measurements and Metrics

by László Kovács

The Fourth DELOS Network of Excellence Workshop on Evaluation of Digital Libraries: Testbeds, Measurements, and Metrics, was held at SZTAKI’s Distributed Systems department in Budapest from 6-7 June 2002.

The workshop was organized by the DELOS Working Group responsible for maintaining a Digital Library Evaluation Forum and a Digital Library Test Suite which will provide researchers with information for the selection of potential digital library testbeds and new research or evaluation targets. 28 participants attended the workshop, 20 of them came from Europe, seven from the US and one from Japan.

Digital libraries (DL) can be viewed from a number of perspectives. They can be new forms of information institutions, multimedia information retrieval systems, or information systems that support the creation, use, and search of digital content. Digital libraries are not an end in themselves; rather, they are enabling technologies for digital asset management, electronic commerce, electronic publishing, teaching and learning, and other activities. Accordingly, digital libraries need to be evaluated in the context of specific applications. The methods and metrics for evaluating digital libraries will vary by whether they are viewed as institutions, as information systems, as new technologies, or as new services.

The DL research communities need large test beds in order to be able to evaluate and demonstrate new concepts. Research results are most valuable when they are compared with other approaches and validated against many sets of data. Considering users, collections and systems as the major components of a digital library, evaluations may involve only a subset of these components. Besides the evaluation context, the underlying criteria have to be specified. Also, for comparing different evaluations, specification of the test material employed is essential. There are no standard description schemes yet and only little work has been done in this area so far.

The programme for the Workshop included ten long papers and six short papers from Europe, Japan and the US, and was presented in five sequential sessions:

• Setting the background on Digital Library Evaluation: reports from working groups
• Users and User Interfaces
• Evaluation in Context
• Metrics and Testbeds
• Evaluation of Digital Library Services and Scalability.

During the first session an overview of some evaluation initiatives was provided. Ron Larsen (US) gave us an insight view of some of the experiences from DARPA’s DLib Test Suite project in the US, and gave the evaluation metrics from the Dlib Metrics Working Group. Michael Mabe (UK) presented a publisher’s view on the conceptual schema developed by the DELOS Evaluation Forum Test Suite team. His message is “WHO is using WHAT and WHY” . The “HOW” is the next question to ask, also in evaluation. Nariko Kando (Japan) reported about the initiative ‘Evaluation of Information Access Technologies’, and the results from three previous evaluation workshops. This was a valuable starting-point for the other presentations where attendees got snapshots of ongoing projects and ideas.

The discussions in the lively ‘mini-panels’ following each session were stimulating. Discussions and results were also the results from the breakout groups, four altogether:

• Users and User Interfaces, chaired by Nicholas Belkin
• Evaluation in Context, chaired by Christine Borgman
• Metrics and Testbeds, chaired by Ingeborg T. Sølvberg
• Strategies for funding of future Evaluation programs, chaired by Ron Larsen.

The results of the breakout groups were reported in plenary sessions, and the written reports will be made available on the homepage of the workshop (http://www.sztaki.hu/conferences/deval). The final proceedings of the Workshop will be published as an ERCIM report. In addition to the papers the proceedings will include reports from the four breakout groups as well. The proceedings and the power point presentations are also available at the workshop homepage.

Links:
Workshop: http://www.sztaki.hu/conferences/deval
DELOS Test Suite: http://www.sztaki.hu/delos_wg21/
SZTAKI, Distributed Systems Department: http://dsd.sztaki.hu

Please contact:
Laszlo Kovacs, SZTAKI
Tel: +36 1 279 6212
E-mail: laszlo.kovacs@sztaki.hu