DELOS NoE
on Digital Libraries
Evaluation Plans for TEL
Norbert Fuhr, University of Duisburg-Essen
(Germany)
Comparative Evaluation
of TEL and Daffodil




Systems: TEL, Daffodil
Evaluation framework
Evaluation method
Conclusion
N. Fuhr: Evaluation Plans for TEL 2/2/06
Systems: TEL, Daffodil
N. Fuhr: Evaluation Plans for TEL 2/2/06
Systems: TEL, Daffodil
N. Fuhr: Evaluation Plans for TEL 2/2/06
TEL: Query formulation
N. Fuhr: Evaluation Plans for TEL 2/2/06
Daffodil: Query formulation
N. Fuhr: Evaluation Plans for TEL 2/2/06
TEL result list
N. Fuhr: Evaluation Plans for TEL 2/2/06
Daffodil: Search progress
N. Fuhr: Evaluation Plans for TEL 2/2/06
Daffodil: Result list
N. Fuhr: Evaluation Plans for TEL 2/2/06
TEL: Detail view
N. Fuhr: Evaluation Plans for TEL 2/2/06
Daffodil: Detail view
N. Fuhr: Evaluation Plans for TEL 2/2/06
Systems: TEL vs. Daffodil

Commonalities:
– Front-end to a federation of heterogeneous digital libraries
– User-friendly interface, supporting mouse-based interactions
– Similar search dialogue

Differences
– TEL is browser-based (Javascript)
• No setup for user
• Limited mouse interaction
– Daffodil uses a client written in Java
• Setup (download & install) required
• Richer mouse interaction
– Daffodil desktop contains various tools
• (some of them are applicable to the TEL domain)
N. Fuhr: Evaluation Plans for TEL 2/2/06
Daffodil: Necessary changes and
extensions

Wrappers for the TEL libraries
– Z39.50, SRU/SRW, OAI

Query processing:
– Incremental generation of result list
(instead of waiting till all DLs have responded)
– Initial result display: grouped by DL
merging upon request

Inclusion/adaption of tools:
– Extraction of authors, keywords
– Clipboard
– Personal library
N. Fuhr: Evaluation Plans for TEL 2/2/06
Evaluation framework
N. Fuhr: Evaluation Plans for TEL 2/2/06
Collection
The European Library’s default collections:











British Library integrated catalogue
Biblioteca Nacional (Portugal)
BN-OPALE PLUS
Online-Katalog Der Deutschen Bibliothek
HELKA
SBN OPAC
General Catalogue Koninklijke Bibliotheek
HELVETICAT
National Library Catalogue (KatNUK)
Croatian National Bibliographies of Books: 1990-2003
Serbian Union Catalogue COBIB.SR
N. Fuhr: Evaluation Plans for TEL 2/2/06
Users & Usage

Users:
DL stakeholders
– End users
– Librarians

Usage:
task scenarios to be developed
N. Fuhr: Evaluation Plans for TEL 2/2/06
Evaluation methods
Analytical evaluation:
Is the design right?

Empirical evaluation:
Is it the right design?

N. Fuhr: Evaluation Plans for TEL 2/2/06
Analytical evaluation
Analytical evaluation:
Is the design right?

Start of evaluation:

Explores how the TEL and Daffodil interface facilities are
useable, learnable and comparable
At the end of evaluation:

Examines how the TEL and Daffodil interface facilities are
in alignment with the user requirements gathered from the
empirical evaluation
N. Fuhr: Evaluation Plans for TEL 2/2/06
Empirical evaluation
Empirical evaluation:
Is it the right design?

Perspectives:

User characteristics, preferences and strategies

Types of activities that the users do (to plan the
task scenarios)

The environments in which TEL and Daffodil are
used, in natural or controlled laboratory settings
N. Fuhr: Evaluation Plans for TEL 2/2/06
Evaluation steps for empirical
evaluation







Acquisition of resources: equipment, instruction
materials, recording technology, selection of
representative users
Briefing meetings with groups of users
Search sessions and interviews with users
Debriefing with groups of users
Transcription of recorded data
Data analysis
Reporting
N. Fuhr: Evaluation Plans for TEL 2/2/06
Conclusion



Comparative evaluation of two similar systems
Will yield valuable output wrt. strengths/weaknesses
of the TEL system
Basis for possible further development
N. Fuhr: Evaluation Plans for TEL 2/2/06

Fulltext