ERCIM News No.46, July 2001 [contents]

Combining Formal and Empirical Evaluation Methods Regarding the Usability of Interactive Systems

by Nico Hamacher

TREVIS is a tool for analysing the usability of interactive systems. This analysis can be done in various ways, eg a formal evaluation using normative user models based on the GOMS theory, and an extensive action log analysis. This tool supports the system engineer to evaluate interactive systems and produces suitable analysis data as a base for decisions while the systems are being developed.

Today, usability engineering of interactive systems is being paid increasing attention. To analyse the usability of interactive systems an empirical evaluation is commonly used. This type of evaluation requires a prototype and a couple of qualified testing subjects. Mostly, this procedure is very expensive and time-consuming. Furthermore, this evaluation is feasible only in late stages of the development process when a prototype is available, so that the results of this evaluation often are not available for redesign. Even though if this empirical evaluation cannot be omitted, because of the generation of plenty of useful information for improving usability, an earlier evaluation would be very helpful for the design and specification of the system.

With normative user models, usability analysis in an early stage of the development process is possible using the GOMS theory, first introduced by Card, Moran and Newell. GOMS is an abbreviation for the components of the model: Goals, Operators, Methods and Selection rules. Over the years it has been shown that GOMS is able to sufficiently describe the interactions of an operator. Its simple and plain structure makes this method easy to understand particularly for development engineers who do not have a psychological background.

An analysis of the user models generates qualitative as well as quantitative predictions, eg execution and learning time. The execution time describes the time to reach the goal whereas the learning time specifies how much time an operator needs to learn the whole task.

System Architecture
TREVIS combines the different usability analysis so that this tool can come into operation in various stages during the development process. The tool includes the following main modules: the user model editor, the device model converter and the analysis module.

Based on the task sequences as one result of the requirement analysis, the user models can be created manually in the user model editor. The tool supports this process, eg, by offering a graphical editor and a library for reusing components. Moreover, user models can be stored in projects, where the project represents the interactive system and the user models describe the tasks which have to be performed.

In contrast to other GOMS-editors which only allow to edit the user models manually, TREVIS is able to generate user models semiautomatically. A device model contains details about the inner work of the device. Using the device model converter the task sequences can be generated semiautomatically. The user models can also be created from these sequences.

In the analysis module the following four different analysis methods are included, which depend on the development phase in which TREVIS will be used:

• The user model analysis generates qualitative as well as quantitative predictions, eg execution and learning time (as already described with NGOMSL).
• A comparison between different user models is implemented in the design analysis module, which can be used as a basis for design decisions. Although this comparison presentation is a helpful functionality, no other tool includes it.
• In the action sequence analysis, action sequences resulting from testing a prototype can be imported and analysed. A grouping of different sequences is possible, eg to perform an analysis of significance. With this feature, TREVIS is also applicable in late stages of a development process.
• A fourth method analyses these action sequences in comparison with the user models. This analysis shows the differences between the actions specified in the user models and the activities the users performed while interacting with a prototype. Based on this analysis, predictions about effectiveness and efficiency can be made.

Some results of the different analysis methods are shown in the figure.

The graphical user interface of the tool TREVIS with the four main analysis methods.

With the possibility to analyse both user models and action sequences, TREVIS is the first tool that can be utilised in early as well as in late phases of the development process. Beside this, a comparison of user models with action sequences has never been implemented before.

Future Work
The tool has reached a first demo status and was tested with a couple of smaller projects. Now, a large-scale evaluation of TREVIS is planned in order to show that the tool can successfully be applied in a real system development process. Furthermore an integration of some interfaces to other popular development tools, eg prototyping and specification tools, is scheduled.

Link:
http://www.techinfo.rwth-aachen.de/Forschung/MMI/Trevis/index.html

Please contact:
Nico Hamacher — Aachen University of Technology, Germany
Tel: +49 241 4402 6105
E-mail: hamacher@techinfo.rwth-aachen.de