ERCIM News No.46, July 2001 [contents]

Interface Development Toolkits for Non-visual and Switch-based Interaction

by Constantine Stephanidis and Anthony Savidis

Commercial user interface development toolkits do not provide support for interaction techniques alternative to mouse/keyboard based graphical interaction. Two toolkits, the HAWK toolkit for non-visual interaction and the SCANLIB toolkit for switch-based interaction have been developed at ICS-FORTH.

The HAWK toolkit provides a set of standard non-visual interaction objects and interaction techniques that have been specifically designed to support high quality non-visual interaction. HAWK is appropriate for developing not only interfaces targeted to blind users, but also interfaces for a variety of situations in which dialogues not relying on visual communication and standard input devices are required (eg, driving, telephone-based applications, home control auditory interaction). A key notion in HAWK is that of a container interaction object. In the HAWK toolkit there is a single generic container class that does not provide any pre-designed interaction metaphor, but supplies appropriate presentation attributes through which alternative representations can be created. The container class has four attributes that enables each distinct container instance to be given a metaphoric substance by appropriately combining messages and sound-feedback (both names and sound effects can have a metaphoric meaning). In addition to generic containers, HAWK provides a comprehensive collection of conventional interaction objects directly supporting non-visual dialogue, namely menus (exclusive choice selector object), lists (multiple choice selector object), buttons (push button analogy for direct command execution), toggles (radio button analogy for on/off state control, edit fields (single line text input) and text reviewers (multi-line read-only text editor, with mark-up facilities).

HAWK also supports a variety of interaction techniques, namely synthesized speech, Braille (2-cell transitory Braille, 40-cell Braille), and digitised audio for output, and standard keyboard, joystick used for gestures independently of visual interaction, touch-tablet (for programmable commands via associated regions) and voice recognition for input. The HAWK toolkit provides all the programming features met in currently available toolkits, such as hierarchical object composition, dynamic instantiation, call back registration, and event handling. The navigation dialogue enables the blind user to move within the interface structure composed of organizations of containers and contained objects in an easy way, through the provision of multi-modal control facilities. For instance, visiting contained objects is possible through joystick-based gestures, voice commands, keyboard short cuts, or via pressing specific regions of the touch-tablet. Container objects may contain other container objects realizing different metaphoric representations, thus supporting fusion of different metaphors in the context of non-visual interactive applications.

The HAWK toolkit has been used in the development of the AVANTI browser (see ERCIM News no. 41), of the NAUTILUS information kiosk (Project Nautilus, funded by the Hellenic Ministry of Development) and of a non-visual digital library for the Hellenic Blind Association.

Switch-based Interaction
In the SCANLIB interface development toolkit, the basic Windows object library has been augmented with scanning interaction techniques. Interfaces implemented through SCANLIB directly support motor-impaired user access, as well as access in other situations in which the keyboard and mouse input devices can not be used. Apart from enabling intra-application interaction control (eg, having access via switches to all interface elements of any interactive application), SCANLIB also supports inter-application interaction control (eg, enabling users to move across different applications). In SCANLIB, basic object classes are classified into five categories, each requiring a different dialogue policy to be designed:

• Top-level windows: these are the objects directly providing window management operations. Since in Windows it is impossible to ‘access’ (in a programmable manner) the built-in controls (icons1) for window management, all top-level windows in SCANLIB have been augmented with an additional toolbar accessible via scanning.
• Container objects: these are object classes with instances present as intermediate nodes in object hierarchies, able to encompass an arbitrary number of contained object instances. Container objects enable sequential scanning of all contained objects.
• Text-entry objects: objects requiring text to be supplied impose the need for keyboard emulation. When the user focuses (via scanning) on a particular text-field object, a special on-screen keyboard automatically appears, through which text input is enabled. Such an augmented dialogue is realized transparently to programmers, for which the augmented text-field object at the programming level appears as a conventional Windows text-field object.
• Composite objects: typical examples of composite objects are the scroll-bar class (composed of two arrow buttons and a slider) and the combo-box class (composed of a drop-down button, a label, and a menu object). Composite objects enable sequential scanning of all component-objects.
• Button categories: these are simple interaction objects supporting direct user actions for executing associated operations or changing state of Boolean parameters. Typical examples from this category are: push buttons, check boxes and radio buttons.

The scanning interaction techniques are based on two fundamental actions: SELECT, and NEXT. Depending on the type of switch equipment required, four scanning modes are supported:

• 1-switch (for SELECT action), time scanning (automatic NEXT action, after a specific time interval)
• 2-switches (one for SELECT action, one for changing scanning direction), time scanning (as before for automatic NEXT action)
• 2-switches (SELECT action, NEXT action), no time scanning
• 5-switches (SELECT action, NEXT action, change direction, exit directly container, re-start with container).

Programming control is provided for the extra attributes introduced by the augmented version of Windows objects, thus enabling objects to differentiate scanning style depending on scanning mode (one of the five alternatives), time interval (when time scanning is supported), and highlighter presentation parameters.

The SCANLIB toolkit has been used in the development of the AVANTI browser (see ERCIM News no. 41), and of the GRAFIS word processor for people with motor impairments (see ERCIM News no. 38).

Please contact:
Constantine Stephanidis — ICS-FORTH
Tel: +30 81 391741
E-mail: cs@ics.forth.gr