@article {366, title = {Human Factors issues in multi-modal interaction in complex design tasks}, year = {2003}, publisher = {Nederlandse Vereniging voor Fonetische Wetenschappen}, address = {Tilburg, The Netherlands}, abstract = {

For human-computer interaction, the naturalness of the dialog is a paramount factor for the acceptance of dialog systems by a large public. It is generally believed that the use of multi-modal communication channels facilitates for natural interactions between a human subject and artificial collocutor. In the European-IST COMIC project (2002-2005, http://www.hcrc.ed.ac.uk/comic/), various scientific and pragmatic aspects of multi-modality are being studied through human factors research and the integration of research findings in a real-life interactive system. The two input modalities considered here are pen gestures (including pointing, handwriting, labeling and sketching) and (continuous) speech. The task of the system is to guide the unexperienced user through a wide range of design options in the highly complex professional domain of bathroom design. The University of Nijmegen is one of the COMIC-participants and focuses on the improvement of the pen and speech recognizers, the use of these recognizers in a real-time application, and the study of human-factors related to the multi-modal human-system interaction. In COMIC, human-factors experiments are carried out to obtain more insight in the complex behavior of humans in interactive design specification tasks. The task of the subjects was to specify shape, size and location of details in bathrooms by using pen input on a Wacom Cintiq 15X LCD tablet and speech through a close-talk microphone. The results of the experiments are presented in this talk. It is shown that there are behavioral differences between subjects, and that the subjects{\textquoteright} behavior is critically dependent on the exact task description. The experimental results yield more insight in the classes of speech dialog acts and pen dialog acts, and in the impact of mode combinations in interaction strategies. These experiments advance the knowledge of cognitive principles underlying the natural use of parallel modalities, and result in concrete guidelines for the design and the development of multi-modal dialog systems.

}, author = {Stephan Rossignol} }