multimedia interfaces

In References on June 28, 2009 at 12:31 pm

Note: what about those with low cognitive functioning?

Crynszpan, O., Martin, J., & Nadel, J. (2008). Multimedia interfaces for uses with high functioning autism: An empirical investigation. International Journal of Human-Computer Studies, 66, 628-639.


This article focuses on issues relevant to human–computer interaction in the case of autism. We designed training software that target specific communicative disorders attributed to autism and defined an empirical protocol to test this software. The experimental software platform that we developed manages each game’s interface modalities and logs users’ actions, for the purpose of exploring the impact of various human–computer interfaces, which involve text, speech and images. Ten adolescents diagnosed with autism used this software during 13 sessions, at the rate of one session per week. The first and last sessions were dedicated for evaluating participants’ skills. The

experiment was also performed by a group of 10 typically developing children matched on developmental age and academic level. Results show that participants with autism had poorer performances on the richer multimedia interfaces. They seemed to lack the initiative of organizing the available multimodal sources of information. In this article, we specifically discuss the impact of executive disorders on the use of multimodal interfaces with an emphasis on Animated Conversational Agents.

In a review on technology for cognitive rehabilitation (LoPresit  et al., 2004) emphasize the advantages of computers for tasks requiring complex attention, prospective memory, self-monitoring of desirable behaviors, inhibition of undesirable behaviors, sequential processing and understanding of social cues.

The goal of this paper was to explore aspects of Human-Computer Interfaces that are relevant for autism, by trying to answer following questions.

Are people with autism influenced by interface modalities in the same way as the general population?

What are the characteristics of suitable multimedia interfaces for autism?

How should human-inspired communicative modalities such as facial expressions be used in the context of human-computer interfaces for autism?

Cognitive alteration relevant for software design

Contextualization disorder + executive dysfunction è software design?

Although people with high functioning autism may have a well-developed vocabulary, they tend to have profound pragmatic difficulties in social interactions. The literature attributes an asymmetric cognitive profile to high functioning autism, with lower performances in language compared to other skills, such as visuospatial competences (Mottron and Belleville, 1993).

Contextualizing problems pervade the entire social disorder in autism. According to literature, autism’s main communicative deficiency relates to pragmatics (Paul,

1987). The authors (e.g. Attwood, 1998) mention that people with autism have a tendency to interpret speech literally rather than in reference to a context. They experience difficulties with pragmatic subtleties such as irony, sarcasm, metaphors and idiomatic expressions.

Jolliffe and Baron-Cohen (1999) conducted an experiment where participants had to understand a short text composed of two or three sentences, one of which contained a homograph. Homographs are words that have the same spelling but different meanings. For example ‘‘lead’’ may be a verb meaning ‘‘to guide’’ or a type of metal. The context given by the text was the only way to disambiguate the meaning of the homograph. Participants had to choose between three possible interpretations: the correct interpretation in regard to the context, an interpretation which would be correct for the homograph left alone, but which was out of context, and an erroneous interpretation. Results indicated that participants with autism selected the second interpretation more often than controls: participants with autism tended to omit context, although it was necessary to understand the context.

Computer interaction could also be influenced by the executive dysfunction that authors link to autism (Russell,1996). Executive functions are involved in the control of behavior during goal-directed actions. The term ‘‘executive functions’’ traditionally refers to a set of cognitive functions that encompasses planning, working memory, impulse control, inhibition, shifting set as well as the initiation and monitoring of action (Hill, 2004).

Educational software for autism

Most projects have been focusing on the communicative and social disorders. Moore and Calvert (2000) compared computer-based vocabulary lessons with similar lessons given by a human teacher. Results indicated better attention, motivation and vocabulary retention when the computer was used.

Bernard-Opitz et al. (2001) studied training with software used for social behavior education. Children had to find a solution to different scenarios involving characters in problematic social conflicts. The performances of children with autism improved, although the progression of children without autism was steadier.

Leonard et al. (2002) designed a virtual reality environment to train social skills for teenagers with high functioning autism. It simulated real-life situations, such as finding a

place to sit in a coffee house. The evaluation of this system indicated that teenagers progressed in dealing with the social situation that had been simulated.

Researchers are investigating various technological options to provide adequate educational software for autism, as for instance network chat (Cheng and Kimberly, 2002), multimedia applications (Barbieri et al., 2004) and virtual reality (Leonard et al., 2002; Takahashi et al., 2004).

Several authors have emphasized the potential advantages of computers for autism. For example, computers are claimed to be reassuring, controllable and adaptable

(Dautenhahn, 2000; Bosseler and Massaro, 2003).

However, authors also suspect problems could arise from computers. Specifically, people with autism could have difficulties in generalizing learning acquired on the computer to everyday life (Bernard-Opitz et al., 2001).

Hetzroni and Tannous (2004) addressed this point in a study assessing the influence on everyday life communication skills of a multimedia application designed for teaching vocabulary to children with autism.

Issue raised for multimedia interfaces

Wong et al. (2004) studied the neurological skills required for adequately using a computer. They recommended reducing visual complexity and using simple language along with audio reading for people with mild cognitive disabilities.

Conflicting results between two studies that involved individuals with autism:

Chen et al. (2005) compared three different ways to display text on a computer: text only, text with images, text and audio. The best text comprehension performances of participants with autism were achieved with the combination of text and images.


Schlosser and Blischak (2004) evaluated a speech-generating device for teaching literacy skills to children with autism. They found that the efficiency of a synthetic speech feedback over a visual feedback depended on the child’s learning style. Although the impact of synthetic voices seemed to vary, there was no indication that it could hinder learning.

The question arises if combining multimedia sources of information such as images, text and speech would foster or hinder comprehension.

Several projects give special emphasis to the inclusion of Animated Conversational Agents (ACA) in interfaces intended for users with autism. The underlying reason is that an ACA communicates through modalities such as speech, facial expressions and gesture that are inspired from human communication. Moreover, while resembling human characters, researchers believe an ACA could enable to control the interaction at a suitable level for people with autism.

Bosseler and Massaro (2003) tested a language-learning tool that used a virtual 3-D talking head. The virtual head could simulate realistically the articulary movements of the mouth and tongue during speech. The speech output relied on a synthetic voice. Eight children with autism were trained during 6 month with this tool. Pre- and post-tests revealed that children acquired new vocabulary and that learning was stable.

Tartaro and Cassell (2006) explored the use of an ACA for training children with autism in collaborative storytelling. They designed an authorable virtual peer, which is a virtual character that looks like a child and can communicate through speech, gesture and gaze. Being authorable, it also enables the child to specify and plan its interactions and control it during storytelling sessions with another person.

Moore et al. (2005) investigated the ability of people with autism to interact with animated characters. People with high functioning autism have been shown to hold average performances in recognizing basic emotional facial expressions (Baron-Cohen et al., 1997).  Questions remain regarding their ability to understand facial expressions displayed by animated agents in relation to a context. In the experiment of Moore et al. (2005), participants had to perform tasks that required associating the animated agent’s facial expressions (happy, sad, angry and frightened) with an emotion or an emotionally connoted social situation. Results showed some evidence that people with high functioning autism could assign the appropriate emotional state to the animated characters. The authors suggest using collaborative virtual environments, which potentially enable users to interact with one another through animated agents.

The fact that people with high functioning autism can recognize animated agents’ facial expressions does not necessarily imply they are able to use them in conjunction with other communicative modalities. à Generalization Issues

Golan and Baron-Cohen (2006) developed and evaluated a multimedia application to train recognition of complex emotions (such as embarrassment, intimacy, etc.) in both visual and auditory channels for people with high-functioning autism. Their software demonstrates each emotion in silent films of faces, faceless voice recordings and videos of situations involving the emotion. Nineteen participants with high-functioning autism were trained with the software during 10–15 weeks. Although they improved for emotion

recognition in faces and voices separately, results did not show any gain on holistic tasks involving integration of facial, vocal and contextual cues in videos. Recognizing emotions in real social situations requires cross-modal processing of different socioemotional cues. The quality of interaction with a facially expressive animated agent is influenced by the ability to integrate information from facial expressions and linguistic sources. This brings forth the need to assess this ability for people with high functioning autism.

The literature suggests that people with high functioning autism face particular difficulties in associating emotions with an ever-moving context.

Nadel et al. (2000) showed that even people with low functioning autism could develop social expectations from others, despite their profound social disorders. But, according to Loveland (2005), people with autism often fail to use perceived social and emotional information to self-regulate their own behavior with an ongoing social situation.


Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: