On “Emotion in Human-Computer Interaction” and Developing Compassion

People are inherently emotional. Emotions, sentiments, and moods effect each aspect of response and interaction in our lives. Indeed, if any interface ignores this, it “risks being perceived as cold, socially inept, untrustworthy, and incompetent.” Part of the truth of this lies in the personification of the computer or software by the user. However, we also have the choice to change our emotions, sentiments, and moods in order to modify our behavior. The key lies in consistent intention and progression.

Brave and Nass reveal the underlying mechanisms of emotion with a figure that demonstrates the connections between the thalamus, the cortex, and the limbic system. The thalamic-limbic pathway is responsible for the primary emotions, while secondary emotions “result from activation of the limbic system by processing in the cortex.” They then go on to cover the debate over whether or not emotion is innate or learned. Regarding this debate, Brave and Nass describe the middle of the two extremes where “the limbic system is prewired to recognize the basic categories of emotion, but social learning and higher cortical processes still play a significant role in differentiation.”

One recent article that came to my attention involves the ability to learn compassion using meditative states. The experiment in the article entitled Regulation of the Neural Circuitry of Emotion by Compassion Meditation: Effects of Meditative Expertise used functional magnetic resonance imaging (fMRI) to measure differences in brain activity between novice and experienced meditators, those having 10,000 plus hours of Buddhist compassion mediation practice. They were asked two alternate between actively generating a condition of compassion mediation and refraining from the practice.

In this case:

The meditative practice studied here involves the generation of a state in which an “unconditional feeling of loving-kindness and compassion pervades the whole mind as a way of being, with no other consideration, or discursive thoughts” ... According to the tradition, as a result of this practice, feelings and actions for the benefit of others arise more readily when relevant situations arise. Our main hypothesis was thus that the concern for others cultivated during this meditation would enhance the affective responses to emotional human vocalizations, in particular to negative ones, and that this affective response would be modulated by the degree of meditation training.

They analyzed the areas of the brain associated with empathy including the insula cortex and the somatosensory cortex. The data support their main hypothesis, namely that “the brain regions underlying emotions and feelings are modulated in response to emotional sounds as a function of the state of compassion, the valence of the emotional sounds and the degree of expertise.”

One interesting way to see these results is to acknowledge that the meditation itself, over time, changes the neurology of the practitioner in such a way that the empathy and compassion become more automatic and spontaneous. In the section on Effects of Affect: Attention, Brave and Nass state, “people also often consciously regulate mood, selecting and attending to stimuli that sustain desired moods or, alternatively, counteract undesired moods.” Compassion meditation would then put people in a more sustained compassionate mood. I say mood because it seems to fit the definition better than emotion. “Moods...are nonintentional; they are not directed at any object in particular and are thus experienced as more diffuse, global, and general.” This would seem to be the case with long-term practitioners of compassion mediation. It becomes a “way of being.”

Brave and Nass support this by saying, “Intense or repetitive emotional experiences tend to prolong themselves into moods.” The Lutz, Johnstone and Davidson study used fMRI results to measure affect, but Brave and Nass also suggest other methods for doing so, including electroencephalogram (EEG) to test neurological responses, autonomic activity, facial expression, voice, self-report measures, and affect recognition by users.

Depending on the conditions of the study, these methods for testing affect differ in efficacy. The Compassion Mediation study did not use behavioral analysis during the testing because the meditators informed them that this would interfere with the meditation process itself. So they has to rely on the fMRI measurements and a certain amount of self-reporting. Self-report measures, in particular, suffer from a problem with temporal relevance. Brave and Nass point out that “questionnaires are capable of measuring only the conscious experience of emotion and mood. Much of affective processing, however, resides in the limbic system and in nonconscious processes.”

The dimensional theories using arousal and valence were also used during the Compassion Mediation study and with correlational results. This would support that for further research, where any self-reporting is necessary, compassion should be tested as a mood emergent from a two-dimensional space of “conscious emotional experience.”

By Christine Rosakranse
Professor Nathan Freier
Theory and Research on Tech Comm and HCI
September 29, 2009

References:
Brave, S., & Nass, C. (2007). Emotion in human-computer interaction. In Sears, A. & Jacko, J. (Eds.). The Human-Computer Interaction Handbook: Fundamentals, Evolving Technologies and Emerging Applications, 2nd Edition. (pp. 77-92). Lawrence Erlbaum.
Fledman Barrett, L. & Russell. (1999). The structure of current affect: Controversies and emerging consensus. Current Directions in Psychological Science, 8(1), 10-14.
Lutz A, Brefczynski-Lewis J, Johnstone T, Davidson RJ (2008) Regulation of the Neural Circuitry of Emotion by Compassion Meditation: Effects of Meditative Expertise. PLoS ONE 3(3): e1897. doi:10.1371/journal.pone.0001897



1 Brave, S., & Nass, C. (2007). Emotion in human-computer interaction. In Sears, A. & Jacko, J. (Eds.). The Human-Computer Interaction Handbook: Fundamentals, Evolving Technologies and Emerging Applications, 2nd Edition. (pp. 77-92). Lawrence Erlbaum.
2 Brave, S., & Nass, C. (2007). Emotion in human-computer interaction. In Sears, A. & Jacko, J. (Eds.). The Human-Computer Interaction Handbook: Fundamentals, Evolving Technologies and Emerging Applications, 2nd Edition. (pp. 77-92). Lawrence Erlbaum.
3 Lutz A, Brefczynski-Lewis J, Johnstone T, Davidson RJ (2008) Regulation of the Neural Circuitry of Emotion by Compassion Meditation: Effects of Meditative Expertise. PLoS ONE 3(3): e1897. doi:10.1371/journal.pone.0001897
4 Brave, S., & Nass, C. (2007). Emotion in human-computer interaction. In Sears, A. & Jacko, J. (Eds.). The Human-Computer Interaction Handbook: Fundamentals, Evolving Technologies and Emerging Applications, 2nd Edition. (pp. 77-92). Lawrence Erlbaum.
5 Brave and Nass, pg. 87.
6 As described by Fledman Barrett, L. Russell, 1999. The structure of current affect: Controversies and emerging consensus. Current Directions in Psychological Science, 8(1), 10-14.

The Rise of Truly Emotional Computing

Brian R Zaik

It’s clear from taking a look at past research in the space that emotional responses to computing is a topic of great interest in the world of HCI. But I would also suggest that these articles, as well as my external knowledge of what exists right now in the world, help me to conclude that emotional computing is a frontier that has yet to be fully explored.

The first questions to answer are the most basic: do people actually develop emotional attachments to computers and computing devices? Do they assign emotional labels and metaphors to their computing experiences? And do they view computers as social actors with whom they may interact in ways similar to how they may interact with human beings? Nass et al. focused on what we might term as more traditional computing experiences (text-based), providing strong evidence that human beings begin to assign social and emotional context to computers as soon as the machines exhibit characteristics that may be compared to human characteristics, such as gender, attitude, disposition, and personality. While today we may take the findings of this 1997 study for granted, I imagine that Nass and his associates made quite an impact on the field of psychology upon the publishing of their research. To me, the question now is not, “Can users view computers as social actors with emotions and personalities?,” but rather, “How can we best design computers to harness emotions for the benefit of their users?”

Clark and Brennan showed that grounding is an essential part of human communication. And the process of grounding, as they defined it, involves the coordinated action of all the participants. Being aware of emotions and the emotional context of a communication is also key to grounding, as it will allow both parties of a two-way exchange to understand the implications of a particular message (though this is not always likely to work out perfectly). In order for computers to truly interact with us on a deeper level, they must be able to both elicit emotions from their users AND respond to those emotions in kind. The research study covered in Nass et al. only examines the first of these design requirements – as they state, every kind of interaction between the user and the computer was scripted for the study. While I can believe that even text-based interactions can generate social responses from users, I find it difficult to view that kind of interaction on the same level as human-to-human interaction. It’s one thing to be able to generate a one-sided social response from the user and another to be able to engage user and machine in a two-way, grounded exchange. I would argue that the true future of emotional computing lies in the design of computers that can both study human reactions AND dynamically deal with the reactions they recognize to improve the effectiveness of the human-machine exchange.

The European Union (EU) recently funded a research program aimed at realizing that kind of future. The Humaine project (1) is founded upon the notion that interfaces haven’t developed enough beyond simple user interface mechanics, despite massive gains in computing power over the years. Humaine is an attempt to expand the modern palette of HCI research beyond those mechanical approaches. To do this, Humaine researchers have started back at the ground level to figure out exactly how human beings should interact with computers, and vice-versa. Humaine’s strength lies in its interdisciplinary nature: psychologists, philosophers, artists, and computer scientists all work together to better understand how emotion can be incorporated into HCI design. And one of the first objectives of the program is to develop computer systems that can recognize human emotions using multiple modalities.

These systems have already been tested in museums in Scotland and Israel (2). In these trials, museum guides were issued to visitors. These handheld computers included earpieces and microphones to monitor visitors’ levels of interest in different types of display and react accordingly. As the Humaine program coordinator, Professor Roddy Cowie, points out, “While this is still at a basic level, it is a big step up from a simple recorded message.” This kind of interaction certainly goes far beyond the simple studies that Nass and his colleagues conducted in 1997.

Perhaps we could build emotion-aware computer systems that operate as closed loop machines. The computer would be capable of recognizing and analyzing how human users react to the user experience in front of them, and later shift its behavior to better suit the user. Nass et al. showed that the strength of the computing experience could be enhanced by strongly matching the user’s personality with a similar computer personality. For example, the data supported the conclusion that users with submissive personalities are generally more attracted to computers they view as submissive. We must realize that the computing experience might need to be finely tuned to the specific personality of the user, and that’s why being able to dynamically adapt the computing experience to the exhibited emotions of the specific user makes so much sense. In the lab, scripting experiences and personalities may suffice, but in order for computers to be truly capable interacting with human users on an emotional level, the experience must dynamically change based on the emotional detections and compensations of both parties.

One other consideration of the Humaine project is how computer representations can be designed to elicit emotions from users most naturally. Professor Cowie claims that Humaine researchers have “identified the different types of signal which need to be given by an agent – normally a screen representation of a person – if it is going to react in an emotionally convincing way.” This brings up another question relevant to our in-class discussions: are graphical avatars effective ways of displaying and exhibiting emotions in a two-way exchange between a human and a computer? Is an avatar that resembles a person necessary to interface properly with a human being on an emotional level? Nass et al. concluded that even simple text messages are enough to elicit emotions from human users, yet researchers are still trying to figure out the true benefits of more sophisticated ways of representing computers. All the way back in 1987, Apple painted a clear vision of the Knowledge Navigator, a type of computer assistant that could interface at a level similar to how two human beings may interact. In that video, Apple showed used a graphical, humanlike avatar for the Navigator’s interactions with human users. Is the computer avatar here to stay?

Emotional computing holds promise in allowing computers to better anticipate and respond to human emotions, which may help us to design breakthrough computer interfaces that can adjust to users. The Humaine program in the European Union may be an important building block for this future. For now, though, we’ll be the only ones who hear all those insults we throw at our computers.

FOOTNOTES:

1. McKie, Robin. "Machine rage is dead ... long live emotional computing." The Observer. Guardian News and Media Limited, 11 Apr. 2004. Web. 28 Sept. 2009.


2. "ICT Results - Emotional Results." ICT Results. European Commission, 3 Apr. 2008. Web. 28 Sept. 2009.