"Affective computing" is defined as the study and development of systems, computational devices, or wearables that can process, interpret, recognize, and detect human affects (Picard, 1997). This area of research is very active and is driven by several promising areas for applications such as virtual reality, smart surveillance, perceptual interfaces, and so on (Tao, 2005). As early as 1960, Page advanced a control-centric theory that proposed that users someday would be able to share their thoughts in real times with machines (Page, 1962). In that side, there is an even wider long term picture in the pursuit of affective computing. 

Affective computing is also related to artificial intelligence. For example, the Hal 9000 computer in the Stanley Kubrick and Arthur C. Clarke film "2001: A Space Odyssey" is an affective computer. Hal could recognize individuals, read lips, understand the aesthetics of art sketches, and recognize the emotions expressed by his astronaut colleagues (Picard, 2001). This example also is a cautionary tale of affective abilities we also might not want to give machines such as the ability to lie and do physical harm to humans. Ultimately though, affective computing is not about giving machines emotions but the ability to have emotional intelligence and learn about the affective state of humans. Picard and others proposed in 2001 that machine intelligence needs to include emotional intelligence and that affective computing is a way to provide this capability to machines by processing physiological signals to infer information about human affects (Picard et al., 2001). Affective computing research is progressing and finding new applications and use cases as time progresses.

Figure 1:  A High-Tech Feather Chip on a Human Palm

The benefits outweigh the risks. For example, besides artificial intelligence, affective computing has been proposed as a way to combat drug addiction (Gustafson, 2005). Researchers have also developed affective wearable devices that reports on social-emotional information in real-time to assist individuals with autism (El Kaliouby, 2006). Research at Kansas State University suggests another area that deserves attention: built environments. The interdisciplinary nature of this research holds promise to each researcher of varying background by the very merit of the multifaceted nature of the research problem.

Affective computing shows promise in eventually teaching machines to be given the ability to sense and recognize expression of human emotions such as interest, distress, pleasure, and stress with the goal of having the machine choose more helpful and less aggravating behavior (Picard, 2004). This article provides the reader an overview of this growing field, the emerging interest at Kansas State University, and a brief discussion of the future potential of the field.

Affective computing research started picking up steam in the late 1990s. In 1997, Picard and Healey demonstrated the use of wearables to detect human affect states using respiration, galvanic skin response (GSR), blood volume pressure (BVP), and electromyogram (EMG, and heart rate (HR) (Picard, 1997). They also outlined several of the general challenges which have persisted in the field ever since in using wearables to infer human emotions. A year later, they demonstrated that the same signals could be used to detect human affect states using pattern recognition via Fischer linear discriminant. They also considered the participants' affective state changes over an average period of three minutes when previous studies were typically one to ten second reactions (Healey, 1998). Researchers have designed experiments which the goal of frustrating the participants on purpose in order to detect their physiometric and emotional state (Cacioppo, 1990). Detecting stress using physiometric signals such as galvanic skin conductivity and real time video to capture participants point of view was pioneered by Healey with StartleCam (Healey & Picard, 1998). StartleCam could detect when a participant was startled by using pattern recognition that relied on a time-reversed filter and convolutional sum. In other words, the device could detect when the participant were startled or stressed by smoothing out noise in the data while preserving the core pattern produced by the sensors.

Interestingly, wearable jewelry, shoes, glasses, and other affective wearable devices were anticipated in the late 1990s by researchers as well (Picard, 2000). Some examples of this nascent research were glasses equipped with embedded sensors to detect confusion and interest from users with moderately high accuracy (Scheirer, 1999). User annotated approaches to studying affective computing were also explored in the offline and online recognition of emotion from physiological data. Specifically, Picard had a trained actress equipped with various sensors that measure EMG, BVP, and GSR display neutral, anger, hate, grief, platonic, romantic, joy, and reverence to be recorded by the computer. Sequential floating forward search (SFFS) and Fisher projection (FP) were applied that achieved a recognition rate of 50% to 80% depending if the algorithm was run offline or online (Picard, 1999). Healey studied driver stress by measuring a user’s face with a camera that recorded their facial expressions approximately once a minute while driving. In addition, she measured BVP, EKG, and GSR. The initial study determined there were many confounding factors and participant’s signals would differ to similar stimuli differently over the period of the study (Healey, 1999).

At the beginning of the early 2000s, research began to expand into other areas and grow more sophisticated. For example, affective medicine, which is teaching computers the recognition of emotion using variables that respond to people with active listening, empathy, and sympathy (Picard, 2002). Work also continued on recognizing the affective state of a human while driving. Healey was able to recognize patterns of about 80% for human stress in drivers by using a linear discriminant function while also recording a participant’s respiration, EMG, HR, and GSR (Healey, 2000). An interesting head nod and head shake detector was created by researchers who used hidden markov models that tracked pupils, pupil position, and directions of head movements to detect when a head nod or shake occurs (Kapoor 2001). In 2002, researchers also started looking into using affective computing to assist individuals diagnosed with autism (Blocher, 2002). Affective computing has also been used in monitoring stress and heart health in real time with a phone and wearable computer as early as 2002 (Picard, 2002). 

Real-time facial feature tracking continued to evolve by tracking eyes and eyebrows in real time extracting features using PCA (Kapoor, 2002). Modeling also was becoming more sophisticated, where researchers used dynamic Bayesian networks (DBN) and mixture of hidden Markov models to classify driver’s speech under stress (Fernandez 2003). By 2004, there was a study by the Defense Advanced Research Projects Agency (DARPA) to study emotional state recognition to determine potential criminal intent which also sparked debate about ethics in affective computing (Reynolds, 2004). Researchers also proposed around this time that affective systems should be minimalist in ubiquitous interface design. That is, the combination of parsimony and transparency where parsimony means a user interface as possible and transparency by being designed to reduce cognitive demands as much as possible (Wren, 2004). It has also been shown that when agents are designed to simulate care and empathy, that individuals both perceive the agents are caring and are willing to continue working with them (Bickmore, 2004).

By 2005, the situation continued to evolve and find many new applications. Research has also been conducted into considering affective computing in adversarial situations such as poker or interviewing experiments (Reynolds, 2005). One interesting result was showing how good typography can induce a good mood, which in turn has been shown to induce individuals to perform better cognitively and more creatively (Larson, 2005). Examination of stress also continue to advance and grow in sophistication. Healey showed that you can use physiological signals to provide a metric of stress for individuals driving a vehicle and collected data for approximately 50 minutes (Healey, 2005). Researchers used Bluetooth and a custom sensor called HandWave to measure skin conductance and used this to detect the emotional, cognitive, and physical arousal of users (Strauss, 2005). Affective computing also was shown in 2005 to have potential in both educational learning environments and sensor fusion, Kapoor and Picard demonstrated that multimodal mixture Gaussian models could achieve accuracy of over 86% in detecting if children were interested or disinterested in solving a puzzle (Kapoor, 2005). 

A few years later, Kapoor went further and demonstrated a similar approach using Gaussian and Bayesian methods has 79% accuracy in detecting user frustration when using an intelligent tutoring application (Kapoor, 2007). Research also continued into looking at the benefits of affective computing in assisting individuals who have autism (Hoque, 2008). Sensor technology itself started to advance in useful ways. In 2009, Poh and Picard developed a wristband for ambulatory assessment of electrodermal activity (EDA) (Poh, 2009). This technology would eventually find its way into products on the commercial market such as those offered by Empatica. Heart rate variability and electrodermal activity to classify children who had sensory impairment versus those who did not using sensors was explored using machine learning such as KNN, Decision Trees, SVM, and Linear Discriminants with some accuracy (Hedman, 2009). This demonstrated the potential viability of applying machine learning to affective computing problems.

Affective Computing since 2010 has observed rapid advances in sensor technology, applications of machine learning, and commercialization of wearables. In 2010, Poh and Picard continued to perfect their wrist worn electrodermal activity (EDA) wrist band and showed it had advantages over FDA approved products at the time (Poh, 2010). Eydgahi and Picard proposed long term continuous monitoring of physiological data and the potential for commercial products to fit this niche (Fletcher, 2010). Also, during the same year Fletcher and Picard also proposed wearable sensors being used with mobile phones as a platform for advances in affective computing and low cost health care (Fletcher, 2010).  

In 2011, researchers pioneered an affective computing furniture in the way of a medical mirror that provided health information related to a person tracking their daily vitals to aid in their health management (Poh, 2011). Researchers also trained computers to classify polite smiles versus actual smiles, the context was a banking scenario with a banker and a client (Hoque, 2011). Cardiovascular monitoring using earphones and a mobile device was also developed which relied on a digital signal controller to process signals and sensors from the headphones (Poh, 2012).

In 2013, a novel mirror sensor system and interface that gave user feedback on their inner physiological responses with the goal of improving scientific understanding of psychophysiology in natural settings (Hernandez, 2013). One of the first applications of affective computing in political science was when researchers measured individual’s candidate preferences based on video clips measuring their spontaneous facial reactions with accuracy of over 74% (McDuff, 2013). Using mobile phones, researchers used data collected from a mobile phone usage and wrist sensor collected over five days to accurately detect stress using binary classification of 75% based on surveys as a baseline (Sano, 2013). In 2014, development in commercial wearables with high precision biosensors began to take off. Picard introduced the Empatica E3 wearable, a commercial product that has photoplethysmograph (PPG), electrodermal activity (EDA), 3-axis accelerometer, and temperature that uses Bluetooth (Garbarino, 2014). Currently, this product has been superseded by the Empatica E4. McDuff (2014) demonstrated the ability to measure cognitive stress remotely via heart rate variability using Naive Bayes and SVM as classifiers.

In 2015, Rivera demonstrated that wearables could be used towards the management of stress and detection by using Naive Bayes and SVM (Hernandez, 2015). Recent research has also begun exploring the potential of affective wearables to detect seizures in outpatient settings. The contemporary pace shows signs that the pace of advancements is accelerating and all evidence points towards more powerful wearable devices and sophisticated machine learning approaches such as deep learning. Recently, at Kansas State University, researchers are considering how affective computing can help us better understand how humans react to different environments in urban settings.

The field of affective computing is very young. Arguably, it is less than 25 years old. There is an emerging group of individuals with diverse backgrounds and disciplines that has formed a group to explore affective computing problems at the Biosecurity Research Institute, Kansas State University. The diverse background in expertise which is indicative of how interdisciplinary affective computing is. Dr. William H. Hsu is Director of the Knowledge Discovery in Databases (KDD) at Kansas State University and is an expert in artificial intelligence and machine learning. Will Baldwin serves as the director of IT at the Biosecurity Research Institute (BRI) and is an expert in software engineering, sensors, and Internet of Things (IoT). Dr. Brent Chamberlain is an expert in visualization, GIS, environmental psychology, and computational approach to landscapes. Heath Yates is a software engineer at the BRI and a research graduate assistant at KDD whose doctoral research is focused on affective computing.

Figure 2: Affective Computing Team at Kansas State University (From left to right are the following:  Dr. William H. Hsu, Heath Yates, Dr. Brent C. Chamberlain, and Will Baldwin

The team is currently working on custom mobile sensors which will record data from participants to indicate their emotional states. This will help the field better ground truth the data being produced by wearables and the team hopes the results lead to improved reproducibility and replication. Also, the team is currently looking into refining machine learning approaches to detecting human emotion using sensor fusion based on the participants’ environment.

Figure 3: Working on a Sensor to be used in an Experiment

The sensor shown above will ultimately be used in affective computing experiments by asking users to indicate the intensity of their emotions. Interest in affective computing is growing and there are other researchers at Kansas State University doing work in affective space who will soon be joining the team.

Figure 4:  Dr. Carlos Castellanos, Interdisciplinary Scientist and Artist at K-State 

Dr. Carlos Castellanos is an interdisciplinary scientist and artist. He is reinterpreting affective computing and exploring its potential as a medium of art. This allows individuals who view the art to also actively participate in it. In short, affective computing is showing promise to evolve art into a more participatory, interactive, and fluid medium. Castellanos implemented a server that responds to a body’s physiological and emotional state to interact through a software program (Castellanos & Schiphorst, 2009).

Figure 5: BodyDaemon Art Demonstration

A video of this may be accessed below.  

Research into affective computing at Kansas State University is young, vibrant, and exploding. New and existing collaborations are quickly forming, deepening, and expanding. The team is looking for collaborations beyond Kansas State University. Specifically, the emerging area will require interested researchers and students working together to solve novel problems in detecting human emotion in a multitude of applications.

While affective computing is a young field, there have been many advances in the last two decades. Wearables have allowed researchers to ask new questions and provided new kinds of data which can be used to infer human emotional states and to teach computers how to recognize them. The applications have found themselves in a diverse range of scenarios from autism, stress detection in driving, recognition of emotion in faces, and so on.

At Kansas State University, researchers are hoping to expand the team with individuals whose core shared interest to further research and exploration of affective computing. It is a nascent field of study that shows promise. Most importantly, it is hoped that this new, emerging, and novel area of research will attract students from all backgrounds and disciplines. The interdisciplinary nature of the area will necessitate the need for students from architecture, art, computer science, electrical engineering, psychology, kinesiology and more. In the immediate future at Kansas State, there will be a lot of work on the refinement of sensor technology and refining machine learning approaches in detecting human emotions.

The future is difficult to predict precisely, but there are currently some undeniable trends which are worth mentioning in context of affective computing. The revolution in artificial intelligence and machine learning currently underway is most visible in the recent advances in computers beating humans in matches at the game of go. Recent trends in industry to make computers more responsive to user needs necessitates that these machines are at least aware of human emotion on a basic level. This has the potential to disrupt how we interact with technology in our vehicles, homes, and hospitals.

L. Ashmore is acknowledged for taking the pictures of the affective computing team.  Thanks to Dr. Carlos Castellanos for his gracious assistance.

Heath Yates is a Ph.D. candidate (in computer science) and software engineer at the Biosecurity Research Institute (BRI). His research interests are in affective computing, wearables, machine learning, statistical learning, and artificial intelligence. He has three masters in computer science, statistics, and mathematics. His passion is research, educational outreach, and finding ways to do data science for social good. When not researching or programming, his passion is spending time with family, reading, and exercising.  He may be reached at hlyates@ksu.edu.  

(Note:  The photo of Heath Yates was taken by Dan Donnert.)