[Article 3] Ever Evolving Artificial Intelligence--AI Interpreting Human Emotions from Their Behavior

Human Centric AI Zinrai is the product of Fujitsu's extensive technical expertise in the AI field. This article features Affective Media technology that is used to interpret human emotions from their behavior. How will our lives be impacted when AI will be able to interpret our feelings and understand when we change our mind?

Real-time Interpretation of Human Thoughts and Emotions

Artificial intelligence has evolved to spawn robots able to communicate with humans. Now interactive robots can provide solace to the lonely and there are even robots that defeat professional human chess players.
Against such a backdrop, Fujitsu is advancing its research on "AI that can provide support in a natural way in human living environments" with an awareness and attentiveness like humans.

Affective Media technology is an AI technology for creating new value by understanding real-world human-human and human-machine interactions. AI can process even our emotions, awareness, and attentiveness by combining media technology to sense human responses and behaviors such as voices, expressions, and small gestures through video and audio with unique information processing algorithms, psychology, sociology, and extensive professionals' knowledge.

AI-Supported Crime Prevention

One of the features of Affective Media technology is to create new value by understanding the communication between humans. The AI wisely monitors humans with both sensing technologies and processing algorithms, and provides optimal encouragement. This application is useful to prevent phone frauds, which are becoming increasingly frequent every year.

Specifically, Fujitsu is proceeding for applying this technology to a fraud warning system which detects suspicious calls that may be associated with furikome fraud calls (remittance-solicitation phone fraud) from the tone of voice and gives warning with a synthesized voice to the victim. By sending an alarm on a suspicious phone call to not only the victim but also his/her family, the police and other related institutions such as banks, the damage caused by phone fraud can be prevented in advance.
This system detects the possibility of a furikome fraud by analyzing the voices of people having a phone conversation using the algorithm which combines the detection of stress levels and keywords unique to phone fraud.
Humans get thirsty when they are stressed and the pitch and volume of their voice change from its ordinary condition. This technology automatically detects these subtle changes in the voice through sensing technology and identifies whether or not the victim is in an overtrust situation where he/she may have a diminished capacity to objectively evaluate information due to the stress caused by a perpetrator.
AI presumes a "situation being solicited phone fraud" by combining the detection of the overtrust situation with the detection number of keywords unique to phone frauds used in the conversation.
In evaluation tests using this technology, phone frauds are accurately detected in more than 90% of cases.
Fujitsu has been proceeding with field trials of this technology in Okayama Prefecture since 2012 to help eliminate furikome fraud. As a result, the number of recognized victims of "It's me" fraud, which had been an average of 2.7 incidents per month in Okayama Prefecture before the field trials, has decreased to between 0 and 1 incidents on average per month during the field trials. Therefore, this technology is expected to be effective in deterring crime.

[Detecting furikome frauds using Affective Media technology]

In previous technologies, in order to detect an overtrust situation in the conversation between a victim and a suspect, it was common for a computer to learn in advance about the person's normal state and then determine an overtrust situation based on the difference from the person's normal state. However, by using AI, it is possible to detect an overtrust situation for an unspecified large number of people by capturing subtle responses from human interactions without learning about the person's normal state in advance.

AI Captures the Human Thoughts from Eye Movements

Another feature of AI using Affective Media technology is that it can be used to provide personalized services by accumulating and analyzing data on human behaviors and responses captured by sensing technology.
For example, a gaze-tracking sensor used at retail stores can detect where people are looking in real-time by capturing minute movements of their eyes while they hesitate between several products.
AI can be used to infer the customers' psychological states from gaze data. This means that not only can we know where the customers are gazing and what products the customers are looking at, we can also infer whether they are "focusing on an item," "undecided about which design to choose" or "interested in an item." Application of this technology makes it possible to predict shoppers' purchase process or hesitation before deciding, such as comparing with other products before deciding to purchase a product. This is expected to contribute to improving marketing activities.

Other than the above applications, AI can be used for ATMs and ticket vending machines at stations to provide support according to each person's needs. For example, by identifying that a person is having trouble in real time from their gaze and finger movements, a system can guide the customer through, for example, the different steps of making a wire transfer or buying a train ticket to Ueno station by providing timely and appropriate information.

Life in the Future That Affective Media Technology Will Bring About

Affective Media technology is expected to be utilized in various scenarios where communication between people or interactions between people and things/machines in the real world take place. Affective Media technology is applicable to the field of retail and distribution as well as the field of education. If AI understands the degree of interest, the process toward understanding, and difficult points to understand by monitoring responses and perceiving changes in facial expressions of children and students, this technology can be used to improve instructions for each child and student according to their respective situations.
Fujitsu plans to develop human behavioral and response data into common knowledge that can be applied to a wide range of services worldwide. Fujitsu aims to create a future where AI has awareness and attentiveness and expresses feelings of trust like humans so as to create value together with humans by extracting and improving wisdom in the spirit of empathy and co-creation.