Analyzing Customers' gaze pattern with AI―Support for More Personalized Customer Service

Customer Service Support Demonstration Experiments at Aoyama Trading Shops

(Photo of the Fujitsu Forum 2018 exhibit)

Technologies such as AI (artificial intelligence) and IoT are being used increasingly in products and services that surround us in our daily lives. Advancements in the development of psychological analysis AI technology, which can assess the psychology of people through subtle changes in speech and behavior, has led to growing possibilities for use in face-to-face tasks such as customer service. Fujitsu and Fujitsu Laboratories aim to utilize this AI technology in major retailers and smaller stores, and apply it to more personalized promotional activities that target individual customers, as well as to more personalized customer service tasks.

Using this psychological analysis AI technology, Fujitsu and Aoyama Trading conducted verification testing for about a month starting on April 6, 2018, at the Aoyama Tailor Ikebukuro Higashiguchi central store.

gaze tracking sensor were installed between two mannequins at the shop's entrance, and movements in the customers' gaze (pattern/direction) were analyzed as they browsed through products, such as jackets, shirts and neckties worn by mannequins. The AI chronologically analyzed each customer's movements in their gaze pattern, including how many times they looked at a product, how long their gaze stayed fixed on a particular product, and in what order they browsed through products. Then, it assessed the customer's state of mind, such as how they were trying to decide between two jackets worn by two mannequins, or if they were interested in neckties as well. Based on those assessment results, additional product information, such as how one necktie would go well with a certain other product, and other recommended product information were displayed on the screens next to the mannequins in real time. This provided a new shopping experience that recommended items to customers based on what they were thinking.

In addition, the results of the customers' psychological assessment were sent to the mobile devices carried by the sales floor staff in real time. This helped staff members understand the customers' intentions even while not standing near them. Staff members were then able to approach customers and make comments relevant to the products they're interested in, allowing for more attentive customer service.

The mobile devices carried by sales floor staff also display the customers' psychological assessment results and recommended product information based on them.

Improving Store Value by Utilizing Psychological Assessment AI Technology

This technology is based on gaze tracking technology, a sensory media processing technology that is one of the core technologies of Human Centric AI Zinrai (referred to as Zinrai below), which is the systemization of Fujitsu's cutting-edge AI technologies. It is also based on gaze pattern and pattern analysis technology. Sensory media processing technology is a form of AI technology that creates new value by capturing people's thoughts and supporting them through their interactions with other people, as well as with various objects and machines in the real world.

By utilizing this psychological assessment AI technology to accumulate analysis data and customer interaction history data of the staff, potential customer needs can be visualized, know-how related to customer interaction can be shared among staff members, and data that is useful when creating new shops and products can be obtained. This can result in various advantages, such as the enhancement of customer service tasks, improvement of store value and the use of data in sales promotions and in developing manufacturing strategies. This technology and vision are organized as 'Affective digital® marketing'. It can be said that these efforts to improve customers' shopping experiences using Affective digital® marketing technologies are one of the key ways of operating physical retail stores in this day and age.

Aiming to Provide Support Using Psychological Analysis AI Technology in Various Business Scenarios

Going forward, Fujitsu aims to use not only line-of-sight analysis, but also analyses of other factors such as subconscious physical movements and satisfaction assessments based on voice pitch to achieve improved usability for customers in various industries and scenarios, as well as for enhanced customer service through optimal combinations of sensory media processing technologies. Even in a variety of other customer interaction situations, such as face-to-face customer guidance, operations through remote screen systems, guidance using digital signage displays and shopping on e-commerce sites using mobile devices, better AI support for customer service that responds to psychological changes regarding products and services can be achieved with appropriate recommendations and timing.

Going forward, Fujitsu and Fujitsu Laboratories aim to achieve business support services that enhance customer service tasks through using AI technology that coexists with people and provides more personalized experiences.

Takuya Uemura
Senior Researcher
Fujitsu Laboratories Sense Computing Research Center