"The eyes are more eloquent than the lips" - this metaphor tells us that we can obtain insight about interests, feelings, and inner motives of a person from their eyes. Often, the movements of the eye can tell us much more about a person's intentions than their words.
People naturally turn their eyes towards things or people they want to deal with or when they want to do something. So Fujitsu focused on these eye movements to create "Gaze Tracking" technology that infers what people want to do and supports them without being obtrusive.
Fujitsu Laboratories has been researching and developing this Gaze Tracking technology to track people's gaze by examining the movements of the eye when they look at something. Fujitsu computers offer a Gaze Assist function to automatically perform various operations like scrolling and zooming. This function has been available on Fujitsu computers since October 2012.
In order to realize this technology, Fujitsu had to overcome two major challenges.
The first challenge was to develop a compact sensor. If the sensor--which consists of an infra-red camera and LED--is large, it becomes difficult to embed in various devices or install easily in different places.
The other major challenge was cost reduction. In order to track a person's gaze with high accuracy, it is necessary to take clear images of the eye. However, using a high-performance camera and multiple LEDs to take clear images makes the sensor expensive and bulky.
In other words, if researchers aimed for a compact and low-cost sensor, the images taken were unclear, and accuracy was compromised. As a result, it was extremely difficult to commercialize such a sensor.
However, a group of researchers at Fujitsu Laboratories, led by Satoshi Nakashima, a Research Manager at the Image Computing Lab within the Media Processing Systems Laboratories, successfully overcame these challenges. Building on their image processing technology developed for face detection and human detection sensors, the team developed technology to detect people's gaze with high accuracy even when the images were blurred.
The video below explains further the key features of Gaze Tracking technology.
The secrets to tracking a person's gaze with high accuracy using a compact, low-cost sensor
The basic method of performing gaze tracking using a camera, is to identify a moving point in the eye and a reference point in the eye region that does not move, and then use the positional relationship between these two points to identify where a person is looking.
This technique can be classified into two major methods, based on differences in the way these two points are selected (Figure A).
As shown in Figure A, the two existing methods both have their own advantages and disadvantages. Focusing on accuracy of detection, Fujitsu Laboratories chose Method 2, which uses an infrared camera and an infrared LED, as a base for its Gaze Tracking technology. However, considering the challenges mentioned previously, images obtained from a compact and low-cost infrared camera are blurred, and subsequently, the accuracy of gaze tracking is also low. For this reason, the team set out to tackle a difficult challenge in developing image processing technology that can track a person's gaze with high accuracy, even with blurred images from a compact, low-cost, infrared camera.
After a considerable amount of research, the team at Fujitsu Laboratories succeeded in developing a unique technology which enables highly accurate gaze tracking, even from blurred images (Figure B).
To achieve this, the Gaze Tracking technology is broken down into four steps as shown in Figure B. (1) The blurred images are obtained from a low-cost, infrared camera. (2) The candidates for corneal reflection and the pupil are extracted. (3) A set of rules based on optics, characteristics of human vision etc. is used to narrow down the most consistent candidates for the corneal reflection and the pupil. (4) After accurately detecting the corneal reflection and the pupil, these points are used to calculate the person's gaze accurately.
This new technology enables highly accurate gaze tracking using small, low-cost infrared cameras and LED. The sensor developed at Fujitsu Laboratories is only 7 mm thick, and can be built into a computer easily, without detracting from the design.
Small gaze tracking sensors create new business opportunities
The compact, low-cost device enables developers to not only embed gaze tracking technology in a computer, but also to use it for gaze detection. For example, it can be installed in a store without impacting the display or showcase design. By installing gaze tracking sensors in various places in-store, store managers can obtain data that shows which products customers are interested in, leading to more effective marketing. Another example is to install these sensors on tourist information boards to display information about spots that catch the visitors' attention, and broadcast relevant information over a speaker. By making the most of Gaze Tracking technology, a whole new range of convenient services will be introduced.
See the video below demonstrating the compact Gaze Tracking sensor, which is at the core of these cutting-edge technologies.
The development of this technology aims to further broaden the potential of Gaze Tracking technology for various applications.
New developments based on the compact Gaze Tracking technology
The compact Gaze Tracking sensor described above enables gaze tracking when a person is relatively close to the sensor - within any distance up to around 80 cm. Based on the technology it developed for the compact Gaze Tracking sensor, Fujitsu Laboratories recently developed a new technology designed to detect a person's gaze from a longer distance as well. This technology uses one camera to detect the location of a person's face, and a pan-tilt-zoom camera to detect where the person is looking after the location of their face has been identified. By understanding how to control high-speed coordination between the two cameras, the researchers were successful in developing long-distance Gaze Tracking technology for a person standing several meters away from the sensor.
Long-distance Gaze Tracking is expected to be applied for a range of purposes, making effective use of its special characteristics. As a person's gaze can be detected even from places relatively far away, this has the potential to be used in places such as museums, art galleries, and store windows. These are places where it is difficult for customers to physically hold the product or get close to the exhibits.
See the video demonstration below for further insight into this exciting new technology.
In addition, by using the Gaze Tracking technology on vehicles, Fujitsu Laboratories can also advance research in support of safer driving, by detecting inattentive behavior like not looking at the road while driving.
Since it uses an infrared LED, the Gaze Tracking technology Fujitsu Laboratories is currently working on is affected by sunlight--which has a considerable amount of infrared light-- making gaze tracking even more difficult. However, by adding new innovative devices to the conventional image processing technology, Fujitsu Laboratories has succeeded in making Gaze Tracking for vehicles stable even under such conditions.
Small components producing great benefits--these are the results of Fujitsu's innovation.
Gaze Tracking technology aims to infer what a person is thinking - the potential of the latest technology.
The possibilities of Gaze Tracking technology are unlimited, when combined with innovative ideas. Research Manager Satoshi Nakashima talks about his enthusiasm for this technology, saying "Our research team aims to develop technologies that understand the intentions of users, with the aim of realizing our vision of a Human-Centric Intelligent Society."
As it develops, what potential does Gaze Tracking technology have for our future business and society?
Examples include detecting simple mistakes during mechanical operations and mistakes on the job, as well as preventing human error that can lead to things such as traffic accidents. One can also imagine technologies that infer what people are thinking and supports smooth communication; for example, robots that fetch objects that a person looks at, or devices that support people when their eye movements indicate that they are confused.
Fujitsu's technology will help realize our vision of a Human-Centric Intelligent Society.