Changing the World of Sports from Japan's Gymnastics--Scoring Support System Using 3D Sensing Technology

[The Power to Create the Future Vol. 3 (Part I)]

Judging a Series of Emerging New Techniques with the Naked Eye Is too Difficult

"Have you heard of 'a Challenge Rules' used to verify the controversial judgement result in sports events? As you may well know, this practice is provided as a rule in many sporting events today, including tennis, baseball, soccer, and volleyball. I think some of you may remember the silver medal won by Kohei Uchimura and other gymnasts in the men's team final during the 2012 London Olympics games. At first, Uchimura's landing (the final technique) in the pommel horse was judged as not having been completed, and the team ended up in the fourth place. However, as a result of a review, the original result was overturned and the team received the silver medal. This is a consequence of the Japan team challenging the call by the judge. There is a rule in gymnastics events that requires a party who is querying a score to pay $300 per challenge. This was something that stirred up significant debate when the process was shown on TV. Since human eyes are not perfect, it should surprise no one that a judgment may be changed later. However, it is also true that a slight difference in score may influence the color of medal a gymnast receives or his or her career. Due partly to revisions of rules every four years, in addition to advances in gymnastic performance, judges face an escalating burden annually. Fujitsu's unique 3D Sensor was developed to make the scoring in gymnastics more transparent, simpler, and more accurate in response to these circumstances."
Hidenori Fujiwara of Fujitsu Limited, in charge of planning and implementing the 3D Sensing Project said.

Making Gymnastics Easy to Understand and More Familiar for Viewers

Demonstration at a general meeting of the International Gymnastics Federation. The process of analyzing techniques performed by the gymnast is successively shown on the monitor (center in the photo).

The 3D Sensing application, which combines the scoring know-how built up by the Japan Gymnastics Association and the advanced technologies of Fujitsu Laboratories, supports scoring in gymnastics. It captures human movements in three dimensions using 3D sensors, recognizes techniques, and scores them.

"Previously, judges visually checked the performance and filled out the score sheet by hand while the performance is underway. Then, after the end of performance, each point of the performance was calculated to find the score. On the other hand, when the 3D Sensing application is used, the name of the technique, degree of difficulty, each score of the performance, etc. are shown on the monitor one after the other at the moment when the technique has been done successfully. This not only reduces the burden on judges, but also allows a problem-free evaluation of a gymnast's optimal performance, allowing him or her to reap the benefits of practice. I also believe that conveying performance information and the extent of athletes' power in real time will make gymnastics a better understood and more attractive sports to viewers." (Fujiwara)

Capturing the Movement of an Athlete with 3D Laser Sensors and Judging the Techniques by Matching It with a Database of the 'Technique Dictionary'

A 3D laser sensor (right) enables the accurate capture of skeletal movements from a distance without needing to place markers on athletes' bodies.

How does 3D Sensing work? Kazuo Sasaki of Fujitsu Laboratories Ltd. explains below the details of the technology.
"First, 3D laser sensor emits 2.3 million fine laser beams per second toward the athlete to accurately capture the position and posture of his/her body based on the time for the beams to return. Next, the movement of skeleton is measured based on the captured data, the positions of the hands and feet, the bending condition of joints, and the number of twists of the body are derived. Then, these data are matched with the database of the ' Technique Dictionary,' which contains gymnastic movements as data to make judgments. Real-time high-accuracy scoring becomes possible by combining 3D laser sensors, skeletal position recognition software, and matching with the 'Technique Dictionary' database."

In what is called motion capture technology, markers are placed on athletes' bodies to capture their movements. However, this is problematic in that athletes are unable to perform as usual because they are uncomfortable with markers and that its use is not permitted during competitions. "Our 3D Sensing application does not require markers, and so it does not interfere with athletes' performance. It can also be used in competitions. The device has appearance similar to a compact speaker, which will not disturb viewers. We are now working on its development, with the aim of putting it into practical use by 2020." (Sasaki)

An Idea Born from My Hobby, Golf, Is Where It All Began

Kazuo Sasaki of Fujitsu Laboratories Ltd., leading the development in the 3D Sensing Project

3D Sensing is an application of human movement sensing technology. That technology combines 3D laser sensor that were originally developed for application to vehicle monitoring devices and skeleton recognition software that was developed for rehabilitation purposes. This came from an idea by Sasaki.
"My hobby, golf, is where it all began. I was wondering what I could do to improve my form. Even after attending a golf school, I could not get an idea of how to improve and was always thinking what I could do to solve this problem. Then, an idea came to me. What if I express my body movements and feelings numerically? So, I created a golf form diagnosis application by combining commercial sensors and my technical knowledge. It was very easy to get an idea of my body movements, and I asked other golfers who were participating in the Fujitsu Ladies golf tournament to try out the application. Many responded by saying, 'it is easy to grasp how I'm doing if I train while checking the figures.' I sensed that there were positive signs that this method could be expanded to other applications. On the other hand, since there were many technical issues in applying the method to other purposes, we mobilized all the different technical specialists in our laboratories and launched the initiative as a 3D Sensing Project in the field of sports." (Sasaki)

Fujitsu's Essence of Craftsmanship: Addressing Challenging Issues

Hidenori Fujiwara of Fujitsu Limited, who leads the planning and implementation of the 3D Sensing Project

Fujiwara found there was great potential in the 3D Sensing technology that Sasaki was developing and approached many parties hoping to find further possible use in the field of sports. Meanwhile, an encounter with the Japan Gymnastics Association led to a joint development of the gymnastics scoring support application.
"Some people around me told me that it is reckless to take up the challenge of gymnastics, whose scoring system is believed to be the most difficult among other sports. Others told me that it would be better to target other major sports with more easy to understand rules. However, the attitude of challenging the unknown and addressing challenges itself represents the essence of Fujitsu's craftsmanship, and this is also the most important thing in innovation. Furthermore, I thought if automated scoring became possible for gymnastics events which involve many basic bodily movements, it would be easy to apply the method to other sports. If it becomes possible for the pommel horse, the scoring of which is the most difficult out of all gymnastic events, in particular, it would be even easier to apply to other sports." (Fujiwara)
In this way, Fujiwara and Sasaki made up their mind to "change the world of sports with Fujitsu's technology" and commenced development.

Fully Learning about Techniques and Rules from Judges

However, immediately after the start of development, a fundamental problem occurred. The project members, including Fujiwara and Sasaki were completely ignorant about gymnastics. So, we started by learning about several hundred types of gymnastics techniques, the degree of difficulty of each, and the scoring criteria.
"When I first had a booklet on scoring rules in my hand, I was surprised by the thickness of the booklet and difficulty of the content. We held a study meeting whenever we had the time... At the start of the project, we were just like students preparing for an examination. But still, our knowledge was not enough to implement the project. We asked judges from the Japan Gymnastics Association to give us lectures and participated in lectures given to judges. On a holiday, we went to see gymnastic competition together with a former national Olympic team coach and were given explanations of the game. Thanks to the truly generous teaching provided by these people involved in gymnastics, we gradually gained knowledge on this sport during this year." (Fujiwara)

Difficulty in Accurately Capturing Gymnasts' Fast and Complex Movements in a Split Second

Sasaki and Fujiwara talking about behind the scenes. Their enthusiam for the project, which they launched from scratch, is strong.

After that, project members carefully analyzed each technique together with judges to develop a unique algorithm and digitized the scoring method. According to the project members, they held a series of discussions due partly to this being the first trial of this technology in the world.
"Our first mission was to realize automated scoring for the pommel horse, which is considered the most difficult among gymnastics events, and we faced difficulties as expected. Techniques such as 'flop' and 'combined elements' for the pommel horse, in particular, are in fact composed of multiple techniques. Therefore, the completion requirements for each technique must be accurately captured. However, the movements of gymnasts were extremely fast and complex... It was very difficult to achieve accuracy and speed of judgment at the same time. Only after countless rounds of trial and error was there hope that our method could be used in the field."

Sasaki says that the stimulation and awareness-raising caused by getting out of the laboratory and visiting a gymnastics arena and the sincere attitude of the gymnasts competing in the event encouraged him to overcome various challenges. In Part II of this interview, we will focus on behind the scenes of development. We will also discuss the team's strengths and awareness of being a part of a team, as well as the future of 3D Sensing, which is expected to be applied to sports other than gymnastics and even outside the field of sport.

Kazuo Sasaki
Head of Life Innovation Laboratories,
Applied Innovation Research Center, Fujitsu Laboratories Ltd.
1969: Born in Shimane Prefecture. Served as the deputy-captain of the Shorinji Kempo (a modern Japanese martial art based on Shaolin Kung Fu) club in his school days.
1994: After joining Fujitsu Laboratories Ltd., Sasaki engaged in research into IoT platforms, and was a pioneer in releasing the results of research on distributed processing technologies, a precursor of modern-day edge computing.
With focusing on the sports field as an area for the application of IoT, Sasaki is currently working on to accelerate improvements achieved by sportspeople by digitizing human movements.

Hidenori Fujiwara
Vice President, Planning and Development Division, Tokyo Olympic and Paralympic
Operations Unit
Fujitsu Limited
1970: Born in Osaka. Previously, Fujiwara worked at a financial institution. After joining Fujitsu Limited, he engaged in sales to major system integrators.
2015: As the vice president for planning and development of sports business at the Tokyo Olympic and Paralympic Headquarters, Fujiwara starts up plans for new businesses. They include entering into ICT partner agreements with B.LEAGUE (Japan Professional Basketball League) and Japan Basketball Association and conducting joint research with the Gymnastics Association.
Currently, he is in charge of the world's first gymnastics ICT project enabled by 3D Sensing technology. While being involved in sports administration as a committee member of the Sports Agency, Fujiwara is striving for the industrialization of sports and sports&health promotion for the public.

Photographs in cooperation with Nippon Sport Science University and Artistic Gymnastics Club