3-D panorama images show the whole picture! Perfect for remote applications

Industry suffers from aging workforce and dearth of experience

General demographic trends in Japan, particularly the shift towards smaller families and the aging population, have led to a severe shortage of the experienced maintenance and inspection personnel needed to maintain key parts of the national infrastructure such as production plants and expressways. One potential solution to this problem is to have the expert engineers work from a remote operations room, providing advice and assistance to less experienced personnel on site.
This approach is already in use to a limited extent, where experienced engineers instruct on-site maintenance or service personnel on the basis of still photographs captured on a tablet or equivalent. But the built-in cameras on these devices tend to have a narrow field of view and are prone to movement and blurring. Often it is difficult for the remote engineer to get a proper appreciation of the site area or workplace environment as a whole. This approach can also be inefficient and time-consuming, particularly if the target work area is outside the field of view, requiring further discussion and explanation before work can resume.

Combination of AR and 3-D imaging shows the entire area in stunning detail

Fujitsu Laboratories has developed a remote work support technology that uses the latest AR* technology in combination with a composite 3-D panorama representation synthesized from pictures supplied by on-site personnel to provide the remote engineer with a detailed and realistic portrayal of the work area that can be used to render the best possible advice. The system allows the engineer to explore the work area from almost any angle or perspective while tracking the operator's location in real time.
Most importantly, the engineer's instructions are tailored to the position and orientation of the on-site operator and transmitted in real time via a tablet or equivalent device, thus ensuring much greater immediacy than current solutions.

* Augmented Reality uses ICT to superimpose digitized data over traditional sensory information to create an enhanced "virtual reality" experience

There are two main components to the remote work support system: online 3D image synthesis technology for field site images, which generates a realistic three-dimensional recreation of the entire work space; and the worker activity monitoring and work instruction AR presentation technology, which translates the remote engineer's advice into an augmented reality display for the on-site operator.

Online 3D image synthesis technology for field site images

The online 3-D image synthesis component uses image data to determine the operator's location and orientation, and stitches together multiple images in three dimensions to create a realistic panorama-style visual representation of the work area. The operator takes photos of the work area on a tablet or equivalent. These are sent to the remote operations room along with real-time location information for the operator, which is based on a combination of camera images and sensor data. The location information is used to coordinate the size and position of the images relative to the distance from the actual work area. The images are then synthesized within a 3-D space to generate a realistic panorama-style representation of the work area, which the remote engineer can explore from any angle or perspective.

Accurate 3-D representation of the entire workspace

Worker activity monitoring and AR presentation technology for work guidance

A virtual rendering of the on-site operator within the panorama display is produced based on location information that is updated in close to real time. This allows the engineer to evaluate the operator's movements within the workspace from any angle and perspective, in much the same way as if they were actually present.

On site: the operator is facing towards the red fire warning panel on the pipe

Remote operations room: the 3-D model renders the operator's movements in real time

The remote engineer uses the pointer and footnote facility to issue instructions to the on-site operator based on the panorama visuals. These instructions are instantly rendered on the operator's tablet or equivalent smart device in the form of an AR display that is coordinated with their precise location and orientation. If the operator is not facing in the required direction, for instance, icons on the AR display will guide them to the target area for the required task. In this way, the AR display helps the remote engineer to provide the best possible advice and instructions to ensure that maintenance and inspection tasks are performed correctly.

On-site operator uses AR display on tablet to locate target area

Target area indicated by Check Here message together with icons

Harnessing the experience of skilled engineers; potential applications in logistics and retail

In this way, the remote work support technology can be used to provide tailored advice and instructions to on-site maintenance and inspection personnel. It is hoped that the system will provide a way to retain and harness the accumulated experience and expertise of older workers who are less able to attend site inspections on a regular basis. In addition, the system is considered to have potential in industries such as logistics and retail. Trials and testing are continuing with a view to full commercialization in FY2016.