Field Service Support with Google Glass and WebRTC
In the past few months I did a thesis at Ericsson Research on the topic contextual communication, in which I investigated the use of Google Glass for field service support.
The focus of the thesis involved a generalized use case where a field service technician has traveled to a remote site to solve a problem. The technician is equipped with Google Glass, or any equivalent head-mounted display. When the technician encounters a problem that is more complicated than expected, or if the technician is unable to solve the issue for any other reason, the technician calls the back office support using the device.
The goal was to research collaborative communication and find ways to tailor a communication system for this specialized kind of call. Different ways to give instructions and display information to the wearer of a head-mounted display were investigated, as well as how these could be implemented.
An experimental prototype was built using WebRTC, WebGL, NodeJS, and other technologies. A framework that has been developed internally at Ericsson Research was used to implement WebRTC on Google Glass, as it runs on Android among other platforms.
The starting point was for the expert to use some form of annotations to give instructions to the technician. Both annotations on live video and still images were considered. In the end, still images were chosen for a number of reasons:
- Still images significantly simplify the problem of overlaying annotations, allowing a useful prototype to be implemented within the limited timeframe.
- Using still images does not have a significant effect on efficiency of communication compared to live video annotations.
- If video annotations were used the processing would have to be offloaded to a nearby device, as the processor in Google Glass is not powerful enough for advanced real time image processing and simultaneous multimedia streaming.
- Still images are less vulnerable to connection issues and require much less bandwidth.
A prototype was created where an expert is able to use a web application to provide guidance for a field technician who is wearing Google Glass. A live video feed is sent from Glass to the expert from which the expert can select individual images. When a still image is selected it is sent to the technician and displayed on Google Glass, and the expert is able to provide instructions using real time annotations.
Three different kinds of annotations can be used: free drawing, text, and highlighting of objects. The annotations can be painted using different colors and the colors can be erased individually. In order to allow highlighting of objects, an algorithm that divides the selected image into segments was implemented using WebGL.
The prototype demonstrated the potential of wearable head-mounted displays for use in field service support. It also shows how web technologies can be used to rapidly prototype applications.
A big concern from the beginning of the work was the limited processing power of Google Glass. Even though the hardware encoder was utilized for encoding video, the prototype could only run for about 10 minutes before the device overheated. The hardware issue prevents Google Glass from being used in any production system, but it will very likely be resolved as newer iterations of Google Glass are released.
— Patrik Oldsberg, Researcher – Ericsson Research