Augmented reality prototyping - Summer in the Labs

Describe your tasks as an intern at Ericsson Research.

As part of the Ericsson Research team, I participated on a project we call AR Collaboration, which involves us using an augmented reality (AR) device to encourage co-working between remote collaborators over the network. It is an excellent way to adapt existing environment and settings, displaying additional information on top of our existing world. This ties into Ericsson Research Media Technologies' work to improve the quality of the media experience – such as refining how audio and video is delivered and consumed. In Silicon Valley, work has been done in content discovery, recommender systems, and even 360° videos.

Name: Jeeeun Kim
Unit: Media Technologies
Education: Ph.D. degree in
Computer Science
Location: Santa Clara, US

Other applications for AR can be found in the home or the office. Suppose you are assembling your furniture, but get stuck at some point, not knowing what to do, no idea even with the instruction. You could call to someone to ask for help, but with limited domain knowledge, explaining the exact problem is yet hard—you might not even be aware of 'what exactly' your problem is. In the meantime, if someone has experience in this domain or at least situation, just seeing the scene gives him/her hints what is going on with you. For your remote collaborator, directly overlaying information to your view will be the best way for him/her to guide and instruct the possible solution. Especially when you are asking a unique problem which is not easy to be solved in written user manual, for example, interior design, is the best benefiter. Where everyone has physical constraints in addition to unique tastes, how can you and your remote collaborator communicate? AR is the answer.

I developed a Microsoft HoloLens application and the Internet service for a remote participant to live stream a user's view and/or the virtual space the user belongs to.

What were your expectations, and what did you learn?

As our internship lasted just twelve weeks, we were not able to plan large projects. My expectation was to build a basic system architecture that employs the essential components of this project, such as an AR device, network platform (server), and a web application to provide an interface for a live stream between remote collaborators over the network. I have experience of developing front-end applications and computer graphics to interpret how the physical world is reproduced in a digital form, but I had no experience with spatial interaction. I have learned how a physical device gets real-world information in real time using depth sensors and camera to overlay digital information on top of the existing world. This project lead me to think about how a network interrelates the transmission of video contents and how we can reduce any negative impact of it for fluid user experiences.

I personally hoped to prototype a haptic device that synchronizes physical experiences between remote participants through Ericsson's 5G network infrastructure. I was pondering how haptic technology will change user experience significantly. I think this is an area for future research.

What challenges have you faced?

We are now integrating our solution into the open source project called HoloJS, a Javascript framework to design 3D graphics renderers for Microsoft HoloLens applications. HoloJS is not a commercial framework, nor does it have a long history where many developers are aware of well-known problems, so every time we encounter random errors there are limited options to look for solutions. Fortunately, the HoloJS community is growing in its backend support and participants, and the original author is very responsive. Thanks to that, we are now approaching the process of 'identifying' the problem and look for alternative solutions.

My mentors' project experience helped teach me where and how to ask for guidance. Sometimes a problem stems from just not matching IDE version, firewall settings, or Ericsson's network policy, failing at synchronizing development environments. Other times it involves reading back at the source code library, since they are not typical, and it also involves mathematical approaches. Thinking out loud with team members helped me identify problems.

Tell us about your education

I am pursuing a Ph.D. degree in Computer Science and I will be entering my fourth year this fall. I got my BE/BS degree in Computer Engineering in Korea.
I am interested in digital fabrication technology and computational design frameworks that introduce human's organic making experiences into computer-aided design systems. I have been researching 3D printing and other digital fabrication techniques to interpolate the gap between virtual conception/design thinking of human and digital models, as well as between such virtual models in computer systems and physical models produced by them. My goal is to deploy AR technology learned from Ericsson to reduce these gaps, by providing seamless information exchange between the virtual world and the physical world to design fabrication artifacts in various domain.

What do you do when you are not studying?

I like traveling. I have visited more than thirty countries so far and have been to all six continents. Traveling and learning new languages are ways for me to keep my eyes and ears open and enlighten my perception towards the world we live in.

Facts about Silicon Valley

• Known as the capital of tech.
• In 2015 there were patent applications for 655 different products.
• The area was first dubbed 'Silicon Valley' in 1971 by American journalist, Donald C. Hoefler.
• The name "Silicon Valley" is derived from the large amount of silicon semiconductor manufacturing companies located in the region.