Enhancing mission-critical XR-based situational awareness over 5G

Have you ever imagined how XR technology and 5G networks can fundamentally transform mission-critical situational awareness use cases?

The growth of XR applications such as augmented reality (AR), virtual reality (VR), and mixed reality (MR) is changing the way we visually experience the everyday world. Situational awareness encompasses knowing what is happening around you and is a way to develop an ever-evolving knowledge of the current situation and surroundings. This is what enables you to assess the situation before making your decision about how to react. It is something we do countless times a day in everyday life, but in emergency situations, the stakes are much, much higher. Such situations are often much more complex, and a decision by emergency services could be a matter of life and death – when choosing how to respond, the way first responders access and process the information is paramount.

The promise of 5G is not just about high-speed data; it’s also about unleashing a new era of connectivity that will redefine industries, elevate our digital interactions to access information, and empower a future that, until now, seemed like a distant dream. The future is ours to build, and as we stand on the precipice of this technological advancement, the possibilities seem limitless.

1. Capability concept vs. mission-critical XR-based solution

The use of AR/VR technology often draws inspiration from science-fiction movies to create immersive experiences. For instance, movies like “Minority Report” showcased gesture-based interfaces such as AR interactions, while “The Matrix” popularized the concept of a fully immersive VR world. These analogies help us envision how the technologies can blend digital elements with the real world (AR) or create entirely virtual environments (VR) for various applications. Furthermore, they provide an interesting insight that highlights the potential of mission-critical XR-based technology in enhancing situational awareness. For example: in "Minority Report”, futuristic technologies allow law enforcement to predict and prevent crimes before they occur.

While the analogies from science fiction movies portray a fictional future, they offer a glimpse into the potential of mission-critical XR-based solutions we can develop. The advancement of these solutions, coupled with the capabilities of 5G networks can evolve over time, introducing the following capabilities.

1. Real-time data visualization: Mission critical XR-based solution enables users to visualize complex datasets, sensor inputs, and live video feeds in real time, enhancing situational awareness and decision-making capabilities.

2. Gesture-based interaction: In mission-critical XR-based solutions, gesture-based interactions can be employed to manipulate XR environments, providing an intuitive and immersive way to interact with situational information or to manipulate a large amount of data into the information feed.

3. Predictive analytics and artificial intelligence (AI)/machine learning (ML): Mission-critical XR-based applications leverage AI-driven analytics to real-time data, predicting potential incidents, and providing users with actionable insights for effective decision-making in critical scenarios.

4. Automation for intelligent content filtering and dynamic resource allocation: Automation can categorize and filter incoming data to present users with the most relevant information. This automation can help to allocate network resources based on the quality of service (QoS) and prioritization of data streams. This prevents information overload and allows users to focus on critical aspects of the situation. When certain conditions are met, automated alerts can be generated and sent to users in real-time.

5. Effective communication and collaborative insights: The mission-critical XR-based solution facilitates the sharing of insights among connected users across organizations or agencies on the collaboration and communication platform. XR interfaces enable real-time communication, shared annotations, and visual cues, fostering collaboration and collective situational awareness of data information regardless of physical location.

By leveraging these capabilities, we intend to embrace the future potential of mission-critical XR-based solutions. The vision is to enhance situational awareness use cases over 5G networks in real-world scenarios and to empower connected users to communicate and respond effectively in critical situations so they can make informed decisions.

2. XR-based situational awareness use case

Along with today’s technological advancements, we can leverage XR applications in mission-critical operations such as AR for field personnel, VR for training and simulation, and the communication and collaboration platform (a.k.a. the command center of the future).

Augmented Reality (AR) for field personnel

Augmented Reality (AR) for field personnel

One of the most exciting aspects of the paradigm shift in mission-critical networks is the integration of AR into field personnel operations. AR technology has the potential to transform the way field personnel interact with their surroundings, enhancing their situational awareness and improving their overall effectiveness in critical scenarios.

AR overlays

Imagine a scenario where firefighters are responding to a complex structure fire. Equipped with AR-enabled devices (hands-free mode), they can instantly access vital information such as floor plans, hazardous material locations, and real-time data feeds from sensors placed within the location. AR overlays can highlight escape routes, point out potential hazards, and provide step-by-step instructions for managing critical situations. This level of augmented situational awareness allows first responders to make quicker, more informed decisions, increasing their safety and the efficiency of their response.

In addition to this specific use case, AR can be applied across a wide range of mission-critical services, including disaster response scenarios or defense operations. The ability to overlay digital maps, navigate unfamiliar terrain, visualize hidden infrastructure, and communicate with the command center enhances the capabilities of field personnel and enables them to operate with greater efficiency, effectiveness, and safety.

As AR technology continues to evolve, we can expect more sophisticated and immersive experiences for field personnel. Collaborative AR platforms will facilitate seamless communication and information sharing among users and with the command center, even when individual team leaders take part in the operation from their respective locations.

 Virtual Reality (VR) for training and simulation

Virtual Reality (VR) for training and simulation

Another captivating use case is the integration of VR for training and simulation purposes. VR technology revolutionizes the way field personnel can be trained, allowing them to immerse themselves in virtual environments that replicate critical scenarios. This immersive training experience enhances their situational awareness, decision-making abilities, and overall preparedness for real-life critical operations. VR enables the visualization of complex scenarios, such as disaster simulations, industrial accidents, or large-scale public events.

Psychological training

VR has the potential to simulate high-stress environments and expose field personnel to challenging situations to develop their psychological resilience. By immersing themselves in virtual scenarios that replicate the emotional and cognitive demands of critical operations, personnel can better prepare themselves mentally, improving their ability to remain calm, focused, and decisive in high-pressure situations.

• This may also be utilized as an aid for veterans in the recovery process where the virtual setup can recreate the situations that influenced the individual mentally as a treatment.
• The healthcare sector can also leverage VR for training medical professionals in critical procedures and emergency response scenarios. Emergency medical personnel can train in realistic virtual environments that simulate mass casualty incidents or complex trauma cases, enabling them to develop effective triage and treatment strategies.

The use of VR for training, simulation, scenario visualization, collaboration, and psychological preparation is poised to revolutionize mission-critical use cases. Its ability to provide immersive and realistic experiences empowers field personnel and control rooms with enhanced communication, situational awareness, decision-making capabilities, and operational effectiveness. With continuous advancements in VR technology, we can expect this transformation to play an increasingly vital role in the future of mission-critical communications.

Communication and collaboration platform (a.k.a. command center of the future)

The way traditional command center operates is based on in-person locations dependent on multiple monitors and people. Since the COVID-19 pandemic, there has been an ongoing shift to facilitate remote operations, but there is still a lack of innovation in terms of finding new ways to collaborate in person and online.

The concept of this solution is to create a virtual space for advanced communication and collaboration centers that can be accessed from anywhere and at any time. This solution enables users with roles and functions to operate the command center virtually, reducing the need for in-person locations dependent on multiple monitors and people in a centralized location.

This command center of the future also aims to support the streaming of high-resolution sensor data, video feeds, and other relevant information to XR devices, giving users up-to-date situational awareness. Accurate time-critical information can be visualized directly in the eye of operators with advanced XR headsets, fostering fast and agile decision-making.

Demo video - 5G-enabled mission critical collaboration

The system solution will evolve along with 5G technology and edge architecture to support low latency and real-time video processing. XR technology provides a powerful human-centric design optimized for fast and simple understanding of complex scenarios and automation.

This seamless integration of XR and 5G brings unprecedented capabilities to public safety, emergency response, defense, and critical industrial operations.

Now, let us explore the profound connection between XR technology and the important role of 5G networks, and how and why 5G is essential for the successful development of mission-critical XR use cases to unlock the full potential of both innovative technologies.

3. 5G network considerations for mission-critical operations

Wireless communication plays a key role in the effectiveness of mission-critical communication to achieve its objectives in any situation, especially public safety. This includes everything from day-to-day routine operations and supporting planned events to responding to major natural disasters.

5G networks for mission-critical operations require enhanced features and capabilities to ensure mission-critical grade performance in the areas outlined below:

• Network availability: always available, reliable, and resilient (can operate locally when and where it is needed), and with extensive coverage including remote areas.
• Multi-network operation: to enable traffic over multiple networks with the required priority and QoS by different networks to extend failover/availability.
• Coverage and capacity: extending network coverage and capacity for mission-critical users beyond what is typically available for commercial users.
• Security and hardening: high level of functionality to ensure access and usage control but also confidentiality and integrity, and by providing multi-layer security and catering for operational and regulatory restrictions.
• Offer special mode of operations: QoS, priority and preemption functions and features for control of application priority to guarantee latency and capacity requirements.

4. Considerations to ensure XR applications work flawlessly

The convergence of mission-critical use cases, XR technology, and the capabilities of 5G networks holds immense promise for revolutionizing various industries. However, deploying mission-critical XR-based technology on 5G networks also presents unique challenges that need to be addressed for future mission-critical 5G networks.

User devices

Mission-critical XR experiences require XR-enabled user devices such as headsets, smart glasses, or smartphones with XR capabilities. These devices integrate XR hardware (displays, sensors) and software (XR rendering engines, tracking algorithms, etc.). While mobile phones are versatile devices with a wide range of applications, XR devices are dedicated to delivering immersive extended reality experiences and designed for user interaction and a level of immersion that allows the augmentation of the real world with virtual elements or creating entirely virtual environments for users to explore.

As technology advances and user needs evolve, these devices must adapt to meet the demands of critical operations with the following unique requirements:

• XR devices must be capable of minimizing latency to imperceptible levels, enabling seamless interactions and data visualization.
• XR devices will require enhanced sensory integration to provide users with a comprehensive view of their environment, including advanced cameras, Light Detection and Ranging (LIDAR) sensors, thermal imaging, and other specialized sensors that augment situational awareness by capturing crucial data in real time.
• Ergonomics and user comfort – lightweight, well-balanced, and designed for extended wear without causing fatigue, as well as ruggedized for tough environments.
• XR devices must support robust connectivity with multi-mode wireless connections, support for various communication standards, and seamless handoffs between networks (such as dedicated and public networks, 5G, Wi-Fi, satellite) to ensure continuous connectivity in dynamic operational environments.
• Extended battery life – future advancement in battery technology, power-efficient processors, and energy management will be essential to ensure XR devices can operate for extended periods without interruption.

5G mission-critical network infrastructure

The architecture relies on a robust 5G network infrastructure with enhanced features to support mission-critical services and applications. This includes end-to-end 5G network components from radio access networks (RAN), transport networks, and core networks to include mobile-access edge computing and end-to-end (e2e) network slicing capabilities.

When applied to a dedicated network, slicing helps differentiate priorities among user groups and enhances isolation between different traffic types or user groups. In a shared public network, slicing allows for the allocation of dedicated resources for critical operations. Whilst it will not replace the need for a fully dedicated, mission-critical core network, it can provide entry-level capabilities for carriers to address and explore opportunities across government and the public sector.

Network slicing architecture enabling XR applications

Network slicing implementation can help mobile network operators provide differentiated services and more quickly deploy new cases. This is a way to advance in 5G network setup to provide a network slice for mission-critical network traffic, with both options of shared and dedicated radio access networks.

5G network slicing allows the partitioning of a physical network into multiple independent logical networks. Each customized slice is designed to meet specific requirements. It enables the allocation of dedicated network resources for different applications, ensuring the right performance characteristics, configurations, and policies. To support mission-critical XR applications, it needs to be set up for high reliability, low latency communications, and QoS guarantees.

Figure 1. 5G network slicing architecture enabling XR applications

Some other general characteristics, features, and developments within the 3rd Generation Partnership Project (3GPP) that will support mission-critical XR-based applications:

1. 5G new radio (NR) enhancements: The 3GPP has been actively working on enhancing the 5G NR technology to meet the requirements for mission-critical communications. These enhancements include improvements in latency, reliability, and QoS, which are crucial for real-time XR-based situational awareness.
2. Ultra-Reliable Low Latency Communication (URLLC) – time-critical communication: URLLC is a key feature of 5G that provides ultra-low latency and high reliability for mission-critical applications. While not specifically tailored for XR, URLLC capabilities can contribute to enabling real-time, high-performance XR experiences in mission-critical scenarios.
3. Edge Computing / multi-access edge computing (MEC): Edge computing/MEC brings computational capabilities closer to the edge of the network, reducing latency and enabling faster processing (i.e., for analytics). By leveraging edge computing infrastructure, mission-critical XR applications can benefit from lower response times and improved real-time data processing.

Network Exposure Function and Global Network Platform roles

The Network Exposure Function is a critical component of 5G core networks that enables secure exposure to external applications and services. The capabilities are specified by 3GPP 5G network functions toward the 3rd Party Partnership (3PP) application services through the application programming interface (API). This way, the underlying 3GPP network interfaces and protocols, can be protected and abstracted by being controlled or mapped to appropriate network interfaces and protocols. In the context of mission-critical network implementation, and especially for use cases like XR-based situational awareness, it plays a pivotal role in the aspects of API integration by providing a centralized front end on which we can expose the APIs to external API consumers, which include other operators, enterprise customers, 3rd party over-the-top (OTT) applications and content providers in a secure, controllable, traceable, scalable, and measurable way.

Figure 2. Network Exposure Function and Global Network Platform role

Global Network Platform (i.e., by Vonage) can offer robust APIs and developer tools that facilitate the integration of communication and collaboration services into XR applications. By leveraging these APIs, mission-critical XR applications can also incorporate real-time voice and video communication, enabling seamless collaboration and information exchange among connected users within one common accessible platform in XR environments.

XR content delivery

To deliver XR content, the architecture includes content delivery mechanisms that leverage both centralized and edge resources, ensuring efficient and low-latency distribution to user devices.

Other key considerations

1. Security and privacy: As mission-critical XR applications involve sensitive information, robust security measures are essential. XR data transmitted over 5G networks must be protected from unauthorized access, interception, or tampering. Encryption, authentication mechanisms, and secure protocols must be implemented to ensure the confidentiality, integrity, and privacy of mission-critical data.
2. Interoperability and standardization: To foster widespread adoption and interoperability, standardized protocols and interfaces for mission-critical XR on 5G networks are also important. Ensuring compatibility between different XR devices, networks, and platforms enables seamless integration and collaboration across diverse systems (if required). Standards also facilitate efficient data exchange, interoperability, and futureproofing of mission-critical XR deployments.
3. Device ecosystem development: The challenges are obvious among device manufacturers, XR platform developers, industry standards bodies, and end-users. From the perspective of hardware compatibility, it will require specialized hardware (such as XR headsets, smart glasses, or wearable devices), and 5G connectivity capable of delivering immersive experiences. Ensuring ergonomic design, lightweight form factors, and comfortable wearability are critical factors for user acceptance and operational effectiveness. Devices must be designed to minimize fatigue, discomfort, or strain, allowing users to focus on the mission-critical tasks at hand. It is also essential to consider large-scale deployments across multiple users or teams and scalability in terms of device provisioning, software updates/security patches, and maintenance. Finally, the cost and affordability need to achieve a balance between performance, reliability, endurance and cost-effectiveness to ensure the affordability and accessibility of XR devices for critical operations.
4. Regulatory and ethical considerations: Mission-critical XR applications should comply with relevant regulatory frameworks and ethical considerations, especially in handling sensitive information, such as emergency response or defense. Ensuring compliance with these regulations and addressing ethical concerns is key for the responsible and safe deployment of mission-critical XR-based use cases.

5. Future enhancement toward virtual command center on deployable 5G networks

Command centers of the future will continue to evolve toward virtualization, with shared spaces where humans and AI systems can collaborate in real-time. 5G technology and edge architecture will play an important role in providing connectivity with the right performance characteristics of the networks. This is where deployable 5G networks with local command center solutions will play a critical role in providing a cost-effective platform.

The deployable 5G networks have now emerged as a groundbreaking solution, from providing rapid and reliable connectivity in challenging environments to ensuring continuous communication and access to XR capabilities for collaboration and coordination.

With the advancement of 5G technology and the need to increase situational awareness, the use of deployable networks as connectivity platforms is considered a key enhancement to establish mission-critical communication with virtual command center capability anywhere and at any time.

These deployable networks can connect and integrate user devices with complex data sets, sensor inputs, and live video feeds at the location. They can also extend the integration with remote sensing technologies and IoT devices to enhance situational awareness by collecting and visualizing data from various sources.

Figure 3. The future of mission-critical 5G deployable with XR-based local command center

Conclusion

The convergence of XR-based situational awareness and 5G networks holds great promise for mission-critical operations. It can improve collaboration and communication in various fields - public safety, defense, critical infrastructure, healthcare, and beyond. It redefines the essence of situational awareness. It’s not just about seeing more; it’s about knowing more, connecting more, assessing more, and responding more effectively.

The capability enhancement of real-time data visualization, seamless communication, effective collaboration, and low-latency connectivity empowers mission-critical industries with an unprecedented edge.

While challenges exist, continuous advancements, collaboration, and adherence to best practices will pave the way for successful deployment of XR-based situation awareness on 5G networks as well as deployable networks, empowering mission-critical personnel with the tools they need to operate effectively and safely in demanding environments.

See how you can prepare for the future that’s not just connected, but also profoundly aware.