Mobile quality of experience: Network readiness for new services

The first wave of 5G primarily brought enhanced user experience for existing applications to smartphone users. New types of applications and use cases are expected in the next wave, bringing to networks new challenges in delivering sufficient mobile QoE to customers. Mobile user experience is a function of both application quality and network quality. Service providers need ways to rate experiences, and to become known in the market for delivering a mobile experience consumers and enterprises can rely on. Ultimately, this will impact how much customers are willing to pay for a service.

Models to predict the quality of mobile experiences

Traditional proxy measures for QoE are generic, and unrelated to a specific application and network combination. What users can expect from the network is typically measured in three ways:

• population coverage (percentage with access to a specific cellular access technology (4G, 5G) in low- or mid-band spectrum)
• radio signal strength (measured on the device and presented as 1–4 bars)
• speed tests (user-initiated peak rate tests of throughput at a given location during a defined time)

These measures indicate network fundamentals for users, but have limited value as input for planning networks for more advanced experiences. An alternative method is to calculate the quality of various mobile experiences using secondary data points retrieved from devices and analyzing by:

• uniformly capturing data across different service, equipment, measurement and device providers
• applying algorithms and standardized models, where a specific set of data points can be measured and correlated with the QoE for a specific service

ITU’s Telecommunication Standardization Sector (ITU-T) set out to standardize and secure a broad anchoring for models to use. The ITU-T Rec. P.1203 is the world’s first standard for measuring the QoE of video streaming services for longer viewing sessions and has been established for years. Models for measuring cloud gaming¹ and video telephony² QoE are under development.

These will rely on a set of data points as input, with a known impact on the QoE, and a calculated overall rating as output.

In collaboration with Ookla, Ericsson conducted a nationwide data-gathering project in the US during the first quarter of 2023. All data points were uniformly captured from smartphones across the three largest service providers’ mobile networks, and were used to rate mobile QoE with the aforementioned approach. The three experiences studied were video streaming (ranging in resolution from 144p to 4K), mobile gaming and video conferencing. All these services are mature and expected by customers to be delivered with excellent quality over a cellular network. The results indicate an ongoing need for network performance improvements to deliver a consistent QoE for these types of applications. General network readiness for delivering good QoE for cloud gaming and extended reality (XR) applications is still in its infancy.

Modeling mobile video streaming QoE

Video is the dominant traffic type in cellular networks, and its use continues to grow. A total of 80 percent of all data traffic in cellular networks is forecast to be video by 2028 (see page 19). Video consumption has gradually shifted from broadcast to streaming, and mobile video quality evolves toward full-HD, 2K and 4K resolutions. However, user experience of mobile video depends on many different measurable aspects, such as intrinsic encoding quality (affected by resolution, frame rate and codec) and dynamic quality effects (such as time-to-content, rebufferings and resolution adaptation to channel capacity).

Figure 31: Video streaming quality model

Video QoE is well-researched and relies on models that were standardized before 5G was introduced, such as the ITU-T P.1203 standard.³ This includes modules for estimating short-term video (P.1203.1) and audio (P.1203.2) quality, and an integration module (P.1203.3) estimating the final session quality due to adaptation and stalling. The short-term video quality scores are fed into the integration module and the final quality score is then presented as a single mean opinion score (MOS) ranged 1–5 for the whole experience. This is an objective model designed to mimic the behavior and perception of humans, producing the MOS values that would result from running a subjective video quality test with a group of individuals in a laboratory environment.

In this study, Ericsson replaced the P.1203.1 module with the open source-based Video Multimethod Fusion Approach (VMAF) algorithm, as P.1203.1 does not support some commonly-used codecs.⁴ Since the test video is known, and pre-encoded, VMAF could be used offline to assess the video encoding quality for the resolutions utilized, while P.1203.3 was used to add the dynamic effects of time-to-content, rebufferings and resolution adaptation. The resulting QoE measure (output) from the model is presented as a single mean opinion score (MOS) in a range of 1–5 for the whole experience (see Figure 31). This figure shows the P.1203 architecture, with P.1203.1 exchanged to VMAF.

The model relies on a theoretical maximum value defined by the resolution, where standard definition (SD) is the lowest possible resolution that gives a good experience (MOS 3.5–4.5) on a smartphone, and excellent experience (MOS 4.5–5) requires at least full HD (see Figure 32).

Insights into mobile video streaming QoE

When applying the model (Figure 31) to mobile video streaming deliveries over commercial networks in the US, it was found that:

• Excellent quality (MOS 4.5–5) was achieved by 61 percent of the mobile video streams measured. The measurements varied 44–72 percent across the 3 large service providers. The differences between service providers relate to spectrum used and network rollout strategies. The premium resolution samples were limited and represented only 12 percent (2K) and 4 percent (4K) of all measured streams.
• Only 13 percent of the streams measured were less-than-good (below 3.5 MOS). Poor radio conditions were the root cause for 40 percent of the less-than-good experiences, with either poor radio frequency (RF) strength, poor RF quality, or a combination of both.
• 5G increases video streaming quality compared to 4G and Wi-Fi. The proportion of streams with excellent quality increased from 58 percent (4G) to 72 percent (5G). The QoE gap compared to Wi-Fi decreased from 22 percent to 8 percent. 5G has reduced Wi-Fi’s previous streaming quality advantage.

Insights into mobile gaming QoE

Two-thirds of mobile app revenues come from mobile games,⁵ and we are at the beginning of the fourth gaming wave (after console, PC, and mobile games) with cloud gaming increasingly offered by service providers. The transition from mobile app games, studied here, to mobile cloud games will materially change network performance requirements. For mobile games executed in an app⁶ on a smartphone or tablet, the QoE depends on latency, packet loss and jitter. A simple evaluation model was applied to the captured data for these parameters to rate the QoE:

• 57 percent of the mobile gaming experiences measured were of excellent quality. Mobile app-based gaming is latency-sensitive. However, the differences in latency between service providers was limited, and all 3 Tier-1 providers fall within the 54–58 percent bracket for an excellent QoE. The shift from 4G to 5G increased gaming sessions with excellent QoE by 6 percentage points.
• Server locations influence mobile gaming QoE due to longer delays. There is a difference in mobile gaming QoE between servers located in the US (82 percent providing excellent quality) and elsewhere (38 percent providing excellent quality).

In this case, a simple evaluation model was used to rate the QoE. More work is needed to develop a deeper understanding of what parameters influence the perceived QoE for mobile gaming.

Insights into mobile video conferencing QoE

COVID-19 led to 2D video conferencing being universally adopted in home offices. Not only is it here to stay in the hybrid workplace, but it will evolve toward immersive 3D communication. While PCs presently dominate as the platform for video conference calls at work and home, mobile devices are growing in importance in the workplace and when commuting. The QoE for mobile video conferencing is dependent on video resolution and round-trip delays. A few aspects make mobile video conferencing different to using a fixed network connection. The high-end resolutions of 4K and 2K are typically not used, and the usage of Full HD (1080p) and lower resolutions vary somewhat between different video conferencing services. Video conferencing is inherently latency-sensitive, but material drops in quality do not occur until after about 100 ms in round-trip delay.

In anticipation of a standardized model, we have used the same resolution base as for video streaming (see Figure 32).

The resolution estimate comes from the available bit rate at the initiation of the video conferencing service. The impact of network delays leverages the ITU-T G.107 model, initially defined for voice-quality predictions, as users tend to be more sensitive to audio delays than video delays. The key results were:

• 79 percent of mobile video conferencing experiences (4G) were of excellent quality, with both throughput and latency variables meeting the threshold for excellent quality
• 88 percent of mobile video conference experiences (5G) were of excellent quality – a leap upward by 9 percentage points from the 79 percent for 4G
• 5G emerged as the best network, even 3 points ahead of Wi-Fi, for overall QoE of mobile video conferencing Network readiness for new types of mobile experiences

Network readiness for new types of mobile experiences

The most significant value of the approach described in this article will be for new types of experiences, for which both application developers and network planners need new tools to rate QoE. However, QoE models are experience-specific and need to be standardized for new types of experiences like cloud gaming and XR.

As part of the analysis, the capabilities of existing networks to meet quality thresholds on downlink, uplink and latency for these types of services were examined. 5G connectivity requirements vary for mobile cloud gaming and AR use cases:

• mobile cloud gaming:⁷ 10 Mbps downlink, 5–9 Mbps uplink, and 30–75 ms one-way latency
• AR:⁸ 2–60 Mbps downlink, 2–20 Mbps uplink, and 5–50 ms one-way latency

Through modeling of measured downlink throughput and round-trip delays, network readiness for these types of services were measured as follows, as a percentage of modeled service access attempts:

• mobile cloud gaming: 40 percent of measured throughput and latency values meet minimum requirements
• AR: 3–32 percent of measured throughput and latency values meet minimum requirements

The large spread in network readiness values for AR depends on the “flavor” of AR in play, and where in the requirement span each flavor falls. The locations of servers required for remote rendering have a high impact on the results for data points collection coupled to latency. As AR flavors, server locations, model development and data point collection mature, the initial spread in network readiness will be replaced by specific values.

The difference between the high percentage (61 percent) of mobile video streaming experiences rated “excellent quality” and the low percentage for meeting the minimum network requirements of mobile cloud gaming (40 percent) and AR (3–32 percent) experiences points to a need for continued 5G network evolution. This will be necessary to meet a large variety of requirements of new types of services, with higher demands on network performance.

Performance capabilities need further improvements

The work to define QoE rating models for new experiences with high network performance requirements is still to be undertaken. However, the work to define the models and the job of capturing datapoints for predicting quality for an experience can happen in parallel, so that both models and robust data sets can guide network evolution plans before standards are complete. A need to continue improving mobile network performance capabilities to meet the requirements of new mobile experiences, and rising user expectation on QoE, will grow with uptake of new services.

1 ITU-T work item P.BBQCG, www.itu.int/ITU-T/workprog/wp_item.aspx?isn=17809.

2 G.CMVTQS, www.itu.int/ITU-T/workprog/wp_item.aspx?isn=17785. Note: Video telephony is not the same as video conferencing, but still similar to a two-party video conference call.

3 Ericsson, Video QoE, leveraging standards to meet rising user expectations (June 2017).

4 GitHub, VMAF.

5 Data.ai, State of Mobile 2023 (January 11, 2023).

6 No video component is transmitted, only metadata in the uplink and downlink.

7 Xbox Game Pass requirements, Microsoft.

8 ”XR and 5G: Extended reality at scale with time-critical communication” (August 24, 2021).