Benchmark measurements in 5G networks

It’s no secret that regulators across Gulf countries have done an amazing job supporting Communication Service Providers (CSPs) with efficient spectrum allocations processes for 5G deployments. I believe that because of this support, some of the first 5G deployments globally were executed by CSPs in the Gulf countries and these CSPs are now global leaders in 5G deployments.

It’s now over a year since the first 5G deployments, and regulators in the Gulf countries are now starting the benchmarking of 5G coverage, to ensure that CSPs are meeting conditions for spectrum allocations. However, I foresee some challenges in terms of how measurements are done. One example being that the methodology typically used for the benchmark measurements in 2G/3G/4G networks might not be suitable for 5G networks and might not provide accurate benchmark results.

In this paper, I explore some of the challenges of benchmark measurements in the 5G networks, outlining system differences between 5G and 4G that cause these challenges, and proposes a preferred way to conduct the benchmark measurements in 5G networks based on actual end-user performance.

5G networks require different benchmark measurements compared to 4G networks

Broadly speaking, we can distinguish three types of channels in the air-interface for every generation of mobile technology:

• Control channels in the idle mode, these are the channels that mobile phones or User Equipment (UE) will monitor for connecting to a network.
• Control channels in the connected mode, these are the channels utilized when UE is sending/receiving user data to control/monitor connections.
• User data channels in the connected mode, these are channels used to send/receive the actual user data.

A common way of conducting benchmark measurements in 4G, as well as 2G/3G, networks is by measuring the control channels in the idle mode. For instance, the network coverage and availability of the mobile service is assessed by measuring the signal strength of the control channels in idle mode.

The benchmark measurements based on the control channels in idle mode are well suitable for 4G, and earlier technologies. They are relatively simple to conduct and there is a strong correlation between the control channels in idle mode and data channels used to transfer the user data. The correlation between the control channels in idle mode and the user data channels in the connected mode for 4G networks is practically defined by standardization and will not vary significantly based on the radio equipment deployed in the network.

Therefore, by measuring the control channels in idle mode it is possible to obtain a good assessment of the actual end-user experiences in the 4G network and conduct fair comparisons between the CSPs deploying radio equipment from different vendors.

In 5G networks, standardization is allowing more flexibility in how the control channels in idle mode are configured. One of the main differences compared to 4G networks is that control channels in idle mode for 5G networks can be “beamformed” or in other words, a signal can be focused to provide higher signal strength but in the smaller, more narrow, area as presented on Figure 1.

Figure 1: Different beam configurations for the control channel

Because a narrow beam for the control channels in idle mode will cover only part of the targeted area, multiple beams need to be sequentially created to cover the whole area, so-called beam sweeping technique.

The standardization is allowing to configure up to eight control channel beams for the Frequency Range 1 (FR1, 410MHz to 7125MHz). There are pros and cons for each configuration of the beam number, optimal configuration can be different based on the implementation.

If we look now at the user data channels in connected mode, that will practically define end-user experiences. The user data channels in a connected mode are typically beamformed. However, these beams are different shapes from the beams used for the control channels and again the shape of the beams will depend on the implementation.

Figure 2: User data beam vs control channel beams configurations

The 5G standardization allows for flexibility on how beams are configured for the control channel in idle mode and the used-data channel in connected mode. This flexibility means that for different 5G deployments we might have a different correlation between the signal of the control channel and the user data channel.

Therefore, it is not possible to conduct fair benchmarking between the CSPs by conducting measurements on the control channels in the idle mode. The benchmarking measurements conducted on the control channel in idle mode, typically used for 4G networks, can not be used for benchmarking the 5G networks.

The benchmark results demonstrate that measurements based on the common channel are not accurate for comparing the performance of 5G networks

We conducted several benchmark measurements between 5G networks with different configurations for common channel and the results demonstrate that it is not accurate to compare 5G networks based on the common channel measurements. Some of the benchmarking results are presented in the following sections.

Benchmark measurements in Gulf Country “A”

Measurements are conducted for the two CSPs that use different configurations for the common channel in idle mode. CSP “A1” has a control channel configuration with beam sweeping while CSP “A2” has a control channel configuration with a wide beam.

Measurements are conducted in scattered locations. On each location, three measurements are conducted, and average results are presented in the tables in Table 1.

Table 1: Static throughput test in Gulf Country “A”

 

As it can be seen from the measurements based on control channels, CSP “A2” will typically have lower signal strength as well as lower signal to interference ratio, however, end-user experiences demonstrated through downlink and uplink throughput are better for the CSP “A2”. This is a consequence of the different configurations these two CSPs use for the control channels in idle mode. The CSP “A1” uses narrow beams for control channels in idle mode and beam sweeping technique that results in higher measured signal strength as well as signal to interference ration however both downlink and uplink throughput are better for the CSP “A2”.

Benchmark measurements in Gulf Country “B”

The 5G benchmark measurements presented in Table 2 are conducted for the two CSPs in Gulf Country “B”. Similarly, to the first example, the two CSPs have different configurations for the common channels in idle mode. Again, it can be seen that CSP that typically has lower signal strength as well as lower signal to interference ratio, CSP “B2”, has a better downlink and uplink throughput.

Table 2: Static throughput test in Gulf Country “B”

 

What measurements can be used for benchmarking of the 5G networks?

The answer to this question is not simple as it is not possible to conduct signal strength measurements in idle mode in the same way it was done for 2G/3G/4G networks. 

The preferred way for 5G networks benchmark measurements is to conduct the end-user performance driven benchmark measurements. These benchmark measurements should be based on the throughput as the main performance indicator that defines end-user experiences when using network services.

The throughput measurements are more complex to conduct then signal strength based measurement and the network traffic level at the time of measurement has an impact on the measurements. However, the throughput measurements will present an accurate snapshot of the real end-user experiences at the time of the measurement.

If one needs to use the signal strength measurements for 5G networks benchmarking, then measurements need to be normalized to reflect different configurations for control channels. Figure 3 illustrates the required normalization for two of the networks that use different configurations for the control channels in idle mode.

Figure 3: Illustration for normalization required to compare CSPs with different SSB configurations

 

As CSPs “A” and “B” use different configurations for SSB (Synchronization Signal Block), the relation between user throughout and signal strengths measured on the control channel in idle mode (SSB-RSRP, Reference Signal Receive Power for SSB) will have a constant shift. Measuring higher signal strength for CSP “B” will not mean higher user throughput and to benchmark these two CSPs, normalization for measured signal strength needs to be implemented.

If signal strength measurements are used for the benchmarking two CSPs, it would require knowledge on the used SSB configuration (SSB beamforming approach and power allocation) for both CSPs to determine normalization value. These normalization values need to be aligned with the CSPs, priory to conducting benchmark measurements based on the signal strength.

The way forward

So, what’s the way forward? I don’t think we can easily identify a perfect solution to the current needs to conduct an objective 5G networks benchmark measurement with similar simplicity and efficiency as it was done for the previous generation of mobile networks.

However, I do think the preferred way to conduct benchmark measurements is through end-user performance driven measurements. This is because the throughput-based measurements will reflect actual end-user experiences in the measured network.

Ericsson is recognizing an industry-wide challenge related to the methodology for 5G benchmark measurement and is currently working on a document for the 5G measurement methodology that will be made public in the near future.

Download the full paper here