Performance verification for 5G NR deployments
This Ericsson whitepaper explains how, in spite of the myriad design solution options and the restrictions associated with some deployments, 5G NR performance verification doesn’t have to be complex. 5G New Radio (NR) network deployments bring significant performance benefits but may involve challenging deployment constrains. By using a framework that identifies solution-specific design dependencies, Key Performance Indicators (KPIs) and measurement verification methodologies can be implemented efficiently and cover the essential use-cases.
Introduction: Strong momentum for 5G
There is strong momentum in the global 5G market . 5G networks are currently being deployed in several regions worldwide and commercial launches are already taking place. On a global level, major 5G network deployments are anticipated from 2020 and, by the end of 2024, Ericsson projects 1.5 billion 5G subscriptions for enhanced mobile broadband. This is in addition to Fixed Wireless Access (FWA) subscriptions, which will be one of the first 5G use cases.
NR will be deployed to help cater for mobile traffic growth which will be driven by both the rising number of smartphone subscriptions and an increasing average data volume per subscription . Through to 2024, video will grow by around 35 percent annually to account for around 74 percent of mobile data traffic, up from 60 percent in 2018 . The full capabilities of NR will extend far beyond previous generations of mobile communication . Examples of these capabilities include very high data rates, very low latency, ultra-high reliability, energy efficiency and extreme device densities. These capabilities will be realized by the development of LTE in combination with new radio-access technologies.
When considering the actual performance of deployed NR networks, it is important to consider the deployment scenario and constraints that apply to the individual network deployment. The performance targets that standards issuing bodies ITU and 3GPP have set for NR technology standardization are often aspirational goals that have been set to allow for the selection of appropriate technologies or configurations from competing options. These targets can also, for example, represent the peak performance that can be achieved under ideal conditions with fully deployed configurations.
Real world deployments will be different so, when verifying NR network performance, it is important to consider the actual deployment scenarios and any constraints that may be placed on the wireless solution and the network rollout. The verification process needs to translate the vision for NR capabilities with the real-world solution and deployment constraints that apply to commercial network rollouts. In addition, where components of the NR solution are shared with legacy networks, it is important to ensure that the existing services are not detrimentally impacted.
This paper proposes a framework that simplifies the process of NR network performance verification and provides a concise view of performance that is aligned with user experience. The use of a NR network performance verification framework is important because networks are becoming increasingly complex – not just in terms of design and optimization, but also in relation to testing and verification. The result of this is a focus shift and introduction of a structured approach to verification that supports simplicity and transparency with an aim of providing rapid time-to-market.
NR performance verification in 3 steps
A structured process for determining the appropriate methodologies, processes, performance metrics and targets for NR network verification can facilitate a rapid network verification and a faster time-to-market. A three-step process can be used to develop a NR network verification process tailored to the deployment scenario as shown in Figure 1 below.
Step 1: Analyze NR drivers, assets and constraints
A method for determining the inputs to the verification framework is shown in Figure 2. Each decision category will be described in more detail in this chapter.
Business requirements direct the NR verification framework
The first stage of the decision flow is to define the business requirements or opportunities that are driving the deployment of the NR network. For example, an operator may be looking to maintain performance leadership in the market, address new or extended business opportunities, achieve a rapid time-to market in order to seek a competitive advantage or to achieve operational saving. Indeed, there may be multiple business drivers for a given NR deployment.
A detailed understanding of the business drivers is needed to ensure that the verification processes and time lines meet the requirements of the operator.
Intended 5G use cases should be clearly defined
NR networks will be used to address one or more use cases. 5G networks will facilitate the introduction of new applications, many of which are not yet clearly defined, however the first three main mobile use cases that 5G technology must support are :
- Enhanced Mobile Broadband (eMBB) - for extended support of conventional MBB through improved peak/average/cell-edge data rates, capacity and coverage,
- Ultra-Reliable Low Latency Communications (URLLC) - for emerging critical applications, and
- Massive Machine Type Communications (mMTC) - to support the envisioned 5G IoT scenario with tens of billions of connected devices and sensors.
In addition, FWA will be one of the first use cases for 5G .
The intended use case of the network deployment and the availability of devices will impact the methodology used for network verification. As an example, FWA applications with predominantly external Customer Premise Equipment (CPE) will be best verified using stationary testing rather than using a traditional drive test approach with terminals inside a test vehicle.
Deployment scenarios should be clearly defined
The deployment scenario will determine the methodology used to verify the performance of the NR network deployment. The method used for testing the performance of a ubiquitous mid-band deployment will be very different to that used to verify high-band performance of hotspot sites. The performance targets of a shared NR/LTE low-band carrier will be very different to those of a high-band FWA network.
Several different deployment scenarios are possible for the rollout of NR networks and these deployment scenarios are discussed in more detail in this section.
Overlay - A typical overlay scenario involves the deployment of NR equipment at all, or the majority of, base station sites within a particular geographical area. When undertaking NR network overlays, it is important to consider the relative propagation characteristics of the NR channels being deployed and the baseline frequency band for which the underlying network was designed.
Hotspot or targeted - Hotspot or targeted deployments, which are sometimes referred to as pepper-pot deployments, selectively identify sites for the installation of NR equipment. These sites may be identified because of the need for capacity offload for existing networks, because of their importance as high traffic areas, such as stadiums, shopping malls or transport hubs, or for competitive marketing purposes.
Ubiquitous - Ubiquitous coverage deployments aim to provide blanket, overlapping coverage using NR equipment within a particular geographical area.
Shared spectrum - Ericsson Spectrum Sharing software , provides the capability to introduce and add 5G within existing 4G carriers. This will allow the introduction of 5G on low- and mid-bands for wide area coverage.
NR introduces many RAN configuration options
The NR physical layer has a flexible and scalable design to support diverse use cases with extreme and sometimes contradictory requirements, as well as a wide range of frequencies and deployment options . The NR solution selected for deployment will have a significant impact on the performance that can be achieved.
The solution design for an NR network will be a compromise between many competing requirements and constraints. These will impact the performance metrics which means that if the network is optimized for one metric, there may be degradation of another metric .
NSA vs SA - The first version of the 5G standard covered Non Stand-Alone (NSA) mode which enabled NR to coexist and interwork with 4G. For initial NR deployments, NSA will provide a means of reducing time-to-market and ensuring good coverage and mobility. The 5G standard for Stand-Alone (SA) mode introduces a new service-based core network architecture which will enable deployments of 5G as an overlay to or independently of 4G coverage.
From a network verification perspective, this means that for NSA NR deployments the performance of the 5G devices from an accessibility, retainability and mobility perspective will be closely aligned to and dependent upon the performance of the underlying LTE network.
Duplex method - The NR frame structure supports Time Division Duplex (TDD) and Frequency Division Duplex (FDD) transmissions and operation in both licensed and unlicensed spectrum. The duplex method and frame structure selected for the NR deployment will influence the integrity KPI performance values that can be achieved in the network. For example, TDD networks share the same radio spectrum for the downlink (DL) and uplink (UL) transmissions which means scaling needs to be applied to the expected throughputs relative to an FDD channel of the same nominal bandwidth. TDD systems also require the configuration of a guard period to protect UL transmissions towards a Base Station (BS) from being interfered by delayed co-channel DL transmissions from another BS. The duration of the guard period, required to avoid interference, will be determined by the expected coverage of the TDD cells and this in-turn will impact the throughput which can be achieved.
Numerology - NR introduces a scalable Orthogonal Frequency Division Multiplexing (OFDM) numerology to enable diverse services on a wide range of frequencies and deployments. The choice of the numerology scaling factor depends on various attributes including the type of deployment, carrier frequency, service requirements (latency, reliability and throughput), mobility and implementation complexity. The selection of the NR numerology for the deployment will impact the latency that can be achieved in the network. This will have consequences for the integrity KPI values which can be achieved during tests such a PING tests which measure Round Trip Time (RTT).
Interworking and spectrum sharing - As previously mentioned, NR has the potential to interwork closely with LTE deployments. In addition, functionality such as Ericsson Spectrum Sharing software , gives the possibility to intelligently, flexibly and quickly introduce and add 5G within existing 4G carriers. When considering the appropriate NR performance targets for verification it is important to consider any constraints that may be introduced by sharing bandwidth and resources between NR and LTE. For example, if spectrum sharing is implemented, the achievable NR throughputs will be impacted by the load and utilization of the LTE network.
Frequency band - Each frequency band has different physical properties, meaning there are trade-offs between capacity, coverage and latency, as well as reliability and spectral efficiency. NR deployments will introduce and extend the use of mid- and high-band spectrum. The coverage capabilities of these frequency bands will need to be taken into consideration when setting verification targets.
Bandwidth - The bandwidth available for NR deployments will have a significant impact on the peak, average and cell edge throughputs that can be achieved. This needs to be fully considered when determining appropriate targets for network verification.
RF power levels - Restrictions on the total Radio Frequency (RF) power emissions allowed from cellular base station sites are common in some regulatory jurisdictions. Power restrictions of RF carriers can limit the coverage capabilities and achievable throughputs for the RF carrier.
Coexistence - Coexistence of NR and LTE deployments from different operators in the same or adjacent frequency bands can be expected to place constraints on the parameters used for some TDD deployments. A number of regulatory bodies  have proposals in place to harmonize the technical conditions for the use of spectrum in NR TDD frequency bands because of the potential for adjacent channel interference. Unsynchronized adjacent channel TDD interference can potentially occur where operators implement different frame structures and numerologies in adjacent frequency networks within the same geographical area. For systems that utilize Massive Multiple-Input Multiple-Output (MIMO) antenna systems, this problem may be exacerbated by the difficulty in implementing high levels of RF filtering within the small space and weight constraints of these antennas. If constraints are placed on the availability of frame structures and/or numerologies for NR systems using TDD spectrum it will need to be considered when the verification targets are set.
Network assets and constraints should be factored into the process
Decisions on where, when and how operators deploy 5G are not only driven by commercial considerations – such as user numbers and distributions, Total Cost of Ownership (TCO) reduction and competitive reasons – but also on the availability of network assets. These assets can include spectrum considerations such as the frequency band, amount or bandwidth of spectrum and the time availability of the spectrum. Network facilities and equipment which need to be considered for NR network performance include site and antenna location, backhaul bandwidth – including upstream aggregation – and core network facilities and equipment.
Each of these factors has the potential to impact the performance and time to market of an NR deployment and must be considered when determining the verification methodology and performance targets. For example, delays in site build permissions may lead to coverage holes in the network for new NR frequency layers because some cells or sites may experience significant deployment delays. In addition, a lack of NR capable devices may mean that reliable operational statistics may not be available to evaluate the performance of the NR network.
Step 2: Align verification milestones to project scope
The NR deployment project will involve multiple activities which comprise the project milestones. The scope and responsibilities of the project will determine which of the typical milestones are applicable to the different project stakeholders. Some milestones will involve performance verification, and these can be typically mapped as illustrated in Figure 4.
A project structured with appropriate milestones can accelerate time-to-market through prompt network performance verification, reduced administration and efficient project delivery procedures.
Step 3: Define the verification environments, KPIs and target values
Use the KPI framework to set realistic performance targets
A typical network-deployment involves several stages as shown in Figure 5, and different verification processes are applicable at each of these stages.
The KPI framework is used to determine:
- Performance areas which need to be measured
- Environments in which testing must be undertaken such as a lab, golden site or golden cluster, whole clusters and others
- Metrics to be measured and the test methods used to make the measurements
- Selection of metrics which will be KPIs and those which will be designated as Performance Indicators (PIs) for performance monitoring and/or troubleshooting
- Target value selection as applicable to the environment and test method
KPI framework in detail
There are many ways of measuring network performance. This chapter proposes a framework that simplifies the process of network performance verification and provides a concise view of performance that is aligned with user experience.
The key characteristics of this framework are that it:
- Complies with ITU and ETSI recommendations for performance reporting
- Uses commercially available terminals (where possible) with standard network parameter settings
- Involves the selection of a manageable number of top-level KPIs, built from many lower-level PIs
- Includes only KPIs that are relevant to user experience
- Specifies how and where each KPI should be measured
- Focuses on verification that is in line with operator priorities and is applicable in strategically selected areas
KPI framework development starts with the use cases
A key input to the development of the verification framework is the intended use case of the NR network. Testing of the overall network capabilities should be conducted in a manner which reflects the applications and terminals used by users.
Many operators will be familiar with test and verification methods which have been used for Mobile Broadband (MBB) services in the past. New use cases such as using FWA to deliver broadband services will introduce new FWA CPE. The form factor of these CPE may range from simple indoor self-install devices to fixed outdoor units with a built-in directional high gain antenna allowing a predictable radio link quality to the selected base station.
Locations and sites selected for verification testing should be representative for the intended site rollout types. Clusters selected for golden cluster testing should have contiguous radio coverage and be representative for the network topography and intended target areas, for example rural, suburban or urban.
In summary, the test and verification methodology selected should be tailored to match the intended network use case.
Solution verification, build verification and environments for testing
The traditional stages used in network verification are displayed in Figure 6. The process moves from solution and product verification, through site build verification and then to cluster and network verification. Once the network is operational and there are a sufficient number of terminals operating on the network, performance can be monitored via operational counters and formulas.
The capabilities of new technologies are often demonstrated in tightly controlled trial scenarios with ideal and highly-optimized network conditions. Trials can be carried out using high-performance User Equipment (UE) prototypes or simulators, with network parameter settings optimized for performance rather than capacity. Whilst these trial environments can be useful for determining appropriate network configuration settings they do not always accurately reflect real network environments and performance. In operational networks, there are trade-offs between performance and capacity that cannot always be taken into account in trials.
In a well-structured verification framework, complex and time-consuming verification procedures should be limited to laboratory or controlled field environments. Such tests can be advantageous when it comes to validating the final set of KPIs and suitable target-value ranges. Golden sites and golden cluster are test environment concepts specially designed to further test the technology in live scenarios, while still in a confined environment.
Typically, the golden site or cluster phase testing is reserved for tests of various configuration options with the aim to simplify – and agree – the configuration and measurement methodology to be used during the wide scale rollout. This approach allows both the customer and Ericsson to achieve cost and time savings by identifying the minimum subset of KPIs that are representative for the site rollout and cluster acceptance.
For cluster tuning and acceptance, a small number of KPIs should be selected to allow for efficient testing and to ensure a sufficient degree of accuracy when measuring user experience. Focused verification can be based on factors such as the location of key users, areas of dense traffic, difficult environments and known trouble spots. This will ensure that the cluster performance is acceptable for commercial launch.
Performance metrics and KPI selection
The next step in the development of a verification framework is to determine which performance to monitor and how to monitor it. For mobile networks, the ITU Telecommunication Standardization Sector (ITU-T) has worked with ETSI to describe a general model for QoS from the user's perspective . The 3GPP-defined QoS categories which are based on this model are service accessibility, retainability, integrity and mobility.
To facilitate a cost-efficient verification process, it is recommended that just a few QoSaligned KPIs be used for subscriber experience and network quality verification. Additional PIs can be recorded and monitored for network analysis and troubleshooting purposes.
Operators need to be assured that a given KPI will reflect the actual user experience accurately after rollout. For example, a radio bearer can be dropped when an always-on LTE device is not transmitting data. Measuring such a KPI does not offer any real understanding of user experience, as sessions are quickly re-established when needed and any delay goes unnoticed by the user.
By using a structured approach, it is possible to consolidate many PI metrics into a small number of KPIs which are reported and can accurately reflect the user experience. This allows an appropriate focus on the vital performance indicators which reflect user experience, for reporting and analysis. This reduces potential wasted effort on collection and analysis of too many indicators with little added benefit. Additional effort can then be spent on addressing any issues quickly, fast tracking improvements in network performance.
KPI target value determination
The target values selected for KPIs should take into consideration the NR network constraints, solution and design. Target values for KPIs should also reflect the different test environments in which the measurements were made.
Once a network is in operation, counter-based KPIs can be utilized to monitor performance as traffic increases. These KPIs based on commercial traffic and network counters are likely to differ from those selected for cluster tuning, which are based on drive test measurements. For example, measuring user perceived latency is straightforward with a drive test, but impractical with network counters.
Benefits of a structured framework
The need for a structured, top-down approach to network verification using a set of standard service-level KPIs has previously been recognized within the industry . These KPIs are grouped into five main categories – accessibility, retainability, integrity, availability and mobility – where measurements use common formulas and methods.
Network-level testing and acceptance should focus on these KPIs, which are based on numerous resource-level PIs that provide system behavior and performance information at a more granular level.
Benefits for operators
- Faster time-to-market for new networks, services and coverage
- Confidence that net work-centric measurements accurately reflect the user experience
- Capacity to focus resources and efforts on areas that will result in the greatest benefit for users
- Ability to achieve targeted network efficiency
Benefits for vendors
- Possibility to focus resources and efforts on tuning networks, maximizing network efficiency and improving user experience
- Capacity to meet aggressive timelines
- Faster network deployment
- Creation of strong operator partnerships
- Opportunity to deliver high-quality networks that can be adapted to changing technology and market conditions
New services, new types of devices and the utilization of multiple technologies are creating a complex environment for NR network deployment. This increase in variables necessitates an efficient approach to network performance verification that encompasses the entire NR roll-out from initial testing through to in-service operation.
Achieving efficiency here means operators are able to focus on rapid deployment of the NR network with assured performance for successful commercial launch and continued operation.
Adopting the correct methodology for NR network performance verification enables operators to be confident that network-centric measurements accurately reflect the user experience. This allows resources and efforts to be focused on areas that will result in the greatest benefit for users, simultaneously helping operators to meet their network efficiency targets.
Mid-band NR eMBB overlay
An operator has a legacy LTE network deployment that involves multiple low and mid-band FDD LTE layers. The operator has acquired mid-band 3600 MHz spectrum with the intention of deploying an overlay TDD NR network using NSA mode. The rollout intention is to commence with an overlay of NR equipment in the center of major cities and gradually increase the footprint of the NR coverage as the rollout progresses.
NSA 5G Option 3 standardization was designed in 3GPP to provide connectivity for combined LTE and NR systems. This option uses LTE as the control plane anchor for NR and uses either LTE or NR for user traffic – utilizing the user plane. The Option allows for the deployment of dual connectivity for data with high throughput in NR downlink and best coverage in LTE uplink, while voice traffic is fully on LTE.
When developing the verification framework for this deployment there are many factors that will need to be considered. Firstly, because the NSA deployment is tied to LTE, it is important to ensure that the existing network is not disrupted.
The NSA Option 3 uses the LTE as the control plane meaning that the performance of the NR network for accessibility, retainability and mobility will be closely tied to the performance of the underlying LTE network. Any testing or verification work undertaken must take into consideration the load and performance of the LTE network at the time of the testing.
The NR layer will provide data connectivity, especially for high throughput DL data transfers. One important point to note is that the NR mid band frequencies will experience a higher pathloss than the lower band LTE carriers and this will impact the DL throughput achievable on the NR layer. The DL throughput on the NR TDD carrier will also be determined by:
- The carrier bandwidth
- The NR TDD frame structure, including the ratio of the DL to UL subframes and the time period allocated for the TDD guard period
- The available UL throughput capacity if testing is to be undertaken using protocols such as Transmission Control Protocol (TCP) which require regular uplink acknowledgements for DL transmissions
The decision as to which NR TDD frame structure to implement and the amount of guard period required, may be driven by factors both internal and external to the operator's network.
Decisions around the frame structure may be influenced by other users which may share the same or an adjacent frequency band in the 3600 MHz frequency range. If two operators are to deploy TDD networks in the same geographical area then, depending on the filtering capabilities of the radio systems and the User Experiences (UEs), the operators may need to synchronize their frame structures in order to avoid adjacent channel interference.
Decisions on the guard period duration will be influenced by the potential for Cross Link Interference (CLI) within the network and also the intended cell range. Factors which will need to be considered are the distances between the furthest base station sites within the geographical area and the potential for base station to base station propagation between these sites. Experience in LTE TDD networks has shown that high, hilltop sites on the outskirts of large cities can be especially problematic when it comes to CLI.
High-band FWA deployment
An operator plans to deploy a high-band NR network designed exclusively for FWA which will operate in SA mode. This means that all control and user plane data will be carried on the NR network. Given that high-band NR carriers will be TDD, if other operators share the adjacent spectrum then consideration may need to be given to network synchronization in order to avoid adjacent channel interference.
In SA mode, the accessibility and retainability performance of the NR network will not be influenced by other networks in the same way that the NSA architecture is. Also, being a network with only FWA terminals, it is probable that mobility will not be implemented within the network.
The throughput capabilities of the FWA NR network will be determined by the bandwidth of the NR carrier and the frame structure allocations to DL, UL and guard period. The values which can be achieved for RTT tests will be influenced by the frame structure which is selected for use. FWA networks offer a number of options for user terminal types. It is possible to use outdoor terminals with high gain antennas in order to improve the radio link performance for subscribers significantly. Alternatively, indoor terminals may be used which can utilize medium gain antennas which may be directional or omni-directional.
When undertaking verification testing for FWA networks it is important to test in a manner that reflects the user terminal characteristics. If the network is designed for outdoor, high gain CPEs then it is important to undertake verification tests with a CPE terminal with a height which reflects that of a typical user CPE installation.
EC Decision 2014_276_EU "Harmonised frequency arrangements for bands 3400-3600 MHz and 3600-3800 MHz"
ETSI, Technical Specification 132 450, 2009-04, Key Performance Indicators (KPIs) for Evolved Universal Terrestrial Radio Access Network (E-UTRAN ): Definitions (3GPP TS32.450 Version 8.0.0 Release 8)
3GPP 3rd Generation Partnership Project
4G 4th Generation Mobile Networks
5G 5th Generation Mobile Networks
BS Base Station
CLI Cross Link Interference
CPE Customer Premises Equipment
eMBB Enhanced Mobile Broadband
EME Electromagnetic Environment
ESS Ericsson Spectrum Sharing
ETSI European Telecommunications Standards Institute
FDD Frequency Division Duplex
FWA Fixed Wireless Access
IOT Internet of Things
ITU International Telecommunication Union
KPI Key Performance Indicator
LTE Long Term Evolution
MBB Mobile Broadband
MIMO Multiple-Input, Multiple-Output
mMTC massive Machine Type Communications
NSA Non Stand-Alone
NR New Radio
OFDM Orthogonal Frequency-Division Multiplexing
PI Performance Indicator
QoS Quality of Service
RF Radio Frequency
RTT Round Trip Time
TCO Total Cost of Ownership
TCP Transmission Control Protocol
TDD Time Division Duplex
UE User Equipment
URLLC Ultra-Reliable Low Latency Communications
The authors to Ericsson's opinion on this topic are Marius Sribu and Greg Harman.
Marius Sirbu drives the overall strategy for network rollout verification. He joined Ericsson in 2005 and has served in roles as consultant, sales and strategic product manager in the area of RAN design, rollout and optimization. Prior to this he worked on 2G and 3G deployments in Europe and the Middle East. He holds a Bachelor of Science degree in Applied Electronics and Telecommunication from the Technical University of Cluj-Napoca, Romania.
Greg Harman joined Ericsson in 2008 and has worked as a radio engineer since 1993. He is currently employed as a principal solution architect working in Ericsson's radio network design and optimization organization. He is involved in radio network consultancy work for 5G, 4G and Massive MIMO antenna systems. Prior to this he has worked on all cellular technology generations starting with 1G analogue systems. He holds both an M.Eng. in Telecommunications Engineering from RMIT University and an MBA from Deakin University in Australia.
The authors would like to thank Tony Guy, Matt Ford, Raul Martin, Ricardo Roman, Michele Cassioli, Salvatore Sesstito, Han Chuan Goh and Thiago Rachadel for their valuable input to this white paper.