CI/CD and CD/D: continuous software delivery explained

What is CI/CD (continuous integration and deployment)?

CI/CD – or continuous integration and continuous deployment is a  framework where software is continuously built, tested, deployed and validated throughout its lifecycle.

While the CI/CD philosophy has been appropriated from the IT industry, it’s now being recognized as essential for the telecom industry’s increasingly digital value chain. For some time now, telecom vendors like Ericsson have been applying CI/CD practices to increase both feature flexibility and software quality. However, continuous integration is still an IT definition, and predominantly falls on the vendor side of the firewall in telecom networks.

What is CD/D (continuous delivery and deployment)?

 CD/D is the much-needed “continuous delivery” and “deployment” model associated with cloud-native software and DevOps. It means that software is developed, delivered, tested, accepted and brought into operation incrementally at a far higher cadence than it was before in a traditional service provider environment. This creates a significant volume of software that needs to be validated for an increasingly dynamic network. The implication then is to bolster support for continuous software validation using what’s known as “CT” or “Continuous Testing”.

So what is Continuous Delivery and Deployment (CD/D) AND Continuous Testing (CT)?

Ericsson operates a framework for continuous software delivery and deployment automation using both IT industry and Ericsson-built tools. The framework implements an automated digital software delivery and testing production line that bridges the vendor/operator divide. This can be adapted for multivendor operations and/or integrated into service provider umbrella CI/CD workflows. Listen to the benefits of Continuous Delivery and Deployment as seen by one of our early adopters.

Why is CD/D important for service providers going forward?

A continuous delivery and deployment framework unlocks many benefits for service providers, the key one being that it allows them to deliver the latest and greatest software in a “continuous” fashion. It also provides the service and business agility that service providers have been looking for –  reducing time to market for new features, allowing security vulnerabilities addressed in time, and reducing OPEX with the automated delivery of software.

However, while the value to upgrade the software “continuously” is easy to imagine, the potential for OPEX efficiencies from automation does demand agile ways of working. In other words, CI/CD cannot be realized without a new seamless software delivery and deployment partnership that should be built between service providers and their portfolio of software vendors.

With cloud native, 5G, IoT and Industry 4.0 technology adoption, value creation will require increasing software change accompanied by the complexity to achieve shorter time to market. As a result, there is an obvious tipping point where traditional ways of software adoption (upgrade, update, rollout) and testing will fail to scale.

Figure 1: The trends require new ways of working. Technical shift, increasing workload and shorter time frames

As 5G Core is rolled out, service providers will face an order of magnitude increase in the demand for software configuration testing and validation. While this volume of testing will take some time to reach its new sustained level of intensity, there is no doubt that automation is essential.

Scale factors include:

• 5G Core network function software updates available every 3 or 4 weeks
• Individual microservice updates impacting multiple network functions utilizing common microservice code
• Network automation policy being rapidly tuned in feedback loops between operations and engineering
• In-house network service creation requiring internal agile CI/CD pipelines

In the common microservice code model that underpins cloud-native software, every time a common code software component is improved it will cause a change impact on all network systems that use that common code. This can bring lightning fast agility and innovation, but leaves today’s legacy bi-annual software test and validate processes entirely unfit for purpose.

The introduction of cloud-native and microservices technology into the network core can only flourish with a high level of automation in continuous delivery, deployment and test validation.

While the CI “development" part is best outsourced with suppliers (i.e. telecom vendors), the CD/D process needs a heavy uplift and transformation project in the service provider environment. This is true regardless of whether it is a vendor-driven software pipeline or vendor-agnostic continuous deployment engine built with open-source tools. Importantly, the digital agility that operators aspire for will come with a significant toll for test and acceptance in the form of “Continuous Testing”, which must be automated.

Figure 2: Automation & selective testing key to scaling at digital speed

What is the greatest barrier to CD/D?

As cited in the 2019 NGMN white paper¹, telecom operators are bound to hit what is described as the “air gap” in their adoption of CD/D. It is caused by the obvious separation of priorities and concerns between the various software lifecycle stakeholders in an operator and its multiple suppliers (telecom vendors).

Indeed, several organizational challenges must be overcome for successful introduction:

• The challenge of agile iteration across vendor/service provider organizational boundaries
• Legacy waterfall software development lifecycles (SDLC) encouraging sequential change in at least two separate engineering and operations (Dev and Ops) domain silos
• Network architects assessing software feature impact and relevance prior to software delivery
• Several substantial test teams deploying and testing software in project-centric test environments
• The “temporal project” philosophy that results in major software update cycles of typically one or two per year.

All the above play an important role in assuring safe software change in the legacy software lifecycle but need to be realigned to support high-cadence CI/CD models.

Addressing the air gap

The adoption of CI/CD methodologies requires service providers to foster a different approach to operations. An agile way of working with close collaboration between engineering and operations is needed to enable rapid innovation and business value in a safe and secure way. Without transformation, the clear OPEX benefits won’t be realized – echoing for many the first era of NFV.

In order to do so, it’s high time to move away from the buzz words (CI/CD, Dev-Sec-Ops) and instead address the culture and mindset changes required within the 3Ps: policy, process and people.

• Policy is the guiding set of decisions that translates strategy, business objectives, risk assessment, key decisions, and measurable business and operational targets for CI/CD into actionable directives
• When it comes to processes, service providers should do an assessment on the scope of opportunity to apply CI/CD over time, and not just the tools to implement it. It’s best to adopt some of the SAFe (Scaled Agile) framework to enable a continuous deployment pipeline with process agility to manage multiple teams and multi-vendor deployments.
• While addressing the third P - people – workforce transformation is needed for organizational readiness. This involves setting up a competence development plan that includes a competence gap assessment, defining the roles needed, their organizational mapping and developing a structured training plan to support CI/CD and agile ways of working.

Essentially, policy guides people and process to adopt a CI/CD-friendly culture supported by technology. Process is the key lever to affect culture change, while people are key to mindset change and adoption.

Figure 3: 5GC is designed to be updated to CI/CD and specifically CDD -Continuous Delivery & Deployment model in an Operator, which underlines the need for a culture change

All of this must be complemented by technology, test strategy and governance. In CD/D, “technology” is the tools framework architected to be the backbone of the software delivery pipeline, the “test strategy” is a well-defined blueprint to implement concepts of comprehensive, selective and passive testing to provide “continuous” software validation as required by the service provider, and as applicable to its telecom environment. Further, an overarching governance mechanism, across multiple pipelines and multiple teams, is needed to ensure visibility and tracking of agile trains, and team velocity for an efficient software deployment lifecycle.

Hence, for automated CD/D to be successful, ways of working transformation must be the mantra for service providers. This will help them become dynamic digital businesses that can address the changing demands of its services and monetize the increase in workloads with speed and agility. However, this starts with enabling the necessary culture and mindset change, and so, it’s important to define the two key pillars of policy and test strategy, and build people’s competence to create agile process change.

In summary, the automation “air gap” can be bridged through CD/D tools and organizational change. At Ericsson, we are also helping customers to address the policy, process and people changes that make our state-of-the-art CD/D framework possible.

Further reading

NGMN Whitepaper – Continuous Delivery in Telecommunication Network Environments