Achieving CI/CD in a cloud native multi-vendor environment

Continuous Integration/Continuous Delivery (CI/CD) is a promising solution with components created and deployed by multiple vendors, such as 5G. Fitting multiple vendors’ components into a deployment and configuration automation is a complex task for communication service providers (CSPs). Each piece must work seamlessly with peer components to deliver an end-to-end working solution. To reap the benefits, CSPs must expedite the solution to these challenges of multi-vendor environments.

The increasing importance of CI/CD

Any vendor or service provider hoping to stay in the customer environment beyond the first deployment must consider the life cycle of the deployed products and services. Manually managing these software and services is error-prone and increasingly impractical. ‘Manual’ is a thing of the past. Instead, CSPs are using tools such as Jenkins, Spinnaker, GitLab and Tekton to: set up the continuous integration of code, build software and test and continuously deliver the built software to all relevant customer lab and production sites. CI/CD enables the vendor to expedite the software delivery while adhering to the customer’s operational SLAs.

Setting up CI/CD

CI/CD tools and procedures are typically set up before the actual software delivery and outlive the software that they deliver. CI pulls in all the software binaries and sources. In a complex solution, it must securely and seamlessly pull in this data from multiple vendors, test each feature, integrate and finally build the final solution. CD, on the other hand, takes the built solution, applies version control, ships it to the customer’s remote destinations, sequences and/or parallelizes the deployment artifacts, deploys the software stack, runs post install tests and provides a result back to the administrators. All these steps are done with visual indicators and the ability to pause, inspect, redo and undo capabilities. The pathway that achieves these steps is called a CI/CD pipeline. Multiple pipelines can exist for just one solution.

In the modern world of software solution delivery, CI/CD becomes the backbone of the much sought-after framework called Development and Operations (DevOps). DevOps strives to connect all stakeholders of the solution to the CI/CD backbone. The result is an informed, expedited development for delivery of small, big or complex solutions.

The challenge of CI/CD in a multi-vendor environment

Hosted or cloud solutions come in a variety of sizes and complexities. No matter the solution, however, vendor lock-in presents a significant risk. Vendor lock-in occurs when the customer signs for the entire solution sourced by a single vendor. That’s great for the vendor, but unquestionably bad for the customer. Vendor lock-in creates a monopoly, increases cost and decreases innovation.

In order to avoid vendor lock-in, customers big and small choose solutions that involve multiple vendors sourcing various components. Because CI/CD is commonplace in software delivery, each vendor uses their own preset CI/CD setup to deliver their software; presenting a significant challenge for the customer. The end solution must work seamlessly not just at the first deployment, but for the full length of the solution’s life cycle. The customer then periodically installs software updates, security fixes and firmware updates, hoping that each update fits the bigger solution without delay and without errors or outages. In reality, none of these updates perfectly sync up with all the vendors and components.

Today’s IT and TELCO applications serving consumers require four- or five-nines availability to see success and stay above the competition. This level of high availability calls for no more than two outages on any feature offered by the solution over a whole year. This standard, in a complex multi-vendor sourced solution, requires a lengthy testing and approval process for each update by the customer’s operations unit. In the absence of a common, unified CI/CD and the related visibility to the customer, each vendor, upon delivery of the update, must implement costly regression test cases—lab deployment re-runs outside their CI/CD systems. Such significant delays for feature-rich updates could give the advantage to more agile competition. And the delays in security fixes could also leave the production solution vulnerable to hackers and consequential legal and consumer ramifications.

A typical multi-vendor and multi-CI/CD customer solution flow and setup includes the following:

• Development environment and labs: A set of labs that is loosely based on production but defined based on development needs. This environment also hosts the tools for CI/CD for the rest of the pathway. Here, developers create or assemble the software with customer configurations from multiple sources and code repositories, perform unit tests and run integration tests. Lack of automated multi-vendor CI pipelines mean that the developers must spend cycles on split automation to bring in software from various vendors and re-run copies of test cases and integrate code.
• System test environment and labs: An exclusive set of labs that are based on production environments. The CD tools would push the un-released but integrated software to labs on this environment. Here, a team of engineers would run automated functional test cases, security tests and pen tests to make sure the yet-to-be-released solution has no flaws or bugs. Most steps in this environment could be automated if the right CI/CD tools are set up correctly. This environment is still under development control.
• Pre-production: In this environment, the labs are mirror copies of production sites and are controlled by customer’s operations unit. The software delivered here must be GA (General Availability) and is treated as production for all intents and purposes. The only difference is that there is no live traffic. Operations uses pre-production to troubleshoot issues uncovered in production. They also run attack simulation based on AL and ML (also controlled by unified CD) software to detect software and solution flaws and anomalies. With multi-vendor CD, patches to the affected software component could be delivered here fast for operational testing before being released into production. This environment is often also used for stress and load testing.
• Production: All customer sites that run live traffic. These sites are fully controlled and operated by the customer’s operations and have strict policies against unauthorized access for any action. Expedited delivery of any feature, patch or update is only possible with well-tested and well-integrated code made possible by a streamlined multi-vendor CD. Any issues identified in this environment must be quickly replicated by the CD tools in lab environments and a fix quickly delivered after the necessary testing and approvals by the operations team.

There are several tools available to setup CI/CD both in the open and proprietary marketplace. Often the challenge is to pick the right tools with proven interoperability. Ericsson provides the following recommendations for help with tool selections:

• GitLab: Provides a full DevOps platform that can host all CI and CD pipelines
• Spinnaker: Provides full DevOps platform like GitLab
• Git: Open-source distributed version control system and widely adopted but communities and organizations
• Dev IDE: This is where most code is written and published to Code repositories. Visual Studio Code is free and has good community support base
• Maven: For software project management
• SonarQube: Automatic code review tool to detect bugs and vulnerabilities
• jFrog Artifactory: Artifact or package repository

Solution Paths to CI/CD

The options for managing CI/CD in a multi-vendor environment can be broadly summarized into three approaches:

Merging vendor provided CI/CD outputs

This approach takes CIs and CDs of each vendor of the solution and merges or integrates the output that come from each pipeline to a common destination as decided by the solution owner or prime integrator. This works best when Ericsson or any vendor who is the prime integrator and has working contracts with other vendors and the solution owners. The advantages of this method are that the end solution can be delivered at speed if there are no significant changes to the deployment mechanism of each vendor software. There are several disadvantages, however. The one-time integration of all the CIs and CDs is a complex task. Every action from each vendor is not standardized and the execution of the pipelines would need a varying set of dependencies which can be costly without a unified CI/CD solution. In addition, any changes to the way the software from a vendor is deployed would trigger a costly re-integration of CI/CD.

Recreating CI/CD automation on a common framework

Since every vendor uses CI/CD automation, it is possible to just pick one CI/CD system, typically from the prime integrator and then require other vendors to recreate or import the CI and CD pipelines and workflows. Benefits of this approach is that a single pipeline would control the third-party (3PP) network functions (NFs) like health checks, traffic starvation, rollbacks etc. The obvious disadvantage of this method is the need to re-create complex CI and CD pipelines on yet another commonly chosen CI/CD system. Because this adds to the time and cost of the program, this option is disliked by both vendors and solution owners. When most vendors use the same or similar CI/CD system and all pre-requisites met, this approach works to deliver the fastest multi-vendor software integration and deployment. But such situations are quite rare in a fast-evolving technology world.

A hybrid approach that meets midway for vendors and solution owners

In this method, each vendor can keep their CI and CD pipelines and workflows without any invasive modifications and yet deliver an integrated solution without the pitfalls of the previous approaches. Here, each vendor packages or wraps their CI/CD pipelines so that it conforms to a particular standard dictated by either the prime integrator, such as Ericsson, or the solution owner. The pipelines and workflows execute within these defined packages as setup by each vendor without intruding upon neighboring vendors. All inputs, outputs and actions that are performed on the solution environment are based on a standardized specification.  The advantage of this novel approach is that each vendor retains their complex CI/CD and workflows when building and deploying their software but still interact with the outside world in a standardized interface. Any changes to the pipelines or the software being deployed does not change the interface thus does not require costly rework of the solution deployment.

CI/CD is never a one-solution-works-for-all problem. Every service provider or enterprise must determine its approach based on its unique circumstances and requirements. Ericsson has a long and successful history of delivering complex solutions to demanding customers worldwide, including pioneering solutions for CI/CD. We encourage you to contact us for help finding the right CI/CD solution for your needs. Learn more about CI/CD.