Service Provider DevOps: Evolving NFV deployment and operations

We introduced the SP-DevOps concept in our blog post “DevOps for service providers – next generation tools”.

A public demonstration at the IFIP/IEEE IM conference in May 2017 will show several of the tools in action as part of observability and troubleshooting processes related to a dynamic network function. Of course, as usual when things get real, there have been many lessons learned through the realization process, we’ll round up this post with the most important insights.

One key result of our work is a multitude of novel algorithmic network features, each advancing the state-of-the-art and solving specific research challenges. Many of the tools have been made available as Open Source in form of the SP-DevOps toolkit, and we could proudly announce the final release at OPNFV 2016 in Berlin. Below, we depict the available UNIFY SP-DevOps tools and map them onto a service lifecycle (as introduced in our earlier blog post).

The tools support operators and developers across all stages of the SP-DevOps pipeline, with functionality targeting verification, observability, and troubleshooting (see color-coding in the figure for categories). A set of novel verification tools addresses a key requirement of software-defined infrastructures: the constant verification of service models, both at design time and runtime. The integration with the orchestration ensures that policies can be consistently deployed to the infrastructure and potential deviations are identified at early stages. Furthermore, a software-defined monitoring approach was developed to take advantage of programmable infrastructure platforms for providing key quality indicators such as congestion notifications and packet loss estimates with a fraction of the overhead of typical network management functions. The tools were designed to be integrated in a software-driven lifecycle management and provide data through a scalable, carrier grade messaging system that eases integration. Their APIs were designed to be familiar for both Developer and Operator roles. Finally, a novel way to automate network debugging processes was implemented through a developer-friendly troubleshooting tool.

Each SP-DevOps tool itselfaddresses an individual opportunity. However, as we also point out in a recent IEEE Comunications Magazine paper , the full potential of SP-DevOps is unleashed once combining them together into powerful, automated processes and workflows supporting verification, observability and troubleshooting. The same SP-DevOps processes are executed by different actors/roles during a service lifecycle, thus establishing a common vocabulary and work routines that foster a DevOps-like approach for managing telecom infrastructure. A simplified quantitative model, explained in our presentation "What Gains for DevOps in Telecom Software-Defined Infrastructure?" points to savings realized through purely technical means of between 32%-77% in terms of OPEX costs associated to certain types of incidents and repair times.

More importantly, we – together with many UNIFY partners – had the ambition to give a more qualitative analysis of our concept in a scenario involving practical network management challenges of the emerging network virtualization era. Consequently, we demonstrated selected SP-DevOps processes integrated with the UNIFY global orchestration platform and the Universal Node, as sketched in the figure below (bold text indicates components part of the public SP-DevOps toolkit). A few key SP-DevOps tools and functions enabled the technical integration with the orchestration and infrastructure layer realization of UNIFY: Our supporting functions for programmability and data distribution (i.e., MEASURE and DoubleDecker) facilitated efficient integration of infrastructure and VNF monitoring functions with entities in all UNIFY layers; the modular design of VeriGraph allowed straight-forward integration of verification features into the global orchestrator; and EPOXIDE proved to be a powerful troubleshooting framework to instrument many types of SP-DevOps monitoring and verification tools as well as third-party debugging tools.

As actual use-case, we chose an elastic router service, which allowed us to explore and verify the set of functional blocks needed to support the main stages during the lifetime of such an NFV-based service. This includes service-specific auto scaling powered by SP-DevOps monitoring and a realization of our troubleshooting process, which directly adopts DevOps principles where a fast cycle of debugging and updating a deployed service is a key asset. Watch this demo video to see the SP-DevOps processes in action, supporting dynamic NFV service deployment and operation.

Based on these prototyping and integration experiences, we gained several practical insights and learnings regarding the deployment and operations of NFV services. For now, allow us to point out two particular key insights:

• Related to the formal description of services (e.g., a service graph), we identified several modifications required to better support automated decomposition during deployment and scaling during operational phases. Specifically, in order to realize an automated, fully-closed control loop, orchestration components need to be able to configure mechanisms beyond resource control. Consequently, we had to extend the basic service description with information regarding the configuration of VNFs that are part of the service, as well as service-specific monitoring requirements (which we realized as MEASURE annotations).
• With respect to the SP-DevOps observability process, we realized that monitoring orchestration and management (implemented as a separate plugin – the MMP) would benefit from tighter integration with service and infrastructure orchestration to facilitate improved life-cycle management. This stems from the observation that service and monitoring components need to synchronize their status and state repeatedly for seamless operation during deployment and scaling phases.

Looking back, we’ve done a tremendous journey with SP-DevOps in UNIFY (detailed extensively in this official project deliverable):

• We started by deriving how IT DevOps principles could be adapted to Service Provider scenarios;
• We then identified unresolved key challenges related to that adaption, and tackled them with a powerful set of individual tools and functions (i.e., the SP-DevOps Toolkit);
• We put all of these novel solutions together by defining detailed processes related to verification, observability and troubleshooting; and finally,
• We actually implemented these processes to support a dynamic NFV use-case during deployment and operations phases, which allowed us to share the insights gained by this experience.

Of course, the story does not end with the termination of the UNIFY project. Each UNIFY partner continues to pursue a subset within the vast scope of research challenges related to DevOps for Software-Defined Telecom Infrastructures, and we at Ericsson work hard on translating these research results into added-value for our product offering and customer engagements. Furthermore, we incorporate the learnings of the UNIFY experiences into our ongoing efforts on defining a 3rd generation cloud system platform.

We will be at the IFIP/IEEE IM conference in Lisbon in May, presenting a paper on our learnings in the experience session, and showing our demo during the first demo session. We would be happy to discuss with you in person, if you plan to be there! Also, if you are interested in designing and prototyping the next-generation clouds, have a look at our job openings.

For further details about the concept, demo and our learnings, there’s more to read:

• NFV Service Dynamicity with a DevOps approach: Insights from a Use-case Realization.
  Experience session paper at the IFIP/IEEE IM 2017
• NFV Service Dynamicity with a DevOps approach: Demonstrating Zero-Touch Deployment & Operations.
  Demo paper at the IFIP/IEEE IM 2017
• Service Provider DevOps.
  Article in IEEE Comm Magazine Volume: 55, Issue: 1, January 2017
• Final Service Provider DevOps concept and evaluation.
  Official project deliverable on arXiv.org

Wolfgang John and Catalin Meirosu
together with the UNIFY SP-DevOps team at Ericsson Research and spread all over Europe