How to set up service excellence teams as part of a DevOps transformation
At Ericsson IoT, our solutions are delivered to our customers as a service. To exceed our customers' expectations, we aim for service excellence.
“Excellence is the unlimited ability to improve the quality of what you have to offer,” Rick Pitino once said. This is also our aim – to relentlessly improve on our journey towards service excellence.
Why we started our service excellence journey
The starting point of our journey involved working in our own silos, which we were happy to do. This included product management, development, and operations – and we worked in the same way any organization does before starting a DevOps transformation, the more commonly used term.
Our transformation, which we’ve called our journey towards service excellence, started by gathering input to form a shared understanding of our customers and organizational pain points. We evaluated feedback from our customers, our customer units, our leadership teams, and our people. But why did we need to go on this journey?
- To have a team structure that supports our business and customer needs with clear end-to-end ownership of our services.
- To continuously grow competence at team level, end-to-end.
- To have a shared understanding of customer needs between product management, development, and operations to shorten lead times for new features and problems.
- To actively monitor performance and act on the results for all our services.
- To align practices, strategies, and tooling across all the services.
Figure 1: The timeline of our journey.
Introducing Service Excellence Teams
Service excellence is a mindset, but what are the means to get there?
Our DevOps transformation started by creating Service Excellence Teams (SETs). Through these SETs, we believed we could reach service excellence by:
- Having product management, development, and operations in one and the same team.
- By each SET having end-to-end ownership of their service to experience the service from the customer’s perspective.
- By achieving top service performance by introducing a service availability related error budget for each service.
As you can see from figure 2, there are three essential changes for SETs. The first is related to the team structure; to bring people from the organizational silos into one and the same team. Interestingly, although some people originally said that they already had a great collaboration between, for example, development and operations, they have since said that being in the same team has improved things even further.
Figure 2: The three essential changes for SETs.
The second essential change is related to end-to-end ownership. What do we mean with end-to-end ownership? We mean that we put the customer first, we see the service from the customer’s perspective, and we really spend time in the customer’s shoes. It also means that the SET handles all aspects of a service, such as security, functional suitability, performance efficiency, usability, and reliability. Lastly, it means that the team carries the development responsibility from an early requirement phase to a phase where the service reaches production and handles the lifecycle management in production. This is what end-to-end means for us!
The third essential change is related to top service performance where we use an error budget to make it possible for us to make courageous and fact-based decisions on the priorities. Error budget is the maximum amount that a service can fail without contractual consequences. When there is no error budget left, we need to concentrate on stability and performance improvements, and must stop developing and deploying new features. Therefore, an error budget will make sure we prioritize stability and performance when there is a need.
We now have close to twenty SETs working side-by-side providing services to our customers.
Verticals
We see the Service Excellence Teams as horizontal teams (see figure 4). Verticals, on the other hand, are the bridges between these teams. Verticals ensure we’re not just building new kinds of silos.
Verticals are nothing new – we’ve had communities such as this in different contexts across the business before. But we now want to highlight the importance of them. We want to make sure that in addition to working, for example, with service performance within their own service, teams also collaborate with the other SETs and Solution Performance vertical team on general performance. Together, they’re able to build a vertical community around performance. A vertical community is an open group of people who are passionate about the same topic, in this case about performance. We need vertical communities to align on practices, strategies, and tooling, to share and build knowledge and to solve problems together.
Besides the community, in some verticals we also have vertical teams, as we do for Solution Performance. These vertical teams offer an internal service for their community. For example, the Solution Performance vertical offers load testing on a solution level in a pre-production environment before production deployments.
In addition, it’s the responsibility of a vertical team to actively build and drive the vertical community. By providing the internal service and by driving the community, the vertical teams contribute to delivering a world class service to our customers.
Figure 3: Vertical team ABC.
Where the verticals are the bridges on our DevOps journey, our product owners (POs) are the glue between the horizontal SETs and the verticals. Each of the SETs and each of the vertical teams has a product owner who coordinates and prioritizes the work in the teams – it’s in this way that they truly glue us together.
Figure 4: Our Service Excellence Teams and verticals.
Start of the journey
Our transformation journey towards service excellence was executed while team members were working remotely and located in several countries. However, starting this journey during the remote working period was a helpful factor, as all team members were on a level playing field.
Kicking off a SET takes a week, with a half day workshop every day.A typical kick off week would be:
Monday: We look at the inspiration behind our DevOps transformation and reflect on it.
Tuesday: We form a common understanding of why we’re doing this transformation, what Service Excellence Teams are all about, and we reflect on what we learned.
Wednesday: It’s time to learn about the Service Level Agreement (SLA), Service Level Objective (SLO), Service Level Indicator (SLI) and error budgets. We also look at our goals, our Objectives and Key Results (OKR).
Thursday: We create a common understanding of our services. We also check the Service Performance of our services, we create the first idea of our Service Level Objective, Service Level Indicator and error budget. At the end, we form our team by agreeing on our roles.
Friday: It’s time to put everything together and agree on how we work together as a Service Excellence Team through our Way of Working (WoW).
Figure 5: The start of our Service Excellence Team journey
The start of the vertical team journey also takes a week where there are ten pitstops on the road and one on the horizon. The start of the journey is the same, but on Tuesday we also learn about what the SETs expect from us (expectations are gathered before the workshop for each vertical separately). On Wednesday, we create a common understanding of what our services are, and the purpose of our community. On Thursday, we create a common understanding of our needs and the needs of our community. We also form our team and agree on our roles. On Friday, we finally agree on how we’ll work together as a vertical team, and we start preparing to build and drive a strong community.
Figure 6: The start of our vertical team journey.
Throughout the SET and vertical journeys, we gradually get to know each other better, both professionally and personally. To succeed with the transformation, it’s essential that we build togetherness, trust, and a feeling of psychological safety within each of our teams. There are different ways to achieve this during remote working conditions. For example:
- Each team member shares a story about who they are and what they care about using an object that represents them as a person.
- Creating a positive working environment by sharing appreciation among team members.
- Starting a team alliance through everybody sharing their expectations of the team and finding expectations that are common for the whole team.
We received questions about whether it was really necessary to use so much time to kick off the teams. For us, a DevOps transformation is a huge change, and change takes time. Therefore, to get the buy in we needed, we decided to show the value of the change being made through looking at why we’re carrying out this transformation.
After that, we wanted to create a solid foundation for all the teams to build on when starting their journey towards service excellence. Polls done after each day of workshops showed us how people’s minds shift from being puzzled or hopeful towards being excited. The workshops give attendees a positive boost for the journey ahead, and help them make changes step by step, and together as one team.
The journey so far
So, how has the journey been for those involved? What is good? What could be improved?
We carried out polls four weeks, and four months after the kick off to produce a total of 159 answers. In general, there are no dramatic changes in how people feel on the journey after the kick off.
Figure 7: How are you feeling now about the SET set-up?
From the verbal answers, we learned that customer focus had been a positive factor on this journey – team members now have a better understanding of how customers use our services. They also get up-to-date information about the customer feedback. This means that our teams are now putting the customer first, taking the first steps in the customer’s shoes.
Taking ownership of the services, having an end-to-end understanding of them, and having a better understanding of the requirements in the first place have also been good.
In addition, collaboration with operations is speeding up development, the quality of deployments has improved and incident solving is faster. To sum up, it seems we’re reaching many of the goals we set when we started this journey.
Figure 8: Attendees answer what is good about the service excellence journey?
As with any change, there’s a learning curve, and as mentioned, change requires time. But what could be improved is the availability of product management and the availability of the team members from operations. It would be important to get the error budget in use for all teams and to use the error budget to prioritize stability for new features. There are also challenges with large team size, many languages, cultures, and time zones, which we need to overcome.
Figure 9: What should be improved?
Besides creating polls to measure the temperature in the teams, we also have some performance indicators related to our transformation. We have agreed to follow up on service level in the following areas:
- Learning and training needs
- Service quality
- Customer defect resolution
- Error budget
We’ll also follow up on overall level of:
- Time to Market (TTM)
- Roadmap stability
- Theme-based, time limited improvements
The future journey
Service excellence is a journey, not a destination, so we’ll be continuing our journey for the foreseeable future. After all, we want to further shape our culture, to sharpen our performance and continue to exceed customer expectations.
“We know we are on the right path. Our journey is not finished, but we have come a long way.” This is what Muhammadu Buhari, President of Nigeria, said in 2018 on the 58th anniversary of Nigerian independence, but it rings true for us too!
Want to join our journey? Check out our open positions and join our team! Or could our journey inspire you to start your own DevOps transformation!
Learn more
Read Outi’s previous blog post, How to grow your Hackathon culture.
