How to survive unpatched vulnerabilities in containers?

Mobile networks, recognized as critical infrastructure, have always been designed according to strict security requirements put in place by competent authorities and regulators and examined by security experts. However, like any software system, even if security has been considered at all stages of the development lifecycle, a few security vulnerabilities might be impossible to discover before deployment. Thus, they might remain invisible for a long time and are only discovered later either after an incident involving these vulnerabilities or via bug bounty programs and crowdsourced cybersecurity campaigns among affected users.

The Cybersecurity and Infrastructure Security Agency (CISA) reported recently that adversaries are now able to exploit a vulnerability within 15 days (on average) of discovery. On the other hand, recent studies show that the average time between the discovery of a vulnerability and the release of its official patch sits at around several hundred days in recent years. The lack of means to mitigate an unpatched vulnerability may force businesses to temporarily shut down their services, which can lead to significant financial loss. Existing post-deployment security solutions, for example, filtering system calls made by a containerized application, can effectively reduce the general attack surface but cannot prevent a specific vulnerability from being exploited. They may also interfere with the normal operation of the software, leading to service disruption.

Modern mobile networks are softwarized and deployed on virtualized infrastructure to benefit from flexibility, performance, and cost reduction. They operate in an evolving and complex threat landscape where attackers are in a constant race to exploit newly discovered or unpatched vulnerabilities before defenders can react. Therefore, it is important to quickly identify unpatched vulnerabilities and efficiently countermeasure them even temporarily before official mitigation is made available.

In collaboration with researchers at Concordia University, we have explored the following research question: How to safeguard cloud-based applications from unknown vulnerabilities as well as known vulnerabilities for which no patch is currently available, while ensuring uninterrupted and timely service delivery?

This research effort was crowned by the development of a novel security countermeasure that prevents attackers from re-exploiting the same vulnerability with neither disruption to the service nor performance overhead costs. Our solution allows the service to operate normally while at the same time temporarily  releasing the pressure on the development of a more secure version. The merits of the solution were highlighted by the community of security experts from industry and academia who recently accepted the publication of our conference paper, Phoenix, in the Network and Distributed System Security Symposium (NDSS), one of the top four security conferences in the world.

Let’s expand a bit on the key idea behind this solution, and how it benefits cloudified services such as mobile networks.

Vulnerabilities in cloudified applications and the role of system calls

Cloudification technology that allows deploying mobile services as containerized applications becomes fundamental to delivering responsive and dynamic services. In cloud environments, containers enable the encapsulation of applications and their dependencies, allowing seamless deployment and scaling across different infrastructures.

System calls, acting as a bridge between containers and the underlying operating system kernel, are integral to the efficiency and functionality of these cloud-native applications. However, this functionality also introduces security considerations as containerized applications share the host kernel, which makes them susceptible to kernel-level vulnerabilities. Inadequate isolation or misconfigured system call permissions could lead to unauthorized access, data breaches, or privilege escalation. Thus, attackers exploiting applications vulnerabilities may abuse the system calls available to the applications to cause security breaches.

What are the challenges to countermeasure the vulnerabilities?

Security measures, such as restricting unnecessary system calls, implementing least privilege principles, and regularly updating container images, are essential to mitigate potential risks and improve the overall security posture of containerized environments. Nevertheless, the cost and technical challenges in establishing a precise mapping between application functionalities and system calls often result in either no system calls being blocked or only the most critical ones being blocked, provided that they do not disrupt the application’s functionality. Additionally, several vulnerabilities may involve benign system calls in their exploits. These calls are necessary for the application to function properly and therefore cannot simply be filtered.

Looking at existing countermeasures, Seccomp and ptrace are two methods that are commonly used.  Seccomp is not precise enough, and therefore incapable of blocking the vulnerability if they exploit harmless system calls. Ptrace is too fine-grained, resulting in a prohibitive delay to the protected container. 

How our solution mitigates unpatched vulnerabilities in Containers

In collaboration with skilled researchers from Concordia University, we have focused on  addressing the following challenge : How to ensure precise but efficient and effective security countermeasures to protect containers from unpatched vulnerabilities without disrupting the service?

Exploited vulnerabilities generally involve more than a single system call, more precisely, a sequence of ordered system calls. We thought about a solution that provides the best of both worlds: the speed of Seccomp to quickly pre-screen system calls at runtime and the precision of Ptrace in performing an in-depth inspection of arguments in specific system calls within a sequence. By combining those two in a smart way, we can ensure a solution that is both accurate and efficient.

Tackling the vulnerability at the system call level has several benefits. For instance, no change is needed to the containerized application and thus the solution can be put in place dynamically and in time. Furthermore, the solution can also be simultaneously applied to several containers that suffer from the same vulnerability.

Preventing exploits of unpatched vulnerabilities: How it works

Let’s take a closer look at the technical features behind the new solution.

Step 1: Malicious Sequence Identification

For each unpatched vulnerability, we first identify the malicious sequence of system calls behind the exploit. This can be performed using human-assisted root cause analysis over the system calls-based provenance graph. The latter can be built from the logs continuously collected in the containerized environment. The security expert, using this tool, can be assisted in quickly identifying the system calls names and their corresponding arguments in a sequence which needs to be inspected as well as the action (allow, warn, or block) to be performed upon a match is found in the right order. Another alternative is to accelerate system call sequence identification and vulnerability exploitation remediation by leveraging crowdsourcing to benefit from the collective expertise and resources of a diverse community including service providers and software vendors.

Step 2: System Calls Sequence State Monitoring

Once the malicious sequence is identified, our solution transparently monitors at the runtime the execution of every system call performed by the vulnerable container, with minimal overhead. It checks whether there is an ordered matching of the system calls with the malicious system call sequence. This is done using a novel mechanism that dynamically changes the Seccomp filters at runtime to follow at each iteration the next system call based on the previously matched system call. Upon matching each system call name in the sequence, arguments of the executing system call are inspected using Ptrace, for a second matching process to ensure a more-fine grained monitoring and thus a more accurate decision.

Step 3: Security Action Enforcement

At each matching of a system call in the sequence, the associated action is applied. The blocked system calls in the sequence will result in preventing the exploit from terminating successfully while preserving the normal execution of the vulnerable container.

Towards efficient protection of containerized services against unpatched vulnerabilities in 5G and future 6G

This research work aims to offer solutions to real-life concerns raised by service providers, operators, and network vendors regarding more effective approaches to managing vulnerability patching in deployed cloud-native services. For vendors, the proposed solution would open an out-of-the-box alternative to effectively and efficiently handle vulnerabilities discovered in services after their deployment while offering more time for developers to productize an officially patched service. For service providers and operators, this would mean reducing security risks faster and preventing monetary losses as they can still serve their customers with services kept up and running securely even with the presence of unpatched vulnerabilities.

As container-based services are foreseen to be in use for a while, it is expected that our solution can be a practical alternative to secure next generation services in 6G while allowing more time to security defenders against rapid pace of attackers.

This work has been carried out as part of the industrial research chair between Ericsson and Concordia University with funding from the Natural Sciences and Engineering Research Council of Canada (NSERC). Read more about it here.

Read more:

Read the research paper behind the work, published in the Network and Distributed System Security Symposium (NDSS) website.

Learn more about Ericsson’s other cyber security initiatives developed together with Concordia University.

Learn more about Ericsson’s future network security research.

Learn more about network function virtualization (NFV), and its role in improving 5G trustworthiness.