Understanding Kubernetes networking, Cilium and eBPF

Understanding Kubernetes networking is not trivial. There are many layers that comprise it, and comprehending how it interacts with the underlying OS networking stack is key for feeling comfortable with it. In addition to these layers, the intricacies of service discovery, load balancing, and network policies further complicate the networking model within Kubernetes. As users deploy more applications and services, the need to grasp these concepts becomes even more critical, as they directly influence performance and security. Moreover, as cloud-native environments evolve, staying current with best practices in Kubernetes networking will ensure that developers and system administrators can build robust, scalable architectures capable of handling dynamic workloads effectively.

Components

Before diving into the meat of this blogpost, I believe it is important to cover the multiple components that build the full Kubernetes network stack. Understanding these various elements, such as the network interfaces, CNI plugins, and service discovery mechanisms, and how they complement one another is paramount to understanding why we use certain configuration options. Each component plays a crucial role in ensuring seamless communication between the services deployed in a Kubernetes cluster, optimizing both performance and security. By grasping the intricacies of the network stack, we can better appreciate the flexibility and scalability that Kubernetes offers, which ultimately leads to more effective management of containerized applications.

Container Network Interface

The CNI providers an interface for developers to implement and build network plugins that follow the Kubernetes network model. This, in summary, guarantees that Pods get automatically assigned their own unique IP address that is routable within the cluster, creating a clean backwards-compatible model where Pods can be treated much like VMs or physical hosts from the perspectives of port allocation, naming, service discovery, load balancing, application configuration, and migration.

Furthermore, the Kubernetes network model also imposes two additional requirements:

Pods can communicate with all other pods on any other node without NAT
Agents on a node (e.g. system daemons, kubelet) can communicate with all pods on that node

It’s usually deployed as a privileged DaemonSet (its pods get scheduled on every node in the cluster). The Container Runtime running on each node is then responsible for calling the CNI plugin whenever a new container is created, delegating the creation of the container network interface to it, in a manner that implements the aforementioned responsibilities.

Source: https://thenewstack.io/container-networking-landscape-cni-coreos-cnm-docker/

For an example of how containerd manages the CNI plugins: https://github.com/containerd/containerd/blob/main/script/setup/install-cni

This is a very brief description of the CNI and how it integrates with Kubernetes (through the Container Runtime). If you want to learn about it in great detail, you can refer to the official spec document found in: https://github.com/containernetworking/cni/blob/main/SPEC.md

Kube-proxy

Additionally to the CNI, the kube-proxy is another network component inherently present in most Kubernetes offerings. Similarly to it, it also gets deployed as a DaemonSet. It is responsible for setting up network rules in each host in order to correctly route service traffic. It works by watching new service creation events and then updating the respective forwarding rules on each host, so that every worker node in the cluster knows how to route service traffic to the corresponding backends, effectively abstracting this behavior from pods and making the forwarding seamless.

There are various ways in which kube-proxy can be deployed in, but the one most typically seen in Kubernetes offerings is through iptables. In this mode, kube-proxy creates NAT rules for incoming traffic and makes it so they get routed to one of the pods that should serve it. It supports various algorithms for load balancing traffic, like round-robin.

Source: https://www.learnsteps.com/how-exactly-kube-proxy-works-basics-on-kubernetes/

It is important to understand that kube-proxy complements the CNI plugins as they each have different responsibilities; whereas CNI plugins are responsible for managing the network stack for Pods, namely for things like IP addressing, kube-proxy is responsible for guaranteeing that the routing rules for services are present on every node.

Cilium

Cilium has in the recent past become the most popular CNI plugin in the Kubernetes space. It is an open-source project with a very active community and release cycle, and provides many configuration options to set the networking stack for a cluster. The biggest novelty introduced by Cilium and what made it so popular was the fact that it used eBPF for routing instead of relying on the traditional routing mechanisms used by other CNIs, like iptables or IPVS.

eBPF

Before further diving into Cilium and its features in more detail, it is also important to first understand what eBPF is so that we can understand how Cilium benefits from it.

eBPF stands for ‚Äúextended Berkeley Packet Filter‚Äù. This is a powerful technology that is built into the Linux kernel starting from version 3.18 (although with several updates since). It allows for users to safely and efficiently extend the capabilities of the kernel by running sandboxed programs in the kernel without needing to modify/recompile it or having to manage kernel modules.

Since the kernel space is privileged in the sense that it is the only component in the whole OS that has full visibility of every underlying component (networking, storage, etc), it is the perfect place to perform several operations like monitoring, tracing, etc. However, since conceptually the evolution of an OS is a complicated and slow process, it is cumbersome to implement programs that take advantage of these capabilities natively. The power of eBPF comes with decoupling this from the OS’ lifecycle by allowing programs to be built to run in the kernel independently of how the kernel itself evolves.

Source: https://ebpf.io/what-is-ebpf/

In sum, it provides:

The ability to run programs within the kernel, reducing overhead and improving its efficiency
Safety and security, as the kernel is responsible for pre-validating the programs before they are executed, which guarantees protection against kernel crashes or memory leaks (effectively, sandboxing)
Flexibility as the programs can be dynamically loaded and unloaded at runtime, meaning it doesn‚Äôt require any reboots or image upgrades to run certain programs
A wide range of applications as programs run in kernel space, including networking, observability, security, performance analysis, etc.

eBPF programs are event-driven and are run when the kernel or an application passes a certain hook point. These hooks can be defined in multiple ways, like system calls, function entry/exit, kernel tracepoints, network events, among others.

Source: https://ebpf.io/what-is-ebpf/

To read in much more detail how eBPF programs work or how to write one, please refer to: What is eBPF? An Introduction and Deep Dive into the eBPF Technology

How does Cilium use eBPF

Cilium’s great selling point was integrating the power of eBPF into the space of CNI plugins. Through it, it can improve the overall networking stack for pods through augmented security and observability. Although for small scale clusters the difference in performance is perhaps negligible, it becomes increasingly more noticeable as the number of pods (and therefore, targets to route traffic to) increases, which in consequence also increases the overall number of active connections across the cluster (benchmark). Using eBPF then has two immediate Performance benefits:

It reduces the overhead, as traditional routing using iptables or IPVS operates in user space, therefore requiring a lot of context-switching between it and kernel space through syscalls
It implements a more sophisticated and performant load balancing as it runs directly in the kernel, which can reduce latency and improves throughput

For more advanced users that want to take full advantage of the capabilities of eBPF, there are two more Advanced Networking benefits:

It allows for direct packet manipulation; Cilium is able to inspect each packet that flows through the host, modify it if needed, redirect it according to some designation, or just drop it altogether

There are a few features that take advantage of this last point, which includes the ability to set up a service mesh or encrypting traffic between pods using mTLS.

It allows to configure granular policy enforcement at L3/L4 (network/transport) and L7 (application)

This gives a lot of flexibility to write network policies, and through eBPF they can be enforced dynamically and efficiently.

In terms of Security, it also provides some benefits:

It uses identity-based security instead of just relying on IP addresses, which are ephemeral and subject to change in a dynamic environment like Kubernetes

In Cilium, an Identity is determined by a set of labels that comprise a Pod. This unique combination is used to unequivocally identify a Pod in the cluster. This allows to enforce security rules based on Identities rather than on the IPs that might get assigned to Pods, making the mapping more efficient. To learn more: Identity-Based — Cilium 1.19.1 documentation

It allows for policies to be dynamically adjusted in real-time without any restarts

If any suspicious activity is detected, a user can dynamically add new enforcement rules through a network policy. This is then picked up by Cilium and dynamically enforced clusterwide without any further intervention being necessary.

Lastly, and as was discussed in the previous section, eBPF also provides capabilities in terms of Observability and Monitoring, which Cilium can also take advantage of through:

Real-time data collection, providing in-detail telemetry about network traffic, specifically for things like latency, throughput, error rates, etc, pulled directly from the kernel
Providing visibility for higher-layer protocols like HTTP and gRPC as packets traverse through the network, which can help debug application errors or improve performance

Complementing Cilium is a tool called Hubble, which is a security and observability platform. It allows to capture many of these traffic-related memories that are captured from the kernel by the eBPF program, making it a very useful tool for debugging, performance improvement and security. You can learn more about Hubble in: Introduction to Cilium & Hubble — Cilium 1.19.0 documentation

Source: https://cilium.io/blog/2020/11/10/ebpf-future-of-networking/

Kube-proxy replacement

Additionally to how Cilium benefits from eBPF as a CNI plugin for managing a Pods network stack, it can also be further empowered and configured as a full replacement for kube-proxy. This means that effectively Cilium manages all the networking within a Kubernetes cluster, including service routing. This benefits from all the advantages listed before in terms of performance and security, and also reduces the complexity by having one less component to manage. The kube-proxy replacement mode for Cilium works conceptually the same as with was previously described in this document, but instead being managed by Cilium.

This is an optional feature you can enable and it works independently from eBPF. In this last case, it works exactly like the standard kube-proxy addon with Cilium managing iptables rules. For more details regarding the kube-proxy replacement in Cilium: Kubernetes Without kube-proxy — Cilium 1.20.0-dev documentation

Conclusion

This is just a small dive into the shallow part of the huge and deep ocean that is Kubernetes networking. We covered a few of the components that comprise it, and we looked into one of the specific controllers, Cilium, that manages networking for a Kubernetes cluster. Cilium, leveraging eBPF technology, enhances the way we handle networking by enabling fine-grained control and observability, making it a powerful tool for developers and operators alike. By integrating seamlessly with Kubernetes, it allows for more efficient communication between microservices, thus improving overall system performance and security. I hope that this helps you understand better how everything comes together and gives you an incentive to further learn about the incredible world of networking. Exploring these technologies not only enhances your skill set but also prepares you to tackle complex challenges in cloud-native environments.