eBPF_primer

My Journey

This section is about my background. Feel free to skip to the next one to read about eBPF.

I started off my career as a network engineer and soon after, got into security. My fascination with optimizing networks and troubleshooting them paved the way to not be afraid of capturing packets using Wireshark or TCPDUMP and diving into them to fish out issues.

Studying and working with all these concepts, protocols and the ability to influence and modify network traffic felt amazing and empowering but at the back of my head, I was always wondering, how are these packets made? How does encapsulation work? How do network devices know how to handle and interpret these packets?

As I gained some courage to learn a bit of programming, I came across Scapy and through that, I discovered the joy of crafting and manipulating packets before sending them off to the wire.

This was all great, yet high-level and language dependent. I wanted more control and perhaps something generic that could also act as a gateway to systems programming.

After seeing articles and reading about the new technical achievements such as XDP, DPDK, as well as eBPF and how the industry is utilizing them to truly influence the future of networking and address conventional difficulties, I decided to finally make some time to study a bit more about them.

Luckily, I came across a series of videos by Mark Pashmfouroush teaching eBPF through a project called flowlat using C and Go programming languages to measure the latency of TCP packets. This was really interesting to me because of my own tcping project which is built around the same idea.

I reached out to Mark and asked for permission to use his project and videos to write a few blog posts and he graciously agreed. So, a big thank you and shoutout to you Mark for sharing your knowledge, making this journey much easier.

Lastly, the Go’s eBPF library made by Cilium and some Go specificities have changed since Mark’s videos and for that, I will upload the new and updated code to my own GitHub account under the name flat while adding some new capabilities to it.

What is eBPF?

eBPF is a low-level technology originating from Linux that enables us to write programs that run in a sandbox environment (virtual machine) inside the kernel space in a safe and controlled manner, allowing us to modify and extend the kernel without having to alter and re-compiling the kernel’s source code.

Windows also supports eBPF. Read more here.

eBPF programs are event-driven and are only run, when a specific event occurs; For instance, when a packet arrives at your NIC or when an application hits a certain hook point.

These pre-defined hook points include:

  • Network events
  • System calls, aka syscall
  • Kernel tracepoints
  • Function entry or exit and a few others

It is also possible to define our own probes; user (uprobe) as well as kernel (kprobe) probes or hooks and attach them to user space applications or almost anywhere in the kernel.

eBPF is an extension of the classic BPF (cBPF), which has been present in the Linux kernel for a long time. Initially, eBPF stood for Extended Berkeley Packet Filter, reflecting its original purpose of filtering network packets. However, eBPF has since evolved to support a wider range of use cases beyond networking. As a result, the acronym no longer stands for anything. Notably, TCPDUMP is a well-known tool that relies on the classic BPF runtime.

Simply put, eBPF really shines in these areas:

  • Networking
  • Security
  • Observability
  • Tracing and profiling

While it is possible to request new kernel features, the process involves rigorous scrutiny and extensive discussions that can take years to complete. As a result, extending the kernel ourselves using eBPF can provide a more efficient solution in most cases.

eBPF programs are incredibly efficient and have a very tiny footprint because unlike other programs, we are not constantly sending data between user and kernel space programs.

Probes

You will come across the term probe a lot in eBPF context. probe is a location in the kernel where an eBPF program is inserted to gather information or change a behavior, say dropping a packet.

There are several types of probes that eBPF supports:

  • Kprobes: Kprobes (Kernel Probes) are a Linux kernel feature that allows you to break into any kernel routine and collect debugging and performance information non-disruptively. You can insert an eBPF program at the entry or exit point of any kernel function using this type of probe.

  • Uprobes: Uprobes (User Probes) are similar to Kprobes but are applied to user space applications. They allow you to insert eBPF programs at any function entry or exit point in a user space application.

  • Tracepoints: Tracepoints are static probe points that exist in predefined locations within the kernel. They are considered safer than Kprobes because they can’t crash the kernel if something goes wrong. An eBPF program attached to a tracepoint will be triggered whenever the tracepoint event occurs.

  • Perf Events: Perf Events are a Linux kernel subsystem that provides a framework for performance analysis. You can attach eBPF programs to perf events to analyze various types of software and hardware events.

Anatomy of eBPF Programs: From Compilation to Execution

eBPF programs are comprised of two main parts:

  • The kernel space program: Usually written in pseudo C or Rust. This program serves as the backend of the eBPF program.
  • The user space program: Typically written in a higher level language like Go or Python. Although it can also be written in Rust or C. This program serves as the frontend and interacts with the kernel space program to deliver the desired functionality.

The user and kernel space programs can communicate with each other using eBPF maps. More on that later.

Using a compiler the eBPF program gets converted to a generic and portable eBPF bytecode which is very similar to x86 instructions. This bytecode is then loaded into the kernel using the bpf() syscall. The kernel takes the program and using the verifier, analyzes it to ensure our program is safe and cannot harm the kernel. For instance:

  • No infinite loops
  • The program is guaranteed to run to completion
  • Or that we cannot read and access memory that is not allocated to us.

If the program is deemed unsafe, the kernel will reject it. Otherwise our generic bytecode goes through the Just In Time (JIT) compiler. This process translates our bytecode into a specific bytecode that is understood by the machine’s CPU, such as ARM, x86, etc.

After compilation, the specific bytecode is loaded into the kernel and attached to a hook point. The hook point is typically an event, such as receiving a packet, which triggers the program to run.

This process is extremely efficient, very similar to re-compiling your kernel! Good stuff.

In short:

  1. The program is compiled into eBPF bytecode
  2. Loaded into the kernel using the bpf() syscall
  3. Analyzed for safety using the Verifier
  4. Translated into machine-specific bytecode through JIT
  5. Attached to a hook point and executed when an event occurs. e.g. on packet transmit or receive

Compilers

There are two types of compilers:

  1. Generic compilers, supported by GCC and LLVM that can take any C code and compile it to eBPF bytecode
  2. Use case specific compilers like BPF Compiler Collection (BCC) or bpftrace that are more intent-based and generate an eBPF program based on what we intend to do.

Think of these use case specific compilers as an abstraction layer in which you define what you need; for instance, observe system’s resource utilization and without needing to write an eBPF directly, these compilers provide the functionality.

In this series, I will be leveraging the LLVM tool-chain to compile our program. However, I might also show BCC and bpftrace in some other posts.

Portability and Maintainability

eBPF programs are completely portable and can be executed on any kernel version (given that it supports eBPF). It is because the program cannot call into arbitrary kernel functions, it has to use stable APIs which are called eBPF helpers that are maintained across all kernel versions.

This is great since we do not need to maintain multiple versions of the same programs for different kernel versions and architectures.

Upcoming posts

In the upcoming posts, I will demonstrate the steps required to write and utilize eBPF programs. Stay tuned.

References

Visit these links to learn more about eBPF: