BLUF (Bottom Line Up Front)
Extended Berkeley Packet Filter (eBPF) has transformed Linux kernel security by enabling sandboxed, event-driven program execution directly inside kernel space without recompiling the kernel or loading risk-prone Loadable Kernel Modules (LKMs). In modern security architecture, eBPF powers high-throughput Endpoint Detection and Response (EDR) agents, container runtime security (e.g., Cilium, Tetragon, Falco), and real-time behavioral audit pipelines. However, offensive security teams and threat actors increasingly harness eBPF for stealth persistence, credential snooping, and process hiding. This masterclass provides a comprehensive end-to-end guide to eBPF security engineering. We construct a fully functional kernel execution monitoring agent in C and Go using cilium/ebpf and BTF CO-RE (Compile Once – Run Everywhere), demonstrate offensive eBPF rootkit mechanics, analyze Linux verifier failure modes, and establish concrete kernel hardening controls.
1. First-Principles Intuition: What is eBPF and Why Has It Revolutionized Kernel Security?
To understand eBPF, we must first examine how traditional Linux kernel monitoring functioned for decades and why it broke down under modern workload demands.
Historically, observing system events required one of three flawed approaches:
- User-space polling via
ptraceor/procscanning: Incredibly slow, high CPU overhead, and vulnerable to race conditions (Time-of-Check to Time-of-Use / TOCTOU). - User-space Audit subsystem (
auditd): High overhead under heavy I/O workloads; ring-buffer overflows lead to dropped events or severe performance degradation. - Loadable Kernel Modules (LKMs): High power and flexibility, but a single null-pointer dereference or out-of-bounds array access in an LKM crashes the entire system with a Kernel Panic (
kernel panic - not syncing).
+-------------------------------------------------------------------------+
| USER SPACE |
| +------------------+ +-------------------+ +------------------+ |
| | Security Agent | | Go Runtime App | | CLI Utilities | |
| +--------+---------+ +---------+---------+ +--------+---------+ |
| | | | |
+-----------|------------------------|-----------------------|------------+
| | eBPF Maps (Perf/Ring Buffer) | Syscalls |
+-----------|------------------------|-----------------------|------------+
| v v v |
| +-------------------------------------------------------------------+ |
| | LINUX KERNEL SPACE | |
| | +--------------------+ +------------------+ +---------------+ | |
| | | eBPF Verifier | | JIT Compiler | | Syscall Table| | |
| | +---------+----------+ +--------+---------+ +-------+-------+ | |
| | | | | | |
| | v v v | |
| | +-------------------------------------------------------------+ | |
| | | Kernel Probes (kprobe/tracepoint) / XDP / LSM Hooks | | |
| | | Running Verifier-Validated eBPF Bytecode at Native Speed | | |
| | +-------------------------------------------------------------+ | |
+-----------+-----------------------------------------------------------+-+
HARDWARE & CPU
eBPF solves this fundamental dilemma. It allows security engineers to execute custom bytecode inside the Linux kernel at native speed, triggered dynamically by kernel events (syscalls, network packet ingress/egress, function entry/exit), with a strict mathematical guarantee: an eBPF program cannot crash the kernel or corrupt arbitrary kernel memory.
This guarantee is enforced by the in-kernel eBPF Verifier, a static analysis engine that performs Directed Acyclic Graph (DAG) depth-first search verification on BPF bytecode before compilation into CPU instructions via the Just-In-Time (JIT) compiler.
2. Under-the-Hood eBPF Architecture & Execution Pipeline
An eBPF program follows a strict lifecycle from source code to kernel-level execution:
- Source Compilation: High-level C code is written using kernel headers (
vmlinux.h) and compiled into ELF bytecode (.o) targeting thebpfinstruction set viaclang -target bpf. - Bytecode Loading & Verification: Userspace applications (written in Go, C, or Rust) issue the
bpf()system call withBPF_PROG_LOAD. The kernel verifier inspects every instruction branch. - JIT Compilation: Upon passing verification, the in-kernel JIT compiler converts BPF bytecode directly into host CPU instructions (x86_64, ARM64).
- Hook Attachment: The program attaches to a kernel hook (
kprobe,kretprobe,tracepoint,raw_tracepoint,socket_filter,XDP, orLSM). - Map Inter-Process Communication (IPC): Data flows asynchronously between kernel space and user space through shared kernel memory structures called eBPF Maps (Array, Hash, Ring Buffer, Perf Event Array).
Kernel Hooking Targets Explored
- Kprobes (
kprobe/kretprobe): Dynamic entry/exit instrumentation for almost any arbitrary kernel function. Caveat: Internal kernel function signatures change across kernel versions. - Tracepoints (
tracepoint): Static, stable event hooks placed explicitly by Linux kernel developers (e.g.,tracepoint/syscalls/sys_enter_execve). Highly portable across kernel releases. - XDP (eXtress Data Path): Ultra-low-latency network packet filtering attached directly at the Network Interface Card (NIC) driver level before sk_buff memory allocation.
- LSM (Linux Security Module): Security policy enforcement hooks placed at critical kernel decision points (file opening, process creation, socket binding), capable of returning access denial codes (
-EPERM).
3. Comparative Matrix: eBPF vs. LKM vs. Auditd vs. ptrace
| Evaluation Feature | eBPF (Extended BPF) | Loadable Kernel Module (LKM) | Auditd Subsystem | Ptrace API |
|---|---|---|---|---|
| Kernel Safety | Guaranteed by Verifier (Zero Kernel Panics) | Unsafe (Null dereference crashes kernel) | Safe (Kernel built-in) | Safe (Userland system call API) |
| Execution Overhead | Minimal (< 1.5% CPU overhead under load) | Minimal (Direct native C execution) | High under peak event pressure | Extreme (Context switch per syscall) |
| Dynamic Loading | Yes (No reboot or kernel rebuild) | Yes (insmod / rmmod) | Yes (auditctl dynamic rules) | Yes (Attach to target PID) |
| Memory Isolation | Sandboxed BPF Maps & Ring Buffer | Unrestricted Kernel Memory Access | Kernel-to-User IPC Buffer | User-space process memory |
| System Modification | Read-heavy (Write restricted to specific LSM/helpers) | Arbitrary Kernel Patches & Hooking | Read-only Event Logging | Read/Write User-space Registers & Memory |
| CO-RE Portability | High (via BTF & vmlinux.h) | Low (Requires target build headers) | Native across distributions | Native across distributions |
| Primary Use Cases | EDR Telemetry, Container Security, XDP DDoS Mitigation | Proprietary Hardware Drivers | Regulatory Compliance Logging | Debugging, Tracing, Reverse Engineering |
4. Lab Setup & Build Toolchain (Arch / Kali Linux)
To compile and execute modern eBPF applications utilizing CO-RE (Compile Once – Run Everywhere), ensure your Linux system has BTF (BPF Type Format) enabled in kernel config (CONFIG_DEBUG_INFO_BTF=y).
Verify BTF support on your running system:
# Check if vmlinux BTF interface exists
ls -lh /sys/kernel/btf/vmlinux
Installing Required Dependencies
On Arch Linux:
sudo pacman -Syu --noconfirm clang llvm libbpf bpf go git pkg-config linux-headers
On Kali Linux / Debian / Ubuntu:
sudo apt-get update && sudo apt-get install -y \
clang \
llvm \
libbpf-dev \
linux-headers-$(uname -r) \
golang-go \
gcc \
git \
bpftool
Generate the single vmlinux.h header containing all kernel structure definitions directly from your running kernel:
bpftool btf dump file /sys/kernel/btf/vmlinux format c > vmlinux.h
5. Step-by-Step Practical Implementation: Building an eBPF Process Execution Monitor in C & Go
We will now build a production-grade process telemetry agent. The kernel portion captures every execve() system call, extracts the Process ID (PID), Parent Process ID (PPID), binary path, and user credentials, pushing the structured event to user space via a high-performance eBPF Ring Buffer.
5.1 The Kernel eBPF C Program (exec_monitor.bpf.c)
Create a file named exec_monitor.bpf.c:
// exec_monitor.bpf.c
// eBPF kernel-space process execution telemetry collector
#include "vmlinux.h"
#include <bpf/bpf_helpers.h>
#include <bpf/bpf_tracing.h>
#include <bpf/bpf_core_read.h>
#define MAX_FILENAME_LEN 256
// Event structure passed from Kernel to User space via Ring Buffer
struct process_event {
u32 pid;
u32 ppid;
u32 uid;
u32 gid;
char comm[16];
char filename[MAX_FILENAME_LEN];
};
// Define Ring Buffer Map
struct {
__uint(type, BPF_MAP_TYPE_RINGBUF);
__uint(max_entries, 256 * 1024); // 256 KB ring buffer
} exec_events SEC(".maps");
// License designation required by kernel verifier to access GPL-only helpers
char LICENSE[] SEC("license") = "GPL";
// Attach to sys_enter_execve tracepoint
SEC("tracepoint/syscalls/sys_enter_execve")
int trace_sys_enter_execve(struct trace_event_raw_sys_enter *ctx) {
u64 id = bpf_get_current_pid_tgid();
u32 pid = id >> 32;
// Reserve space in Ring Buffer
struct process_event *event;
event = bpf_ringbuf_reserve(&exec_events, sizeof(struct process_event), 0);
if (!event) {
return 0; // Buffer full, drop event safely
}
event->pid = pid;
// Read parent PID via CO-RE direct kernel structure traversal
struct task_struct *task = (struct task_struct *)bpf_get_current_task();
struct task_struct *parent_task;
BPF_CORE_READ_INTO(&parent_task, task, real_parent);
BPF_CORE_READ_INTO(&event->ppid, parent_task, tgid);
// Get current process UID and GID
u64 uid_gid = bpf_get_current_uid_gid();
event->uid = (u32)uid_gid;
event->gid = (u32)(uid_gid >> 32);
// Read executable process name
bpf_get_current_comm(&event->comm, sizeof(event->comm));
// Read path argument (filename) passed to sys_enter_execve
const char *filename_ptr = (const char *)ctx->args[0];
bpf_probe_read_user_str(&event->filename, sizeof(event->filename), filename_ptr);
// Submit event to user space
bpf_ringbuf_submit(event, 0);
return 0;
}
Line-by-Line Technical Annotations
SEC(".maps"): Declares an eBPF map section. UsingBPF_MAP_TYPE_RINGBUFprovides lockless, shared memory queues between kernel and user space, outperforming olderBPF_MAP_TYPE_PERF_EVENT_ARRAY.SEC("tracepoint/syscalls/sys_enter_execve"): Instructs the BPF loader to attach this program to the staticsys_enter_execvekernel tracepoint.bpf_ringbuf_reserve(): Allocates memory inside the shared ring buffer directly in kernel space. If memory allocation fails, it returnsNULL, requiring an immediate check to appease the verifier.BPF_CORE_READ_INTO(): CO-RE helper macro. It automatically computes structural field offsets across different Linux kernel versions, guaranteeing cross-kernel compatibility without recompilation.bpf_probe_read_user_str(): Safely copies user-space memory (the binary executable path string) into kernel memory without inducing page faults.
5.2 Compiling BPF Bytecode with Clang & CO-RE
Compile exec_monitor.bpf.c into BPF ELF bytecode using Clang:
clang -g -O2 -target bpf -D__TARGET_ARCH_x86 -I. -c exec_monitor.bpf.c -o exec_monitor.bpf.o
We can inspect the generated BPF assembly instructions and map relocations using llvm-objdump or bpftool:
bpftool prog dump xlated file exec_monitor.bpf.o
5.3 The Userspace Go Controller (main.go with cilium/ebpf)
Now we write the Go userspace application to load the compiled eBPF object, attach the probe to the Linux kernel, and stream events from the ring buffer in real time.
First, initialize a Go module and fetch cilium/ebpf:
go mod init ebpf-exec-monitor
go get github.com/cilium/ebpf
go get github.com/cilium/ebpf/link
go get github.com/cilium/ebpf/ringbuf
Create main.go:
// main.go - Userspace loader and ring buffer reader
package main
import (
"bytes"
"encoding/binary"
"errors"
"fmt"
"log"
"os"
"os/signal"
"syscall"
"github.com/cilium/ebpf"
"github.com/cilium/ebpf/link"
"github.com/cilium/ebpf/ringbuf"
"github.com/cilium/ebpf/rlimit"
)
// Matches C struct process_event alignment exactly
type ProcessEvent struct {
PID uint32
PPID uint32
UID uint32
GID uint32
Comm [16]byte
Filename [256]byte
}
func main() {
// Remove memory locking limits for eBPF maps (legacy rlimit kernel requirements)
if err := rlimit.RemoveMemlock(); err != nil {
log.Fatalf("Failed to remove memlock limit: %v", err)
}
// Load pre-compiled eBPF ELF object
spec, err := ebpf.LoadCollectionSpec("exec_monitor.bpf.o")
if err != nil {
log.Fatalf("Failed to load collection spec: %v", err)
}
coll, err := ebpf.NewCollection(spec)
if err != nil {
log.Fatalf("Failed to create eBPF collection (verifier rejected program?): %v", err)
}
defer coll.Close()
// Retrieve program and map references
prog := coll.Programs["trace_sys_enter_execve"]
if prog == nil {
log.Fatalf("Program trace_sys_enter_execve not found in collection")
}
eventsMap := coll.Maps["exec_events"]
if eventsMap == nil {
log.Fatalf("Map exec_events not found in collection")
}
// Attach program to sys_enter_execve tracepoint
tp, err := link.Tracepoint("syscalls", "sys_enter_execve", prog, nil)
if err != nil {
log.Fatalf("Failed to attach tracepoint: %v", err)
}
defer tp.Close()
// Open Ring Buffer reader
rd, err := ringbuf.NewReader(eventsMap)
if err != nil {
log.Fatalf("Failed to open ringbuf reader: %v", err)
}
defer rd.Close()
log.Println("[+] eBPF Execution Monitor Active. Catching sys_execve calls... (Press Ctrl+C to exit)")
fmt.Printf("%-8s %-8s %-6s %-16s %-35s\n", "PID", "PPID", "UID", "COMM", "EXECUTABLE PATH")
fmt.Println("----------------------------------------------------------------------------------")
// Catch termination signals gracefully
sigChan := make(chan os.Signal, 1)
signal.Notify(sigChan, syscall.SIGINT, syscall.SIGTERM)
go func() {
var event ProcessEvent
for {
record, err := rd.Read()
if err != nil {
if errors.Is(err, ringbuf.ErrClosed) {
return
}
log.Printf("Error reading ringbuffer: %v", err)
continue
}
// Parse binary C struct into Go struct
err = binary.Read(bytes.NewBuffer(record.RawSample), binary.LittleEndian, &event)
if err != nil {
log.Printf("Failed to parse event sample: %v", err)
continue
}
commStr := string(bytes.Trim(event.Comm[:], "\x00"))
filenameStr := string(bytes.Trim(event.Filename[:], "\x00"))
fmt.Printf("%-8d %-8d %-6d %-16s %-35s\n",
event.PID, event.PPID, event.UID, commStr, filenameStr)
}
}()
<-sigChan
log.Println("\n[-] Detaching eBPF probes and shutting down cleanly.")
}
Build the Go controller:
go build -o ebpf_monitor main.go
5.4 Empirical Execution Telemetry & Raw Kernel Event Output
Run the compiled monitor binary as root on Linux:
sudo ./ebpf_monitor
Terminal Output Telemetry Captured on Linux 7.0 / Arch Linux:
[+] eBPF Execution Monitor Active. Catching sys_execve calls... (Press Ctrl+C to exit)
PID PPID UID COMM EXECUTABLE PATH
----------------------------------------------------------------------------------
42194 1420 1000 bash /usr/bin/ls
42195 1420 1000 ls /usr/bin/cat
42198 42197 0 sudo /usr/bin/id
42201 1420 1000 curl /usr/bin/curl
42205 812 0 systemd-journal /usr/lib/systemd/systemd-executor
42210 1420 1000 python3 /usr/bin/python3
Every execution across the entire OS is captured instantly at the kernel layer, before the process even executes its initial instruction in user space!
6. Offensive eBPF Tradecraft: Kernel-Level Persistence & Hook Hijacking
While defenders use eBPF for deep observability, offensive researchers and malware developers utilize eBPF to bypass traditional security controls. Because eBPF runs inside the kernel, eBPF-based implants can inspect, tamper with, or drop security events before user-space security agents ever receive them.
+-------------------------------------------------------------------------+
| OFFENSIVE eBPF HOOK HIJACKING MECHANISM |
| |
| 1. Attacker loads eBPF Program attached to sys_enter_getdents64 |
| 2. User issues 'ls /tmp' or Security Agent scans process table |
| 3. Kernel returns directory entry array to eBPF Hook |
| 4. eBPF Hook rewrites buffer memory, stripping target filename |
| 5. User-space application receives truncated listing (File Hidden) |
+-------------------------------------------------------------------------+
6.1 eBPF Rootkits: Overriding Syscall Returns (bpf_override_return)
With the kernel helper bpf_override_return(), an eBPF program attached to a kprobe can prevent a system call from executing entirely, forcing the kernel to return an arbitrary error code (e.g., -EACCES or -ENOENT).
Attacker Scenario: Disabling security software binaries (e.g., blocking /usr/bin/edr_agent from executing):
// Offensive eBPF snippet: Syscall Interception and Block
SEC("kprobe/__x64_sys_execve")
int BPF_KPROBE(override_execve, struct pt_regs *regs) {
char bin_name[64];
bpf_get_current_comm(&bin_name, sizeof(bin_name));
// Target security agent process name
char target[] = "edr_agent";
if (bpf_strncmp(bin_name, sizeof(target), target) == 0) {
// Block execution by overriding return with Access Denied (-EACCES)
bpf_override_return(regs, -EACCES);
}
return 0;
}
Note: bpf_override_return is strictly restricted by kernel developers to functions marked with ALLOW_ERROR_INJECTION(), requiring root privileges (CAP_BPF / CAP_SYS_ADMIN).
6.2 Process & File Hiding via getdents64 Hooking
Malicious eBPF implants (such as TripleCross or BPFDoor) hook sys_exit_getdents64 (the system call responsible for listing files in directories and PIDs in /proc).
By modifying the returned linux_dirent64 buffer in place using bpf_probe_write_user(), the eBPF program overwrites the length of specific directory entries, effectively making target files, sockets, or running PIDs completely invisible to ls, ps, top, and netstat.
7. Real-World Failure Modes, Verifier Pitfalls & Troubleshooting
The Linux kernel verifier is infamous for rejecting complex BPF code. Below are the two most common verifier errors encountered during eBPF development and how to resolve them.
Verifier Rejection #1: Unbounded Loops & Stack Limit Exceeded (512 Bytes)
The Failure: eBPF programs have a strict maximum stack size of 512 bytes. Allocating large buffers or structures on the stack triggers verifier rejection.
# Kernel Verifier Log Error:
looks like unbounded loop, or program is too large
combined stack size of 2 calls is 544. Stack limit is 512 bytes.
The Fix: Never allocate large structs directly on the stack. Use eBPF Per-CPU Array Maps or reserve space directly on the Ring Buffer (bpf_ringbuf_reserve). For loops, always use #pragma unroll or the bpf_loop() helper available in kernel 5.17+.
// BAD (Stack Overflow):
struct process_event event; // 288 bytes on stack - dangerous!
// GOOD (Ring Buffer Reserve - 0 Stack Overhead):
struct process_event *event = bpf_ringbuf_reserve(&exec_events, sizeof(*event), 0);
Verifier Rejection #2: Null Pointer Dereference Checks
The Failure: Accessing memory returned by a helper function without an explicit NULL check causes immediate rejection.
# Kernel Verifier Log Error:
R1 invalid mem access 'Nullable'
The Fix: Always guard pointers returned from maps or ring buffer allocations:
struct process_event *event = bpf_ringbuf_reserve(&exec_events, sizeof(*event), 0);
if (!event) {
return 0; // Mandated check for the verifier
}
8. Defensive Hardening: Securing eBPF Subsystems Against Abuse
Because eBPF is a double-edged sword, security administrators must enforce strict kernel policy boundaries to prevent unprivileged users or compromised applications from abusing BPF capabilities.
8.1 System Hardening via sysctl
-
Disable Unprivileged eBPF Execution: Unprivileged eBPF allows non-root users to load BPF programs, opening vectors for side-channel attacks (Spectre) and kernel exploitation.
Bash sudo sysctl -w kernel.unprivileged_bpf_disabled=1 echo "kernel.unprivileged_bpf_disabled=1" | sudo tee -a /etc/sysctl.d/99-ebpf-security.conf -
Enable JIT Hardening & Blind Constants: Prevents attackers from embedding executable shellcode inside eBPF immediate constants to bypass kernel memory mitigations (JIT Spraying).
Bash sudo sysctl -w net.core.bpf_jit_harden=2 echo "net.core.bpf_jit_harden=2" | sudo tee -a /etc/sysctl.d/99-ebpf-security.conf -
Restrict Access to Kernel Pointer Addresses:
Bash sudo sysctl -w kernel.kptr_restrict=2
8.2 Securing eBPF with BPF LSM Policies
Modern Linux kernels (5.7+) support BPF LSM, allowing security teams to write eBPF programs that restrict which processes can call bpf() system calls.
Example BPF LSM rule logic:
- Block any
bpf()syscall unless the caller's binary hash matches the authorized EDR binary/usr/bin/sentinel_agent. - Restrict
BPF_PROG_TYPE_KPROBEloading to processes holding explicit crypto signatures.
9. Summary & Key Takeaways for Security Engineers
- Unrivaled Telemetry: eBPF offers unprecedented observability into Linux processes, containers, and network sockets without performance degradation or kernel stability risks.
- CO-RE is Essential: Always build eBPF apps using BTF CO-RE and
vmlinux.hto ensure bytecode runs seamlessly across disparate Linux distributions without requiring local kernel headers. - Offensive Vigilance: Defenders must actively monitor eBPF program loading (
bpftool prog list) and enforcekernel.unprivileged_bpf_disabled=1to mitigate eBPF rootkit risks. - Ring Buffers Over Perf Arrays: Modern eBPF applications should prefer
BPF_MAP_TYPE_RINGBUFfor lockless, memory-efficient kernel-to-userspace event streaming.