The Linux isolation stack in fourteen working demos

Isolation in Linux is not one technology. It spans five layers (hardware, hypervisor, kernel primitives, kernel policy, and runtime), each addressing a different threat model, none sufficient on its own. Most teams deploying containers understand mechanisms from three or four of those layers. This project picks fourteen mechanisms across all five and runs them: one working demo each, on a stock Linux machine, no Kubernetes or cloud account required.

The impetus was a piece on why a microVM is not a sandbox: a Firecracker guest is strong on process and filesystem isolation and leaves the network dimension open by default. Writing that piece made the gap in available working examples clear. Most isolation explainers describe the abstractions. Almost none run them. These fourteen demos run, and each one's output explains what is happening as it happens.

The stack

The demos are independent. Each directory has a setup.sh that downloads its own dependencies and a run.sh that runs the demo with narrated output. The only system-wide requirements are Linux 5.15+ and Zig 0.13+ in $PATH. KVM is needed only for Firecracker. Everything else (user namespace, Landlock, seccomp, capabilities, cgroups, gVisor, and the TEE demo's simulation mode) runs without root.

What the fourteen actually are

Not all fourteen are kernel primitives. Calling them that flattens a distinction the stack diagram is trying to surface. By actual category:

Three internal relationships are worth naming. The time namespace is a namespace variant (CLONE_NEWTIME sits alongside CLONE_NEWUSER and CLONE_NEWPID in the same unshare(2) interface), but its use case (correcting monotonic time after restoration) is specific enough to warrant a dedicated demo. The network namespace is also a namespace variant; its demo focuses on the veth wiring mechanism that connects namespaces together rather than the isolation property itself. And seccomp BPF and seccomp user-notify are two modes of the same seccomp(2) syscall: BPF attaches a filter that enforces inline; user-notify freezes the syscall and routes to a supervisor. They are split into separate demos because the control model inverts: one blocks, the other consults.

The fourteen represent a cross-section of the stack, not a complete enumeration. The count is the number of runnable demos. It is not a claim about how many isolation mechanisms exist in Linux.

Namespaces: the linchpin

Almost every isolation primitive requires elevated privilege. The user namespace is the exception. It lets an unprivileged process map its own UID to 0 inside the namespace, gaining a full capability set scoped to that namespace. Every other namespace type then becomes available without root. This is the entire foundation of rootless containers.

// unshare(NEWUSER) → write uid/gid map → have CAP_SYS_ADMIN inside the ns
if (sc1(SYS_unshare, CLONE_NEWUSER) != 0) return error.UnshareUser;
try writeProc("/proc/self/uid_map", "0 1000 1\n");
try writeProc("/proc/self/gid_map", "0 1000 1\n");

// now unshare everything else, capabilities from user ns cover these
const other = CLONE_NEWPID | CLONE_NEWNS | CLONE_NEWNET | CLONE_NEWUTS | CLONE_NEWIPC;
if (sc1(SYS_unshare, other) != 0) return error.UnshareOther;

// fork: child is PID 1 in its own PID namespace
const child = try std.posix.fork();

After the fork, the child process is PID 1 in its own PID namespace, UID 0 in its own user namespace mapping to the real non-root UID on the host, has a fresh network stack with no interfaces, and sees only its own /proc. The host process count, network topology, and hostname are invisible to it. The demo verifies all of this live: pid=1, uid=0, hostname changed to container-demo, /proc showing only the namespaced process tree.

The UID mapping carries an important subtlety. UID 0 inside the namespace maps to a real non-root UID on the host. Code that checks for UID 0 sees root. The kernel's permission enforcement uses the host UID. A container that runs as apparent-root inside but is mapped to uid 1000 outside cannot escalate beyond what uid 1000 can do on the host, regardless of what it does inside.

cgroups: resource containment, not isolation

cgroups v2 limits what processes can consume: memory, CPU quota, process count. The demo configures pids.max=12 and verifies that the thirteenth fork() returns EAGAIN from the kernel. The v2 unified hierarchy exposes all resource axes under a single /sys/fs/cgroup/<scope>/ path.

The critical distinction: cgroups limit resource consumption, not privilege. A process inside a cgroup can still read any file it has permission to, make any syscall not filtered elsewhere, and reach any network endpoint it can route to. Combine cgroups with namespaces and seccomp to get actual isolation; use cgroups alone and you have a governor, not a jail. This distinction matters when security teams claim that a service is "containerised" when it only has resource limits applied.

Capabilities: subdividing root

Linux root privilege decomposes into roughly 40 independent capability bits. The capabilities demo makes this concrete by verifying that creating a raw socket requires exactly CAP_NET_RAW and fails immediately when that bit is cleared:

// drop only CAP_NET_RAW (bit 13), keep everything else
const no_net_raw: u32 = 0xFFFFFFFF & ~(@as(u32, 1) << 13);
caps[0] = .{ .effective = no_net_raw, .permitted = no_net_raw, .inheritable = 0 };
if (!capSet(&caps)) return error.CapSetFailed;

// AF_INET=2, SOCK_RAW=3, IPPROTO_ICMP=1 → EPERM
const rc: isize = @bitCast(sc3(SYS_socket, 2, 3, 1));
// rc is -1, errno = EPERM

Docker and Podman drop around 30 of the 40+ capabilities by default. An attacker who exploits a container process and gains code execution starts with roughly ten effective capability bits: enough to do useful work inside the container, not enough to reconfigure the network, mount filesystems, ptrace other processes, or load kernel modules.

Clearing the full set (capset(effective=0, permitted=0)) is irrevocable. Once permitted is zero, no code execution path restores it. This is the correct posture for agent processes that need no special kernel access, which is most of them. The bounding set provides an additional hard ceiling: dropped capabilities cannot be re-acquired by acquiring file capabilities or executing a setuid binary.

Landlock + seccomp: the kernel policy layer

seccomp-bpf filters syscall numbers before the kernel handles them. Landlock (kernel 5.13+) whitelists which filesystem paths the process can reach, regardless of file permissions or UID. The two layers are orthogonal: seccomp restricts what the process can ask the kernel to do; Landlock restricts where in the filesystem those requests are permitted. Used together, they are the production sandbox profile deployed in Chromium, Firefox, and most container runtimes.

The demos implement a fork → sandbox.apply() → execve pattern that matters particularly for agent workloads: the sandbox is applied before execve, so the executed program is born inside the policy. The policy survives the exec boundary:

// runner.zig: apply sandbox, then exec the agent binary.
// If sandbox fails to apply, refuse to exec, fail closed.
sandbox.apply(.strict_worker, &safe_paths) catch |err| {
    std.process.exit(126);
};
// The execve'd program starts inside the sandbox.
// No moment between launch and confinement exists.
_ = std.os.linux.syscall3(@enumFromInt(SYS_execve), probe_z, argv, envp);

Landlock's restrict_self() call is irrevocable. After it returns, no code path (including code injected via ptrace or shared library) can open paths outside the whitelist. The policy is attached to the process's credentials, not its code. The blocked syscall list in the strict profile includes ptrace, process_vm_writev, mount, unshare, kexec_load, and bpf, the last entry preventing the filter itself from being modified at runtime.

Four demo modes run in sequence to show the layers independently and combined:

seccomp user-notification: freeze, inspect, respond

SECCOMP_RET_USER_NOTIF (kernel 5.0) is seccomp's most underappreciated feature. Instead of allowing or killing a matched syscall, the kernel freezes the calling thread and sends a notification to a file descriptor held by a supervisor process. The supervisor reads the syscall arguments, applies policy, and injects a response. The worker thread sees the outcome as if the kernel handled it normally: either the syscall proceeds, or it gets back EPERM.

// supervisor loop: block until next frozen syscall
var notif = std.mem.zeroes(SeccompNotif); // must zero on every call
_ = sc3(SYS_ioctl, notif_fd, SECCOMP_IOCTL_NOTIF_RECV, ¬if);

// worker announced the path before calling openat, read it now
const path = readFromPipe(pipe_rd);

const resp = SeccompNotifResp{
    .id    = notif.id,
    .val   = 0,
    .err   = if (allow(path)) 0 else -@as(i32, EPERM),
    .flags = if (allow(path)) SECCOMP_USER_NOTIF_FLAG_CONTINUE else 0,
};
_ = sc3(SYS_ioctl, notif_fd, SECCOMP_IOCTL_NOTIF_SEND, &resp);

This mechanism is what gVisor builds on in ptrace mode. The freeze-inspect-inject loop the supervisor demo shows at the syscall level is the loop gVisor's sentry runs for every syscall a guest process makes. The rootless container use case is equally important: when an unprivileged container calls newuidmap, a supervisor with the necessary capability can validate and execute it on the container's behalf without granting the container elevated privilege directly.

Firecracker: hardware boundary and snapshot fork

Firecracker is a KVM VMM built by AWS for Lambda. It boots a full Linux kernel in under 125 ms and exposes a REST API over a Unix socket. The isolation story is the hardware VM boundary: the guest kernel runs in ring 0 of a KVM virtual machine with no shared memory or direct syscall path to the host. The microVM sandbox article covers why this boundary is necessary but not sufficient.

The snapshot fork is the more interesting capability for understanding serverless scaling. A running VM is snapshotted (its complete memory and disk state written to files) and then restored as multiple independent children that share the same initial state and diverge from that point:

# boot parent → snapshot to disk → fork 3 independent children
bash scripts/start-vm.sh /tmp/fc-parent.sock
FC_SOCK=/tmp/fc-parent.sock bash scripts/snapshot.sh /tmp/fc-snapshot
bash scripts/fork.sh /tmp/fc-snapshot 3
# three VMs running, each with its own API socket, none modifying the snapshot

A warm-started Lambda function is not cold-booted on each invocation. It is restored from a snapshot of a running function environment that already completed its initialization. The kernel, runtime, and application startup are in the memory image. Restoration from snapshot is faster than boot because the work is already done. The demo shows the raw Firecracker API calls that implement this; the mechanism is less magical than it sounds once you have seen it run.

gVisor: remove the host kernel surface

The controls above all operate on a guest that has a real Linux kernel. seccomp filters which syscalls reach it; capabilities limit what succeeds; Landlock restricts filesystem paths. A zero-day in an unfiltered syscall path can still reach the kernel directly. gVisor takes a different structural position: a user-space kernel (the sentry, written in Go) interposes between guest processes and the host kernel, so guest syscalls never reach the host Linux kernel at all.

The demo runs a static Zig binary inside a gVisor OCI container managed by runsc. From inside the container, the guest kernel's identity is visible immediately:

uname()         →  Linux 4.19.0-gvisor #1 SMP
/proc/version   →  Linux 4.19.0-gvisor #1 SMP ...
/proc/net/dev   →  (synthesised stub, not host kernel output)
/proc           →  directory synthesised by Go code

The sentry reimplements the Linux syscall interface. An exploit that breaks out of the guest sentry still needs to break out of the sentry process before reaching the host kernel. The sentry runs with a stripped capability set on the host. The host kernel's direct syscall surface for the guest is structurally absent, not just filtered.

gVisor runs in ptrace mode (no root, works inside VMs, used in the demo) or KVM mode (/dev/kvm required, lower overhead). Google Cloud Run and Fly.io run production workloads on it. The overhead is real: all guest syscalls route through the sentry rather than the kernel path. For multi-tenant agent workloads where the guest kernel surface is the primary concern, it is the structural answer that seccomp filters cannot provide.

Hardware TEE: excluding the cloud operator

Every technique above protects the host from the guest. A TEE inverts the question: it protects the guest from the host, including from the cloud operator who owns the hardware. CPU memory encryption keeps guest memory opaque to the hypervisor. An attestation report, signed by the silicon vendor's PKI, proves which firmware, kernel, and application are running inside the TEE.

On TDX or SEV-SNP hardware the demo fetches and parses a real attestation report. On other hardware it runs in simulation mode, walking through the data structures and trust chain without the hardware. The measurement registers show what can be attested:

MRTD   (SHA-384)  initial TD memory: firmware + kernel image
RTMR0             UEFI firmware events
RTMR1             bootloader / shim
RTMR2             OS kernel
RTMR3             application

The trust model shift is worth being explicit about. A TEE removes the cloud operator from the trust boundary. It does not eliminate a trust root; it moves it from the cloud operator to the silicon vendor. The attestation report is signed by Intel's PCK certificate chain (TDX) or AMD's VCEK chain (SEV-SNP). Side-channel surfaces (shared L3 cache, DRAM refresh timing, simultaneous multithreading) remain in most configurations. A TEE is appropriate when the threat model includes a compromised or malicious cloud operator; it is not appropriate as the sole isolation control for an untrusted workload running on the same host as other tenants.

CRIU and the time namespace: checkpoint/restore

CRIU snapshots the complete state of a running process (file descriptors, memory maps, execution context, pending signals) to disk and restores it later as if nothing happened. The process being checkpointed has no knowledge of CRIU and requires no modification. The demo runs a counter that logs one tick per second, freezes it mid-count, waits two seconds, restores it, and shows the log: the sequence is continuous with a gap marking the freeze.

The time namespace (CLONE_NEWTIME, kernel 5.6) exists specifically for the restoration case. When a process is restored, its CLOCK_MONOTONIC has jumped by the duration of the freeze. Any code that uses monotonic time for timeouts, intervals, or TTL checks sees the jump and behaves incorrectly. The time namespace lets the restored process see a shifted view of the clock such that no time appears to have passed:

// write offset to /proc/self/timens_offsets BEFORE forking the child
// (offsets become immutable once a process enters the namespace)
// offset of +100,000 seconds shifts CLOCK_MONOTONIC by ~1.15 days
const offset_str = "monotonic 100000 0\n";
try writeProc("/proc/self/timens_offsets", offset_str);

// child is forked after, it sees the shifted clock
const child = try std.posix.fork();
// child: CLOCK_MONOTONIC ≈ parent + 100,000 seconds

The three serverless demos connect directly: Firecracker snapshot fork produces the same restoration problem as CRIU at the VM level, and the same time namespace correction applies. Seeing the mechanism in isolation at the process level makes the VM-level solution obvious.

How the layers compose

No production system uses a single layer. Three representative stacks from the codebase, showing which demos contribute to each:

Each layer closes what the others leave open. Understanding why each one exists requires understanding what the layers below it cannot do.

The stack order matters for threat modelling. The hardware TEE excludes the cloud operator from the trust model but says nothing about what the guest software can do. The hypervisor layer provides the kernel boundary. The kernel primitives isolate resources. The kernel policy layer restricts what the process is allowed to ask for. The runtime layer (WASM, CRIU, time namespace) addresses correctness and lifecycle questions that the security layers do not cover. Each layer answers a different question, and none of them answers the others'.

What running all fourteen teaches

The main lesson is that most of the mechanisms in this stack are available without root. User namespaces, seccomp, Landlock, cgroups in the user slice, the time namespace, and gVisor in ptrace mode all work on a stock Linux machine without elevated privilege. The barrier to applying them is not permission. It is knowing what each one does and how to apply it correctly. The barrier is knowledge, and knowledge is removable.

The second lesson is that the layers compose but do not substitute. Running all fourteen in sequence reveals how many common deployment configurations leave obvious gaps: a container runtime that applies cgroups without seccomp, an agent sandbox that has the microVM boundary but no egress policy, a CRIU-based system that does not apply time namespace offsets on restore. Each of these is a case where someone reached for one layer, got the property that layer provides, and assumed that was sufficient. Running the full stack makes the assumptions explicit.