I care a lot about reliability - specifically, deterministically simulating bad conditions to test how systems behave. Last year for example when I was learning database internals I spent so much time studying the correctness of fsync(2) and its behaviour. Then when I started building my database I spent more time building a "framework" for simulating block-io errors than I have spent on the database itself (I might write a blog on that later).

I was recently building a network protocol and I was looking for ways to simulate network failure and slowness, that's when I learned about toxiproxy.

toxiproxy: A TCP proxy to simulate network and system conditions for chaos and resiliency testing

Toxiproxy lets you setup a proxy, connect it to the upstream server (e.g. a database) and configure delay, packet loss or jitter. Then, instead of directly connecting to the database you connect to the proxy.

While setting up toxiproxy all I could think about was: is it possible to do something similar transparently? (i.e. without a proxy)

Now to be clear - I have no problem using a proxy, I don't have a reason not to. This project is more of a challenge and a fun experiment.

For this experiment to be considered successful, I should be able to at least simulate:

• packet loss
• packet delay
• and dynamically apply these conditions

I immediately thought of using eBPF as it's the right tool for selectively dropping packets, but it can't delay them because you can't sleep in eBPF hooks. I did some searching and that's when I learned about Traffic Control (I'll call it tc from now on).

eBPF is a technoloy that lets your write sandboxed programs to extend the kernel. Due to the nature of its restricted environment, you can't block, sleep or have unbounded loops in it.

TC is responsible for packet scheduling in the Linux kernel. Each network interface has a queue discipline (qdisc), when you send a packet, it it first enqueued to the qdisc for that interface and the kernel then tries to dequeue packets from the qdisc to send to the network driver. Different qdiscs implement different scheduling algorithms.

In the kernel, qdiscs live under net/sched. Each module must register a struct Qdisc_ops, exposing two main operations: enqueue and dequeue.

1
static struct Qdisc_ops my_qdisc_ops = {
2
    .id         =   "qdisc_id_here",
3
    .enqueue    =   enqueue_impl,
4
    .dequeue    =   dequeue_impl,
5
};
6

7
static int enqueue_impl(struct sk_buff *skb, ...) {
8
    /* save packet for later dequeue */
9
}
10
static struct sk_buff *dequeue_impl(struct Qdisc *sch) {
11
    /* implementation selects and returns next packet */
12
}

As you can imagine, qdiscs can be used to ensure fairness, rate-limit, or control bandwidth. But they can also be used to simulate network conditions. I could, for example, write a custom kernel module that implements a qdisc that only dequeues a packet after a certain amount of time has passed. Luckily, I don't have to, because there already exists a qdisc called netem that does exactly that. When you attach it to a network device, you can pass parameters like delay, jitter, and loss.

A quick demo

If you'd like to see tc-netem in action, follow these steps:

First create a root qdisc of type netem for the loopback device, with a 1-second delay.

1
λ sudo tc qdisc add dev lo root netem delay 1s

Then start any service on localhost, I'll use a python http server:

1
λ python -m http.server -b 127.0.0.1 8002

In another terminal window, run curl and measure the tcp handshake time

1
λ curl -w "TCP Handshake: %{time_connect}s\n" 127.0.0.1:8002
2
TCP Handshake: 2.000233s

The two-second delay happens because TC controls egress traffic. Since both the client and server send packets over the loopback interface, each direction is delayed by one second, resulting in a total connect time of two seconds.

run tcpdump -i lo -nn 'port 8002' -tttt to see packet delay for yourself.

you will see duplicate packets (re-transmissions) because this delay is greater than RTO.

Let's remove the qdisc now

1
λ sudo tc qdisc del dev lo root

Classful qdiscs

With TC we can transparently simulate network conditions, but our approach right now has one limitation: it's affecting all egress traffic on the interface.

Luckily, TC has the concept of classful qdiscs: These qdiscs have multiple classes within, where each class contains a leaf qdisc, and when a packet enters the classful qdisc, it has to be classified to one of the classes. Classification is done by by filters, the available filters include: basic, cgroup and bpf.

The idea now is to have a classful qdisc attached to lo with 2 classes within, a class with a regular fifo qdisc (no simulation), and a class with a netem qdisc. Then have a TC filter choose which packets to delay (to send to netem class) and which packets to route to the fifo class.

The filters "basic" and "cgroup" are probably perfect for most setups, you could for example put your database in a cgroup and have the netem qdisc match that cgroup only. But.. I'm always looking for a reason to use eBPF :)

Jolt

It's now time to introduce my newest project, called jolt.

1
Usage: jolt [OPTIONS]... [PORT]...
2

3
Simulate network conditions on connections made to the
4
speicied ports.

Jolt it intended to abstract away all of the details I mentioned in the article, and provide a simple interface to affect(e.g. delay) local connections from either direction.

Jolt will setup qdiscs on the loopback interface, and load a bpf tc-filter to delay traffic dynamically. The project is still in early development, and I need to solve a couple of issues first.

The biggest issue now is that a netem qdisc has a fixed delay, so if I want to affect different connections differently, I need to insert a new qdisc in the interface for each connection and have the bpf filter route packets accordingly. To be honest, that's not a big deal, because how many connections are you going to simulate and would they have different conditions? But I can see this being useful to see for example how a consensus algorithm behaves when a group of nodes are all affected differently.

To achieve true flexibility I need to stop using netem and write my own qdisc, but I didn't want to write a kernel module so I started wondering if it's possible to implement a qdisc in eBPF.

Luckily, a recent patch made writing a qdisc in eBPF possible, more details here.

I expected this project to be difficult, but the kernel had all the building blocks i needed: TC, a netem qdisc, and recently qdisc in eBPF. Anyway, I'll update this article whenever I improve Jolt, but for now you can see the code in github: https://github.com/mohammedgqudah/jolt