NAME

fireperf-design - How fireperf works, and why it works that way

DESCRIPTION

fireperf(8) is a tool to measure what a network - and above all, what is in between two hosts - can do: how much data it can move, how many connections it can open, how long that takes, and whether it keeps the connections that it has. It has been written to test firewalls, which is what IPFire is.

That is a different question from what most tools for this have been built for. They measure a link: they open a connection or a couple of them, send as much as they can, and report a number. A firewall is not a link. It keeps state for every connection, it has a table of them that can fill up, it may forget about a connection that has been quiet for a while, and the number of new connections per second matters to it at least as much as the number of bytes. A tool that tests a firewall has to be able to push the firewall harder than the firewall can take, which means that the tool itself must never be what is holding the test back. And it has to tell when it is.

This page explains how fireperf is built to do that.

ONE WORKER PER CPU, AND NOTHING SHARED

fireperf starts one worker thread for every CPU (see --workers). Every worker runs its own event loop and has its own connections. Workers don’t share any locks, lists or buffers on the way of the data, so that adding a CPU adds a worker that can do as much as the others.

The server opens one listening socket per worker on the same port (with SO_REUSEPORT), and the kernel spreads new connections across them. That means that all clients connect to a single port, which keeps firewall rules simple, and that no worker has to hand connections to another one. The client spreads the connections that it should open across its workers in the same way.

What the workers count (bytes, connections, failures, how long it took to establish them) is kept in counters that only they write to. Once a second, the main thread adds them up. It is the only thread that talks to the terminal, handles signals (through a signalfd) and keeps time, so that the workers never wait for any of that.

TWO WAYS TO DO I/O: IO_URING AND EPOLL

Every worker does its I/O through a backend (see --backend):

io_uring

The kernel is being asked to do things (accept, receive, send, connect, close) and tells us when they are done, without a system call for every one of them. fireperf accepts all connections of a listening socket with a single request, and receives on a connection with a single request for as long as it is open, from a ring of buffers that the kernel picks from. It also opens and closes connections through the ring. This is what is being used if it is available.

epoll

The classic way of being told when a socket is ready and then doing what has to be done with a system call. It works everywhere, which io_uring does not: it is often disabled in containers and on older kernels.

Both backends do the same things. Everything else in fireperf does not know which one is being used.

SENDING WITHOUT COPYING

Most of what it costs to send data is copying it from the program into the kernel. fireperf doesn’t copy:

  • What it sends comes from a pool of 32 MiB of random data (or zeroes, see --zero), so that nothing in between can compress it or recognise it. Random data that does not fit into a cache is also what a real network carries, unlike the same couple of kilobytes being sent again and again.

  • The pool is a file in memory (a memfd). With epoll, the kernel is being asked to send from that file (sendfile(2)), which never copies.

  • With io_uring, the pool is registered with the kernel once, and the kernel sends straight from it (IORING_OP_SEND_ZC) and tells us when it is done with it. The kernel also tells us when it had to copy after all - which is what happens over loopback and with some virtual devices - and then fireperf goes back to copying, which is less work for the same result (see --zerocopy).

Registering memory counts towards how much memory a user may lock, which is only a couple of MiB by default. fireperf registers as much of the pool as it may, and if that is not enough, it copies rather than failing.

Received data is thrown away by the kernel without being copied out (MSG_TRUNC), as there is nothing that fireperf would do with it.

OPENING CONNECTIONS

Testing how many connections a firewall can handle needs a client that can open many more than that. fireperf keeps a number of connections open (--parallel), opens them at a fixed rate (--rate), closes them straight away (--close), resets them (--reset), or exchanges a request and a response on every one of them, as a web server would (--exchange).

With --rate, connections are being opened on a schedule, whether the ones before have been established or not. A client that waits for every connection before it opens the next one measures how fast the slowest part is, not whether everything keeps up with the rate. fireperf reports how many connections it has opened late, or not at all.

Every connection that could not be established is counted, and why: it has timed out (see --connect-timeout), it has been rejected, or the client itself has run out of something (ports, file descriptors). Established connections that end without being closed properly are counted as reset, which is what happens if a firewall loses its state. With --keepalive, the client checks every connection that it keeps open once in a while, and counts those that don’t answer any more as lost.

How long it takes to establish a connection is recorded for every connection in a histogram that is accurate to about six percent, so that the median, the 99th percentile and the maximum can be shown for any number of connections without keeping all of them.

Before the test, fireperf checks whether the system can do what it has been asked to do (file descriptors, local ports, the connection tracking table, the listen backlog) and says so if it cannot. With --bind, connections are opened from more than one address, so that a client is not limited by the number of its local ports.

KNOWING WHAT HAS BEEN MEASURED

A result is only worth something if it is clear what has limited it. fireperf measures how much of a CPU every worker uses. If one has been busy all of the time, the result shows what fireperf could do on this side, not what the network or the firewall can do, and fireperf says so.

On the server, fireperf counts how many connections the kernel has dropped because they could not be accepted in time.

ONE PROTOCOL, AND THE OTHERS, TOO

When a client connects, it sends 37 bytes that tell the server what kind of test it wants. The server answers, and both start. That is all there is to the protocol. The server does not need any options for any test, and can run different tests for different clients at the same time.

Those 37 bytes look like what iperf3 sends first, and they start in a way that makes an iperf (version 2) server answer. So the client finds out what the server is from what it answers: fireperf, iperf3 or iperf. The server finds out what a client is from what it sends first. fireperf speaks the protocol of iperf3 (including its JSON messages), and it understands what iperf sends. So a fireperf client can test against an iperf3 server, and an iperf3 client can test against a fireperf server, without anybody having to say so.

SHOWING WHAT IS GOING ON

On a terminal, fireperf shows a graph of the traffic, its numbers, and every address that it is talking to (see fireperf(8)). Without a terminal, it prints a line every second, and with --json, it prints a JSON object every second for other programs to read.

COMPARED TO IPERF3 AND IPERF

iperf3 and iperf (version 2) are the tools that most people use to measure a network, and they are good at measuring a link. Where fireperf is different:

  • An iperf3 server runs one test at a time and turns everybody else away until it is done. A fireperf server takes everybody, and every client can run a test of its own.

  • iperf3 and iperf measure how much data a couple of connections move. Neither can open connections at a fixed rate, tell why connections could not be established, find out whether connections that are being kept open still work, or show how long it takes to establish connections across many of them.

  • By default, iperf3 and iperf copy what they send. Sending from another host over a network, fireperf needs less than half of the CPU time per GiB that iperf3 needs with --zerocopy, and about a seventh of what iperf3 and iperf need by default (see benchmarks/results.csv in the source).

  • fireperf can be the client or the server for both of them, so it can be used where one of them is already running.

SEE ALSO

fireperf(8), iperf3(1), iperf(1), io_uring(7), epoll(7), sendfile(2)

AUTHORS

Michael Tremer