NAME

fireperf - A Network Benchmarking Tool

SYNOPSIS

fireperf --server ...
fireperf --client HOST ...

DESCRIPTION

fireperf can perform network benchmarks to test throughput and quality of a network between a fireperf client and fireperf server.

The client decides what kind of test is being run and tells the server when it connects. A server does not need any options for that and can serve different tests at the same time. fireperf can also talk to iperf (version 2) and iperf3, see INTEROPERABILITY below.

COMMON OPTIONS

--ipv4
-4

Only uses IPv4. The server only listens on IPv4, and the client connects to an IPv4 address of HOST.

--ipv6
-6

Only uses IPv6. The server only listens on IPv6, and the client connects to an IPv6 address of HOST.

--backend=NAME
-b NAME

Selects how fireperf performs its I/O. NAME is one of:

  • auto: Use io_uring if it is available, and epoll otherwise (this is the default).

  • io_uring: Use io_uring. Over a network, it sends without copying at less than half of the CPU time of epoll. It is not available if fireperf has been built without it, on old kernels, or where it has been disabled (which is common in containers). Over loopback, the kernel copies what it sends anyway, which makes epoll the faster one there.

  • epoll: Use epoll, which works everywhere. It sends without copying, too (see --zerocopy).

--debug
-d

Enables debugging output
If fireperf has been compiled with debugging support, this command will print various debug messages that will help to find bugs in the program.

--help
-h

Shows a short summary of all options and exits.

--json
-j

Prints one JSON object per line for other programs to read instead of showing the interactive terminal interface. See JSON OUTPUT below.

--theme=MODE

Chooses colours that can be read on a terminal with a dark or a light background. MODE is one of auto (this is the default), dark and light. With auto, fireperf asks the terminal what its background looks like, or looks at COLORFGBG if that has been set, and uses dark colours if neither tells.

--no-tui
-n

Prints plain text reports instead of showing the interactive terminal interface. See OUTPUT below.

--zerocopy=MODE

Most of what it costs to send data is copying it from the program into the kernel, which fireperf avoids. How depends on the backend:

  • epoll keeps what it sends in a file in memory and asks the kernel to send it from there (sendfile(2)). This always works.

  • io_uring asks the kernel to send straight from the memory of fireperf. That pays off with a network card that can take the data as it is. Over loopback and some virtual devices the kernel has to copy it after all, which is more work than copying it in the first place.
    MODE is one of:

  • auto: Send without copying, but go back to copying if the kernel says that it had to copy most of the data anyway (this is the default)

  • always: Never go back to copying

  • never: Copy what is being sent like most other programs do, which is useful to see what difference it makes
    The memory that io_uring sends from has to be locked while the kernel is busy with it. Unless fireperf runs as root, that counts towards how much memory a user may lock (ulimit -l), which is only a couple of MiB by default. fireperf then sends from as much of its data as it may lock, and copies if it runs out. Raising the limit (for example with LimitMEMLOCK= in systemd or in /etc/security/limits.conf) allows to send from more data, but makes no difference to how fast it is.

--parallel=N
-P N

If this option is set on the client side, it will instruct the client to open N connections and keep them open (the default is one).
On the server side, it will set a limit how many connections the server is willing to accept. N can be up to 1048576.

--port=N
-p N
--port=N:M
-p N:M

Both server and client take a single port (N) or a port range (N:M).
The server will open a listening socket the port given or each port in the range and waits for the client to connect.
The client will spread its connections evenly across all ports in the range or use the specified port to connect to the server.
A single port is all that is needed for almost everything: all workers of the server listen on the same port and the kernel spreads the connections across them. A range is only there for one thing: a client cannot have more connections open to the same address and port of a server than it has local ports available (about 28,000 by default, see net.ipv4.ip_local_port_range), and a port range multiplies that. More source addresses on the client (--bind) do the same and leave you with a single port to allow on a firewall.

--rate=N
-r N

Makes the client open N connections per second (up to ten million).
Without this option, the client opens a new connection whenever one has been closed, as fast as it can. If whatever it is talking to slows down, the client slows down with it, and the result only tells which one of them was the slowest.
With this option, every connection is opened when it is due, no matter what has happened to the ones before. If the other side cannot keep up, connections take longer to be established, pile up and fail, all of which is shown in the output. To find out how many connections per second something can handle, increase the rate until that happens.
--parallel limits how many connections may be open at the same time. It is not limited by default if a rate is given. Connections that are due while that limit has been reached are skipped. With --connections, the client makes exactly that many attempts, whether they succeed or not.
At the end, the client shows how well it has been able to follow the schedule:
---- Schedule: 10000 due, 9990 on time, 10 late, 0 skipped ----
A connection is late if it was opened more than ten milliseconds after it was due, and skipped if the client was more than a second behind or was not allowed to open another connection. If more than one percent were not on time, the client itself was the limit and the result does not say much about anything else.

--timeout=T
-t T

If set, fireperf will automatically terminate itself after T seconds. Otherwise it will run for forever.

--version
-V

Prints the version of this program

--workers=N
-w N

Sets the number of worker threads. By default, fireperf launches one worker for every processor, but never more than the client has connections to open.
Every worker handles its own share of the connections independently of all others. The server opens one listening socket per worker for every port and the kernel distributes incoming connections across them.

--close
-x

If set, all connections will be closed as soon as they have been established. No data will be sent.
This is useful to test how many TCP handshakes per second are possible. The client will open a new connection for every one that has been closed, so that it is enough to pass this option to either the server or the client.
NOTE: This opens tens of thousands of connections per second. Every stateful firewall on the way (including connection tracking on the client and the server themselves) has to keep an entry for each of them for a while after it has been closed, and will start to drop packets of any other connection once its table is full. Use --connections or --timeout to limit the test.

--zero
-z

Sends zeroes instead of random data, in both directions.
This is useful for testing compression between the client and server. On the server, it applies to all clients, whether they have asked for it or not.

--keepalive-interval=T

How often connections that are being kept open (--keepalive) are checked, in seconds (1 to 32767). The default is 10. See --keepalive.

--keepalive-count=N

How many TCP keepalives may go unanswered before the kernel gives up on a connection (1 to 127). The default is 3.

SERVER-SPECIFIC OPTIONS

--server
-s

This starts the program in server mode which makes it listen on the specified port and wait for clients to connect.
Without -4 or -6, the server listens on IPv6 and IPv4. If the system does not have IPv6, it listens on IPv4 only.

--keepalive
-k

Instead of sending data, this will configure fireperf to only send keepalive packets.
If a large number of connections is being created, it might become undesirable to saturate the link between client and server. This option will send some packets to keep the connection alive through for example NAT gateways.
This is an option of the client like any other: it asks the server not to send any data on its connections. Given to the server, it makes the server never send any data, whatever its clients ask for.
A connection that nothing is sent over does not tell if it still works: if a firewall in between has forgotten about it, both ends still have it open. So the client sends one byte over every connection once per interval (see --keepalive-interval), which the server sends back. A connection is lost if that has not come back by the time the next one is due. It is then closed and the client opens a new one. The Alive column shows how many of the open connections have last been found to work.
This only works if the server is fireperf. The kernel is being asked to send TCP keepalives as well.

CLIENT-SPECIFIC OPTIONS

--client HOST
-c HOST

This starts the program in client mode. In this mode, the client will connect to the server at HOST and start a throughput test.
HOST can be an IPv4 or IPv6 address, or a name. If a name has more than one address, the client connects to the first one that the system prefers, unless -4, -6 or --bind tell otherwise. If that one does not work, it tries the next one, until it has reached the server. After that, all connections go to the address that has worked.

--bind=ADDRESS
-B ADDRESS

Makes the client open its connections from ADDRESS, which must be an address of this host. This option may be given more than once and the connections are then spread across all of the addresses.
All addresses must be of the same family (IPv4 or IPv6), and the client connects to an address of the server of that family.
Every address can open as many connections to the same address and port of the server as there are local ports (see --port). More addresses allow more connections, and what is in between sees connections from more than one client.
The client then shows what has happened for every one of these addresses instead of for the server (see OUTPUT), including how many connections could not be established from it - which shows if something in between treats one of them differently.

--connections=N
-C N

Limits the number of connections that the client opens in total. The client will terminate as soon as it has opened N connections and all of them have been closed.
Together with --close, this makes the client perform exactly N handshakes.

--connect-timeout=T
-T T

The client gives up on a connection if it could not be established after T seconds and counts it as failed (timed out). The default is 5 seconds. 0 leaves it to the kernel, which keeps trying for about two minutes.
A SYN that is lost on the way is being sent again after one second, then after another two, and so on. A timeout of one second therefore counts every connection that has lost a single packet as failed, and with the default the first two attempts may be lost.

--exchange=N[:M]
-X N[:M]

The client sends a request of N bytes on every connection, waits for the server to answer with a response of M bytes (or N, if only one number has been given), and closes the connection. Like with --close, it then opens a new one.
A connection only counts if all of the response has arrived. If it is closed before that, it is counted as failed (incomplete).
This is what a firewall gets to see from a short transaction like a DNS query over TCP or a small HTTP request, and it is a better test than --close: there, a connection is established and closed without a single byte going through, which is the easiest case there is. Both sizes can be zero, which still exchanges the two messages in which the client and the server tell each other what they want. Neither can be larger than one megabyte.
At the end, the client shows how many exchanges have been completed and how long they took, from asking for the connection until the last byte of the response:
---- Exchanges: 10000 completed, min 71us, median 119us, 99% 1.24ms, max 1.03s ----
The server does not need this option, but it has to be fireperf.

--reset
-R

Makes the client reset connections (RST) instead of closing them properly (FIN) when it is done with them, which is with --close and --exchange.
Something that keeps state for a connection may forget about it straight away when it has been reset, but has to remember it for a while after it has been closed. The server counts these connections as Reset.

--duplex
-D

Duplex mode enables the client to send data instead of only receiving it from the server. The same TCP connection will be used for traffic in both directions.

OUTPUT

If fireperf is started on a terminal, it shows an interactive interface:

  • At the top, what is running, and for how long.

  • A graph of the traffic, with the latest second on the right hand side. What has been received grows upwards and what has been sent grows downwards. If only one direction carries any data, it gets all of the space.

  • On terminals that are at least 100 columns wide, the current, average and peak rates and the totals in both directions, the connections, how long it takes to establish them, and how much of a CPU the busiest worker is using.

  • Below that, every address that we are talking to: the clients of a server, or the server of a client, with what it is, its connections, and its traffic. The busiest one comes first, and those without any open connections are grey.

  • The latest messages.

Press q or Ctrl-C to quit, p or the space bar to pause and resume, and Ctrl-L to redraw the screen. If there are more addresses than fit, the arrow keys (or j and k), Page Up, Page Down, Home and End scroll through them. s sorts them by the next column, and n shows their names instead of the addresses (which are being looked up in the background, so it might take a moment until they show up). With a UTF-8 locale, lines and the graph are drawn with what UTF-8 has for them.

To find out who has connected, the server asks the kernel for the address of every connection that it accepts. That costs a system call, which does not make a difference that can be measured, even with hundreds of thousands of connections per second.

If standard input or standard output is not a terminal (for example when the output is redirected into a file or fireperf has been launched by a script), or if --no-tui has been passed, one line is printed every second instead:

     Time     Open    Alive      Total      New/s   Failed    Reset    Median       99%   CPU        RX Rate     RX Total        TX Rate     TX Total
     1.0s        1        0          1       1.00        0        0      85us      85us   52%      9.41 Gb/s     1.10 GiB       0.00 b/s       0.00 B

The columns show:

  • the time since the start,

  • the number of connections that are currently open,

  • how many of those are known to work (see --keepalive),

  • the number of connections that have been opened in total and per second,

  • the number of connections that could not be established (Failed),

  • the number of connections that were established, but then failed instead of being closed properly (Reset). This is what happens if a firewall in between loses its state of a connection - or if the other side terminates while data is on its way,

  • how long it took the client to establish the connections that have been opened since the previous line: the median, and the time that 99% of them stayed below,

  • how much of a CPU the busiest worker has been using,

  • the current rate and the total amount of data that has been received (RX) and sent (TX).

When fireperf terminates, a final line shows the totals and the average rates over the whole runtime, followed by a summary. The client shows what has happened to the connections it has tried to open:

Workers:     16 using io_uring, the busiest one was using 23% of a CPU
Transferred: 0 bytes sent, 1181116006 bytes received
Connections: 1000 attempted, 990 established, 10 failed (8 timed out, 2 rejected, 0 local limit, 0 incomplete), 0 reset, 0 lost
Handshake:   min 45us, median 120us, 99% 1.03s, max 3.02s

The server shows its workers, and how many connections the kernel has dropped because they have not been accepted in time. That is counted for the whole host and includes all other programs.

Both show everybody that they have been talking to - the clients of a server, or the server of a client - with what it is, how many connections it has had, how many of them have been reset, and how much has been received from it and sent to it, the one that has transferred the most first. The client also shows how many connections could not be established, and with --bind, it shows the addresses that it has connected from instead of the server:

Clients:     2
             Address    Type           Total   Reset      Received          Sent
             127.0.0.1  fireperf           2       0        0.00 B     10.51 GiB
             ::1        fireperf           1       0      2.92 GiB      3.07 GiB

Every worker keeps up to 64 addresses apart and adds up everything else, which is shown as Others.

A connection has timed out if nobody has answered, it was rejected if somebody has answered that it cannot be established (refused, reset, unreachable), and local limit means that the client itself ran out of something (ports, file descriptors, memory) before anything was sent. incomplete is a connection that has been closed before the response to a request has arrived (see --exchange), and a connection is lost if it was open, but the server has stopped answering on it (see --keepalive).

The time of a handshake is measured from the moment the client asks for the connection until it is established, and is accurate to about six percent. If a packet of the handshake is lost, it is being sent again after one second, which shows up as a 99th percentile or a maximum of slightly more than that.

JSON OUTPUT

With --json, fireperf prints one JSON object per line (JSON Lines) to standard output, and all messages go to standard error. Every object has a type:

start

is printed first and describes the test: version (of the format, currently 1), fireperf (the version of the program), mode (client or server), backend, workers, port and ports. The client adds the server as it has been given (host) and the address that it connects to, and the options of the test: parallel, rate, connections, timeout, duplex, keepalive, close, zero and exchange (with the request and response sizes).

interval

is printed every second.

summary

is printed last and covers the whole runtime.

Intervals and the summary have the same fields:

  • time is the time since the start and interval the time this object covers, both in seconds,

  • connections has the number of connections that are open and alive, how many are new in this interval, the total, new_per_s, and how many have been attempted, have failed, have been resets or lost,

  • received and sent have the bytes of this interval, the total_bytes and the rate in bits_per_s,

  • handshake has the count, min_us, median_us, p99_us and max_us of the handshakes of this interval in microseconds, or is null if there were none,

  • cpu is how much of a CPU the busiest worker has been using (1.0 is one CPU),

  • remotes has everybody that we have been talking to so far, the one that has transferred the most first: its address (or null for everything that did not fit in), what it is (type: fireperf, iperf2, iperf3, unknown, or null if we don’t know, yet), its connections (open, total, failed and resets), and the received_bytes and sent_bytes in total. With --bind, these are the addresses that the client has connected from, which the start object lists as bind.

The summary of the client also has the kind of server it has been talking to, why connections have failed (timeout, rejected, local and incomplete), the number of exchanges and their latency in exchange (see --exchange), and whether it has kept its schedule (due, late and skipped, see --rate). The summary of the server has the listen_drops.

{"type":"interval","time":1.000000,"interval":1.000000,"connections":{"open":1,...},"received":{"bytes":5165447638,...},...}

New fields may be added at any time, so programs should ignore what they do not know. Fields will not be removed or change their meaning without changing version. Numbers are always written the same way, no matter what the language is set to.

INTEROPERABILITY

The first thing that a client sends on a connection is a short message that says what it wants, which the server answers with what it is going to do. Both sides also use it to find out what they are talking to, so that nothing has to be configured for this. The client opens a single connection to find out before it opens all others.

fireperf

The client and the server do what the client has asked for.

iperf (version 2)

iperf only ever sends data from its client to its server.
A fireperf server receives what an iperf client sends.
A fireperf client that finds an iperf server sends data to it (as if --duplex had been given), says so, and does not expect to receive anything. Recent versions of iperf identify themselves straight away. Older ones do not answer at all, which the client takes for an answer after one second.

iperf3

A fireperf server serves iperf3 clients that run a test over TCP: by default (the client sends), with --reverse, with --bidir and with --parallel. It can serve many of them at the same time, and fireperf clients next to them. The results that iperf3 shows for the server are what fireperf has counted. It does not know anything about retransmits or the usage of the CPU.
A fireperf client that finds an iperf3 server runs a test with it: the server sends (which is what iperf3 calls reverse), and with --duplex the client does, too. --parallel sets the number of connections, which is doubled with --duplex as iperf3 only sends in one direction on each of them. The test runs until fireperf terminates.
An iperf3 test has an extra connection that controls it, which is counted like all others. All of its connections are handled by one worker of the client, so the CPU column tells whether that was enough. An iperf3 server only talks to one client at a time, and --keepalive, --close, --rate and --connections have no meaning with it.

Connections that are closed as soon as they have been established (--close) never carry any of this. Connections that exchange a request and a response (--exchange) and the checks of connections that are being kept open (--keepalive) need fireperf on both sides.

TESTING WHAT IS IN BETWEEN

fireperf is made to test something that sits between two hosts, like a firewall or a NAT gateway. The client and the server must run on hosts on either side of it and not on the device itself. If they are running on the same host, or are connected to each other directly, the numbers only show how fast that host, its network stack and fireperf are.

That is still worth doing once: connect the client and the server back to back and run the test that you are planning to run. The result is the most this pair of hosts can generate. Anything close to it that you measure later with the device in between says nothing about the device.

Is it the device or is it me?

The client and the server run into limits of their own long before a decent firewall does. fireperf prints a warning at start for the ones it knows about, and the output shows a few things that tell you who was the limit:

  • CPU: If a worker uses almost all of a CPU, this side cannot go any faster. Use more workers or a faster host. Time that the kernel spends on packets outside of fireperf is not included, so a low number does not prove that the host has capacity to spare.

  • local limit in the summary of the connections: The client ran out of local ports or file descriptors before it has sent anything.

  • Schedule (with --rate): If connections were late or have been skipped, the client could not open them at the rate that it has been asked to.

  • Accept in the summary of the server: The server did not accept connections fast enough.

These settings of the hosts matter:

  • A client can only open as many connections to one address and port of the server as it has local ports (net.ipv4.ip_local_port_range, about 28,000 by default). Use a port range (--port) on both sides and more source addresses (--bind). A connection that the client closes keeps its port for another minute, so let the server close the connections if you are opening many.

  • Every connection needs a file descriptor (ulimit -n, fs.nr_open).

  • If the client or the server are tracking connections themselves (if they are running a firewall with connection tracking, which most distributions do), that has a limit of its own (net.netfilter.nf_conntrack_max), slows them down, and will be full long before the table of a dedicated firewall is. Turn it off on both hosts or exclude the test traffic from it.

  • The server can only have so many connections waiting to be accepted (net.core.somaxconn, net.ipv4.tcp_max_syn_backlog).

How much data can it forward?

server# fireperf --server
client# fireperf --client SERVER --parallel 16 --duplex

The server sends data to the client and, with --duplex, the client sends data to the server, as fast as they can. One connection is usually handled by one CPU on every device on the way, so use as many connections as you want to see CPUs being used. Random data is being sent by default; --zero shows the effect of compression.

How many connections can it keep open?

server# fireperf --server --keepalive --port 5001:5064
client# fireperf --client SERVER --keepalive --port 5001:5064 --parallel 1000000

The client opens connections until it has as many as it has been asked for and no data is being sent. Watch Open go up and where it stops, and whether connections start to fail (Failed) or are being lost again (Reset).

The client checks every connection once per interval (ten seconds by default, see --keepalive-interval) by sending a byte that the server sends back. Alive shows how many connections have passed that check, and a connection that does not is counted as lost, closed and opened again. If the device silently forgets about connections, Alive falls behind Open and the number of lost connections goes up. Keep the test running for a while after the number of open connections has stopped changing: a firewall might only drop what it has not seen any traffic on for some time.

How many connections per second can it take?

server# fireperf --server
client# fireperf --client SERVER --exchange 100:1000 --rate 10000 --timeout 30

The client opens connections at a fixed rate, sends a small request on each of them, waits for the response and closes it. Run this a couple of times and increase the rate. The device is at its limit when:

  • connections fail (Failed), or

  • it takes longer to establish them (Median and 99%). A step to about one second means that packets are being dropped and sent again.

Do not use the highest rate at which nothing failed as the result without checking that the client was able to open connections at that rate (Schedule and CPU).

Without --rate, the client opens a new connection whenever one has been closed. That is easier, but it only finds whatever is slowest and slows down with it: you get a number, but you cannot tell whose it is.

Everything on the way that keeps state has to remember each of these connections for a while after it has been closed - on a Linux firewall for up to two minutes. A test with 10,000 connections per second needs space for more than a million of them, and when a table is full, connections of everybody else are being dropped, too. Do not run this through anything that others depend on.

LANGUAGE

fireperf speaks the language of the user, as far as it has been translated (see LANG and LC_ALL in locale(7)). That includes the plain text output, in which numbers are written the way the language writes them, too. Scripts that read the output should run fireperf with LC_ALL=C.

SIGNALS

fireperf terminates cleanly when it receives SIGINT or SIGTERM.

EXIT STATUS

0

The test has finished: the timeout was reached, the user has quit, or the process has been asked to terminate by a signal.

1

An error has occurred.

2

The command line was invalid.

BUGS

Please report all bugs to the bugtracker at https://bugzilla.ipfire.org/

SEE ALSO

fireperf-design(7), which explains how fireperf works and how it compares to iperf3(1) and iperf(1).

AUTHORS

Michael Tremer