NAME

pakfire-reproducible-builds - Building the same package twice

DESCRIPTION

A build is reproducible if building the same source package in the same build environment creates packages that are identical, bit by bit. Anybody can then rebuild a package and compare it with the one that has been distributed, and a difference means that something has changed which should not have.

What has to be the same is what is in the package: the metadata and every file, uncompressed. The compressed file does not have to be, because it depends on the version and the settings of the compressor.

Test builds rely on this, too: if two builds of the same source package do not produce the same files, a difference between them cannot be used to find out what a change did.

This page describes what Pakfire does to make builds reproducible, what it does not do yet, what a package has to take care of itself, and how to find out what makes two packages differ.

WHAT THE BUILD DOES

SOURCE_DATE_EPOCH

Every build has a time which is what SOURCE_DATE_EPOCH says in the commands of the build and in the package. It is the number of seconds since 1970-01-01 00:00:00 UTC, as described at https://reproducible-builds.org/specs/source-date-epoch/.

It is the value of the variable if it is set when the build is started. That is how a caller can choose the time, for example the time of the last commit. Otherwise it is the time at which the source package was made, which is as close to the last modification of the source as the build can get, and which makes building the same source package again use the same time. A variable that is not an integer makes the build fail, because it is not known what was meant by it. An empty variable is the same as none.

The time is then

  • in the environment of every command that the build runs (and not changed by the build),

  • the time of the build in the metadata of the package,

  • the timestamp of the files that carry the metadata (pakfire-format and .PKGINFO), and

  • the latest time that any packaged file can have (see below).

Building a makefile that has not been made into a source package first creates the source package right then, so it has the current time unless the variable is set. Set it to build the same makefile again.

The compiler cache does not get in the way of this: the time is only part of what ccache looks at when the source uses __DATE__ or __TIME__. Everything else is cached as before, and what uses them can be cached now, as long as the time is the same.

Contents of the package

  • The files in a package are always listed in the order of their paths, no matter in which order the filesystem returns them.

  • The files of a source package are all made readable by everybody and owned by the user and the group root. The files of the other packages keep the permissions and the owner that the build has given them.

  • The modification time of a file is never later than the time of the build (SOURCE_DATE_EPOCH). A file that is older keeps its time.

  • The UUID of a package is not random. It is derived from the name and the version, the time, the host and the ID of the build, and the path, mode, size, modification time and link target of every file, so that building the same package again creates the same UUID, and a package that is different has a different one. (This applies to the packages that a build creates.)

The build environment

Every build runs in a new jail with the same settings, so these things do not depend on the machine that does the build:

  • The directory that the source is extracted to and built in is /build/source/NAME-VERSION, and the files are installed to the same BUILDROOT every time.

  • LANG is set to C.utf-8, HOME to /root, and PATH, TERM and PS1 are fixed. A build does not inherit the environment of the user who started it, so the time zone (TZ) and the locale (LC_*) are not those of the user either. The time zone is the one that the build environment has in /etc/localtime. Only the interactive shell copies TERM and LANG.

  • Python and Perl are given a fixed seed for their hash tables (PYTHONHASHSEED, PERL_HASH_SEED and PERL_PERTURB_KEYS are 0). They would choose a random one on every run, and the order in which they list the entries of a dictionary or a hash would be different every time.

  • The name of the host is builder, whatever the machine is called that the build runs on.

  • The umask is 0022.

  • The compiler is told to replace the path of the sources with /usr/src/debug in the debugging information and everywhere else, for example in __FILE__ and so in the message of a failed assert() (-ffile-prefix-map). It is only used for C and C++. Programs that are built with another compiler have to do this themselves.

  • The build ID that the linker adds to ELF files is a hash of their contents, so identical programs have the same build ID, and so have the files that the debugging information is stored in.

  • The compiler is given a seed for the random names that it puts into objects, for example the sections for link-time optimisation in static libraries. The seed is the name of the file that is being compiled, so that it is the same every time, and different for every file, which it has to be: if two objects that are linked into one with ld -r had the same seed, the compiler could not read the result any more. A build that sets -frandom-seed itself keeps its own value. This is done with a specification for the compiler in /usr/lib/pakfire/random-seed.specs, which Pakfire puts into the build environment, and -specs= in CFLAGS.

Recording the environment

The metadata of a package lists the version of every package that was installed in the build environment, in the field packages of the build. A rebuild is only expected to be identical if the same versions are installed again.

WHAT IS NOT REPRODUCIBLE YET

As of today, two builds of the same source package with the same SOURCE_DATE_EPOCH in the same environment still differ in the places below. In a test with a small C program, the files that the build created were identical and only the metadata differed. That does not mean that the files of a larger package are identical.

  • The id of the build in the metadata. It is chosen for every build and it is meant to be different for every build. Since the UUID of the package is derived from it, the UUID differs, too. --build-id gives the build an ID of your choice, and then they are the same (see below).

  • The host that the package has been built on. The UUID of the package is derived from it, too. --build-host says what the package is told, so that building the same package on another machine can make the same one (see below).

  • The jail shows the number of processors of the build machine. Anything that is written into a file because of it (for example the number of parallel jobs) differs between machines.

  • The same source package is only reproducible if the source package itself is. Source packages are written in the same way, so they differ in the same places.

WHAT A PACKAGE HAS TO DO

Pakfire cannot know which of the things that a program puts into its output are meant to be there. The following helps:

  • Use SOURCE_DATE_EPOCH instead of the current time wherever the time has to be stored. Many tools support this directly.

  • Never store the path of the build directory, the name of the host, the name of the user, or the number of CPUs in a file. If a compiler would add the path of a source file and CFLAGS does not reach it (for example rustc), tell it to replace the path, e.g. with --remap-path-prefix.

  • Sort whatever is read from a directory, a hash table or the output of find before it is used for something that ends up in the package.

  • Build in parallel only if the result does not depend on which job finishes first.

  • Do not download anything while the package is being built, because a build that fetches something does not build the same package again once it has changed.

HOW TO CHECK A PACKAGE

Build the package twice with the same SOURCE_DATE_EPOCH, build ID and name of the build host, each time into a different directory, and compare what is in the packages. The compressed files are not compared:

$ export SOURCE_DATE_EPOCH=$(date +%s)
$ ID=$(uuidgen)
$ pakfire-builder build --build-id=$ID --build-host=builder --target=/tmp/build-1 package.nm
$ pakfire-builder build --build-id=$ID --build-host=builder --target=/tmp/build-2 package.nm
$ for i in 1 2; do zstd -dc /tmp/build-$i/*.pfm > /tmp/package-$i.tar; done
$ diff <(tar -tvf /tmp/package-1.tar --full-time) \
        <(tar -tvf /tmp/package-2.tar --full-time)
$ cmp /tmp/package-1.tar /tmp/package-2.tar

The first command lists the owners, modes and times of the files, which extracting them would not show.

diffoscope does the same for the files in the archive, and for the contents of the files that are in it, so that it is possible to see where two executables differ. The differences listed in section WHAT IS NOT REPRODUCIBLE YET are to be expected until they have been fixed.

SEE ALSO

pakfire(8), pakfire-archive-format(5), pakfire-bootstrap(7)