4.3 KiB
repliqate - reproducible builds
Repliqate is a tool for testing and confirming reproducible builds processes based on Qemu, and a Debian Cloud image.
Step 0: Install Qemu for your System
Qemu is usually available via your package manager, of can be compiled from source.
Step 1: Obtain a Debian Cloud Image
Official cloud images can be found here: https://cloud.debian.org/images/cloud/ - you will need a nocloud to work with Qemu e.g. debian-11-nocloud-amd64-20221219-1234.qcow2
Optional Step: Resize Cloud Images
Depending on how many packages need to be installed as part of your build script you may need to resize the disk size of the provided image e.g. qemu-img resize debian-11-nocloud-amd64-20221219-1234.qcow2 +10G
Step 2: Define a Build Script
A build script sets up dependencies, folder locations, and source code repositories needed to produce a builds. See the Cwtch repliqate-scripts repository folder for examples and inspiration.
Step 3: Run Repliqate
Basic example: repliqate cloud-img.qcow2 build.script
Writing a Build Script
By default, each line is interpreted as a shell command to run in the build environment e.g. echo "Hello World will result in echo "Hello World" being run in the VM.
Lines starting with # are treated as comments and ignored.
Lines starting with @% e.g. @%echo "Hello World" indicate to repliqate that it should output the result of the command. This is useful when debugging build steps.
Lines starting with @! are treated as repliqate meta commands, these commands extend the execution environment with functionality useful for constructing and checking reproducible builds:
@!check <filename> <sha512hash to check>- executes asha512sumcommand on the given file, checks the result against the given hash, and terminates the build on failure.@!preserve- repliqate will skip deleting temporary disk images at shutdown (see FAQ for more information)@!setup-secondary- repliqate will format and mount the secondary disk image at/mount@!extract <filename>- repliqate will extract the given file from the virtual machine and into the current working directory.netcatneeds to be installed on the virtual machine for this to work (this can be done by addingapt install -y --no-install-recommends netcatto your build script (assuming you have also runapt updatepreviously).
FAQ
Answers to a few possible questions:
What are the inuse-*qcow2 and vd.img files
These are temporary disk images that are used while repliqate is running. By default, repliqate deletes these once the script has run, however adding the metacommand
@!preserve anywhere in the build script will cause repliqate to skip the cleanup - these disk images can then be mounted and explored further (e.g. to debug an issue)
Why not use Docker?
We do use Docker as part of our continuous build processes. However, we have chosen to develop repliqate to meet two specific requirements:
- Standlone and Unprivileged - Our goal with repliqate was to provide a standalone way of reproducing builds. Docker requires a whole host of permissions to be granted, and privileged services to be setup, in order for containers to be run by non-root users.
- Redundancy - Having a tool that is entirely separate from our build containers (and the Docker infrastructure that supports them) allows us have both keep each other in check. Many of the containers we rely on in our build process are managed and maintained by other teams. By maintaining our own reproducible build scripts we can narrow down the exact environments needed while still taking advantage of open source infrastructure for our day-to-day builds.
Builds are slow. Can they be sped up?
Virtualization is inherently slower than running on bare metal. We consider the speed overhead worth it for complete isolation of the build environment. However, if you are considering using repliqate for something more speed-sensitive then you may want to consider enabling and using qemu with kvm enabled.
Note: This is now enabled by default. To disable kvm acceleration see the note in src/main.rs