Repliqate is a tool for testing and confirming reproducible builds processes based on Qemu, and a Debian Cloud image.
You can not select more than 25 topics Topics must start with a letter or number, can include dashes ('-') and can be up to 35 characters long.
Sarah Jamie Lewis 7045f99aa6
Formatting Fixes
3 weeks ago
src Formatting Fixes 3 weeks ago
.gitignore Initial Commit of Repliqate 3 weeks ago
Cargo.lock Initial Commit of Repliqate 3 weeks ago
Cargo.toml Initial Commit of Repliqate 3 weeks ago
LICENSE Initial Commit of Repliqate 3 weeks ago
README.md Add extact documentation 3 weeks ago
rustfmt.toml Add Support for extract and setup-secondary meta commands 3 weeks ago

README.md

repliqate - reproducible builds

Repliqate is a tool for testing and confirming reproducible builds processes based on Qemu, and a Debian Cloud image.

Step 0: Install Qemu for your System

Qemu is usually available via your package manager, of can be compiled from source.

Step 1: Obtain a Debian Cloud Image

Official cloud images can be found here: https://cloud.debian.org/images/cloud/ - you will need a nocloud to work with Qemu e.g. debian-11-nocloud-amd64-20221219-1234.qcow2

Optional Step: Resize Cloud Images

Depending on how many packages need to be installed as part of your build script you may need to resize the disk size of the provided image e.g. qemu-img resize debian-11-nocloud-amd64-20221219-1234.qcow2 +10G

Step 2: Define a Build Script

A build script sets up dependencies, folder locations, and source code repositories needed to produce a builds. See the Cwtch repliqate-scripts repository folder for examples and inspiration.

Step 3: Run Repliqate

Basic example: repliqate cloud-img.qcow2 build.script

Writing a Build Script

By default, each line is interpreted as a shell command to run in the build environment e.g. echo "Hello World will result in echo "Hello World" being run in the VM.

Lines starting with # are treated as comments and ignored.

Lines starting with @% e.g. @%echo "Hello World" indicate to repliqate that it should output the result of the command. This is useful when debugging build steps.

Lines starting with @! are treated as repliqate meta commands, these commands extend the execution environment with functionality useful for constructing and checking reproducible builds:

  • @!check <filename> <sha512hash to check> - executes a sha512sum command on the given file, checks the result against the given hash, and terminates the build on failure.
  • @!preserve - repliqate will skip deleting temporary disk images at shutdown (see FAQ for more information)
  • @!setup-secondary - repliqate will format and mount the secondary disk image at /mount
  • @!extract <filename> - repliqate will extract the given file from the virtual machine and into the current working directory. netcat needs to be installed on the virtual machine for this to work (this can be done by adding apt install -y --no-install-recommends netcat to your build script (assuming you have also run apt update previously).

FAQ

Answers to a few possible questions:

What are the inuse-*qcow2 and vd.img files

These are temporary disk images that are used while repliqate is running. By default, repliqate deletes these once the script has run, however adding the metacommand @!preserve anywhere in the build script will cause repliqate to skip the cleanup - these disk images can then be mounted and explored further (e.g. to debug an issue)

Why not use Docker?

We do use Docker as part of our continuous build processes. However, we have chosen to develop repliqate to meet two specific requirements:

  1. Standlone and Unprivileged - Our goal with repliqate was to provide a standalone way of reproducing builds. Docker requires a whole host of permissions to be granted, and privileged services to be setup, in order for containers to be run by non-root users.
  2. Redundancy - Having a tool that is entirely separate from our build containers (and the Docker infrastructure that supports them) allows us have both keep each other in check. Many of the containers we rely on in our build process are managed and maintained by other teams. By maintaining our own reproducible build scripts we can narrow down the exact environments needed while still taking advantage of open source infrastructure for our day-to-day builds.

Builds are slow. Can they be sped up?

Virtualization is inherently slower than running on bare metal. We consider the speed overhead worth it for complete isolation of the build environment. However, if you are considering using repliqate for something more speed-sensitive then you may want to consider enabling and using qemu with kvm enabled.