Initial Commit of Repliqate

This commit is contained in:
Sarah Jamie Lewis 2023-01-18 11:39:40 -08:00
commit 7cd0199623
6 changed files with 343 additions and 0 deletions

5
.gitignore vendored Normal file
View File

@ -0,0 +1,5 @@
/target
*.qcow2
*.img
*.script
.idea/

7
Cargo.lock generated Normal file
View File

@ -0,0 +1,7 @@
# This file is automatically @generated by Cargo.
# It is not intended for manual editing.
version = 3
[[package]]
name = "repliqate"
version = "0.1.0"

8
Cargo.toml Normal file
View File

@ -0,0 +1,8 @@
[package]
name = "repliqate"
version = "0.1.0"
edition = "2021"
# See more keys and their definitions at https://doc.rust-lang.org/cargo/reference/manifest.html
[dependencies]

8
LICENSE Normal file
View File

@ -0,0 +1,8 @@
MIT License
Copyright (c) 2022 Open Privacy Research Society
Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions:
The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.

53
README.md Normal file
View File

@ -0,0 +1,53 @@
# repliqate - reproducible builds
Repliqate is a tool for testing and confirming reproducible builds processes based on Qemu, and a Debian Cloud image.
## Step 0: Install Qemu for your System
Qemu is usually available via your package manager, of can be compiled from source.
## Step 1: Obtain a Debian Cloud Image
Official cloud images can be found here: https://cloud.debian.org/images/cloud/ - you will need a `nocloud` to work with Qemu e.g. **debian-11-nocloud-amd64-20221219-1234.qcow2**
### Optional Step: Resize Cloud Images
Depending on how many packages need to be installed as part of your build script you may need to resize the disk size of the provided image e.g. `qemu-img resize debian-11-nocloud-amd64-20221219-1234.qcow2 +10G`
## Step 2: Define a Build Script
A build script sets up dependencies, folder locations, and source code repositories needed to produce a builds. See the [examples](./examples) folder for inspiration.
## Step 3: Run Repliqate
Basic example: `repliqate cloud-img.qcow2 build.script`
## Build Script
By default, each line is interpreted as a command to run in the build environment.
Lines starting with `#` are treated as comments and ignored.
Lines starting with `@%` e.g. `@%echo "Hello World"` indicate to repliqate that it should output the result of the command. This is useful when debugging build steps.
Lines starting with `@!` are treated as repliqate meta commands:
* `@!check <filename> <sha512hash to check>` - executes a `sha512sum` command on the given file, checks the result against the given hash, and terminates the build on failure.
* `@!preserve` - repliqate will skip deleting temporary disk images at shutdown (see FAQ for more information)
## FAQ
Answers to a few possible questions:
### What at the `inuse-*qcow2` and `vd.img` files
These are temporary disk images that are used while repliqate is running. By default, repliqate deletes these once the script has run, however adding the metacommand
`@!preserve` anywhere in the build script will cause repliqate to skip the cleanup - these disk images can then be mounted and explored further (e.g. to debug an issue)
### Why not use Docker?
We do use Docker as part of our continuous build processes. However, we have chosen to develop repliqate to meet two specific requirements:
1. Standlone and Unprivileged - [Docker requires a whole host of permissions](https://docs.docker.com/engine/install/linux-postinstall/) to be granted, and privileged services to be setup, in order for containers to be run by non-root users. Our goal with repliqate was to provide a standalone way of reproducing builds.
2. Redundancy - Having a tool that is entirely separate from our build containers (and the Docker infrastructure that supports them) allows us have both keep each other in check.

262
src/main.rs Normal file
View File

@ -0,0 +1,262 @@
use std::fs::{remove_file, File};
use std::io::{BufRead, Read, Write};
use std::path::Path;
use std::process::{exit, Child, Command, Stdio};
use std::sync::{Arc, Mutex};
use std::thread::sleep;
use std::time::Duration;
use std::{fs, thread};
pub struct QemuProcess {
child: Box<Child>,
stdout: Arc<Mutex<Vec<u8>>>,
len: usize,
}
/// Pipe streams are blocking, we need separate threads to monitor them without blocking the primary thread.
fn child_stream_to_vec<R>(mut stream: R) -> Arc<Mutex<Vec<u8>>>
where
R: Read + Send + 'static,
{
let out = Arc::new(Mutex::new(Vec::new()));
let vec = out.clone();
thread::Builder::new()
.name("child_stream_to_vec".into())
.spawn(move || loop {
let mut buf = [0];
match stream.read(&mut buf) {
Err(err) => {
println!("{}] Error reading from stream: {}", line!(), err);
break;
}
Ok(got) => {
if got == 0 {
break;
} else if got == 1 {
vec.lock().expect("!lock").push(buf[0])
} else {
println!("{}] Unexpected number of bytes: {}", line!(), got);
break;
}
}
}
})
.expect("!thread");
out
}
impl QemuProcess {
pub fn new(path: &str) -> QemuProcess {
let mut child = Command::new("qemu-system-x86_64")
.args([
"-nographic",
"-smp",
"4",
"-m",
"2048",
"-net",
"nic",
"-net",
"user",
"-drive",
format!("if=virtio,format=qcow2,file={}", path).as_str(),
"-drive",
"if=virtio,format=qcow2,file=vd.img",
])
.stdin(Stdio::piped())
.stdout(Stdio::piped())
.spawn()
.unwrap();
let out = child_stream_to_vec(child.stdout.take().expect("!stdout"));
QemuProcess {
child: Box::new(child),
stdout: out,
len: 0,
}
}
pub fn wait_for_shell(&mut self) {
println!("Starting Qemu...\x1b[1;32m OK \x1b[0m");
let child_stdin = self.child.stdin.as_mut().unwrap();
child_stdin.write_all(format!("\n").as_ref()).unwrap();
child_stdin.flush().expect("could not flush qemu stdin");
print!("waiting for vm to start (this can take up to a minute) ...");
std::io::stdout()
.flush()
.expect("could not flush actual stdout");
loop {
let stdout = self.stdout.lock().unwrap().len();
if self.len > 0 && self.len == stdout {
println!("\x1b[1;32m OK \x1b[0m");
child_stdin.write_all(format!("root\n").as_ref()).unwrap();
child_stdin.flush().expect("could not flush qemu input");
break;
}
sleep(Duration::from_secs(5));
self.len = stdout;
print!(".");
std::io::stdout()
.flush()
.expect("could not flush actual stdout");
}
}
pub fn read_until_shell(&mut self, output: bool) -> Vec<String> {
sleep(Duration::from_millis(500));
loop {
{
let stdout = self.stdout.lock().unwrap();
let tbd =
String::from_utf8(stdout.as_slice()[self.len..stdout.len()].to_vec()).unwrap();
let parts: Vec<String> = tbd.split("\n").map(|x| String::from(x)).collect();
let last = &parts[parts.len() - 1];
if last.contains("root@debian:") && last.ends_with("# ") {
let mut cleaned_lines = vec![];
println!(". \x1b[1;32mOK\x1b[0m");
// skip the last element (which is the shell prompt)
// skip the first element (which is the command)
for part in parts.split_last().unwrap().1.iter().skip(1) {
if output {
println!(" {:?}", part);
}
let rparts: Vec<&str> = part.split("\r").collect();
if rparts.len() == 0 {
cleaned_lines.push(part.clone())
} else if rparts.len() == 1 {
// remove end \r
cleaned_lines.push(String::from(rparts[0]))
} else if rparts.len() == 2 {
// remove bracketed paste \u{1b}[?2004l\r2 from start of line
cleaned_lines.push(String::from(rparts[0]))
} else if rparts.len() == 3 {
// remove bracketed paste \u{1b}[?2004l\r2 from start of line
cleaned_lines.push(String::from(rparts[1]))
}
}
self.len = stdout.len();
return cleaned_lines;
} else {
print!(".");
std::io::stdout().flush().expect("could not flush output")
}
}
sleep(Duration::from_millis(2000));
}
}
pub fn execute_command(&mut self, cmd: &str) {
let child_stdin = self.child.stdin.as_mut().unwrap();
child_stdin
.write_all(format!("{}\n", cmd).as_ref())
.unwrap();
child_stdin.flush().expect("could not write");
print!("{} ", cmd);
}
pub fn close(&mut self) {
self.child.kill().expect("could not kill qemu process");
}
}
// The output is wrapped in a Result to allow matching on errors
// Returns an Iterator to the Reader of the lines of the file.
fn read_lines<P>(filename: P) -> std::io::Result<std::io::Lines<std::io::BufReader<File>>>
where
P: AsRef<Path>,
{
let file = File::open(filename)?;
Ok(std::io::BufReader::new(file).lines())
}
fn main() {
let args: Vec<String> = std::env::args().collect();
if args.len() != 3 {
println!("usage: repliqate <cloud-image.qcow2> <build script>");
exit(1);
}
let cloud_image = &args[1];
let build_script = &args[2];
let inuse_cloud_image = format!("inuse-{}", cloud_image);
let _ = remove_file("vd.img");
fs::copy(
cloud_image.as_str(),
inuse_cloud_image.as_str(),
)
.expect("copying of clean image failed...");
Command::new("qemu-img")
.args(["create", "-f", "qcow2", "vd.img", "30G"])
.output()
.expect("error creating secondary harddisk image...");
let mut qemu_process = QemuProcess::new(inuse_cloud_image.as_str());
qemu_process.wait_for_shell();
let mut purge = true;
if let Ok(lines) = read_lines(build_script.as_str()) {
// Consumes the iterator, returns an (Optional) String
for line in lines {
if let Ok(line) = line {
let mut command = line.clone();
let mut output = false;
if line.trim().starts_with("#") {
// comment - skip
continue
}
if line.trim().starts_with("@%") {
command = line.replacen("@%", "", 1);
output = true;
}
if line.trim().starts_with("@!") {
command = line.replacen("@!", "", 1);
let parts: Vec<&str> = command.split(" ").collect();
if parts[0] == "preserve" {
purge = false;
println!("preserve metacommand activated. Virtual Disk Images will not be purged at end of run")
} else if parts[0] == "check" {
if parts.len() != 3 {
println!("check metacommand needs 2 arguments: check file hash")
}
// maybe we need to sleep to let the slow...disk update...
sleep(Duration::from_secs(1));
qemu_process
.execute_command(format!("sha512sum {}", parts[1].trim()).as_str());
let sha512result = qemu_process.read_until_shell(false);
println!("{:?}", sha512result);
if sha512result[0].starts_with(parts[2]) {
println!("confirmed build hash of {}", parts[1])
} else {
println!(
"check hashes do not match {}. {} != {}",
parts[1], parts[2], sha512result[0]
);
break;
}
}
} else {
qemu_process.execute_command(command.as_str());
qemu_process.read_until_shell(output);
}
}
}
}
qemu_process.close();
if purge {
println!("cleaning up...");
let _ = remove_file("vd.img");
let _ = remove_file(inuse_cloud_image.as_str());
}
println!("script finished");
}