bindings draft
This commit is contained in:
parent
0527e4e137
commit
1c75001c47
|
|
@ -0,0 +1,107 @@
|
|||
---
|
||||
title: Making Cwtch Bindings Reproducible
|
||||
description: "How Cwtch Bindings are currently built, the changes we have made to Cwtch Bindings to make future distributions verifiable, and the next steps we will be taking to make all Cwtch builds reproducible."
|
||||
slug: cwtch-bindings-reproducible
|
||||
tags: [cwtch, cwtch-stable, reproducible-builds, bindings]
|
||||
image: /img/devlog3_small.png
|
||||
hide_table_of_contents: false
|
||||
toc_max_heading_level: 4
|
||||
authors:
|
||||
- name: Sarah Jamie Lewis
|
||||
title: Executive Director, Open Privacy Research Society
|
||||
image_url: /img/sarah.jpg
|
||||
---
|
||||
|
||||
From the start of the Cwtch project the source code for all components making up Cwtch has been freely available for anyone to inspect, use, and modify.
|
||||
|
||||
But open source code is only one defense against malicious actors who might seek to undermine your privacy and security. This is why, as part of our ongoing Cwtch Stable work, we are working towards making all parts of the Cwtch chain reproducible and verifiable.
|
||||
|
||||
In this devlog we will talk about how Cwtch Bindings are currently built, the changes we have made to Cwtch Bindings to make future distributions verifiable, and the next steps we will be taking to make all Cwtch builds reproducible.
|
||||
|
||||
<!--truncate-->
|
||||
|
||||
## How Cwtch Bindings are Built
|
||||
|
||||
Since we launched Cwtch Beta we have used Docker containers as part of our continuous build process.
|
||||
|
||||
When a new change is merged into the repository it kicks off the Cwtch Bindings build pipeline which result in the new source tree being downloaded, inspected, compiled, tested, and eventually packaged for different platforms.
|
||||
|
||||
The Cwtch Bindings build pipeline results in four compiled libraries:
|
||||
|
||||
- **libcwtch.so** – For Linux Platforms, built using the [official golang:1.19.X Docker Image](https://hub.docker.com/_/golang)
|
||||
- **libcwtch.dll** – For Windows Platforms, built using our own [mingw-go Docker Image](https://git.openprivacy.ca/openprivacy/mingw-go)
|
||||
- **libcwtch.ld** – For OSX Platforms, built using dedicated OSX build server (Big Sur 11.6.1)
|
||||
- **cwtch.aar** – For Android Platforms, built using our own [Android/GoMobile Docker Image](https://git.openprivacy.ca/openprivacy/android-go-mobile)
|
||||
|
||||
These compiled libraries eventually make their way into Cwtch-based applications, like the Cwtch UI.
|
||||
|
||||
## Making libCwtch Reproducible
|
||||
|
||||
Docker containers alone aren't enough to guarantee reproducibility. On inspection of several builds of the same source tree,
|
||||
we noticed a few elements that were distinct to each build:
|
||||
|
||||
* **Go Build ID**: By default, Go includes a Build ID as part of compiled binary. When using CGO this build id is non-deterministic and differs for every build. We made the decision to override this build id for all outputs, setting it to the version of the code being built.
|
||||
* **Build Paths and Go Environment Variables**: By default, Go includes full filesystem paths, and many Go-specific environment variables in the compiled binary – ostensibly to aid with debugging. These can be removed using the `trimPath` option, which we now specify for all bindings builds.
|
||||
|
||||
### Linux Specific Considerations
|
||||
|
||||
After the general fixes for Go builds are applied, the main variable input that impacts reproducibility is the version of libc that the bindings are compiled against.
|
||||
|
||||
Our Drone/Docker build environments are based on [Debian Bullseye](https://www.debian.org/releases/bullseye/) which provides [libc6-dev version 2.31](https://packages.debian.org/bullseye/i386/libc6-dev). Other development setups will likely link libc-dev 2.34+.
|
||||
|
||||
libc6-dev 2.34 is notable [because it removed dependencies on libpthread and libdl](https://developers.redhat.com/articles/2021/12/17/why-glibc-234-removed-libpthread) – neither are used in libCwtch, but they are currently referenced – this increases the number of sections (and thus the virtual addresses of those sections) defined in the produced ELF file.
|
||||
|
||||
This means that in order to reproduce libCwtch linux bindings it is necessary to have a development environment that will link libc 2.31. We have provided a small, standalone environment which can be used for this purpose (see the section on [Next Steps](#next-steps) for more information).
|
||||
|
||||
### Windows Specific Considerations
|
||||
|
||||
The headers of PE files technically contain a timestamp field. In recent years an [effort has been made to use this field for other purposes](https://devblogs.microsoft.com/oldnewthing/20180103-00/?p=97705), but by default `go build` will still include the timestamp of the file when producing a DLL file (at least one using CGO).
|
||||
|
||||
Fortunately this field can be zeroed out through passing `-Xlinker –no-insert-timestamp` into the `mingw32-gcc` process.
|
||||
|
||||
With that, and the universal go fixes outlined above, Windows bindings are now reproducible using the same standalone linux environment.
|
||||
|
||||
### Android Specific Considerations
|
||||
|
||||
With the above universal go fixes, Android builds artifacts become repeatable. However, achieving full reproducibility for Android builds requires a number of specific dependencies:
|
||||
|
||||
* Cwtch makes use of [GoMobile](https://github.com/golang/mobile) for compiling Android libraries. We pin to a specific version `43a0384520996c8376bfb8637390f12b44773e65` in our Docker containers.
|
||||
* We still use [sdk-tools](https://developer.android.com/studio/releases/sdk-tools) instead of the new [commandline-tools](https://developer.android.com/studio/command-line). The latest version of sdk-tools is `4333796` and available from: https://dl.google.com/android/repository/sdk-tools-linux-4333796.zip
|
||||
* Finally, official Cwtch Android builds use OpenJDK 8.
|
||||
|
||||
### OSX Specific Considerations
|
||||
|
||||
Perhaps surprisingly, OSX builds present the created reproducibility challenge. Unlike Linux, Windows, and Android builds we do not have a Dockerized build environment for OSX builds - relying instead on a dedicated machine to perform the builds.
|
||||
|
||||
As with Linux above, the general fixes for setting go build id and trimming paths are enough to ensure repeatability on the same machine.
|
||||
|
||||
In order to fully guarantee reproducibility OSX libraries need to be built on the same version of OSX with the same version of Xcode. For reference our current build system uses: Big Sur 11.6.1 with Xcode version 13.2.1.
|
||||
|
||||
In an ideal world we would be able cross-compile OSX libraries on Linux the same way we do for Windows and Android. While there are no technical limits, compiling for OSX is dependent on [proprietary SDK](https://www.apple.com/legal/sla/docs/xcode.pdf). There is no way to trustfully obtain this SDK from anyone except Apple, and the license appears to strictly prohibit transferring the SDK to non-Apple hardware.
|
||||
|
||||
We will continue to look for ways to bring OSX builds to the same level as the rest of our distributions.
|
||||
|
||||
## Next Steps
|
||||
|
||||
Reproducible bindings are a big achievement, but there is obviously much more to do. In the coming weeks we are committed to undertaking the same process with our Cwtch UI builds to determine what needs to be done to make this as reproducible as bindings.
|
||||
|
||||
We also want to make it easier for you to check the reproducibility of our builds yourself! The whole point of reproducible builds is that you no longer have to trust binaries provided by the Cwtch Team because you can independently verify that the binaries we release are built from the source code that we release.
|
||||
|
||||
To make this easier we are releasing instructions on building a canonical testing environment based on a standard Debian Live distribution.
|
||||
|
||||
These instructions come in the form of a script that runs a live cd in Qemu, downloads the specific versions of Go used in Cwtch official builds, grabs a copy of the source code for Cwtch bindings, and compiles the latest tagged version, checking the hash against the same version that is fetched from [builds.openprivacy.ca](https://build.openprivacy.ca/). This process can be used to check Linux, Windows and Android Builds.
|
||||
|
||||
You can (and I want to highly encourage you to) perform all these steps yourself (either on a Debian live distribution, or a setup with the same specifications) and report back. We want to know if there are any other barriers to reproducing cwtch bindings, and anything that we can do to make the process easier.
|
||||
|
||||
## Help us go further!
|
||||
|
||||
We couldn't do what we do without all the wonderful community support we get, from [one-off donations](https://openprivacy.ca/donate) to [recurring support via Patreon](https://www.patreon.com/openprivacy).
|
||||
|
||||
If you want to see us move faster on some of these goals and are in a position, please [donate](https://openprivacy.ca/donate). If you happen to be at a company that wants to do more for the community and this aligns, please consider donating or sponsoring a developer.
|
||||
|
||||
Donations of **$5 or more** can opt to receive stickers as a thank-you gift!
|
||||
|
||||
For more information about donating to Open Privacy and claiming a thank you gift [please visit the Open Privacy Donate page](https://openprivacy.ca/donate/).
|
||||
|
||||
|
||||
|
||||
Loading…
Reference in New Issue