cwtch.im
/
docs.cwtch.im
4
0
Fork 3
Code Issues 2 Pull Requests Projects Releases Wiki Activity

Erinn Comments

This commit is contained in:
Sarah Jamie Lewis 2023-01-19 20:01:58 -08:00
parent 1c75001c47
commit 291eb173ae
2 changed files with 41 additions and 28 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

4
blog/2023-01-13-cwtch-stable-api-design.md
View File

@ -190,11 +190,11 @@ All we need to know from CwtchLib is what methods to export to C bindings, and w
As these changes are made, and these goals met we will be posting about them here! Subscribe to our [RSS feed](/blog/rss.xml), [Atom feed](/blog/atom.xml), or [JSON feed](/blog/feed.json) to stay up to date, and get the latest on, all Cwtch development.
## Help us get there!
## Help us go further!
We couldn't do what we do without all the wonderful community support we get, from [one-off donations](https://openprivacy.ca/donate) to [recurring support via Patreon](https://www.patreon.com/openprivacy).
If you want to see us move faster on some of these goals and are in a position, please [donate](https://openprivacy.ca/donate). If you happen to be at a company that wants to do more for the community and this aligns, please consider donating or sponsoring a developer.
If you want to see us move faster on some of these goals and are in a position to, please [donate](https://openprivacy.ca/donate). If you happen to be at a company that wants to do more for the community and this aligns, please consider donating or sponsoring a developer.
Donations of **$5 or more** can opt to receive stickers as a thank-you gift!

65
blog/2023-01-20-reproducible-builds-bindings.md
View File

@ -1,8 +1,8 @@
---
title: Making Cwtch Bindings Reproducible
description: "How Cwtch Bindings are currently built, the changes we have made to Cwtch Bindings to make future distributions verifiable, and the next steps we will be taking to make all Cwtch builds reproducible."
description: "How Cwtch bindings are currently built, the changes we have made to Cwtch bindings to make future distributions verifiable, and the next steps we will be taking to make all Cwtch builds reproducible."
slug: cwtch-bindings-reproducible
tags: [cwtch, cwtch-stable, reproducible-builds, bindings]
tags: [cwtch, cwtch-stable, reproducible-builds, bindings, repliqate]
image: /img/devlog3_small.png
hide_table_of_contents: false
toc_max_heading_level: 4
@ -12,11 +12,13 @@ authors:
image_url: /img/sarah.jpg
---
From the start of the Cwtch project the source code for all components making up Cwtch has been freely available for anyone to inspect, use, and modify.
From the start of the Cwtch project, the source code for all components making up Cwtch has been freely available for anyone to inspect, use, and modify.
But open source code is only one defense against malicious actors who might seek to undermine your privacy and security. This is why, as part of our ongoing Cwtch Stable work, we are working towards making all parts of the Cwtch chain reproducible and verifiable.
In this devlog we will talk about how Cwtch Bindings are currently built, the changes we have made to Cwtch Bindings to make future distributions verifiable, and the next steps we will be taking to make all Cwtch builds reproducible.
The whole point of reproducible builds is that you no longer have to trust binaries provided by the Cwtch Team because you can **independently verify** that the binaries we release are built from the Cwtch source code.
In this devlog we will talk about how Cwtch bindings are currently built, the changes we have made to Cwtch bindings to make future distributions verifiable, and the next steps we will be taking to make all Cwtch builds reproducible. This will be useful to anyone who is looking to reproduce Cwtch bindings specifically, and to anyone who wants to start implementing reproducible builds in their own project.
<!--truncate-->
@ -24,23 +26,24 @@ In this devlog we will talk about how Cwtch Bindings are currently built, the ch
Since we launched Cwtch Beta we have used Docker containers as part of our continuous build process.
When a new change is merged into the repository it kicks off the Cwtch Bindings build pipeline which result in the new source tree being downloaded, inspected, compiled, tested, and eventually packaged for different platforms.
When a new change is merged into the repository it kicks off the Cwtch bindings build pipeline which result in the new source tree being downloaded, inspected, compiled, tested, and eventually packaged for different platforms.
The Cwtch Bindings build pipeline results in four compiled libraries:
- **libcwtch.so** – For Linux Platforms, built using the [official golang:1.19.X Docker Image](https://hub.docker.com/_/golang)
- **libcwtch.dll** – For Windows Platforms, built using our own [mingw-go Docker Image](https://git.openprivacy.ca/openprivacy/mingw-go)
- **libcwtch.ld** – For OSX Platforms, built using dedicated OSX build server (Big Sur 11.6.1)
- **libcwtch.ld** – For OSX Platforms, built using our dedicated OSX build server (Big Sur 11.6.1)
- **cwtch.aar** – For Android Platforms, built using our own [Android/GoMobile Docker Image](https://git.openprivacy.ca/openprivacy/android-go-mobile)
These compiled libraries eventually make their way into Cwtch-based applications, like the Cwtch UI.
## Making libCwtch Reproducible
Docker containers alone aren't enough to guarantee reproducibility. On inspection of several builds of the same source tree,
Docker containers alone aren't enough to guarantee reproducibility. On inspection of several builds of the same source tree,
we noticed a few elements that were distinct to each build:
* **Go Build ID**: By default, Go includes a Build ID as part of compiled binary. When using CGO this build id is non-deterministic and differs for every build. We made the decision to override this build id for all outputs, setting it to the version of the code being built.
* **Go Build ID**: By default, Go includes a build ID as part of compiled binaries. When using CGO this build ID is non-deterministic and differs for every build. We made the decision to override this build ID for all outputs, setting it to the version of the code being built.
* **Build Paths and Go Environment Variables**: By default, Go includes full filesystem paths, and many Go-specific environment variables in the compiled binary – ostensibly to aid with debugging. These can be removed using the `trimPath` option, which we now specify for all bindings builds.
### Linux Specific Considerations
@ -49,21 +52,21 @@ After the general fixes for Go builds are applied, the main variable input that
Our Drone/Docker build environments are based on [Debian Bullseye](https://www.debian.org/releases/bullseye/) which provides [libc6-dev version 2.31](https://packages.debian.org/bullseye/i386/libc6-dev). Other development setups will likely link libc-dev 2.34+.
libc6-dev 2.34 is notable [because it removed dependencies on libpthread and libdl](https://developers.redhat.com/articles/2021/12/17/why-glibc-234-removed-libpthread) – neither are used in libCwtch, but they are currently referenced – this increases the number of sections (and thus the virtual addresses of those sections) defined in the produced ELF file.
libc6-dev 2.34 is notable [because it removed dependencies on libpthread and libdl](https://developers.redhat.com/articles/2021/12/17/why-glibc-234-removed-libpthread) – neither are used in libCwtch, but they are currently referenced – which increases the number of sections (and thus the virtual addresses of those sections) defined in the produced ELF file.
This means that in order to reproduce libCwtch linux bindings it is necessary to have a development environment that will link libc 2.31. We have provided a small, standalone environment which can be used for this purpose (see the section on [Next Steps](#next-steps) for more information).
This means that in order to reproduce libCwtch Linux bindings it is necessary to have a development environment that will link libc 2.31. We have provided a small, standalone environment which can be used for this purpose (see the section on [Next Steps](#next-steps) for more information).
### Windows Specific Considerations
The headers of PE files technically contain a timestamp field. In recent years an [effort has been made to use this field for other purposes](https://devblogs.microsoft.com/oldnewthing/20180103-00/?p=97705), but by default `go build` will still include the timestamp of the file when producing a DLL file (at least one using CGO).
The headers of PE files technically contain a timestamp field. In recent years an [effort has been made to use this field for other purposes](https://devblogs.microsoft.com/oldnewthing/20180103-00/?p=97705), but by default `go build` will still include the timestamp of the file when producing a DLL file (at least when using CGO).
Fortunately this field can be zeroed out through passing `-Xlinker –no-insert-timestamp` into the `mingw32-gcc` process.
With that, and the universal go fixes outlined above, Windows bindings are now reproducible using the same standalone linux environment.
With that, and the universal Go fixes outlined above, Windows bindings are now reproducible using the same standalone Linux environment.
### Android Specific Considerations
With the above universal go fixes, Android builds artifacts become repeatable. However, achieving full reproducibility for Android builds requires a number of specific dependencies:
With the above universal Go fixes, Android build artifacts become repeatable. However, achieving full reproducibility for Android builds requires a number of specific dependencies:
* Cwtch makes use of [GoMobile](https://github.com/golang/mobile) for compiling Android libraries. We pin to a specific version `43a0384520996c8376bfb8637390f12b44773e65` in our Docker containers.
* We still use [sdk-tools](https://developer.android.com/studio/releases/sdk-tools) instead of the new [commandline-tools](https://developer.android.com/studio/command-line). The latest version of sdk-tools is `4333796` and available from: https://dl.google.com/android/repository/sdk-tools-linux-4333796.zip
@ -71,33 +74,43 @@ With the above universal go fixes, Android builds artifacts become repeatable. H
### OSX Specific Considerations
Perhaps surprisingly, OSX builds present the created reproducibility challenge. Unlike Linux, Windows, and Android builds we do not have a Dockerized build environment for OSX builds - relying instead on a dedicated machine to perform the builds.
Perhaps surprisingly, OSX builds present the biggest reproducibility challenge. Unlike Linux, Windows, and Android builds we do not have a Dockerized build environment for OSX builds - relying instead on a dedicated machine to perform the builds.
As with Linux above, the general fixes for setting go build id and trimming paths are enough to ensure repeatability on the same machine.
As with Linux above, the general fixes for setting Go build id and trimming paths are enough to ensure repeatability on the same machine.
In order to fully guarantee reproducibility OSX libraries need to be built on the same version of OSX with the same version of Xcode. For reference our current build system uses: Big Sur 11.6.1 with Xcode version 13.2.1.
In order to fully guarantee reproducibility, OSX libraries need to be built on the same version of OSX with the same version of Xcode. For reference our current build system uses: Big Sur 11.6.1 with Xcode version 13.2.1.
In an ideal world we would be able cross-compile OSX libraries on Linux the same way we do for Windows and Android. While there are no technical limits, compiling for OSX is dependent on [proprietary SDK](https://www.apple.com/legal/sla/docs/xcode.pdf). There is no way to trustfully obtain this SDK from anyone except Apple, and the license appears to strictly prohibit transferring the SDK to non-Apple hardware.
In an ideal world we would be able to cross-compile OSX libraries on Linux the same way we do for Windows and Android. While there are no technical limits, compiling for OSX is dependent on a [proprietary SDK](https://www.apple.com/legal/sla/docs/xcode.pdf). There is no way to trustfully obtain this SDK from anyone except Apple, and the license appears to strictly prohibit transferring the SDK to non-Apple hardware.
We will continue to look for ways to bring OSX builds to the same level as the rest of our distributions.
Because of these limitations we cannot yet offer a way to automatically verify OSX builds, in the same way that we can for Linux, Windows, and Android. We will continue to look for ways to bring OSX builds to the same level as the rest of our distributions.
## Introducing Repliqate!
With all the above changes, Cwtch Bindings are fully reproducible!
That alone is great, but we also want to make it easier for **you** to check the reproducibility of our builds yourself! As we noted in the introduction, the whole point of reproducible builds is that you no longer have to trust binaries provided by the Cwtch Team.
To make this process accessible we are releasing a new tool called [repliqate](https://git.openprivacy.ca/openprivacy/repliqate).
Repliqate makes it easy to construct isolated build environments, powered by Qemu and a standard Debian Cloud Image distribution.
Repliqate runs [build-scripts](https://git.openprivacy.ca/openprivacy/repliqate#writing-a-build-script) to perform actions like downloading the specific versions of Go used in Cwtch official builds, grabbing a copy of the source code for Cwtch bindings, compiling the latest tagged version, and checking the hash against the same version that is available from [builds.openprivacy.ca](https://build.openprivacy.ca/files/).
We now provide [Repliqate build-scripts](https://git.openprivacy.ca/cwtch.im/repliqate-scripts) for checking Linux, Windows, and Android Builds of libCwtch bindings.
You can (and I want to highly encourage you to) perform all these steps yourself (either via Repliqate, or a setup with the same specifications) and report back. We want to know if there are any other barriers to reproducing cwtch bindings, and anything that we can do to make the process easier.
## Next Steps
Reproducible bindings are a big achievement, but there is obviously much more to do. In the coming weeks we are committed to undertaking the same process with our Cwtch UI builds to determine what needs to be done to make this as reproducible as bindings.
We also want to make it easier for you to check the reproducibility of our builds yourself! The whole point of reproducible builds is that you no longer have to trust binaries provided by the Cwtch Team because you can independently verify that the binaries we release are built from the source code that we release.
To make this easier we are releasing instructions on building a canonical testing environment based on a standard Debian Live distribution.
These instructions come in the form of a script that runs a live cd in Qemu, downloads the specific versions of Go used in Cwtch official builds, grabs a copy of the source code for Cwtch bindings, and compiles the latest tagged version, checking the hash against the same version that is fetched from [builds.openprivacy.ca](https://build.openprivacy.ca/). This process can be used to check Linux, Windows and Android Builds.
You can (and I want to highly encourage you to) perform all these steps yourself (either on a Debian live distribution, or a setup with the same specifications) and report back. We want to know if there are any other barriers to reproducing cwtch bindings, and anything that we can do to make the process easier.
As we go through this process we also expect to add additional functionality to Repliqate. If you have any feedback or would like to contribute to Repliqate development then please get in touch!
## Help us go further!
We couldn't do what we do without all the wonderful community support we get, from [one-off donations](https://openprivacy.ca/donate) to [recurring support via Patreon](https://www.patreon.com/openprivacy).
If you want to see us move faster on some of these goals and are in a position, please [donate](https://openprivacy.ca/donate). If you happen to be at a company that wants to do more for the community and this aligns, please consider donating or sponsoring a developer.
If you want to see us move faster on some of these goals and are in a position to, please [donate](https://openprivacy.ca/donate). If you happen to be at a company that wants to do more for the community and this aligns, please consider donating or sponsoring a developer.
Donations of **$5 or more** can opt to receive stickers as a thank-you gift!