Resolves #21 #145

GBHC · 2018-10-15T12:09:30Z

GBHC commented

2018-10-15 12:09:30 +00:00

Adds a Dockerfile and dependencies (server/torrc, server/docker-entrypoint) allowing one to create a docker image containing tor and cwtch.

Both Tor and Cwtch are downloaded and compiled at build time, only the build artifacts are present in the final image.

A .gitlab-ci.yml file is present for compatibility with Gitlab CI pipelines allowing one to automatically push new Cwtch container images to Gitlab registry and Docker Hub.

If using Gitlab the CI Environment variables DOCKER_HUB_ID and DOCKER_HUB_PASSWORD must be present to push to docker hub.

Adds a Dockerfile and dependencies *(server/torrc, server/docker-entrypoint)* allowing one to create a docker image containing tor and cwtch. Both Tor and Cwtch are downloaded and compiled at build time, only the build artifacts are present in the final image. A .gitlab-ci.yml file is present for compatibility with [Gitlab CI pipelines](https://gitlab.com/BrassHornComms/cwtch/pipelines) allowing one to automatically push new Cwtch container images to [Gitlab registry](https://gitlab.com/BrassHornComms/cwtch/container_registry) and [Docker Hub](https://hub.docker.com/r/brasshorncomms/cwtch/). If using Gitlab the CI Environment variables `DOCKER_HUB_ID` and `DOCKER_HUB_PASSWORD` must be present to push to docker hub.

buildbot commented

2018-10-15 12:09:52 +00:00

Drone Build Status: failure

https://build.openprivacy.ca/cwtch.im/cwtch/218

Drone Build Status: failure https://build.openprivacy.ca/cwtch.im/cwtch/218

buildbot commented

2018-10-15 16:59:27 +00:00

Drone Build Status: success

https://build.openprivacy.ca/cwtch.im/cwtch/220

Drone Build Status: success https://build.openprivacy.ca/cwtch.im/cwtch/220

sarah commented

2018-10-15 17:06:49 +00:00

This looks awesome! Assigning @dan to look at it.

dan commented

2018-10-15 18:01:43 +00:00

General: Omg wow, thanks, this is amazing work!

Comments:

.gitlab-ci.yml:
mkdir /go/src/cwtch.im
ln -s /builds/BrassHornComms/cwtch /go/src/cwtch.im/cwtch
cd /go/src/cwtch.im/cwtch/server/app/
go get

This seems a bit specific to one setup, anyway we can generalize this a little more?

Also will this work on a clean system? it seems like you are cding into server/app before go getting the cwtch codebase, might want to cd into cwtch.im/cwtch then go get server/app ? Not sure, just eyeballing this

You will find in testing we have two scripts tests.sh and quality.sh. Tests runs unit tests on the whole code base, and quality does vet and lint. Additionally running go test testing will run our integration test which spins up 3 peers and a servers and confirms they can interact. May want to use those for your tests, not sure?

Dockerfile:

Omg you have it build the latest Tor? That's great!!!
I may borrow that for our build server test images, currently we have it wget-ting a static tor build... -_-; nice work

This may be something we need to work together on: when it launches, does it give any feed back on the .onion address its working on? I believe right now we do have it print it out on boot, but I think we could do a little more to expose it and make it clear. Is it good enough for your uses now?

k8s/cwtch.yml:

image: registry.gitlab.com/brasshorncomms/cwtch:TAGNAME

I am getting a 404 on this and it may be a specific name. I'm unfamiliar with Kubernetics and registry.gitlab.com. What is this? Is this a place to store built images?

So yeah, this looks amazing, just a few this as outlined above I'd like to investigate if we can generalize.
Thanks!

General: Omg wow, thanks, this is amazing work! Comments: .gitlab-ci.yml: mkdir /go/src/cwtch.im ln -s /builds/BrassHornComms/cwtch /go/src/cwtch.im/cwtch cd /go/src/cwtch.im/cwtch/server/app/ go get This seems a bit specific to one setup, anyway we can generalize this a little more? Also will this work on a clean system? it seems like you are cding into server/app before go getting the cwtch codebase, might want to cd into cwtch.im/cwtch then go get server/app ? Not sure, just eyeballing this You will find in `testing` we have two scripts `tests.sh` and `quality.sh`. Tests runs unit tests on the whole code base, and quality does vet and lint. Additionally running `go test testing` will run our integration test which spins up 3 peers and a servers and confirms they can interact. May want to use those for your tests, not sure? Dockerfile: Omg you have it build the latest Tor? That's great!!! I may borrow that for our build server test images, currently we have it wget-ting a static tor build... -_-; nice work This may be something we need to work together on: when it launches, does it give any feed back on the .onion address its working on? I believe right now we do have it print it out on boot, but I think we could do a little more to expose it and make it clear. Is it good enough for your uses now? k8s/cwtch.yml: image: registry.gitlab.com/brasshorncomms/cwtch:TAGNAME I am getting a 404 on this and it may be a specific name. I'm unfamiliar with Kubernetics and registry.gitlab.com. What is this? Is this a place to store built images? --- So yeah, this looks amazing, just a few this as outlined above I'd like to investigate if we can generalize. Thanks!

GBHC commented

2018-10-15 19:59:22 +00:00

Whoops, yes those are both in need of a cleanup.

TAGNAME will 404 as it gets replaced by sed with the commit hash, but again this is probably a bit too specific to my setup so will remove it.

Docker captures STDOUT and STDERR so you would need to issue docker logs CONTAINERNAME to get your address if demonized (-d) or if running in the foreground with docker run cwtch:latest you'd see; https://twitter.com/NetworkString/status/1051808658605006848

Whoops, yes those are both in need of a cleanup. TAGNAME will 404 as it gets [replaced by sed with the commit hash](https://git.openprivacy.ca/cwtch.im/cwtch/pulls/145/files#diff-ecda7ec2ebdc68e2df854b322ff0438f03f06fbR72), but again this is probably a bit too specific to my setup so will remove it. Docker captures STDOUT and STDERR so you would need to issue `docker logs CONTAINERNAME` to get your address if demonized (`-d`) or if running in the foreground with `docker run cwtch:latest` you'd see; https://twitter.com/NetworkString/status/1051808658605006848

GBHC commented

2018-10-15 20:10:39 +00:00

registry.gitlab.com

This is a docker registry built into gitlab.com (github clone) which as you suggest is a place to store built docker images. FWIW we can actually remove .gitlab-ci.yml since you have drone (which I wasn't aware of at the time).

I'm not familiar with drone but one of you could register at hub.docker.com (the defacto docker registry) and take these lines from the .gitlab-ci.yml to build a docker image and push it straight to docker hub;

docker login -u ${DOCKER_HUB_ID} -p ${DOCKER_HUB_PASSWORD} registry.hub.docker.com
docker build -t registry.hub.docker.com/${DOCKER_HUB_ID}/cwtch:latest -t registry.hub.docker.com/${DOCKER_HUB_ID}/cwtch:${CI_COMMIT_SHA:0:8} .
docker push registry.hub.docker.com/${DOCKER_HUB_ID}/cwtch:latest 
docker push registry.hub.docker.com/${DOCKER_HUB_ID}/cwtch:${CI_COMMIT_SHA:0:8}

> registry.gitlab.com This is a [docker registry](https://docs.docker.com/registry/) built into gitlab.com (github clone) which as you suggest is a place to store built docker images. FWIW we can actually remove .gitlab-ci.yml since you have drone (which I wasn't aware of at the time). I'm not familiar with drone but one of you could register at hub.docker.com *(the defacto docker registry)* and take these lines from the .gitlab-ci.yml to build a docker image and push it straight to docker hub; docker login -u ${DOCKER_HUB_ID} -p ${DOCKER_HUB_PASSWORD} registry.hub.docker.com docker build -t registry.hub.docker.com/${DOCKER_HUB_ID}/cwtch:latest -t registry.hub.docker.com/${DOCKER_HUB_ID}/cwtch:${CI_COMMIT_SHA:0:8} . docker push registry.hub.docker.com/${DOCKER_HUB_ID}/cwtch:latest docker push registry.hub.docker.com/${DOCKER_HUB_ID}/cwtch:${CI_COMMIT_SHA:0:8}

GBHC commented

2018-10-15 20:21:18 +00:00

Also will this work on a clean system? it seems like you are cding into server/app before go getting

gitlab spins up a fresh container, checks out the source and then executes the commands, $GOWORKDIR handles everything being pulled into the right place;

Step 1/21 : FROM golang as server-build-stage
Status: Downloaded newer image for golang:latest
 ---> be13d0c67b75
Step 2/21 : ENV CGO_ENABLED=0 GOOS=linux
 ---> Running in 20685f18ada3
Removing intermediate container 20685f18ada3
 ---> 94eb8dba8951
Step 3/21 : WORKDIR /go/src/cwtch.im/cwtch
 ---> Running in fde1ca039812
Removing intermediate container fde1ca039812
 ---> 7f6ed17821a7
Step 4/21 : COPY . .
 ---> 07878ffa2aeb
Step 5/21 : RUN go get -d -v ./...
 ---> Running in 42851325bb61
github.com/yawning/bulb (download)
 Fetching https://golang.org/x/net/proxy?go-get=1
 Parsing meta tags from https://golang.org/x/net/proxy?go-get=1 (status code 200)
get "golang.org/x/net/proxy": found meta tag

I had issues doing go get or go test in any dir other than server/app or client/app but I'm not sure if this is down to my build env or not?

> Also will this work on a clean system? it seems like you are cding into server/app before go getting gitlab spins up a fresh container, checks out the source and then executes the commands, $GOWORKDIR handles everything being pulled into the right place; Step 1/21 : FROM golang as server-build-stage Status: Downloaded newer image for golang:latest ---> be13d0c67b75 Step 2/21 : ENV CGO_ENABLED=0 GOOS=linux ---> Running in 20685f18ada3 Removing intermediate container 20685f18ada3 ---> 94eb8dba8951 Step 3/21 : WORKDIR /go/src/cwtch.im/cwtch ---> Running in fde1ca039812 Removing intermediate container fde1ca039812 ---> 7f6ed17821a7 Step 4/21 : COPY . . ---> 07878ffa2aeb Step 5/21 : RUN go get -d -v ./... ---> Running in 42851325bb61 github.com/yawning/bulb (download) Fetching https://golang.org/x/net/proxy?go-get=1 Parsing meta tags from https://golang.org/x/net/proxy?go-get=1 (status code 200) get "golang.org/x/net/proxy": found meta tag I had issues doing `go get` or `go test` in any dir other than `server/app` or `client/app` but I'm not sure if this is down to my build env or not?

GBHC commented

2018-10-15 20:34:57 +00:00

OK.

So you should be good to checkout this branch and in the root issue;

docker build -t cwtch:latest

If all is successful then;

docker run cwtch:latest

I've pushed this commit to my gitab repo and it has triggered gitlab pipeline https://gitlab.com/BrassHornComms/cwtch/pipelines/33046937 which should result in a new docker image in gitlab and docker hub

At this point you could either configure drone to mimic what's in the .gitlab-ci.yml or signup for gitlab, configure a repository mirror for Gogs and create a Docker Hub organisation for pushing to Docker Hub.

The bonus of pushing your images to docker hub is that people only need to issue docker run openpriv/cwtch (or possibly just docker run cwtch) and it'll work without them needing to do any other config!

OK. So you should be good to checkout this branch and in the root issue; `docker build -t cwtch:latest` If all is successful then; `docker run cwtch:latest` I've pushed this commit to my gitab repo and it has triggered gitlab pipeline https://gitlab.com/BrassHornComms/cwtch/pipelines/33046937 which should result in a new docker image in [gitlab](https://gitlab.com/BrassHornComms/cwtch/container_registry) and [docker hub](https://hub.docker.com/r/brasshorncomms/cwtch/) At this point you could either configure drone to mimic what's in the .gitlab-ci.yml or signup for gitlab, configure a [repository mirror](https://docs.gitlab.com/ee/workflow/repository_mirroring.html) for Gogs and create a Docker Hub organisation for pushing to Docker Hub. The bonus of pushing your images to docker hub is that people only need to issue `docker run openpriv/cwtch` *(or possibly just `docker run cwtch`)* and it'll work without them needing to do any other config!

buildbot commented

2018-10-15 20:35:41 +00:00

Drone Build Status: success

https://build.openprivacy.ca/cwtch.im/cwtch/221

Drone Build Status: success https://build.openprivacy.ca/cwtch.im/cwtch/221

buildbot commented

2018-10-15 20:36:59 +00:00

Drone Build Status: success

https://build.openprivacy.ca/cwtch.im/cwtch/222

Drone Build Status: success https://build.openprivacy.ca/cwtch.im/cwtch/222

dan commented

2018-10-16 22:03:57 +00:00

Drone is out build system (http://docs.drone.io/) and uses docker compose to make images to compile and run tests on

I don't know kubernetics so I dont know how compatible they are, there may be value in having that still around?

We really don't have enough documentation about running Cwtch and now you're adding a bunch of lovely new ways, can you maybe add a new section at the bottom of README.md on how to use these new containers and then we should be good to merge it all!

thanks so much

Drone is out build system (http://docs.drone.io/) and uses docker compose to make images to compile and run tests on I don't know kubernetics so I dont know how compatible they are, there may be value in having that still around? We really don't have enough documentation about running Cwtch and now you're adding a bunch of lovely new ways, can you maybe add a new section at the bottom of README.md on how to use these new containers and then we should be good to merge it all! thanks so much

GBHC commented

2018-10-17 09:24:29 +00:00

Kubernetes is just dockerd on steroids. Underneath it's still docker but with lots of other bits on top (which run as containers) that allows you to 'declare' a configuration (this many containers, running this image, with these values, and a load balancer please) and it goes and does it for you.

In order for a kubernetes manifest to be useful we need to know where to pull the image(s) from.

I'd suggest that once I've added the README changes someone at OpenPrivacy signup to Docker Hub and configure Drone to push the image there ( http://docs.drone.io/publish-docker-images/ seems to be lacking :/ ), once there's a "public" image available I can submit another PR with a k8s manifest?

Kubernetes is just dockerd on steroids. Underneath it's still docker but with lots of other bits on top (which run as containers) that allows you to 'declare' a configuration (this many containers, running this image, with these values, and a load balancer please) and it goes and does it for you. In order for a kubernetes manifest to be useful we need to know where to pull the image(s) from. I'd suggest that once I've added the README changes someone at OpenPrivacy signup to Docker Hub and configure Drone to push the image there ( http://docs.drone.io/publish-docker-images/ seems to be lacking :/ ), once there's a "public" image available I can submit another PR with a k8s manifest?

buildbot commented

2018-10-17 17:26:42 +00:00

Drone Build Status: success

https://build.openprivacy.ca/cwtch.im/cwtch/223

Drone Build Status: success https://build.openprivacy.ca/cwtch.im/cwtch/223

dan closed this pull request

2018-10-17 23:18:27 +00:00

dan commented

2018-10-17 23:18:33 +00:00

awesome!
thanks!
and will do

awesome! thanks! and will do

Sign in to join this conversation.