Merge pull request 'Small Updates and Clarifications' (#6) from kngako-updates into trunk

Reviewed-on: #6

book.toml

@@ -7,4 +7,4 @@ title = "Cwtch Secure Development Handbook"
 
 [output.html]
 mathjax-support = true
-default-theme = "navy"
+default-theme = "ayu"

src/SUMMARY.md

@@ -1,14 +1,16 @@
 # Summary
 
-- [Overview](./overview.md)
+- [Overview and History](./overview.md)
 - [Risk Model](./risk.md)
+- [Cwtch Technical Basics](./cwtch-overview.md)
 - [Open Questions](./open-questions.md)
-- [Connectivity](./connectivity.md)
-- [Tapir](./tapir.md)
-    - [Packet Format](./packet_format.md)md
-    - [Authentication Protocol](./authentication_protocol.md)
 - [Cwtch Library](./cwtch.md)
+    - [Message Formats](./message_formats.md)
     - [Groups](./groups.md)
+    - [Tapir](./tapir.md)
+      - [Packet Format](./packet_format.md)
+      - [Authentication Protocol](./authentication_protocol.md)
+    - [Connectivity](./connectivity.md)
 - [Cwtch UI](./ui.md)
     - [Profile Encryption & Storage](./profile_encryption_and_storage.md)
     - [Android Service](./android.md)

src/android.md

@@ -1,3 +1,28 @@
 # Android Service
 
-(Currently under active development, will be documented here once it has been merged into trunk)
+[Adapted from: Discreet Log #11: Integrating FFI processes with Android services](https://openprivacy.ca/discreet-log/11-android-ffi-service-integration/)
+
+In addition to needing to make plain ol’ method calls into the Cwtch library, we also need to be able to communicate with (and receive events from) long-running Cwtch goroutines that keep the Tor process running in the background, manage connection and conversation state for all your contacts, and handle a few other monitoring and upkeep tasks as well. This isn’t really a problem on traditionally multitasking desktop operating systems, but on mobile devices running Android we have to contend with shorter sessions, frequent unloads, and network and power restrictions that can vary over time. As Cwtch is intended to be metadata resistant and privacy-centric, we also want to provide notifications without using the Google push notification service.
+
+The solution for long-running network apps like Cwtch is to put our FFI code into an Android Foreground Service. (And no, it’s not lost on me that the code for our backend is placed in something called a ForegroundService.) With a big of finagling, the WorkManager API allows us to create and manage various types of services including ForegroundServices. This turned out to be a great choice for us, as our gomobile FFI handler happened to already be written in Kotlin, and WorkManager allows us to specify a Kotlin coroutine to be invoked as the service.
+
+If you’d like to follow along, our WorkManager specifications are created in the handleCwtch() method of [MainActivity.kt](https://git.openprivacy.ca/cwtch.im/cwtch-ui/src/branch/trunk/android/app/src/main/kotlin/im/cwtch/flwtch/MainActivity.kt), and the workers themselves are defined in [FlwtchWorker.kt](https://git.openprivacy.ca/cwtch.im/cwtch-ui/src/branch/trunk/android/app/src/main/kotlin/im/cwtch/flwtch/FlwtchWorker.kt).
+
+Our plain ol’ method calls to FFI routines are also upgraded to be made as WorkManager work requests, which allows us to conveniently pass the return values back via the result callback.
+
+One initial call (aptly named Start) gets hijacked by FlwtchWorker to become our eventbus loop. Since FlwtchWorker is a coroutine, it’s easy for it to yield and resume as necessary while waiting for events to be generated. Cwtch’s goroutines can then emit events, which will be picked up by FlwtchWorker and dispatched appropriately.
+
+FlwtchWorker’s eventbus loop is not just a boring forwarder. It needs to check for certain message types that affect the Android state; for example, new message events should typically display notifications that the user can click to go to the appropriate conversation window, even when the app isn’t running in the foreground. When the time does come to forward the event to the app, we use LocalBroadcastManager to get the notification to MainActivity.onIntent. From there, we in turn use Flutter MethodChannels to forward the event data from Kotlin into the frontend’s Flutter engine, where the event finally gets parsed by Dart code that updates the UI as necessary.
+
+Messages and other permanent state are stored on disk by the service, so the frontend doesn’t need to be updated if the app isnt open. However, some things (like dates and unread messages) can then lead to desyncs between the front and back ends, so we check for this at app launch/resume to see if we need to reinitialize Cwtch and/or resync the UI state.
+
+Finally, while implementing these services on Android we observed that WorkManager is very good at persisting old enqueued work, to the point that old workers were even being resumed after app reinstalls! Adding calls to pruneWork() helps mitigate this, as long as the app was shut down gracefully and old jobs were properly canceled. This frequently isn’t the case on Android, however, so as an additional mitigation we found it useful to tag the work with the native library directory name:
+
+    private fun getNativeLibDir(): String {
+        val ainfo = this.applicationContext.packageManager.getApplicationInfo(
+                "im.cwtch.flwtch", // Must be app name
+                PackageManager.GET_SHARED_LIBRARY_FILES)
+        return ainfo.nativeLibraryDir
+    }
+
+…then, whenever the app is launched, we cancel any jobs that aren’t tagged with the correct current library directory. Since this directory name changes between app installs, this technique prevents us from accidentally resuming with an outdated service worker.

src/cwtch-overview.md

@@ -0,0 +1,61 @@
+# Cwtch Technical Basics
+
+This page presents a brief technical overview of the Cwtch protocol.
+
+## A Cwtch Profile
+
+Users can create one of more Cwtch Profiles. Each profile generates a random ed25519 keypair compatible with
+ Tor.
+
+In addition to the cryptographic material, a profile also contains a list of Contacts (other Cwtch profile public keys +
+associated data about that profile like nickname and (optionally) historical messages), a list of Groups (containing the group cryptographic material in addition to other associated data like the group nickname and historical messages).
+
+## 2-party conversions: Peer to Peer
+
+![](./BASE_3.png)
+
+For 2 parties to engage in a peer-to-peer conversation both must be online, but only one needs to be reachable via
+their onion service. For the sake of clarity we often label one party the "inbound peer" (the one who hosts the onion service) and the other party the
+"outbound peer" (the one that connects to the onion service).
+
+After connection both parties engage in an authentication protocol which:
+
+* Asserts that each party has access to the private key associated with their public identity.
+* Generates an ephemeral session key used to encrypt all further communication during the session.
+
+This exchange (documented in further detail in [authentication protocol](./authentication_protocol.md)) is *offline deniable*
+i.e. it is possible for any party to forge transcripts of this protocol exchange after the fact, and as such - after the
+fact - it is impossible to definitely prove that the exchange happened at all.
+
+After, the authentication protocol the two parties may exchange messages with each other freely.
+
+## Multi-party conversations: Groups and Peer to Server Communication
+
+**Note: Metadata Resistant Group Communication is still an active research area and what is documented here
+will likely change in the future.**
+
+When a person wants to start a group conversation they first randomly generate a secret `Group Key`. All group communication will be encrypted using this key.
+
+Along with the `Group Key`, the group creator also decides on a **Cwtch Server** to use as the host of the group.
+For more information on how Servers authenticate themselves see [key bundles](./key_bundles.md).
+
+A `Group Identifier` is generated using the group key and the group server and these three elements are packaged up
+into an invite that can be sent to potential group members (e.g. over existing peer-to-peer connections).
+
+To send a message to the group, a profile connects to the server hosting the group (see below), and encrypts
+their message using the `Group Key` and generates a cryptographic signature over the `Group Id`, `Group Server`
+and the decrypted message (see: [wire formats](./message_formats.md) for more information).
+
+To receive message from the group, a profile connected to the server hosting the group and downloads *all* messages (since
+their previous connection). Profiles then attempt to decrypt each message using the `Group Key` and if successful attempt
+to verify the signature (see [Cwtch Servers](./server.md)  [Cwtch Groups](./groups.md) for an overview of attacks and mitigations).
+
+### Servers are Peers
+
+In many respects communication with a server is identical to communication with a regular Cwtch peer,
+all the same steps above are taken however the server always acts as the inbound peer, and the outbound
+peer always uses newly generated **ephemeral keypair** as their "longterm identity".
+
+As such peer-server conversations only differ in the *kinds* of messages that are sent between the two parties,
+with the server relaying all messages that it receives and also allowing any client to query for older messages.
+

src/cwtch.md

@@ -37,7 +37,7 @@ version of the protobufs framework.
 We have removed protobufs entirely from the project by porting this
 functionality over the Tapir.
 
-### PoW Spam Prevention as a Metadata Vector
+### Proof of Work (PoW) Spam Prevention as a Metadata Vector
 
 **Status: Outdated**: Cwtch now uses Token Based Services to separate challenges like PoW from
 resolving the tokens. 
@@ -58,7 +58,7 @@ Additionally, Token Based Services and Peer-based Groups are both potential
 
 ## Thread Safety
 
-**Status: Mitigated **
+**Status: Mitigated**
 
 The Cwtch library evolved from a prototype that had weak checks around
 concurrency, and the addition of singleton behavior around saving profiles to

src/groups.md

@@ -57,4 +57,4 @@ chain (which if they were attempting to censor other messages would
  reveal such censorship)
  
 Finally: We are actively working on adding non-repudiation to Cwtch servers such
-that they themselves are restricted in what they can censor efficiently.
+that they themselves are restricted in what they can censor efficiently.

src/input.md

@@ -3,7 +3,7 @@
 
 ## Risk: Interception of Cwtch content or metadata through an IME on Mobile Devices
 
-**Status: Unmitigated**
+**Status: Partially Mitigated**
 
 Any component that has the potential to intercept data between a person, and the Cwtch app is a 
 potential security risk.
@@ -19,4 +19,7 @@ ecosystem.
 
 A similar risk exists on desktop through the use of similar input applications (in addition to software keyloggers),
 however we consider that fully outside the scope of Cwtch risk assessment (in line with other attacks on the security of the underlying 
-operating system itself).
+operating system itself).
+
+This is partially mitigated in Cwtch 1.2 through the use of `enableIMEPersonalizedLearning: false`. See
+[this PR](https://git.openprivacy.ca/cwtch.im/cwtch-ui/pulls/142) for more information.

src/message_formats.md

@@ -0,0 +1,57 @@
+# Message Formats
+
+## Peer to Peer Messages
+
+    PeerMessage {
+        ID      string // A unique Message ID (primarily used for acknowledgments)
+        Context string // A unique context identifier i.e. im.cwtch.chat
+        Data    []byte // The context-dependent serialized data packet.
+    }
+
+### Context Identifiers
+
+* `im.cwtch.raw` - Data contains a plain text chat message (see: [overlays](./overlays.md) for more information)
+* `im.cwtch.acknowledgement` - Data is empty and ID references a previously sent message
+  
+* `im.cwtch.getVal` and `im.cwtch.retVal` - Used for requesting / returning specific information about a peer. Data
+contains a serialized `peerGetVal` structure and `peerRetVal` respectively.
+    
+        peerGetVal struct {
+                Scope string
+                Path string
+        }
+
+        type peerRetVal struct {
+            Val    string // Serialized path-dependent value
+            Exists bool
+        }
+
+## Plaintext / Decrypted Group Messages
+
+    type DecryptedGroupMessage struct {
+        Text      string // plaintext of the message
+        Onion     string // The cwtch address of the sender
+        Timestamp uint64 // A user specified timestamp
+        // NOTE: SignedGroupID is now a misnomer, the only way this is signed is indirectly via the signed encrypted group messages
+        // We now treat GroupID as binding to a server/key rather than an "owner" - additional validation logic (to e.g.
+        // respect particular group constitutions) can be built on top of group messages, but the underlying groups are
+        // now agnostic to those models.
+        SignedGroupID      []byte 
+        PreviousMessageSig []byte // A reference to a previous message
+        Padding            []byte // random bytes of length = 1800 - len(Text)
+    }
+
+DecryptedGroupMessage contains random padding to a fixed size that is equal to the length of all fixed length fields + 1800.
+This ensures that all encrypted group messages are equal length.
+
+## Encrypted Group Messages
+    
+    // EncryptedGroupMessage provides an encapsulation of the encrypted group message stored on the server
+    type EncryptedGroupMessage struct {
+        Ciphertext []byte
+        Signature  []byte // Sign(groupID + group.GroupServer + base64(decrypted group message)) using the senders cwtch key
+    }
+
+Calculating the signature requires knowing the groupID of the message, the server the group is associated with
+and the decrypted group message (and thus, the Group Key). It is (ed25519) signed by the sender of the message, and can be
+verified using their public cwtch address key.

src/overlays.md

@@ -2,6 +2,9 @@
 
 [Adapted from: Discreet Log #8: Notes on the Cwtch Chat API](https://openprivacy.ca/discreet-log/08-chatapi/)
 
+**Note: This section covers overlay protocols on-top of the Cwtch protcol. For information on the Cwtch Protocol
+messages themselves please see [Message Formats](./message_formats.md)**
+
 We envision Cwtch as a platform for providing an authenticated transport layer to higher-level applications. 
 Developers are free to make their own choices about what application layer protocols to use, 
 whether they want bespoke binary message formats or just want to throw an HTTP library on top and call it a 
@@ -17,9 +20,7 @@ And because serialised JSON objects are just dictionaries, we can easily add mor
 ## Chat overlays, lists, and bulletins
 
 The original Cwtch alpha demoed "overlays": different ways of interpreting the same data channel, 
-depending on the structure of the atomic data itself. W
-
-e included simple checklists and BBS/classified ads as overlays that could be viewed 
+depending on the structure of the atomic data itself. We included simple checklists and BBS/classified ads as overlays that could be viewed 
 and shared with any Cwtch contact, be it a single peer or a group. The wire format looked like this:
 
 ```
@@ -64,6 +65,10 @@ Where O stands for `Overlay` with the current supported overlays documented belo
 
 The most simple over is a chat message which simply contains raw, unprocessed chat message information.
 
+```
+{o:1,d:"got milk?"}
+```
+
 ## Invitations (Overlays 100 and 101)
 
 Instead of receiving the invite as an incoming contact request at the profile level, new inline invites are shared with a particular contact/group, where they can be viewed and/or accepted later, even if they were initially rejected (potentially by accident).
@@ -76,3 +81,7 @@ The wire format for these are equally simple:
 ```
 
 This represents a departure from our original "overlays" thinking to a more action-oriented representation. The chat "overlay" can communicate that someone *did* something, even if it's paraphrased down to "added an item to a list," and the lists and bulletins and other beautifully chaotic data can have their state precomputed and stored separately. 
+
+## Lists / Bulletin Boards
+
+**Note: Expected to be Defined in Cwtch Beta 1.5**

src/overview.md

@@ -9,6 +9,20 @@ processes.
 ![](https://docs.openprivacy.ca/cwtch-security-handbook/2.png)
 
 
+## What is Cwtch?
+
+Cwtch (/kʊtʃ/ - a Welsh word roughly translating to “a hug that creates a safe place”) is a decentralized, privacy-preserving, multi-party messaging protocol that can be used to build metadata resistant applications.
+
+* **Decentralized and Open**: There is no “Cwtch service” or “Cwtch network”. Participants in Cwtch can host their own safe spaces, or lend their infrastructure to others seeking a safe space. The Cwtch protocol is open, and anyone is free to build bots, services and user interfaces and integrate and interact with Cwtch.
+* **Privacy Preserving**: All communication in Cwtch is end-to-end encrypted and takes place over Tor v3 onion services.
+* **Metadata Resistant**: Cwtch has been designed such that no information is exchanged or available to anyone without their explicit consent, including on-the-wire messages and protocol metadata.
+
+### A Video Explainer
+
+ <video width="99%" controls>
+   <source src="https://cwtch.im/cwtch-explainer.mp4" type="video/mp4">
+ </video>
+
 ## A (Brief) History of Metadata Resistant Chat
 
 In recent years, public awareness of the need and benefits of end-to-end

src/packet_format.md

@@ -8,5 +8,5 @@ All tapir packets are fixed length (8192 bytes) with the first 2 bytes indicated
 Once encrypted, the entire 8192 byte data packet is encrypted using [libsodium secretbox](https://libsodium.gitbook.io/doc/secret-key_cryptography/secretbox) using the standard structure (
 note in this case the actual usable size of the data packet is 8190-14 to accommodate the nonce included by secret box)
 
-For information on how th secret key is derived see the [authentication protocol](./authentication_protocol.md)
+For information on how the secret key is derived see the [authentication protocol](./authentication_protocol.md)
 

src/profile_encryption_and_storage.md

@@ -1,6 +1,6 @@
 # Profile Encryption & Storage
 
-Profiles are stored on locally on disk and encrypted using a key derived from user-known password (via pbkdf2).
+Profiles are stored locally on disk and encrypted using a key derived from user-known password (via pbkdf2).
 
 Note that once encrypted and stored on disk, the only way to recover a profile is by rederiving the password - as such
 it isn't possible to provide a full list of profiles a user might have access to until they enter a password.

src/risk.md

@@ -14,7 +14,7 @@ use of discardable, untrusted, anonymous infrastructure.
 
 Since then, Cwtch has evolved into a protocol in its own right, this section
 will outline the various known risks that Cwtch attempts to mitigate and will
-nbe heavily referenced throughout the rest of the document when discussing the
+be heavily referenced throughout the rest of the document when discussing the
 various sub-components of the Cwtch Architecture.
 
 ## Threat Model

src/tapir.md

@@ -5,7 +5,7 @@ framework for building anonymous applications.
 
 It is divided into a number of layers:
 
-* Identity - An ed25519 keypair, required for established a Tor v3 onion service
+* Identity - An ed25519 keypair, required to establish a Tor v3 onion service
 and used to maintain a consistent cryptographic identity for a peer.
 * Connections - The raw networking protocol that connects two peers. Connections
 are so far only defined over Tor v3 Onion Services (see: [connectivity](./connectivity.md))
@@ -23,7 +23,7 @@ application.
 
 ### Identity
 
-An ed25519 keypair, required for established a Tor v3 onion service
+An ed25519 keypair, required to establish a Tor v3 onion service
 and used to maintain a consistent cryptographic identity for a peer.
 
 * InitializeIdentity - from a known, persistent keypair:  \\(i,I\\)
@@ -98,25 +98,6 @@ we expect it to be protected via a preceeding app in an `ApplicationChain` e.g.
 
 * No direct testing (tested via integration tests and unit tests)
 
-### Ephemeral Connections
-
-Occasionally it is desirable to have a peer conenct to another / a service
-without using their long term identity (e.g. in the case of connecting to
-a Cwtch Server).
-
-In this case we want to enable a convenient way to allow connecting with an
-ephemeral identity. 
-
-It turns out that doing this securely requires maintaining a completely separate
-set of connections and applications in order to avoid side channel around avoid
-duplicate connections (i.e. if we did mix them up then a service might be able
-to exploit the fact that clients avid duplicate connections by attempting to
-connect to known-online peers and observing if they reject the connection
- because they already have an outbound ephemeral connection open.)
- 
-Because of this, we don't provide an explicit Ephemeral Connect api and instead
-recommend that peers maintain one long term service and multiple ephemeral
- services. 
 
 ## Known Risks  
 
@@ -125,7 +106,7 @@ recommend that peers maintain one long term service and multiple ephemeral
 **Status: Mitigated** 
 
 By default, tor v3 onion services only provide one-way authentication, that
-is the client can verify a metadata resistant connection to the server by the
+is the client can verify a metadata resistant connection to the server but the
 server obtained no information about the client.
 
 Tapir provides a peer-to-peer interface over this client-server structure
@@ -170,6 +151,22 @@ handling of such instances (such as returning an error when they are found
 allowing a handling application to retry the request if a connection with a
 given capability isn't returned)
 
+### Ephemeral Connections
+
+Occasionally it is desirable to have a peer connect to a service
+without using their long term identity (e.g. in the case of connecting to
+a Cwtch Server).
+
+In this case we want to enable a convenient way to allow connecting with an
+**ephemeral identity**.
+
+It turns out that doing this securely requires maintaining a completely separate
+set of connections and applications in order to avoid side channels caused by duplicate connections handling.
+
+As such the Cwtch Protocol Engine maintains two disctinct connection pools, one for avowed connections and another
+for ephemeral connections. All connections to known Cwtch Servers are made through the ephemeral pool.
+
+
 ## Testing Status
 
 Tapir features a number of well-defined integration tests which exercise not