Erinns Comments

blog/2023-01-13-cwtch-stable-api-design.md

@@ -1,6 +1,6 @@
 ---
 title: Cwtch Stable API Design
-description: "The post outlines the general principles that are guiding the development of Cwtch Stable, the obstacles that prevent a stable Cwtch release, and closes with an overview the next steps and a timeline to tackle them."
+description: "The post outlines the technical changes we are planning on making to the core Cwtch API in preparation for Cwtch Stable "
 slug: cwtch-stable-api-design
 tags: [cwtch, cwtch-stable, planning, api]
 image: /img/devlog1_small.jpg
@@ -27,8 +27,8 @@ In this post we will outline planned new changes to the Cwtch API that realign r
 Over the years we have evolved how we talk about the various parts of the Cwtch ecosystem. To make this document clear we have revised and clarified some terms.
 
 - **Cwtch** refers to the overall ecosystem including all the component libraries, bindings, and the flagship Cwtch application. 
-- **Cwtchlib** refers to the reference implementation of the Cwtch Protocol / Application framework, currently written in Go.
-- **Bindings** refers to C/Java/Kotlin/Rust bindings (primarily libcwtch-go) that act as an interface between Cwtchlib and downstream applications.
+- **Cwtchlib** refers to the [reference implementation of the Cwtch Protocol](https://git.openprivacy.ca/cwtch.im/cwtch) / Application framework, currently written in Go.
+- **Bindings** refers to C/Java/Kotlin/Rust bindings (primarily [libcwtch-go](https://git.openprivacy.ca/cwtch.im/libcwtch-go)) that act as an interface between Cwtchlib and downstream applications.
 - `CwtchPeer` is where the reference Cwtch API is defined. It is responsible for managing the state of a single Cwtch Profile, persistence (e.g. storing messages), and automatically reacting to certain messages like message acknowledgements and providing public profile attributes (e.g. profile display name).
 - `ProtocolEngine` is responsible for maintaining networking resources like listening threads, peer connections, ephemeral server connections. At present, `ProtocolEngine` is also responsible for automatically responding to certain kinds of messages like providing file chunks for shared files.
 
@@ -37,7 +37,7 @@ Over the years we have evolved how we talk about the various parts of the Cwtch
 
 Based on the tenets we have laid out for the Path to Cwtch Stable, we have adopted the following guiding principles for a new API design:
 
-- **Flexibility** -  new features and functionality can be implemented in Cwtch without adding new functions or dependencies to existing Cwtch interfaces.
+- **Robustness** -  new features and functionality can be implemented in Cwtch without adding new functions or dependencies to existing Cwtch interfaces.
 - **Completeness** - all behaviour is either defined in the official library, or explicitly deferred to applications, no special behaviour is implemented by intermediate wrappers.
 - **Security** – experiments should not compromise existing Cwtch functionality - and should be able to be turned on/off at any time without issue.
 
@@ -45,11 +45,11 @@ Based on the tenets we have laid out for the Path to Cwtch Stable, we have adopt
 
 A summary of the experiments that are currently implements or in design, and the changes to the code that were required to support them.
 
-- **Groups** – the very first prototypes of Cwtch were designed around group messaging and, as such, multi-party chats are the most integrated experiment within Cwtch sharing interfaces with P2P chat and requiring specialized Protocol Engine functionality to manage ephemeral connections and antispam tokens, including the introduction of new peer events like NewMessageFromGroup. 
+- **Groups** – the very first prototypes of Cwtch were designed around group messaging and, as such, multi-party chats are the most integrated experiment within Cwtch sharing interfaces with P2P chat and requiring specialized `ProtocolEngine` functionality to manage ephemeral connections and antispam tokens, including the introduction of new peer events like NewMessageFromGroup. 
   - **Hybrid Groups**  - we have plans to upgrade the Groups experience to a more flexible “hybrid-groups” protocol which requires additional custom hook-response that needs to be tightly controlled and isolated from other parts of the system.
 - **Filesharing** – like Groups, Filesharing is a cross-cutting feature that required new APIs, Hooks into Peer Events, and additional capability in the ProtocolEngine itself.
 - **Profile Images** – based on Filesharing and the core get/val functionality, there are only a few small parts of the codebase that are explicitly dedicated to profile images, and these are all event-based reactions that currently reside in the event-decoration module of licwtch-go, but could easily be moved to a standalone module if a hook-based API was available.
-- **Server Hosting** – the only example of an Application-level experiment in Cwch at present. This functionality requires no changes to the cwtchlib module, but is mainly implementing in the libcwtch-go bindings themselves. Ideally this functionality would be moved into a standalone package.
+- **Server Hosting** – the only example of an Application-level experiment in Cwch at present. This functionality requires no changes to the cwtchlib module, but is mainly implemented in the libcwtch-go bindings themselves. Ideally this functionality would be moved into a standalone package.
 - **Message Formatting** – notable as the the main example of a former experimental-functionality that was promoted to an optional feature, but because it is entirely UI based in implementation there are few insights that can be gained from its history
 - **Search / Microblogging** – proposed features that would require database access/changes in order to implement fully and efficiently, any proposed changes to the Cwtch API should allow for the possibility of new functionality at all layers of the Cwtch stack, including storage.
 - **Status / Profile Metadata** – proposed features that only require specific APIs / hooks for saving requested information for the purposes of caching.
@@ -62,15 +62,15 @@ We have also worked to package experimental functionality into so-called **Gated
 
 Together, these form the current basis for implementing to Cwtch features in the official libraries, but they are not without problems:
 
-- The scope of a functionality is rather broad, and can only be passed a complete Cwtch profile or a denoted subset of functionality e.g. `SendMessages` – there is no current way to scope a function to a given conversation, or to a given set of attributes.
-- The implementation of experiments has mostly been delegated to libCwtch-go and, as such, the gating inside CwtchLib is limited, often relying on state to be passed into it by libcwtch-go, or relying on libcwtch-go to explicitly disable the functionality.
-- This lack of ownership over experiments by the official CwtchLib means that libraries based on Cwtch instead of libcwtch-go do not have access to the safeguards provided by libcwtch-go.
+- The scope of a functionality is rather broad, and can only be passed a complete Cwtch profile or a denoted subset of functionality e.g. `SendMessages` – there is no current way to scope a function to a specific conversation, or to a given zone (e.g. filesharing code is technically able to update attributes unrelated to filesharing).
+- The implementation of experiments has mostly been delegated to bindings and, as such, the gating inside CwtchLib is limited, often relying on state to be passed into it by the bindings, or relying on the bindings explicitly disable the functionality.
+- This lack of ownership over experiments by the official CwtchLib means that libraries based on CwtchLib instead of bindings do not have access to the safeguards provided by the bindings.
 
 ### Restricting Powerful Cwtch APIs
 
 To carefully expand Cwtch out using additional experimental APIs we must work to limit the impact further e.g. restricting actions to a given type of conversation, or only executing actions at registered times. To do this we require three separate but related strands of work:
 
-- Assume responsibility for experiments and features in Cwtch itself so that Cwtchlib has direct access to which experiments are enabled at any given time. Doing this allows changes to settings to always flow through Application and, (as currently happens with Anonymous Communication Network (ACN) state), provides a natural point at which to interface those changes into a Cwtch Profile.
+- Assume responsibility for experiments and features in Cwtch itself so that Cwtchlib has direct access to which experiments are enabled at any given time. Doing this allows changes to settings to always flow through `Application` and, (as currently happens with Anonymous Communication Network (ACN) state), provides a natural point at which to interface those changes into a Cwtch Profile.
 - Finer-grained Interfaces that allow restricting actions to preregistered conversation types e.g. a `RestrictedCwtchConversationInterface` which decorates a Cwtch Profile interface such that it can only interact with a single conversation – these can then be passed into hooks and interface functions to limit their impact.
 - Registered Hooks at pre-specified points with restricted capabilities – to allow experimental functionality to register interest in certain events, and act on them at the correct time, and to allow `CwtchPeer` to control which experiments get access to which events at a given time.
 
@@ -86,8 +86,8 @@ We are introducing a new set of Cwtch APIs designed for this purpose:
 - `OnPeerMessageConfirmed` – hooked after a peer message has been inserted into the database.
 - `OnEncryptedGroupMessage` – hooked after receiving an encrypted message from a group server.
 - `OnGroupMessageReceived` – hooked after a successful decryption of a group message, but before inserting it into the database.
-- `OnContactRequestValue` – hooked on request of a scoped,zoned, and keyed value from a peer.
-- `OnContactReceiveValue` – hooked on receipt of a requested scoped,zoned, and keyed  value from a peer.
+- `OnContactRequestValue` – hooked on request of a scoped (the permission level of the attribute e.g. `public` or `conversation` level attributes), zoned ( relating to a specific feature e.g. `filesharing` or `chat`), and keyed (the name of the attribute e.g. `name` or `manifest`) value from a contact.
+- `OnContactReceiveValue` – hooked on receipt of a requested scoped,zoned, and keyed  value from a contact.
 
 Including the following APIs for managing hooked functionality:
 
@@ -95,18 +95,18 @@ Including the following APIs for managing hooked functionality:
 - `RegisterExperiments` - returns a set of experiments that the extension is interested in being notified about
 - `OnEvent` - to be called by `CwtchPeer` whenever an event registered with `RegisterEvents` is called (assuming all experiments registered through `RegisterExperiments` is active)
 
-#### Protocol Engine Subsystems
+#### `ProtocolEngine` Subsystems
 
-As mentioned in our experiment summary, some functionality needs to be implemented directly in the Protocol Engine. The protocol engine is responsible for managing networking clients, and sending/receiving packets from those clients to/from a CwtchPeer (via the event bus).
+As mentioned in our experiment summary, some functionality needs to be implemented directly in the `ProtocolEngine`. The `ProtocolEngine` is responsible for managing networking clients, and sending/receiving packets from those clients to/from a CwtchPeer (via the event bus).
 
-Some types of data are too costly to send over the event bus e.g. requested chunks from shared files, and as such we need to delegate the handling of such data to a Protocol Engine.
+Some types of data are too costly to send over the event bus e.g. requested chunks from shared files, and as such we need to delegate the handling of such data to a `ProtocolEngine`.
 
-At the moment is this done through the concept of informal “subsystems”, modular add-ons to Engine that process certain events. The current informal nature of this design means that there are not hard-and-fast rules regarding what functionality lives in a subsystem, and how subsystems interact with the wider protocol engine ecosystem. 
+At the moment is this done through the concept of informal “subsystems”, modular add-ons to Engine that process certain events. The current informal nature of this design means that there are not hard-and-fast rules regarding what functionality lives in a subsystem, and how subsystems interact with the wider `ProtocolEngine` ecosystem. 
 
 We are formalizing this subsystem into an interface, similar to the hooked functionality in `CwtchPeer`:
 
 - `RegisterEvents`  - returns a set of events that the subsystem needs to consume to operate.
-- `OnEvent` – to be called by `ProtocolEngine` whenever an event registered with `RegisterEvents` is called (assuming all experiments registered through `RegisterExperiments` is active)
+- `OnEvent` – to be called by `ProtocolEngine` whenever an event registered with `RegisterEvents` is called (when all the experiments registered through `RegisterExperiments` are active)
 - `RegisterContexts` - returns the set of contexts that the subsystem implements e.g. `im.cwtch.filesharing`
 
 This also requires a formalization of two *engine specific* events (for use on the event bus):
@@ -120,18 +120,18 @@ And the introduction of three **additional** engine specific events:
 - `StopEngineSubsystem` – replaces subsystem specific stop event mechanisms, can be driven by functionalities to stop all protocol specific handling.
 - `SubsystemStatus` – a generic event that can be published by subsystems with a collection of fields useful for debugging
 
-This will allow us to move the following functionality, currently part of Protocol Engine itself, into generic subsystems:
+This will allow us to move the following functionality, currently part of `ProtocolEngine` itself, into generic subsystems:
 
 - **Attribute Lookup Handling** -  this functionality is currently part of the overloaded `handlePeerMessage` function, filtered using the `Context` parameter of the `CwtchPeerMessage`. As such it can be entirely delegated to a subsystem. 
 - **Filesharing Chunk Request Handling** – this is also part of handlePeerMessage, also filtered using the `Context` parameter, and is already almost entirely implementing in a standalone subsystem (only routing is handled by `handlePeerMessage`)
 - **Filesharing Start File Share/Stop File Share** – this is currently part of the `handleEvent` behaviour of `ProtocolEngine` and can be moved into an `OnEvent` handler of the file sharing subsystem (where such events are already processed).
 
-The introduction of pre-registered hooks in combination with the formalizations of Protocol Engine subsystems will allow the follow functionality, currently implemented in `CwtchPeer` or libcwtch to be moved entirely to additional functionality:
+The introduction of pre-registered hooks in combination with the formalizations of `ProtocolEngine` subsystems will allow the follow functionality, currently implemented in `CwtchPeer` or libcwtch-go to be moved to standalone packages:
 
 - **Filesharing**  makes heavy use of the getval/retval functionality, we can move all of this into a hooked-based functionality extension. 
-  - Filesharing also depends on the file sharing subsystem to be enabled in Engine. This engine is responsible for processing chunk requests. See Registered Engine Subsystems.
-- **Profile Images** – we treat profile images as a specialization of the file sharing function, as such the experiment can operate entirely over apis provided by the filesharing experiment. (Right now this specialization lives in libcwtch as hooks into the relevant functions)
-- **Legacy Groups** – while groups themselves are a first-class consideration for Cwtch, the actual process of constructing and receiving group messages relies heavily on processing of events, or interpreting generic conversation attributes, and as such this functionality can be moved entirely to hooked-based functionality. By doing this we also open the path towards additional group constructions implemented over the same interface.
+  - Filesharing also depends on the file sharing subsystem to be enabled in a `ProtocolEngine`. This subsystem is responsible for processing chunk requests.
+- **Profile Images** – we treat profile images as a specialization of the file sharing function, as such the experiment can operate entirely over apis provided by the filesharing experiment. (Right now this specialization lives in libcwtch-go as hooks into the relevant functions)
+- **Legacy Groups** – while groups themselves are a first-class consideration for Cwtch, the actual process of constructing and receiving group messages relies heavily on processing of events, or interpreting generic conversation attributes, and as such this functionality can be moved entirely to hooked-based functionality. By doing this we also open the path towards introducing new group protocols based on the same interface.
 - **Status/Profile Metadata** – status depends entirely on OnPeerRequestValue / OnPeerReceiveValue  and requires little Cwtch Peer interaction other than saving the result.
  
 #### Impact on Enabling (Powerful) New Functionality
@@ -151,7 +151,7 @@ This functionality also doesn’t belong in the bindings either. They should be
 
 ## Bindings
 
-The last problem to be solved is how to interface experiments with the bindings (libcwtch) and ultimately downstream applications.
+The last problem to be solved is how to interface experiments with the bindings (libcwtch-go) and ultimately downstream applications.
 
 We can split the bindings into four core areas:
 
@@ -164,7 +164,7 @@ The flip side of the bindings is the event bus handing which is responsible for
 
 In an ideal future, all of these bindings could be **generated automatically** from the Cwtchlib interface definitions i.e. there should be no special functionality in the bindings themselves. The generation would need to include C bindings (untyped with automatic checks) and the Dart library calling convention (type safe)
 
-We can define two types of C/Java/Kotlin interface function templates:
+We can define three types of C/Java/Kotlin interface function templates:
 
 - `ProfileMethodName(profilehandle String, args...)` – which directly resolves the Cwtch Profile and calls the function.
 - `ProfileExperimentalMethodName(profilehandle String, args...)`  – which checks the current application settings to see if the experiment is enabled, and then resolves the CwtchProfile and calls the function - else errors.
@@ -176,7 +176,7 @@ All we need to know from CwtchLib is what methods to export to C bindings, and w
 
 - **Freeze any changes to the bindings interface**  - we have made minimal changes to the bindings in the Cwtch 1.9 and 1.10 – until we have implemented the proposed changes into cwtchlib.
 - As part of Cwtch 1.11 and 1.12 Release Cycles
-  - Implement the Protocol Engine Subsystem Design as outlined above.
+  - Implement the `ProtocolEngine` Subsystem Design as outlined above.
   - Implement the Hooks API.
   - Move all special behaviour / extra functionalities in the libcwtch-go bindings into cwtchlib – with the exception of behaviour related to Application Experiments (i.e. Server Hosting).
   - Move event handling from the bindings into Application.
@@ -218,10 +218,10 @@ The following is an exhaustive list of functionality currently provided by libcw
   - UI Server Struct Definition
 - Server Hosting Experiment Functionality - creating/deleting/managing the server hosting experiment for desktop Cwtch clients.
 - "Unencrypted" Profile Handling - replacing a blank password with a default password where the underlying API expects a password but the profile has been designated "unencrypted".
-- Image Previews Experiment Handling (eventbus) - automatically starting the downloading of certain file types (when the experiment is enabled).
+- Image Previews Experiment Handling - automatically starting the downloading of certain file types (when the experiment is enabled).
 - Cwtch UI Reconnection Handling (for Android) - restarting various Cwtch subsystems when the UI attempts to reconnect in circumstances where the Android kernel has killed the underlying process.
-- Cwtch Profile Engine Activation (eventbus) - starting/stopping ProtocolEngines when requested by the UI, or in response to changes in ACN state.
-- UI Profile Aggregation (eventbus) - aggregating information related to Profiles for the UI (e.g. network connection status / unread messages) into a single event.
-- File sharing restarts (eventbus)
+- Cwtch Profile Engine Activation - starting/stopping a `ProtocolEngine` when requested by the UI, or in response to changes in ACN state.
+- UI Profile Aggregation - aggregating information related to Profiles for the UI (e.g. network connection status / unread messages) into a single event.
+- File sharing restarts 
 - UI Event Augmentation - augmenting various internal Cwtch events with information that the UI needs but that isn't directly embedded within the event (e.g. converting `handle` to a `conversation id`). Much of this augmentation is legacy, implemented before recent changes to internal Cwtch structs, and likely can either be removed entirely, or delegated into Cwtch itself.
 - Debug Information - special information available to Cwtch debug builds (memory use / active goroutines etc.)