New Storage Refactor #404

Merged
sarah merged 33 commits from p2p-interim-new-storage into master 2021-11-25 23:56:51 +00:00
Owner

Significant refactor for the new storage engine plus laying the ground work for hybrid groups.

  • Refactor old-style encrypted file storage to use encrytped sqlite
  • Moves storage to be the responsibility of Cwtch Peer
  • Redesigns Cwtch Peer interface around the concept of conversations and channels instead of contacts and groups
  • Deprecates Profile and PublicProfile
  • Repurposes Groups as LegacyGroups
  • Significantly alters event bus APIs to output ConversationID instead of RemotePeer or GroupID i.e. "a move towards a hybrid interface"
  • Automatically upgrades profiles from old style storage to new database storage - we do not delete the old profiles yet for safety, that will come in a future PR.
  • Deletes a lot of deprecated code, significantly reduces functionality of storage and profile to the bare minimum needed for import.

There are libcwtch-go, server and ui changes associated with this work.

This change is backwards compatible as far as the network interface goes.

This chaange makes significant enough changes to warrant a major version increase of Cwtch and libCwtch-go

Still some minor todos (see below), but ready for general comments.

TODO:

  • Minor Updates to the Integration Test
  • Wire up the UpdateMessageAttributes API into the UI
Significant refactor for the new storage engine plus laying the ground work for hybrid groups. - Refactor old-style encrypted file storage to use encrytped sqlite - Moves storage to be the responsibility of Cwtch Peer - Redesigns Cwtch Peer interface around the concept of `conversations` and `channels` instead of `contacts` and `groups` - Deprecates `Profile` and `PublicProfile` - Repurposes `Groups` as `LegacyGroups` - Significantly alters event bus APIs to output `ConversationID` instead of `RemotePeer` or `GroupID` i.e. "a move towards a hybrid interface" - Automatically upgrades profiles from old style storage to new database storage - we do not delete the old profiles yet for safety, that will come in a future PR. - Deletes a lot of deprecated code, significantly reduces functionality of storage and profile to the bare minimum needed for import. There are libcwtch-go, server and ui changes associated with this work. This change is backwards compatible as far as the network interface goes. This chaange makes significant enough changes to warrant a major version increase of Cwtch and libCwtch-go Still some minor todos (see below), but ready for general comments. TODO: - [ ] Minor Updates to the Integration Test - [ ] Wire up the `UpdateMessageAttributes` API into the UI
dan was assigned by sarah 2021-11-19 07:53:52 +00:00
erinn was assigned by sarah 2021-11-19 07:53:52 +00:00
sarah added 13 commits 2021-11-19 07:53:53 +00:00
continuous-integration/drone/push Build is pending Details
8c80340a3d
Interim Work - P2P now Works on New Storage Model
continuous-integration/drone/push Build is pending Details
406d900029
First Cut of P2P and Groups using new Storage APIs!
continuous-integration/drone/push Build is passing Details
5c47dd789a
Deleting Unused Profile Code
continuous-integration/drone/push Build is pending Details
e296c30818
libcwtch-go first cut integration / message timelines etc
continuous-integration/drone/push Build is pending Details
cfff858fe1
First cut of Importing Legacy Profiles
continuous-integration/drone/push Build is pending Details
0614d31366
Staticheck Pass
continuous-integration/drone/push Build is pending Details
2caaa7eb87
More Deletions
continuous-integration/drone/push Build is pending Details
continuous-integration/drone/pr Build is pending Details
72ac4099d5
Fixes from Cwtch UI Integration
sarah added 1 commit 2021-11-19 07:59:45 +00:00
continuous-integration/drone/push Build is pending Details
continuous-integration/drone/pr Build is pending Details
dfb4f7c14e
Sent new message total for Group Messages
sarah added 1 commit 2021-11-19 08:06:55 +00:00
continuous-integration/drone/push Build is pending Details
continuous-integration/drone/pr Build is pending Details
c1428762f8
Merge remote-tracking branch 'origin/master' into p2p-interim-new-storage
sarah added 1 commit 2021-11-19 08:09:30 +00:00
continuous-integration/drone/push Build is passing Details
continuous-integration/drone/pr Build is failing Details
cb8960f893
Fixups from merging AddServer PR
sarah requested review from dan 2021-11-19 08:12:12 +00:00
sarah requested review from erinn 2021-11-19 08:12:12 +00:00
Member
Drone Build Status: failure https://build.openprivacy.ca/cwtch.im/cwtch/499
sarah added 1 commit 2021-11-19 19:49:16 +00:00
continuous-integration/drone/push Build is pending Details
continuous-integration/drone/pr Build was killed Details
f1caca3adf
Closing Down Database + Delete Peer
sarah added 1 commit 2021-11-19 20:28:10 +00:00
continuous-integration/drone/push Build is pending Details
continuous-integration/drone/pr Build is pending Details
847b04e4fc
More comments + UpdateMessageAttribute public API
dan reviewed 2021-11-19 20:40:32 +00:00
app/app.go Outdated
@ -147,3 +122,3 @@
defer app.appmutex.Unlock()
if app.storage[onion].CheckPassword(password) {
if app.peers[onion].CheckPassword(password) {
Owner

delete commented out code?

delete commented out code?
dan marked this conversation as resolved
app/app.go Outdated
@ -166,3 +137,4 @@
log.Debugf("Delete peer for %v Done\n", onion)
app.appBus.Publish(event.NewEventList(event.PeerDeleted, event.Identity, onion))
return
Owner

ditto delete

ditto delete
dan marked this conversation as resolved
app/app.go Outdated
@ -166,3 +137,4 @@
log.Debugf("Delete peer for %v Done\n", onion)
app.appBus.Publish(event.NewEventList(event.PeerDeleted, event.Identity, onion))
return
Owner

return will never happen as delete above it

return will never happen as delete above it
dan marked this conversation as resolved
@ -481,53 +64,53 @@ func getRandomness(arr *[]byte) {
}
}
// GenerateRandomID generates a random 16 byte hex id code
Owner

dup? it's also in app

dup? it's also in app
Owner

GenerateRandomID is still dupped in model/profile and app/app

GenerateRandomID is still dupped in model/profile and app/app
dan marked this conversation as resolved
sarah added 1 commit 2021-11-19 22:04:57 +00:00
continuous-integration/drone/push Build is pending Details
continuous-integration/drone/pr Build is failing Details
4f5b1fa106
Fixups for Integration Test
Member
Drone Build Status: failure https://build.openprivacy.ca/cwtch.im/cwtch/505
sarah added 1 commit 2021-11-19 22:10:59 +00:00
continuous-integration/drone/push Build is pending Details
continuous-integration/drone/pr Build is failing Details
824cb3b951
Fixing small lints
Member
Drone Build Status: failure https://build.openprivacy.ca/cwtch.im/cwtch/507
sarah added 1 commit 2021-11-19 22:13:40 +00:00
continuous-integration/drone/push Build is pending Details
continuous-integration/drone/pr Build is failing Details
54e6122af7
Remove outdated tests in test script
Member
Drone Build Status: failure https://build.openprivacy.ca/cwtch.im/cwtch/509
sarah added 1 commit 2021-11-19 22:32:00 +00:00
continuous-integration/drone/push Build is passing Details
continuous-integration/drone/pr Build is failing Details
a4e62fe902
Lock app for CreateTaggedPeer
Member
Drone Build Status: failure https://build.openprivacy.ca/cwtch.im/cwtch/511
sarah added 1 commit 2021-11-19 23:10:39 +00:00
continuous-integration/drone/push Build is pending Details
continuous-integration/drone/pr Build is failing Details
41dbd6da39
Fixing locking in ACN event bus interface
Member
Drone Build Status: failure https://build.openprivacy.ca/cwtch.im/cwtch/513
Member
Drone Build Status: failure https://build.openprivacy.ca/cwtch.im/cwtch/514
erinn approved these changes 2021-11-19 23:48:36 +00:00
@ -110,0 +89,4 @@
func GenerateRandomID() string {
randBytes := make([]byte, 16)
rand.Read(randBytes)
return path.Join(hex.EncodeToString(randBytes))
Owner

uhhh

uhhh
sarah marked this conversation as resolved
app/app.go Outdated
@ -203,0 +168,4 @@
} else { // On failure attempt to load a legacy profile
profileStore, err := storage.LoadProfileWriterStore(profileDirectory, password)
if err != nil {
continue
Owner

log

log
sarah marked this conversation as resolved
app/app.go Outdated
@ -228,0 +200,4 @@
count++
} else {
// Otherwise shutdown the connections
profile.Shutdown()
Owner

why?

why?
sarah marked this conversation as resolved
@ -25,7 +25,7 @@ type Functionality struct {
// FunctionalityGate returns contact.Functionality always
Owner

comment seems wrong lol

comment seems wrong lol
sarah marked this conversation as resolved
@ -0,0 +64,4 @@
// GetAttribute is a helper function that fetches a conversation attribute by scope, zone and key
func (ci *Conversation) GetAttribute(scope attr.Scope, zone attr.Zone, key string) (string, bool) {
if value, exists := ci.Attributes[scope.ConstructScopedZonedPath(zone.ConstructZonedPath(key)).ToString()]; exists {
Owner

this syntax is a little painful

this syntax is a little painful
sarah marked this conversation as resolved
@ -0,0 +75,4 @@
if _, exists := ci.Attributes[attr.LocalScope.ConstructScopedZonedPath(attr.LegacyGroupZone.ConstructZonedPath(constants.GroupID)).ToString()]; exists {
return true
}
return false
Owner

....if true return true, else return false....

....if true return true, else return false....
Author
Owner

I agree, but current linting in quality.sh does not.

I agree, but current linting in quality.sh does not.
sarah marked this conversation as resolved
@ -0,0 +83,4 @@
if _, exists := ci.Attributes[attr.PublicScope.ConstructScopedZonedPath(attr.ServerKeyZone.ConstructZonedPath(string(BundleType))).ToString()]; exists {
return true
}
return false
Owner

"

"
sarah marked this conversation as resolved
@ -0,0 +10,4 @@
func CalculateContentHash(author string, messageBody string) string {
content := []byte(author + messageBody)
contentBasedHash := sha256.Sum256(content)
return base64.StdEncoding.EncodeToString(contentBasedHash[:])
Owner

add timestamp or something? otherwise this generates lotsa duplicates

add timestamp or something? otherwise this generates lotsa duplicates
Author
Owner

we cannot as content hash must be the same for all members of a conversation.

we cannot as content hash must be the same for all members of a conversation.
sarah marked this conversation as resolved
@ -0,0 +50,4 @@
channelGetMessageBySignatureStmts map[ChannelID]*sql.Stmt
channelGetCountStmts map[ChannelID]*sql.Stmt
channelGetMostRecentMessagesStmts map[ChannelID]*sql.Stmt
channelGetMessageByContentHashStmts map[ChannelID]*sql.Stmt
Owner

statements are threadsafe but these maps are not

statements are threadsafe but these maps are not
sarah marked this conversation as resolved
@ -0,0 +141,4 @@
return nil, err
}
acceptConversationStmt, err := db.Prepare(acceptedConversationSQLStmt)
Owner

accept/accepted

accept/accepted
sarah marked this conversation as resolved
@ -0,0 +264,4 @@
return int(conversationID), nil
}
// GetConversationByHandle is a convienance method to fetch an active conversation by a handle
Owner

convenience

convenience
sarah marked this conversation as resolved
@ -0,0 +332,4 @@
}
}
// GetConversation looks up a particular conversation by handle
Owner

by id*

by id*
sarah marked this conversation as resolved
sarah added 1 commit 2021-11-19 23:55:13 +00:00
continuous-integration/drone/push Build is pending Details
continuous-integration/drone/pr Build is passing Details
6101e4e031
Fix Filesharing Integ Test
Member
Drone Build Status: success https://build.openprivacy.ca/cwtch.im/cwtch/516
Owner
  • peer.GetOnion is marked deprecated but no replacement and use still in App. I think we need to offer an alternative before we can mark something "deprecarted". maybe change to "disliked in 1.5" lol
  • ? so we're now kind of making a distinction between data we want to persist that goes into cwtchProfileStorage, and data we don't want to persist, which is so far the stand alone state map. I know it's one thing right now but might we want to stub out a LiveContactData class or something and put connection state in it just to future proof a bit?
  • // Generate a random number and use it as the signature
    signature := event.GetRandNumber().String()
    why?
  • It's unclear to me how we're respecting DeleteHistory[Default|Confirmed] as everything seems to be sent to SQLite and I can't find any code responding to those attributes. as far as I can tell we now store everything always regardless? (might be a second use for the above mentioned "liveData" class, and continuation of the Timeline class, for just in memory use?)
  • with this large refactor, wasn't now the time to drop local.x.name and only use public.x.name for the profile (keeping it for contacts and groups cus its a local setting for them)
  • GetMostRecentMessages since it takes a limit and offset might just be named GetMessages? with default of Most recent if no offset specified, and since go doesnt support default params, we have to specify an offset so it's a bad fit name. We can fake default parameters by reordering limit and offset and having two calls, one with offset and one without if its last param
  • const serverPrefix = "server:" const tofuBundlePrefix = "tofubundle:" const groupPrefix = "torv3" const importBundlePrefix = "importBundle"
    • I don't know if it's good form to bury consts 700 lines deep in a file, maybe move to top?
  • ResyncServer ? todo?
  • GenerateRandomID is still dupped in model/profile and app/app
- [x] peer.GetOnion is marked deprecated but no replacement and use still in App. I think we need to offer an alternative before we can mark something "deprecarted". maybe change to "disliked in 1.5" lol - ? so we're now kind of making a distinction between data we want to persist that goes into cwtchProfileStorage, and data we don't want to persist, which is so far the stand alone state map. I know it's one thing right now but might we want to stub out a LiveContactData class or something and put connection state in it just to future proof a bit? - [x] // Generate a random number and use it as the signature signature := event.GetRandNumber().String() why? - [x] It's unclear to me how we're respecting DeleteHistory[Default|Confirmed] as everything seems to be sent to SQLite and I can't find any code responding to those attributes. as far as I can tell we now store everything always regardless? (might be a second use for the above mentioned "liveData" class, and continuation of the Timeline class, for just in memory use?) - [x] with this large refactor, wasn't now the time to drop local.x.name and only use public.x.name for the profile (keeping it for contacts and groups cus its a local setting for them) - [x] GetMostRecentMessages since it takes a limit and offset might just be named GetMessages? with default of Most recent if no offset specified, and since go doesnt support default params, we have to specify an offset so it's a bad fit name. We can fake default parameters by reordering limit and offset and having two calls, one with offset and one without if its last param - [x] const serverPrefix = "server:" const tofuBundlePrefix = "tofubundle:" const groupPrefix = "torv3" const importBundlePrefix = "importBundle" - I don't know if it's good form to bury consts 700 lines deep in a file, maybe move to top? - [x] ResyncServer ? todo? - [x] GenerateRandomID is still dupped in model/profile and app/app
sarah added 1 commit 2021-11-23 20:17:30 +00:00
continuous-integration/drone/push Build is passing Details
continuous-integration/drone/pr Build is failing Details
6ab11fc929
Purge message history for not-saved conversation on Close + other review comments
Author
Owner

RE: GetOnion - the point of deprecation is that it's an API that shouldn't be assumed to exist in the future. If clients want onion they should get it via the Attributes API.

App currently uses it to detect unique profiles. Ideally in the future it should define a uuid or something, and eventually the ui should phase out ProfileOnion for that uuid. But that is a bigger change that is beyond the scope.

RE: GetOnion - the point of deprecation is that it's an API that shouldn't be assumed to exist in the future. If clients want onion they should get it via the Attributes API. App currently uses it to detect unique profiles. Ideally in the future it should define a uuid or something, and eventually the ui should phase out ProfileOnion for that uuid. But that is a bigger change that is beyond the scope.
Member
Drone Build Status: failure https://build.openprivacy.ca/cwtch.im/cwtch/518
sarah added 1 commit 2021-11-23 20:32:50 +00:00
continuous-integration/drone/push Build is pending Details
continuous-integration/drone/pr Build is passing Details
b45aec6271
Constant Comments
Member
Drone Build Status: success https://build.openprivacy.ca/cwtch.im/cwtch/520
sarah added 1 commit 2021-11-23 22:30:46 +00:00
continuous-integration/drone/push Build is pending Details
continuous-integration/drone/pr Build is pending Details
6e2e67d26f
More efficient content hash fetch
sarah added 1 commit 2021-11-23 22:45:37 +00:00
continuous-integration/drone/push Build is passing Details
continuous-integration/drone/pr Build is failing Details
781f4a919b
Deduplicate Random ID
Member
Drone Build Status: failure https://build.openprivacy.ca/cwtch.im/cwtch/524
sarah added 1 commit 2021-11-23 22:54:15 +00:00
continuous-integration/drone/push Build is pending Details
continuous-integration/drone/pr Build is failing Details
e7191f5d57
comments
Member
Drone Build Status: failure https://build.openprivacy.ca/cwtch.im/cwtch/526
sarah added 1 commit 2021-11-23 23:00:28 +00:00
continuous-integration/drone/pr Build is passing Details
continuous-integration/drone/push Build is passing Details
1b9a9a0b72
quality
Member
Drone Build Status: success https://build.openprivacy.ca/cwtch.im/cwtch/528
sarah added 1 commit 2021-11-25 22:35:22 +00:00
continuous-integration/drone/pr Build is pending Details
continuous-integration/drone/push Build is passing Details
e9f986cc2e
Update Server Attribute. Fix Profile Attribute Updates. Add UNIQUE constraint to type/key in profile attributes
sarah added 1 commit 2021-11-25 23:30:50 +00:00
continuous-integration/drone/push Build is passing Details
continuous-integration/drone/pr Build is failing Details
af8322b734
ContentHash should return offset not count
Member
Drone Build Status: failure https://build.openprivacy.ca/cwtch.im/cwtch/532
sarah added 1 commit 2021-11-25 23:39:31 +00:00
continuous-integration/drone/push Build is pending Details
continuous-integration/drone/pr Build is passing Details
5a3d393472
Purge on Startup + Fix SetSZA eventbus safety
dan approved these changes 2021-11-25 23:45:06 +00:00
Member
Drone Build Status: success https://build.openprivacy.ca/cwtch.im/cwtch/534
sarah changed title from WIP: New Storage Refactor to New Storage Refactor 2021-11-25 23:56:33 +00:00
sarah merged commit bc8a13b707 into master 2021-11-25 23:56:51 +00:00
Sign in to join this conversation.
No description provided.