Timeline storage and wiring #202

dan · 2019-01-29T22:12:48Z

dan commented

2019-01-29 22:12:48 +00:00

created new "SteamStore" that acts like a storage stream for timeline storage

under the hood, it holds ~16kb (as per secret box recommendation) buffer and appends messages to that and saves to file, until full, then rotates files and starts again.

this completes a vast improvement, where as before we were storing all timeslines in one giant profile file which was terrible for performance (every message could trigger a full json serialization of all timelines and the profile) and far agaisnt secret box usages of 16kb chunks

the profile store initializes and manages streamstores for timelines

some basic wiring for events is now included. things should be stored and loaded properly

- created new "SteamStore" that acts like a storage stream for timeline storage under the hood, it holds ~16kb (as per secret box recommendation) buffer and appends messages to that and saves to file, until full, then rotates files and starts again. this completes a vast improvement, where as before we were storing all timeslines in one giant profile file which was terrible for performance (every message could trigger a full json serialization of all timelines and the profile) and far agaisnt secret box usages of 16kb chunks the profile store initializes and manages streamstores for timelines some basic wiring for events is now included. things should be stored and loaded properly

buildbot commented

2019-01-29 22:13:30 +00:00

Drone Build Status: failure

https://build.openprivacy.ca/cwtch.im/cwtch/394

Drone Build Status: failure https://build.openprivacy.ca/cwtch.im/cwtch/394

buildbot commented

2019-01-29 22:22:16 +00:00

Drone Build Status: failure

https://build.openprivacy.ca/cwtch.im/cwtch/395

Drone Build Status: failure https://build.openprivacy.ca/cwtch.im/cwtch/395

buildbot commented

2019-01-30 19:09:38 +00:00

Drone Build Status: success

https://build.openprivacy.ca/cwtch.im/cwtch/396

Drone Build Status: success https://build.openprivacy.ca/cwtch.im/cwtch/396

dan commented

2019-01-30 22:32:07 +00:00

updated with unit tests testing

stream store in isolation. it writes a line and a new one can read it
profile store:
- tests basics like name and attribute save/load
- tests group creation and message saving and loading

important breaking change for ui

there is no message sent when UI creates a group, so storage engine doesn't initialize a storage engine for the group, and when it gets messages for the group that ui created but it didn't initialize for, it seg faults...

updated with unit tests testing - stream store in isolation. it writes a line and a new one can read it - profile store: - tests basics like name and attribute save/load - tests group creation and message saving and loading **important breaking change for ui** there is no message sent when UI creates a group, so storage engine doesn't initialize a storage engine for the group, and when it gets messages for the group that ui created but it didn't initialize for, it seg faults...

buildbot commented

2019-01-30 22:35:24 +00:00

Drone Build Status: success

https://build.openprivacy.ca/cwtch.im/cwtch/397

Drone Build Status: success https://build.openprivacy.ca/cwtch.im/cwtch/397

sarah closed this pull request

2019-01-31 21:34:02 +00:00

sarah commented

2019-01-31 21:34:12 +00:00

Awesome!

Sign in to join this conversation.

No reviewers