File Sharing MVP #384

Merged
sarah merged 52 commits from filesharing into master 2021-09-30 00:57:14 +00:00
Owner
No description provided.
dan was assigned by sarah 2021-09-02 18:03:31 +00:00
erinn was assigned by sarah 2021-09-02 18:03:31 +00:00
Member
Drone Build Status: failure https://build.openprivacy.ca/cwtch.im/cwtch/273
sarah force-pushed filesharing from 6d22c4ed67 to d8a12433c9 2021-09-02 18:21:32 +00:00 Compare
Member
Drone Build Status: failure https://build.openprivacy.ca/cwtch.im/cwtch/275
sarah force-pushed filesharing from d8a12433c9 to f4caddbc1b 2021-09-02 19:35:18 +00:00 Compare
sarah added 2 commits 2021-09-02 19:58:26 +00:00
continuous-integration/drone/push Build is pending Details
a2e3f6ee18
File Sharing MVP
continuous-integration/drone/pr Build is pending Details
continuous-integration/drone/push Build is passing Details
603632b206
Merge branch 'filesharing' into origfile5
sarah added 1 commit 2021-09-02 20:01:23 +00:00
continuous-integration/drone/push Build is passing Details
continuous-integration/drone/pr Build is failing Details
c3376e9d3a
Fix merge conflicts
sarah added 1 commit 2021-09-02 20:41:40 +00:00
continuous-integration/drone/pr Build is pending Details
continuous-integration/drone/push Build is passing Details
ade5f944b3
Check maximum manifest size + error flow
sarah added 1 commit 2021-09-02 23:18:58 +00:00
continuous-integration/drone/pr Build is pending Details
continuous-integration/drone/push Build is passing Details
9bff4cef26
Update Overlay Message to use Explictly Hex Encoded Strings
sarah added 1 commit 2021-09-02 23:56:13 +00:00
continuous-integration/drone/pr Build is pending Details
continuous-integration/drone/push Build is passing Details
5ba1410a3b
Bugfix for larger manifests
sarah added 1 commit 2021-09-02 23:57:11 +00:00
continuous-integration/drone/pr Build is pending Details
continuous-integration/drone/push Build is passing Details
b2bb520fea
Manifest Chunk ID Fix
sarah added 1 commit 2021-09-03 00:01:26 +00:00
continuous-integration/drone/pr Build is pending Details
continuous-integration/drone/push Build is passing Details
c373f68fbc
Manifest fix
sarah added 1 commit 2021-09-03 18:26:58 +00:00
continuous-integration/drone/pr Build is pending Details
continuous-integration/drone/push Build is passing Details
d91d5ffdd3
Streamline sending peer messages and unify error handling
sarah added 1 commit 2021-09-03 19:02:54 +00:00
continuous-integration/drone/pr Build is pending Details
continuous-integration/drone/push Build is passing Details
edd130b56f
Add goroutine checking to integration test
sarah added 1 commit 2021-09-03 19:29:09 +00:00
continuous-integration/drone/pr Build is pending Details
continuous-integration/drone/push Build is passing Details
85a45a27f6
Report deserialization error...
sarah added 1 commit 2021-09-03 19:31:02 +00:00
continuous-integration/drone/pr Build is pending Details
continuous-integration/drone/push Build is passing Details
76175b48a3
Report casting errors
sarah added 1 commit 2021-09-03 19:32:59 +00:00
continuous-integration/drone/push Build is pending Details
continuous-integration/drone/pr Build is pending Details
19088c1734
Report debug of plaintext length
dan reviewed 2021-09-03 19:40:18 +00:00
@ -0,0 +41,4 @@
// DownloadFile given a profile, a conversation handle and a file sharing key, start off a download process
// to downloadFilePath
func (f *Functionality) DownloadFile(profile peer.CwtchPeer, handle string, downloadFilePath string, key string) {
profile.SetAttribute(attr.GetLocalScope(key), downloadFilePath)
Owner

what's this for?

what's this for?
dan marked this conversation as resolved
@ -0,0 +1,19 @@
package model
// MessageWrapper is the canonical Cwtch overlay wrapper
type MessageWrapper struct {
Owner

do we want to more formally move this into Cwtch core and expose it a bit more, and have cwtch-ui use that directly reather than dupping it there?

do we want to more formally move this into Cwtch core and expose it a bit more, and have cwtch-ui use that directly reather than dupping it there?
Author
Owner

Yes, that is why it is now in model.

Yes, that is why it is now in model.
dan marked this conversation as resolved
@ -0,0 +18,4 @@
type Chunk []byte
// DefaultChunkSize is the default value of a manifest chunk
const DefaultChunkSize = 4096
Owner

this seems a little low? most torrents I see anyways chunk in 1MB to 8mb chunks. dependingly a single jpg would only be like 1-3 chunks. with 4kb instead even a jpg is gonna be a ton of chunks, that may be a lot of needless overhead?

this seems a little low? most torrents I see anyways chunk in 1MB to 8mb chunks. dependingly a single jpg would only be like 1-3 chunks. with 4kb instead even a jpg is gonna be a ton of chunks, that may be a lot of needless overhead?
Author
Owner

We are limited by tapir upper bound here plus the over head of json + encryption. We may increse the tapir limit at some point but it is currently also bound to server message max size.

We are limited by tapir upper bound here plus the over head of json + encryption. We may increse the tapir limit at some point but it is currently also bound to server message max size.
dan marked this conversation as resolved
@ -0,0 +71,4 @@
break
}
hash := sha256.New()
hash.Write(buf[0:n])
Owner

this is inside a loop reading variable n bytes at a time? how can we then keep writing 0:n ? shouldnt it be like 0,k , k,m , m,n etc?

this is inside a loop reading variable n bytes at a time? how can we then keep writing 0:n ? shouldnt it be like 0,k , k,m , m,n etc?
Author
Owner

because the buffer is a fixed size and we read n bytes at a time int the buffer

because the buffer is a fixed size and we read n bytes at a time int the buffer
dan marked this conversation as resolved
@ -0,0 +75,4 @@
rootHash.Write(buf[0:n])
chunkHash := hash.Sum(nil)
chunks = append(chunks, chunkHash)
fileSizeInBytes += uint64(n)
Owner

yeah we inc filesize by n here

yeah we inc filesize by n here
dan marked this conversation as resolved
@ -0,0 +102,4 @@
}
// Seek to Chunk
offset, err := m.openFd.Seek(int64(id*m.ChunkSizeInBytes), 0)
Owner

i'm just picturing thrashing if we're sharing the same file to multiple parties and chunk sizes are 4k. and a lot of lock waiting

i'm just picturing thrashing if we're sharing the same file to multiple parties and chunk sizes are 4k. and a lot of lock waiting
dan marked this conversation as resolved
@ -0,0 +127,4 @@
}
manifest, err := files.CreateManifest("cwtch.out.png")
if hex.EncodeToString(manifest.RootHash) != "8f0ed73bbb30db45b6a740b1251cae02945f48e4f991464d5f3607685c45dcd136a325dab2e5f6429ce2b715e602b20b5b16bf7438fb6235fefe912adcedb5fd" {
Owner

can't we get the compare to value from the pre send file / manifest? so that we dont have to change this code if the ref file ever changes?

can't we get the compare to value from the pre send file / manifest? so that we dont have to change this code if the ref file ever changes?
@ -0,0 +134,4 @@
queueOracle.Shutdown()
app.Shutdown()
acn.Close()
Owner

i think you want a 10-30 sleep here before measuring go routines, as we do in the other test, as close and shutdown could take a few seconds to wind down connections I believe?

i think you want a 10-30 sleep here before measuring go routines, as we do in the other test, as close and shutdown could take a few seconds to wind down connections I believe?
sarah added 1 commit 2021-09-03 19:43:51 +00:00
continuous-integration/drone/push Build is pending Details
continuous-integration/drone/pr Build is pending Details
1938f14837
Debugging value gettings
sarah added 1 commit 2021-09-03 20:03:17 +00:00
continuous-integration/drone/push Build is pending Details
continuous-integration/drone/pr Build is pending Details
cef991a5ef
sync map woes
sarah added 1 commit 2021-09-03 20:07:54 +00:00
continuous-integration/drone/push Build is pending Details
continuous-integration/drone/pr Build is pending Details
c0eb013fcc
error on failed map lookup
sarah added 1 commit 2021-09-03 20:19:36 +00:00
continuous-integration/drone/push Build is pending Details
continuous-integration/drone/pr Build is pending Details
c728987259
more context for failed retval
sarah added 1 commit 2021-09-03 21:12:57 +00:00
continuous-integration/drone/push Build is pending Details
continuous-integration/drone/pr Build is pending Details
8e9ed132d8
Split download progress and verification
sarah added 1 commit 2021-09-03 21:14:50 +00:00
continuous-integration/drone/push Build is pending Details
continuous-integration/drone/pr Build is pending Details
902b759e81
Map logging
sarah added 1 commit 2021-09-03 21:39:42 +00:00
continuous-integration/drone/push Build is pending Details
continuous-integration/drone/pr Build is pending Details
1d3f44a79f
Move state out of peerApp
sarah added 1 commit 2021-09-03 21:42:45 +00:00
continuous-integration/drone/push Build is pending Details
continuous-integration/drone/pr Build is pending Details
963b99b5e6
Fix map return
sarah added 1 commit 2021-09-03 21:47:10 +00:00
continuous-integration/drone/push Build is pending Details
continuous-integration/drone/pr Build is pending Details
120ebe1312
Fixing getVal
sarah added 1 commit 2021-09-03 21:49:52 +00:00
continuous-integration/drone/push Build is pending Details
continuous-integration/drone/pr Build is passing Details
8998025305
Fixing retVal
sarah added 1 commit 2021-09-03 21:59:24 +00:00
continuous-integration/drone/pr Build is failing Details
continuous-integration/drone/push Build is passing Details
62f0cfaad3
Remove all engine state from peerHandle
Member
Drone Build Status: failure https://build.openprivacy.ca/cwtch.im/cwtch/283
Member
Drone Build Status: success https://build.openprivacy.ca/cwtch.im/cwtch/323
Member
Drone Build Status: failure https://build.openprivacy.ca/cwtch.im/cwtch/325
sarah added 1 commit 2021-09-08 18:45:49 +00:00
continuous-integration/drone/push Build is pending Details
continuous-integration/drone/pr Build is pending Details
e038da335e
Deprecated SendMessageToPeer and SendMessageToGroupTracks
New Generic SendMessage function that does basic handle checking and
error reporting
sarah added 1 commit 2021-09-08 20:38:46 +00:00
continuous-integration/drone/push Build is pending Details
continuous-integration/drone/pr Build is failing Details
a28896c293
Upgrade Tapir to v4.5
Member
Drone Build Status: failure https://build.openprivacy.ca/cwtch.im/cwtch/329
sarah added 1 commit 2021-09-09 19:05:33 +00:00
continuous-integration/drone/pr Build was killed Details
continuous-integration/drone/push Build was killed Details
271ecdb8ec
Fixup Token Requests
sarah added 1 commit 2021-09-09 21:50:26 +00:00
continuous-integration/drone/push Build is passing Details
continuous-integration/drone/pr Build is failing Details
9254c9bc07
Upgrade Log to 1.0.3
Member
Drone Build Status: failure https://build.openprivacy.ca/cwtch.im/cwtch/333
sarah added 1 commit 2021-09-09 22:06:32 +00:00
continuous-integration/drone/push Build is passing Details
continuous-integration/drone/pr Build is failing Details
0f09a5ddcf
Maintain Tokens
Member
Drone Build Status: failure https://build.openprivacy.ca/cwtch.im/cwtch/335
sarah added 1 commit 2021-09-09 22:15:38 +00:00
continuous-integration/drone/push Build is passing Details
continuous-integration/drone/pr Build was killed Details
32c57a07ae
Remove explicit MakePayment from SendMessageToGroup
sarah added 1 commit 2021-09-09 22:23:27 +00:00
continuous-integration/drone/push Build is passing Details
continuous-integration/drone/pr Build is failing Details
2dc4325e1f
MakePayment inline to avoid connection contention
Member
Drone Build Status: failure https://build.openprivacy.ca/cwtch.im/cwtch/339
sarah added 1 commit 2021-09-09 22:42:46 +00:00
continuous-integration/drone/push Build is passing Details
continuous-integration/drone/pr Build is failing Details
83bf6559a2
Use a different check for PoWApp casting
Member
Drone Build Status: failure https://build.openprivacy.ca/cwtch.im/cwtch/341
sarah added 1 commit 2021-09-09 23:27:19 +00:00
continuous-integration/drone/pr Build is failing Details
continuous-integration/drone/push Build was killed Details
7682351753
Check for Duplicates...
Member
Drone Build Status: failure https://build.openprivacy.ca/cwtch.im/cwtch/343
sarah added 1 commit 2021-09-09 23:41:39 +00:00
continuous-integration/drone/pr Build is failing Details
continuous-integration/drone/push Build was killed Details
b389a3421a
Fetch more Tokens
Member
Drone Build Status: failure https://build.openprivacy.ca/cwtch.im/cwtch/346
sarah added 1 commit 2021-09-10 16:12:33 +00:00
continuous-integration/drone/push Build is passing Details
continuous-integration/drone/pr Build is failing Details
38dd63abd9
Don't automatically run file sharing integration test (for now)
We either need to reduce the time these tests take or bump up drone timeouts
Member
Drone Build Status: failure https://build.openprivacy.ca/cwtch.im/cwtch/348
sarah added 1 commit 2021-09-10 16:18:33 +00:00
continuous-integration/drone/push Build is passing Details
continuous-integration/drone/pr Build is failing Details
b2646d0f8e
fmt
Member
Drone Build Status: failure https://build.openprivacy.ca/cwtch.im/cwtch/350
sarah added 1 commit 2021-09-10 16:53:18 +00:00
continuous-integration/drone/push Build is passing Details
continuous-integration/drone/pr Build is failing Details
0f71da3fb6
Allow SendMessageToGroups to silently retry prior to failing
Member
Drone Build Status: failure https://build.openprivacy.ca/cwtch.im/cwtch/352
sarah added 1 commit 2021-09-10 17:07:13 +00:00
continuous-integration/drone/push Build is passing Details
continuous-integration/drone/pr Build is failing Details
e2204a52dc
Simplify MakePayment
Member
Drone Build Status: failure https://build.openprivacy.ca/cwtch.im/cwtch/354
sarah added 1 commit 2021-09-10 17:19:03 +00:00
continuous-integration/drone/push Build is passing Details
continuous-integration/drone/pr Build is failing Details
e967f151ee
increased error reporting
Member
Drone Build Status: failure https://build.openprivacy.ca/cwtch.im/cwtch/356
sarah added 1 commit 2021-09-10 18:02:34 +00:00
continuous-integration/drone/push Build is passing Details
continuous-integration/drone/pr Build is failing Details
d20d4b18c2
deep logging
Member
Drone Build Status: failure https://build.openprivacy.ca/cwtch.im/cwtch/358
sarah added 1 commit 2021-09-10 18:23:54 +00:00
continuous-integration/drone/push Build is passing Details
continuous-integration/drone/pr Build is passing Details
e4aac53622
Upgrade Tapir to 0.4.6
Member
Drone Build Status: success https://build.openprivacy.ca/cwtch.im/cwtch/360
sarah changed title from WIP: File Sharing MVP to File Sharing MVP 2021-09-10 18:33:10 +00:00
erinn added 3 commits 2021-09-23 20:47:15 +00:00
Member
Drone Build Status: failure https://build.openprivacy.ca/cwtch.im/cwtch/362
dan approved these changes 2021-09-23 23:34:48 +00:00
dan left a comment
Owner

.drone.yml needs a new step for the fileshare integ test. drone is sad when any one step is over 10 min but over all run time can exceed 10 min. the problem we had before was both integ tests together in 1 step was causing the problem

.drone.yml needs a new step for the fileshare integ test. drone is sad when any one step is over 10 min but over all run time can exceed 10 min. the problem we had before was both integ tests together in 1 step was causing the problem
@ -0,0 +23,4 @@
}
// FunctionalityGate returns contact.Functionality always
func FunctionalityGate(experimentMap map[string]bool) (*Functionality, error) {
Owner

shouldnt FunctionalityGate stuff be defined a level up in /functionality/ ? not in functionality/fileshareing? so it could be reused in the future.

Also we're moving functionality gate up into cwtch then? makes sense. double cehcking, with the assumption of porting libcwtch-go to this then?

shouldnt FunctionalityGate stuff be defined a level up in /functionality/ ? not in functionality/fileshareing? so it could be reused in the future. Also we're moving functionality gate up into cwtch then? makes sense. double cehcking, with the assumption of porting libcwtch-go to this then?
Author
Owner

This is because of golangs (very bad) naming convention. The true name for this struct reads filesharing.FunctionalityGate but if you try to call it FilesharingFunctionalityGate go quality complains because it "stutters".

This is because of golangs (very bad) naming convention. The true name for this struct reads `filesharing.FunctionalityGate` but if you try to call it FilesharingFunctionalityGate go quality complains because it "stutters".
Owner

aaaaah i see. This seems like an interface we could be implementing defined in functionality/? i know it's just convention now but something to codify it to clear it up and make further use quicker easier more obvious for anyone else stepping into the code? or maybe over kill?

aaaaah i see. This seems like an interface we could be implementing defined in functionality/? i know it's just convention now but something to codify it to clear it up and make further use quicker easier more obvious for anyone else stepping into the code? or maybe over kill?
@ -7,3 +15,1 @@
"encoding/base64"
"encoding/json"
"errors"
model3 "cwtch.im/cwtch/protocol/model"
Owner

model3?

model3?
sarah marked this conversation as resolved
@ -481,3 +510,2 @@
}
} else if numtokens < 5 {
// we failed to post, probably because we ran out of tokens... so make a payment
go tokenApp.MakePayment()
Owner

keep making a payment each time? (this can loop 5 times now?) and why put it in a go routine we we sleep right after? any reason not let it by synchronous at least and remove or keep the sleep for back off anyways?

keep making a payment each time? (this can loop 5 times now?) and why put it in a go routine we we sleep right after? any reason not let it by synchronous at least and remove or keep the sleep for back off anyways?
@ -493,2 +530,3 @@
log.Debugf("New message from peer: %v %v", hostname, context)
if context == event.ContextGetVal {
if context == event.ContextAck {
Owner

switch statement?

switch statement?
@ -2,3 +2,3 @@
import (
"cwtch.im/cwtch/event"
model2 "cwtch.im/cwtch/protocol/model"
Owner

model2?

model2?
@ -0,0 +63,4 @@
i := 0
for {
Owner

for ; i >= len(cs) ; {
?

for ; i >= len(cs) ; { ?
@ -0,0 +287,4 @@
buf := make([]byte, m.ChunkSizeInBytes)
chunkI := 0
for {
n, err := reader.Read(buf)
Owner

go's only use of for loops i feel fails it here... for looping this is a bit ugly... so ... not a request, just a evaluation put forward

for n, err := reader.Read(buf) ; err != nil ; n, err = reader.Read(buf) {
...
}
if err != io.EOF {
return err
}

go's only use of for loops i feel fails it here... for looping this is a bit ugly... so ... not a request, just a evaluation put forward for n, err := reader.Read(buf) ; err != nil ; n, err = reader.Read(buf) { ... } if err != io.EOF { return err }
sarah added 1 commit 2021-09-27 22:11:46 +00:00
continuous-integration/drone/push Build is pending Details
continuous-integration/drone/pr Build is pending Details
62c4724ada
Upgrade Tapir
sarah added 1 commit 2021-09-28 21:16:11 +00:00
continuous-integration/drone/pr Build is failing Details
continuous-integration/drone/push Build is passing Details
c45a6f2fd0
Upgrade Tapir and Connectivity
Member
Drone Build Status: failure https://build.openprivacy.ca/cwtch.im/cwtch/366
sarah added 1 commit 2021-09-28 21:17:55 +00:00
continuous-integration/drone/push Build is passing Details
continuous-integration/drone/pr Build is passing Details
f4af0b3c63
API Fixes
Member
Drone Build Status: success https://build.openprivacy.ca/cwtch.im/cwtch/368
sarah added 1 commit 2021-09-29 20:46:40 +00:00
continuous-integration/drone/push Build is pending Details
continuous-integration/drone/pr Build is pending Details
a15dee2b76
Filesharing Integration Test + Drone
sarah added 1 commit 2021-09-29 20:55:18 +00:00
continuous-integration/drone/push Build is passing Details
continuous-integration/drone/pr Build is failing Details
740f3aff34
Merge branch 'master' into filesharing
Member
Drone Build Status: failure https://build.openprivacy.ca/cwtch.im/cwtch/379
sarah added 1 commit 2021-09-29 21:07:28 +00:00
continuous-integration/drone/push Build is passing Details
continuous-integration/drone/pr Build is passing Details
f690825c8d
Integration Test Tor Path
Member
Drone Build Status: success https://build.openprivacy.ca/cwtch.im/cwtch/381
erinn added 1 commit 2021-09-30 00:14:42 +00:00
continuous-integration/drone/push Build is passing Details
continuous-integration/drone/pr Build is passing Details
ae075ed3fe
redact filepath when sending manifests, and add original suggestion for android notifications
Member
Drone Build Status: success https://build.openprivacy.ca/cwtch.im/cwtch/383
sarah merged commit 907a7ca638 into master 2021-09-30 00:57:14 +00:00
sarah referenced this issue from a commit 2021-09-30 00:57:14 +00:00
Sign in to join this conversation.
No description provided.