Tweaking with Network Protocol while building AWS s3 because I'm sad boy

Use docker Image

docker pull ronakgh97/rdrive:latest (<60 mb)
docker run -d -p 3000:3000 -v rdrive-storage:/home/rdrive/.rdrive/storage --name rdrive ronakgh97/rdrive:latest

then you can use the CLI to push/pull files

rdrive push --file dummy.bin --port 3000 --protocol v1       
Enter a lock key: ronak
↪ Starting upload: dummy.bin (1180000000 bytes)
↪ File hash: ef5bfea558b31b8ecf673a0445ec035394f9a3a40fad69cd8a9ad1c5f5aaf56b...
File ID: 2e8e2c5e-9f36-4369-802f-81d6b7fc0e69 - Time took: 3.350482605

rdrive pull --port 3000 --protocol v1
Enter file ID: 2e8e2c5e-9f36-4369-802f-81d6b7fc0e69
Enter file key: ronak
↩ Downloading: dummy.bin (1180000000 bytes)
Saved to: .\dummy.bin

layering/CAS (WIP)

running 1 test
hash of old layers:
Layer 0: hash=d30cb8c6e507680f667757850464aa41df951d3db917cd849c53098d205eade8, offset=0, size=67108864
Layer 1: hash=7c3c8ffc3a009485d796b2958ca5518a1aea848f009a66afb51c7aabf46f2fb5, offset=67108864, size=67108864
Layer 2: hash=00f3ebdf08b2c200e25b2cf2ea10f7eff9a3cc69986b5a2795de234efbbe9fb7, offset=134217728, size=67108864
Layer 3: hash=16b484824dbd5e232acc48c5225714ebe991a00524763ddc24e0463dccb3cec9, offset=201326592, size=67108864
hash of new layers:
Layer 0: hash=abbf42262a83cfc231444dae845d186df367f89ba736eab2eb3dee3de7cc3585, offset=0, size=67108864
Layer 1: hash=fa6735573375624fa6b197847c647ea68028d7a17121fa5b4c11d9edaab48a98, offset=67108864, size=67108864
Layer 2: hash=d6d958e0d0b247225fc7db068163bcabc841738267d4b1fc66e940646bf7718e, offset=134217728, size=67108864
Layer 3: hash=44bd5c864d8788b797ecd2f81d6ee3f52bed6c9a9353cc6915ec8b3c355f16c6, offset=201326592, size=67108864
Changed layers:
Layer 0: hash=d30cb8c6e507680f667757850464aa41df951d3db917cd849c53098d205eade8, offset=0, size=67108864
Layer 1: hash=7c3c8ffc3a009485d796b2958ca5518a1aea848f009a66afb51c7aabf46f2fb5, offset=67108864, size=67108864
Layer 2: hash=00f3ebdf08b2c200e25b2cf2ea10f7eff9a3cc69986b5a2795de234efbbe9fb7, offset=134217728, size=67108864
Layer 3: hash=16b484824dbd5e232acc48c5225714ebe991a00524763ddc24e0463dccb3cec9, offset=201326592, size=67108864
test layer::experimental_layer_test ... ok

TODO

• Better Encryption for storage and metadata
• Better Bandwidth tracking and limits
• Better Error handling and logging
• Thread pool for better concurrency and resource management
• Better file management and cleanup strategies
• Authentication and access control
• Uhm...what else?
• Fix and improve the buffering and streaming for large files (diff hashing, chunking, chopping, etc.)
• More protocol features like file listing, metadata retrieval, more commands etc.
• Graceful shutdown and cleanup
• Little bit client polish
• Protocol v2 meant to be use UDP, but skill issues...
• Encrypted share feature between clients (stateless relay server) without sharing the master key, maybe using some kind of temporary keys or something, idk
• Migrate to async architecture (TOKIO)
• Too many repetitive code, need to refactor and clean up the codebase
• Still some buffering issues, data gets stalls, does not flush properly
• Multi-port support for better concurrency
• rsync support (rolling hashing, delta transfers, etc.) CDC LAYERING like docker
• Serialized headers, rm fragile parsing
• Add proper user-space (multiple users)
• DO some CAS magic for better storage efficiency and deduplication

ISSUE

• There is a RACE CONDITION
• HASH MISMATCH UNDER HIGH LOAD, FUCK
• Cross upload/donwload protocol corrupts the server somehow, I don't know, this shouldn't happen Server is stateless