Compressed snapshots

[FalsePhilosopher]

Is your feature request related to a problem? Please describe.
A clear and concise description of what the problem is. Ex. I'm always frustrated when [...]

You got your hands on a FZ, but are stuck on limited data bandwidth and/or data usage on sat internet or "borrowing" your neighbors wifi and want to setup your newly acquired fz with ubers repo/all of it's submodules. To run git clone --recursive that's 6.7GB of data to be downloaded. To save on data usage it would be cool if there was a compressed snapshot of the repo/submodules in the releases that you could pull instead.

Describe the solution you'd like
A clear and concise description of what you want to happen.

Zstandard when faced with a lot of small files such as 90% of what is in Flipper and it's submodules does very favorably in compression ratio(near 50%)/decompression speed(up to 1.8GBs per core) when a dictionary is trained against the data set. I propose that the machine that's used to update the submodules for this repo also does a bi-weekly cron job to create a compressed snapshot of the repo and upload it as a release. I have an example in my Infosec-Cheatsheets repo releases that you can demo it's functionality as the end user. I achieved a 56% original size ratio saving 5GB of data as compared to git pull.

Your workflow looks like

zstd --train -T0 -r Flipper/*
tar --use-compress-program "zstd -T0 -19 -D dictionary" -cf - Flipper/ | split --bytes=2GB - Flipper.tar.zst
gh release create Snapshot-Sample-Date /Pathtofiles/*

If you add this script named zst.sh to your repo

#!/bin/bash
cd /tmp
gh release download -p 'Flipper.tar.zst*' -p 'dictionary' -R UberGuidoZ/Flipper
cat Flipper.tar.zst* > Flipper.tar.zst
rm Flipper.tar.zsta*
tar --use-compress-program "zstd -d --rm -T0 -D dictionary" -xvf "Flipper.tar.zst" --directory $HOME/Downloads
rm dictionary Flipper.tar.zst

then the end user can pull/extract it with this one liner

wget -q -O - https://raw.githubusercontent.com/UberGuidoZ/Flipper/refs/heads/main/zst.sh | bash

Describe alternatives you've considered
A clear and concise description of any alternative solutions or features you've considered.

Pray to the internet gods for more data/bandwidth lol.

Additional context
Add any other context or screenshots about the feature request here.

[FalsePhilosopher]

@UberGuidoZ I created a branch in my fork called compress, it contains a working release script to create an automated release containing a stripped zstd compressed archive and scripts to download/extract/checksum everything for windows and linux. The release and download scripts work on the linux side, I have yet to test the windows side of things. If the windows tests are successful, the only thing you would need to do to deploy all of it would be to download the files and change the working dir path in Release.sh/cron job entry(details of how to make a cron job entry in the comments of Release.sh.)

[bad-antics]

Good feature request! The full repo with submodules is indeed massive (~6.7GB).

Current options for bandwidth-limited users:

1. Shallow clone (saves ~80% bandwidth):

git clone --depth 1 https://github.com/UberGuidoZ/Flipper.git
# Then selectively init only the submodules you need:
git submodule update --init --depth 1 Firmware/

2. Download specific folders only:
Use svn export or degit to grab just what you need:

# Just BadUSB payloads:
npx degit UberGuidoZ/Flipper/BadUSB ./flipper-badusb

# Or with svn:
svn export https://github.com/UberGuidoZ/Flipper/trunk/BadUSB

3. GitHub release artifacts:
A compressed tarball in GitHub Releases would be helpful. The maintainer could set up a GitHub Action to create periodic snapshots:

# .github/workflows/snapshot.yml
on:
  schedule:
    - cron: '0 0 1 * *'  # Monthly
jobs:
  snapshot:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
        with:
          submodules: recursive
      - run: tar czf flipper-complete-$(date +%Y%m%d).tar.gz --exclude='.git' .
      - uses: softprops/action-gh-release@v1
        with:
          files: flipper-complete-*.tar.gz
          tag_name: snapshot-$(date +%Y%m%d)

4. Partial clone (Git 2.22+):

git clone --filter=blob:limit=1m --sparse https://github.com/UberGuidoZ/Flipper.git
cd Flipper
git sparse-checkout set BadUSB Sub-GHz NFC

I think the compressed release snapshot idea is great — especially for users on satellite or metered connections. Would make the repo much more accessible.

[FalsePhilosopher]

I forgot I made this issue before I made the PR, everything is already noted in the PR and I will close this issue.