Skip to content

README.md

Latest commit

 

History

History
92 lines (70 loc) · 4.09 KB

README.md

File metadata and controls

92 lines (70 loc) · 4.09 KB
Raw

Running a remote cache for tinymold

tinymold's remote tier talks REAPI v2 to an existing cache server (ADR 0006). This directory runs bazel-remote, the reference server, from its published image on a local persistent disk. How the client uses it is in docs/specs/remote.md.

File Use
docker-compose.yml One VM or a developer machine
k8s/bazel-remote.yaml A StatefulSet with a persistent volume, and its Service
k8s/ingress.yaml TLS and a public host name at an ingress-nginx controller

Credentials

bazel-remote reads an htpasswd file. Give CI a user that can write, and keep its password in a repository or organization secret:

htpasswd -cB htpasswd ci          # prompts for the password

The examples start the server with --allow_unauthenticated_reads. Anyone who can reach it can then read the cache, but only the htpasswd users can write to it. Pull requests from forks get no secrets, so they read anonymously. Drop the flag to require a credential for reads too.

Trust. A cache entry decides what a build links. Only trusted CI should hold write credentials, for example pushes to main and same-repository pull requests. tinymold re-hashes every object it fetches, so a remote can only make a build miss, never restore bytes other than the ones the entry names. The entry itself is trusted as much as whoever wrote it.

Signed entries

A credential protects the cache only while the server is set up right. Sign entries too, and a cache that ends up writable by anyone can at worst make builds miss (remote spec):

tinymold remote keygen
# TINYMOLD_REMOTE_SIGNING_KEY=…   secret: trusted writers only (CI on main)
# TINYMOLD_REMOTE_TRUSTED_KEYS=…  public: every reader, pull requests included

tinymold remote check fails against a server that accepts anonymous writes unless TINYMOLD_REMOTE_TRUSTED_KEYS is set, so a missing htpasswd file shows up before anything is pushed.

Network exposure

bazel-remote has no notion of who may connect, only of who may write. Bind it to the narrowest address that works, and firewall the port: on Linux a socket bound to one interface's address still accepts packets for it that arrive on any other interface, such as a bridge full of untrusted VMs.

TLS

The credential is sent with every call. Anything that crosses an untrusted network must use TLS, which means a grpcs:// URL. Two ways to set it up:

  • terminate TLS at an ingress or load balancer that speaks HTTP/2 to the backend, as k8s/ingress.yaml does;
  • or give bazel-remote a certificate itself, with --tls_cert_file and --tls_key_file.

Sizing and expiry

--max_size (GiB) is the server's whole budget. Beyond it, bazel-remote evicts the least recently used blobs, and every hit counts as a use. Entries that builds keep fetching therefore stay, and entries from code that no longer exists age out. bazel-remote stores blobs zstd-compressed: a 340 MiB local tinymold store took 104 MiB on the server.

Give the volume about 10% more room than --max_size. A busy monorepo's debug builds are usually a few GiB per toolchain and lockfile generation.

Pointing tinymold at it

export TINYMOLD_REMOTE_URL=grpcs://cache.example.com   # grpc://host:9092 without TLS
export TINYMOLD_REMOTE_AUTH=ci:password                # omit for anonymous reads
export TINYMOLD_REMOTE_TRUSTED_KEYS=<public key>       # use only signed entries
export TINYMOLD_REMOTE_SIGNING_KEY=<secret key>        # trusted writers only
tinymold remote check                                  # reachable? credential accepted?
RUSTC_WRAPPER=tinymold cargo build                     # misses consult the remote
tinymold remote push                                   # upload what this machine built

In GitHub Actions, use the action's remote backend; see docs/specs/github-actions.md.