Skip to content

README.md

Latest commit

 

History

History
244 lines (198 loc) · 15.4 KB

README.md

File metadata and controls

244 lines (198 loc) · 15.4 KB
Raw

source.coop mirror — campaign plan

Mirrors the canonical NRP public-* datasets to Source Cooperative for discoverability, the way catalog/sync/k8s/sync-public-*.yaml mirrors them to MinIO. NRP S3 stays canonical; source.coop is a downstream public copy.

Scope policy (what we mirror)

A bucket/collection is mirrored only if both hold:

  1. Catalogued — reachable from the STAC root catalog (public-data/stac/catalog.json). Raw/uncatalogued buckets are not datasets and are not published.
  2. License-clear — the upstream license permits redistribution by a third party. NC / share-alike are fine (we label them); no-redistribution licenses are not.

Per-collection license verdicts live in license-inventory.md (OK · OK-NC · NO · HOLD). That file + the REPOS/EXCLUDES arrays in gen-source-sync.sh are the single source of truth for scope.

Currently mirrored: 27 repos

Repo name = NRP bucket minus public- (1:1). Two get a new repo while the older differently-versioned legacy repo is left untouched:

NRP bucket source.coop repo legacy repo kept as-is
public-padus (PAD-US 4.1) cboettig/padus cboettig/pad-us-3
public-rivers cboettig/rivers cboettig/us-rivers

Two buckets are mirrored partially — HOLD sub-collections (license unconfirmed) are excluded via per-repo --exclude until terms are confirmed:

repo excluded sub-paths reason
rivers american-rivers/{campaigns,ira-watersheds,roo-cjest} American Rivers org layers, no clear license
high-seas mpa-candidates candidate-MPA source/license unidentified

NOT mirrored

  • License prohibits redistribution → NRP-only: hydrobasins (HydroSHEDS v1c: "no stand-alone redistribution"; re-import v2 on release — issue #223), wdpa (+ WD-OECM), icca (UNEP-WCMC Protected Planet), iucn (IUCN Red List — incl. derived richness/ranges).
  • Not a catalogued dataset / WIP: datacenters, im3 (raw files, no STAC); landfire (WIP, issue #203); ca30x30, wlfw, working-lands (real data but not yet in the STAC catalog — catalogue them first, then add here).
  • tpl — misfiled state-agency data, no clear-license remainder: the only license-clear collection in public-tpl was wcb-approved-projects (CDFW BIOS ds672, CA Wildlife Conservation Board) — a CDFW state-agency product, not a Trust for Public Land one. It is being relocated to the planned public-cdfw bucket (geo-agent-ops #19; data-workflows #228), after which mirror it as cboettig/cdfw. The rest of public-tpl (Conservation Almanac, LandVote) is HOLD pending TPL terms, so cboettig/tpl was never created.
  • ca-wolves — live product, not a license issue: license is clear (CC0-1.0), but it is a real-time updated product (wolf_*_latest.geojson + snapshots/); a static mirror would be a stale point-in-time copy presented as current. Held until we decide whether to publish only the dated snapshots/ (not the _latest feeds).
  • Infra/internal: public-{test,output,requests,boettiger-lab,data,grids}; public-tnc empty.

Non-commercial datasets (mirrored, must be labeled NC)

carbon, gfw, mobi, mappinginequality, high-seas hydrothermal-vents (CC-BY-NC), rfmo (CC-BY-SA), meow (CC-BY-NC), and the GBIF aggregate. Plus pre-existing source.coop repos fishbase (CC-BY-NC) and carbon. The source.coop product page / STAC for these MUST state the NC/SA license — we are a non-commercial project and downstream users must know.

Adding a repo later (the maintainable loop)

  1. Confirm it's catalogued and license-clear (add a row to license-inventory.md).
  2. Add the bucket to REPOS in gen-source-sync.sh (and EXCLUDES/MODE/new-repos.md if needed).
  3. ./gen-source-sync.sh → emits ../k8s/source-sync-<repo>.yaml and regenerates the scheduled-backup scope (../k8s/source-sync-cron-config.yaml).
  4. Create the cboettig/<repo> product in the source.coop web UI (it's not automatable — see below).
  5. ./dry-run-local.sh <repo> then ./run-source-sync.sh <repo>.
  6. Roll the new repo into the weekly backup: kubectl apply -f ../k8s/source-sync-cron-config.yaml (the source-sync CronJob picks up the new line on its next run — see Scheduled weekly backup below).

⚠️ Credentials are account-wide

The source rclone remote (k8s rclone-config secret) is a long-term AWS key with access to the entire source.coop account (every tenant in the shared bucket). Every job hard-codes the source:us-west-2.opendata.source.coop/cboettig/<repo> dest and refuses to run against any other path, so a typo can't rclone sync (delete-extras) the bucket root or another account. Always dry-run-local.sh before the first real sync.

Phase 1 — data mirror

Create repos (manual, web UI): the create API (POST /api/v1/products/{account_id}) is 501 Not implemented in production (the documented /repositories/ API is stale), so create each repo in the web UI first (visibility: public). The 23 still-to-create repos (id / title / description, with NC + partial-mirror notes) are in new-repos.md; 7 already exist (carbon, cpad, fire, gbif, mappinginequality, mobi, social-vulnerability) and only refresh.

Generate jobs: ./gen-source-sync.sh (30 jobs; minio recipe exactly — --transfers 2 --checkers 4 --bwlimit 50M --tpslimit 5 --retries 5, opportunistic, 2 cpu / 4 Gi, one pod/bucket).

Preview then sync (sequentially):

./dry-run-local.sh rivers          # preview adds/updates/DELETES + excludes (no writes, no cluster)
./run-source-sync.sh rivers        # apply one job, wait for completion
./run-source-sync.sh               # all repos, smallest -> largest (gbif ~1.2 TB last)

Run sequentially (the runner does this): 30 jobs at --bwlimit 50M in parallel ≈ 1.5 GiB/s of NRP egress — the opposite of gentle. Monitor: kubectl -n biodiversity get jobs | grep source-sync.

Mirror-with-delete: rclone sync makes the dest an exact copy, deleting stale source.coop files. For the pre-existing repos this replaces older structure with NRP-canonical content — dry-run first. This is the riskiest step: several pre-existing repos were populated from an older layout of their NRP bucket that has since been restructured/re-versioned, so a blind sync deletes the old-vintage files. That is usually intended (the data lives on NRP under new paths), but it breaks any citable cboettig/<repo>/… URL to the old paths, so confirm before running. The 2026-06 refresh dry-ran all 7 and split them:

  • Safe (0–2 benign deletions): social-vulnerability, mappinginequality, cpad — synced.
  • carbon (~445 GiB), fire (~6 GiB): full sync; old cogs/*_2010|2018.tif / 2022 CALFIRE vintages and build cruft removed, replaced by current NRP layout (decision: match NRP).
  • gbif (~1.2 TB): dry-run separately before refreshing (the long pole).

copy vs sync (per-repo MODE in gen-source-sync.sh): a repo holding content that exists only on source.coop (not a stale version of any NRP file) must NOT be mirror-with-deleted. Set MODE[repo]="copy" so the job uses rclone copy (additive, never deletes). Currently:

repo mode why
mobi copy A 27k-tile tiles/** XYZ pyramid, a whole range-size-rarity-all/ layer, the original SpeciesRichness_All/RSR_All source rasters, and LICENSE.txt live only on source.coop (NRP public-mobi has just the reprocessed COG + hex). A sync would wipe them.
(all others) sync mirror-with-delete (default)

Scheduled weekly backup (CronJob) — keeps the mirror fresh

Once the one-time backfill above is done, the source-sync CronJob keeps every in-scope repo current automatically — this is the original goal of #158. ./gen-source-sync.sh emits two extra files alongside the per-repo jobs:

  • ../k8s/source-sync-cron-config.yaml — a source-sync-scope ConfigMap holding repos.txt (one line per repo: <repo> <verb> [exclude globs…]), generated from the same REPOS/MODE/EXCLUDES arrays. This is the policy/scope — regenerate + re-apply it whenever scope changes.
  • ../k8s/source-sync-cron.yaml — the CronJob itself (mechanism, ~static). One pod loops repos.txt sequentially at the same gentle 50 MB/s recipe, continue-on-error (one bad repo doesn't block the rest; the Job still exits non-zero so a failure shows up). Schedule: Sundays 08:00 UTC (0 8 * * 0), concurrencyPolicy: Forbid (a still-running weekly backup is never overlapped), opportunistic priority, 3-run history.

Keeping scope in the generated ConfigMap (not baked into the CronJob) is deliberate: the per-repo jobs and the weekly cron read the same source of truth and can't drift.

kubectl apply -f ../k8s/source-sync-cron-config.yaml -f ../k8s/source-sync-cron.yaml   # install/update
kubectl -n biodiversity get cronjob source-sync                                        # status
# one-off run now (real sync):
kubectl -n biodiversity create job --from=cronjob/source-sync source-sync-manual
# dry-run a one-off (lists changes, writes nothing): create the job then flip env DRYRUN=true,
# or just run ./run-source-sync.sh <repos…> which honors DRYRUN.
kubectl -n biodiversity logs -f job/source-sync-manual

The per-repo source-sync-<repo>.yaml jobs + run-source-sync.sh remain for manual/backfill use (a single repo, a fresh repo's first sync, or a targeted re-run); the CronJob is the standing backup. The same DEST safety guard (refuse any non-cboettig/<repo> path) runs per repo inside the loop.

Phase 2 — STAC on source.coop (DONE; runs after every mirror)

As mirrored, the STAC *.json files carry NRP hrefs. rewrite-stac-hrefs.py makes them self-referential for everything that exists on source.coop:

  • Rewrites any href https://s3-west.nrp-nautilus.io/public-<X>/<path> → https://data.source.coop/cboettig/<X>/<path> iff <X> is a mirrored repo (read from source-sync-cron-config.yaml repos.txt — the same single source of truth). Covers self / child / describedby / all asset hrefs, including cross-bucket references.
  • Leaves root/parent pointing at the NRP canonical root (public-data/stac/catalog.json is the global catalog spanning non-mirrored buckets; it is not mirrored). source.coop collections stay navigable up to the canonical root. (Decision in #158; a source.coop root catalog is a possible future enhancement.)
  • Drops dangling child links to the excluded HOLD sub-paths that were never mirrored (rivers/american-rivers/{campaigns,ira-watersheds,roo-cjest}, high-seas/mpa-candidates).
  • Licenses are unchanged (already correct SPDX on NRP, carried over by the mirror).

Idempotent (a 2nd run is a no-op — an already-source.coop href won't re-match) and topology-agnostic (does not assume a fixed root path; e.g. rivers has no top-level stac-collection.json). Writes to source.coop ONLY (refuses any non-cboettig/<repo> path); never touches NRP. It reads/writes via rclone (cat/rcat --s3-no-check-bucket), so it works identically on a laptop and in-pod.

./rewrite-stac-hrefs.py --dry-run          # report changes, write nothing
./rewrite-stac-hrefs.py                      # rewrite in place on source.coop
./rewrite-stac-hrefs.py --repos rivers high-seas   # limit to some repos

Why it's wired into the weekly CronJob: a plain rclone sync makes source.coop an exact copy of NRP, so the weekly data mirror would clobber the rewrite back to NRP-pointing hrefs. The source-sync CronJob therefore runs this script after each sync (fetched from main, scope from the mounted ConfigMap), re-applying it after each sync. So source.coop self-consistency is maintained automatically — no manual re-run needed.

Phase 3 — unified GLEN root catalog (#351; runs after phase 2)

Phase 2 makes each mirrored collection self-consistent, but nothing ties them together into a navigable catalog. gen-root-catalog.py publishes the root: cboettig/glen/{catalog.json, README.md} — the source.coop analog of NRP's public-data/stac/catalog.json.

  • Child list = the NRP root's top-level child links, filtered to the mirror scope, hrefs rewritten to source.coop. The STAC catalog is a tree: this root links only the curated top-level collections; nested/cross-collection nodes (e.g. gfw under high-seas, the CAL FIRE leaves under fire-perimeters) are reached by descending. The scope (repos.txt) is the license gate, so restricted collections (wdpa/iucn/hydrobasins/icca/tpl/…) can never leak in — enforced by construction. It reuses phase-2's mirrored_repos() + rewrite_string() (imported from rewrite-stac-hrefs.py).
  • Navigation is downward-only by design: mirrored collections keep their root/parent pointing at canonical NRP (#158). glen is an entry point you descend from.
  • Fails hard (won't publish) if the NRP root can't be fetched or yields fewer than --min-children (default 15) in-scope collections — a circuit-breaker against a truncated catalog.
  • One-time owner step: register cboettig/glen on source.coop (web UI — the create-repo API is disabled). Writes fail until the repo exists; the account-wide-cred guard already scopes writes to cboettig/.
SCOPE_FILE=/config/repos.txt ./gen-root-catalog.py --dry-run --readme glen-README.md   # preview
SCOPE_FILE=/config/repos.txt ./gen-root-catalog.py --readme glen-README.md             # publish
./gen-root-catalog.py --dry-run --nrp-root /tmp/cached-root.json  # test offline / when Ceph is down

Existing-content audit (2026-06-16)

Audited all pre-existing cboettig/* (+ berkeley-dse, espm-288) repos: none contain non-redistributable data (no WDPA/IUCN/ICCA/HydroBASINS). Several are NC (fishbase, carbon, mobi, mappinginequality, gbif) — keep, but ensure NC labels. Several hold older versions than NRP (pad-us-3, old ca30x30/cpad/fire) — stale, not a license issue.

Total in current scope ≈ 2.5 TB / ~44 k objects (gbif alone ≈ 1.2 TB).

Private backup (MinIO) for datasets NOT on source.coop

Anything we can't publish to source.coop still needs an off-NRP backup — that's MinIO (catalog/sync/k8s/sync-public-*.yaml). Status:

  • License-prohibited (source.coop impossible — MinIO is the ONLY off-NRP copy): wdpa, icca, iucn, hydrobasins — ✅ all have MinIO sync jobs. Keep these.
  • Already MinIO-backed (uncatalogued/clipped, no source.coop): wyoming, datacenters, im3, ca30x30 (wyoming/ca30x30 restructuring tracked in #225).
  • NOT backed up yet — and shouldn't be until fixed upstream (don't back up a mess):
    • landfire — partial import; complete it first (#203), then catalogue + back up.
    • wlfw — migrate into public-working-lands first (#226), don't back up standalone.
    • working-lands — back up after the wlfw consolidation + proper STAC (#226).
  • Redundant-backup candidates (space optimization, not urgent): large public datasets ALSO mirrored to source.coop (gbif ≈1.2 TB, carbon, wetlands, rap, …) now have an off-NRP copy on source.coop, so their MinIO copy is redundant and could be dropped to save private-system space. Not needed yet ("so far we are fine") — revisit if MinIO fills up.