GepMap: S3 hosted terrain tiles

[DonLakeFlyer]

Hey Don,

The terrain assets infra is up. Here's what you need:

Bucket: qgc-terrain-assets (us-west-2, Dronecode account). Your existing IAM credentials (don user) already have full access through the QGroundControl group, nothing new to configure. Upload as usual:

aws s3 cp ./terrain-tiles/ s3://qgc-terrain-assets/ --recursive

The bucket is fully private, no public access, no ACLs. Everything is served through CloudFront only.

Delivery: the URL for everything (QGC code included) is https://terrain.assets.qgroundcontrol.com. The TLS cert is validating right now, so give it a few minutes before testing. Behind it is CloudFront distribution E2AZ33YAOTUR6: HTTPS only (HTTP redirects), GET/HEAD only, compression enabled, standard CachingOptimized policy, reading from the bucket via Origin Access Control, so direct S3 URLs won't work by design. There's a healthcheck.txt at the root you can use to sanity-check; feel free to delete it once you've got real assets up.

Keep in mind cached objects live on the edge for up to 24h by default, so if you overwrite an existing key you'll need an invalidation to see it immediately.

Ramón

[DonLakeFlyer]

Plan: Copernicus GLO-30 → terrarium tiles → qgc-terrain-assets

Infra verified: https://terrain.assets.qgroundcontrol.com/healthcheck.txt returns 200 through CloudFront. Source dataset: copernicus-dem-30m on AWS Open Data (eu-central-1, ~26k 1°×1° COGs + tileList.txt, anonymous access works). Design decisions carried over from #14892.

0. Decisions to lock in

Decision	Value	Why
Max zoom	z12	Native 30 m of GLO-30; deeper is upsampling. Client already clamps deeper requests to the ancestor.
Coverage	Full globe (no land mask)	Ocean tiles are ~1 KB; removes coastline logic; clean 200s through CloudFront, no 404 storms.
Key layout	v1/terrarium/{z}/{x}/{y}.png	Versioned prefix per Copernicus edition; overwriting keys would need CloudFront invalidations (24 h edge TTL).
Encoding	terrarium RGB PNG, 256×256, h = R*256 + G + B/256 − 32768	Zero client decode changes (TerrariumTileFetcher).
Object metadata	Content-Type: image/png, Cache-Control: public, max-age=31536000, immutable	CloudFront CachingOptimized honours origin Cache-Control.
Nodata / voids	Fill with 0 m (sea level)	Matches what the heightfield assumes for missing data.

Note: GLO-30 is a DSM (surface incl. canopy/buildings), not a DTM. Same as what the Auterion carpet endpoint already serves QGC, so behaviour is consistent — worth stating in attribution/docs.

1. Pipeline design (run in eu-central-1, next to the source bucket)

Generation on an EC2 instance / container job in eu-central-1 so the ~26k COG reads are free; output pushed cross-region to qgc-terrain-assets (us-west-2) — one-time ~400 GB transfer.

Per 1° source cell (embarrassingly parallel):

1. gdalwarp the cell COG (EPSG:4326, float32) → EPSG:3857 at z12 resolution (38.2 m/px), -r bilinear, -dstnodata 0, target extent expanded to full z12 tile boundaries so cells stitch seamlessly.
2. Encode float32 → terrarium RGB (v = h + 32768; R = v>>8; G = v&255; B = frac(v)*256).
3. Cut 256×256 PNGs for the z12 tiles covered by the cell; write directly to S3 with the metadata above.
4. Ocean tiles (z12 keys not covered by any cell in tileList.txt): one precomputed constant-sea-level PNG, uploaded per key (copy_object from a template key to avoid re-sending bytes).
5. Overview zooms 0–11: build bottom-up by decoding four terrarium children → float → 2×2 box average → re-encode. Never average the RGB bytes directly (terrarium is not linear in RGB).

Tooling: a small Python package (rasterio/numpy/Pillow/boto3) under tools/terrain/, reusing tools/common/aws.py + proc.py. PUBLIC_BUCKETS allowlist gains qgc-terrain-assets; uploads must not pass --acl public-read (bucket blocks ACLs — the current upload_public_file would fail). Considered and rejected: rio-rgbify (unmaintained, overviews wrong for this case) and gdal2tiles (8-bit PNG output loses the encoding).

2. Validation before the global run

• Generate 3–4 representative cells locally via Docker (ghcr.io/osgeo/gdal): Alps N46_E007, US plains, a coastline cell, a Himalayan cell (exercises the 32768 offset range).
• Round-trip: decode generated PNG at sample points vs gdallocationinfo on the source COG (tolerance ≤ 1/256 m quantum + bilinear error).
• Seam check: tile-border heights between two adjacent source cells must agree.
• Compare against the current Tilezen tile at the same z12 key to confirm orientation (y increases southward) and no half-pixel shift — client sample-center convention assumes Tilezen's layout.
• Measure the real tile size distribution → firm up the Self-hosted terrain elevation tiles: S3/CloudFront setup and cost analysis (z11 vs z12) #14892 storage/PUT estimates before the 22 M-object upload.
• Upload these cells to v1/ and point a local QGC build at the CloudFront domain to eyeball in GeoMap.

3. Global generation + upload

• c7g/c7i-class instance with NVMe scratch in eu-central-1; worker pool across cells; run is PUT-bound (22.4 M objects ≈ ~$112 PUT fees per Self-hosted terrain elevation tiles: S3/CloudFront setup and cost analysis (z11 vs z12) #14892).
• Resumable: per-cell done marker (e.g. v1/_manifest/cells/N46_E007.done).
• Overviews as a second pass once all z12 tiles exist.
• Write v1/tilejson.json (bounds, minzoom 0, maxzoom 12, attribution) and v1/ATTRIBUTION.txt with the required notice: "Produced using Copernicus WorldDEM-30 © DLR e.V. 2010-2014 and © Airbus Defence and Space GmbH 2014-2018 provided under COPERNICUS by the European Union and ESA; all rights reserved."
• Keep healthcheck.txt — useful for a CI smoke check.

4. Client changes (one PR, land once v1/ is complete)

1. ElevationMapProvider.h: TerrariumElevationProvider::kProviderURL → https://terrain.assets.qgroundcontrol.com; ElevationMapProvider.cpp URL pattern → %1/v1/terrarium/%2/%3/%4.png; update doc comment + attribution.
2. TerrariumTileFetcher.h: kMaxTileZoom 15 → 12 (tests are written against the constant).
3. UrlFactoryTest.cc: expected URL string.
4. kAvgElevSize (45000) → measured average from step 2 (drives offline-cache size estimates).
5. Docs: docs/en/qgc-user-guide/fly_view/geoview.md attribution paragraph, src/GeoMap/README.md Elevation section.
6. Keep the provider key "Terrarium" — it's persisted in the tile cache DB as the provider type; renaming would orphan cached tiles. Cached Tilezen z13–15 tiles simply stop being requested once the ceiling is 12.

5. Ops

• Invalidations only if a key is overwritten; the versioned prefix avoids that.
• New Copernicus edition → generate to v2/, flip the client URL, delete v1/ after a release cycle.
• Optional monthly CloudFront usage check against the free-tier egress assumption in Self-hosted terrain elevation tiles: S3/CloudFront setup and cost analysis (z11 vs z12) #14892.

Out of scope (adjacent)

The 2D terrain-query path still hits Auterion's terrain-ce.suite.auterion.com/api/v1/carpet (TerrainTileCopernicus). Since the new tile set is the same Copernicus data at native resolution, a follow-up could serve TerrainQuery from the same z12 terrarium tiles and drop the Auterion dependency — separate client change to the TerrainTile format, shouldn't block this.