Direction and sequencing for the PortalJS revamp. Living document — decisions here are revisited as we learn. For the conceptual model behind the product, see Core concepts; for the architecture decision layer, see the decision framework.
PortalJS is an agentic skills framework that helps data teams build, develop, and ship data portals — and the data infrastructure underneath them. It is not only a frontend layer. The skills do two jobs:
- Advise — given what you're building, what your data is, and what it's for, recommend an architecture (storage, compute, catalog, access, hosting).
- Build — scaffold the chosen stack as plain, editable code with no lock-in.
The product is organized into three layers:
| Layer | What it does | Built from |
|---|---|---|
| Decision | Advises on architecture from the team's needs | Advisory skills (e.g. /portaljs-architect) |
| Build | Scaffolds + wires the chosen stack | Generative skills + the contracts below |
| Presentation | The portal users see | The three surfaces (Home / Catalog / Showcase) |
Everything in the Build layer plugs into one of four file-declarative, pluggable contracts. Each has a static/none default and richer backend implementations, so simple stays simple.
| Contract | Static default | Scales to | Drives |
|---|---|---|---|
| Metadata profile | Frictionless Tabular Data Package | extend → custom → multi-profile; DCAT interop | catalog facets, showcase metadata |
| Data provider | git files (datasets.json) |
git+LFS, lakehouse, CKAN, OpenMetadata | what feeds all three surfaces |
| Data query | flat CSV fetch | DuckDB-Wasm → server/remote DuckDB | showcase preview + views |
| Auth / RBAC | none (repo permissions) | CKAN / OIDC / OpenMetadata | visibility + edit across surfaces |
The recurring axis through all four is build-time (static) vs request-time (runtime). We stay static-first; runtime is an explicit opt-in mode, unlocked when a portal needs private data, live search, or write access — never a rewrite.
The data provider and data query contracts together span a spectrum the advisory layer reasons over — open/composable/cheap on the left, traditional/heavy on the right:
| Flat files | Git-LFS + R2 | Open lakehouse | Warehouse / datastore | |
|---|---|---|---|---|
| Storage | repo | R2 via giftless (Git LFS) | Parquet on R2 | Postgres / CKAN datastore / Snowflake |
| Catalog | datasets.json |
git + Frictionless | DuckLake (catalog in SQL) | warehouse-native |
| Compute | papaparse | DuckDB-Wasm | DuckDB (Wasm or server) | warehouse SQL engine |
| Versioning | git | git + LFS | git + table snapshots | none / CDC |
| Best for | a few small files | large files, gitops workflow | analytics-grade, open, cheap | large internal/SQL, RBAC-native |
Opinionated modern default: git + giftless/R2 + Parquet + DuckLake + DuckDB —
open formats, object storage, bring-your-own-compute, no warehouse lock-in. The
warehouse/CKAN column stays a fully supported choice for teams that need it. DuckDB is
the through-line: it queries Parquet/CSV directly off object storage, client (Wasm) or
server.
giftless is Datopian's pluggable Git LFS
server. It is the git tier's ingest + versioning plane: git versions metadata and
LFS pointers (Frictionless packages, profiles, config) and giftless moves the actual
bytes to blob storage. It keeps large data out of the repo, makes a git push / PR the
way data enters and is versioned, and is the on-ramp to the lakehouse — the same R2
bucket that holds LFS blobs holds the Parquet that DuckLake + DuckDB query.
We champion Cloudflare as the default substrate, but keep storage S3-compatible so R2 is never a hard lock-in.
| Cloudflare | Role |
|---|---|
| R2 | object storage — LFS blobs + Parquet (S3-compatible; swappable for S3/GCS/Azure) |
| Workers | the opt-in runtime — SSR, auth/RBAC, server-side DuckDB, LFS endpoints |
| D1 | DuckLake catalog / metadata index |
| Pages | static hosting for the build-time portal |
The static-first → runtime fork is concretely Pages → Workers.
- Advisory / decision layer (next — Phase 1) — the needs→architecture framework
and an
/portaljs-architectinterview skill. Defines what the build layer produces. See decision-framework.md. - Data-provider contract — the storage+compute+catalog seam every surface consumes;
refactor the static path onto it; design-in giftless/LFS+R2 (wire later); refactor
/portaljs-connect-ckanonto it. - Metadata profile layer ∥ DuckDB data query — parallel; both static-friendly, high visible value.
- DCAT interop — export then import, layered on the profile layer (interop over a Frictionless-native model, not the native model).
- RBAC + runtime mode — auth provider + visibility + Pages→Workers. Last; only advanced/multi-team portals need it.
- Three-surface product model — Home
/, Catalog/search, Showcase/@<namespace>/<slug>(#1536). - Interactive skills — interview in rounds, never dead-end on missing input (#1537).
- Docs teaching the three concepts + aligned routes (#1538).