refactor!: compositional pipeline API, path-free engine, and pipeline unification

[sjawhar]

Summary

Replace the entire pipeline definition and execution model. Stages are now plain Python functions composed via ArtifactHandle wiring instead of verbose Annotated[T, Dep(...)] / Out(...) annotations. File paths are eliminated as identity throughout the stack — the DAG, worker, lock files, state DB, and skip detection all operate on structured ArtifactIdentity(producer, key) objects, with paths resolved late by Store.

246 files changed, +17,512 / −64,643 (net −47K lines)

What Changed

1. Compositional Pipeline API (compose.py)

New @stage decorator + Pipeline context manager. Stages are pure functions, the DAG emerges from data flow:

with Pipeline("my_pipeline") as p:
    raw = fetch_data(p.input("source"))
    result = transform(raw, params=TransformParams(...))
    report = generate_report(result)

• ArtifactHandle objects wire stages together — no paths, no loader annotations on consumers
• Multi-output via TypedDict return types with attribute access on handles
• Variants are just Python loops (no matrix expansion machinery)
• Cross-pipeline composition via pipeline.include(other_pipeline)
• Format inference from return types (DataFrame → JSONL, dict → YAML, Figure → PNG)

2. Path-Free Engine (ArtifactIdentity)

ArtifactIdentity(producer, key) replaces file paths as the canonical identity:

• DAG/graph: Nodes and edges reference identities, not paths
• Lock files: Store identity objects with hash info (schema v2)
• StateDB: Generation keys use canonical identity_key() encoding
• Skip detection: Compares identity-keyed hashes
• Worker: Resolves identities to paths via Store at execution time
• Presentation layer (storage/presentation.py): Symlink tree materializes CAS outputs into conventional workspace paths

3. Pipeline Unification

Eliminated the dual-Pipeline-class architecture:

• Defined PipelineLike Protocol for the engine contract
• compose.Pipeline is now the sole implementation
• Deleted pipeline.Pipeline, pipeline/yaml.py, and the build() bridge
• Deleted resolve_external_dependencies() — compose handles all dep resolution via handles, include(), and p.input()

4. Dead Code Removal

Deleted modules with zero production imports now that the declarative API is gone:

• matrix.py (143 lines) — matrix expansion, replaced by Python loops
• dvc_import.py (880 lines) — DVC import compatibility
• dvc_compat.py (389 lines) — DVC format compatibility
• pipeline/pipeline.py (642 lines) — old Pipeline class
• pipeline/yaml.py (765 lines) — YAML pipeline parsing

5. CLI Migration

All CLI commands (diff, checkout, verify, sync, restore, show, completion) updated to resolve targets via identity keys instead of file paths.

6. Test Overhaul

• Deleted ~5,700 lines of tests for removed features (matrix, DVC import, dep injection, status, run history, placeholder deps, file/directory out integration)
• Added test_compose.py (+1,542 lines), test_discovery.py, test_names.py covering the new API
• Updated all integration tests for identity-based interfaces

Breaking Changes

• Registration: Only compose.Pipeline + @stage — annotation-based registration path removed
• Lock file schema: v2 with identity objects (no migration — pre-alpha)
• RPC contract: Returns structured identity objects instead of path strings
• pivot.yaml discovery: Now discovers pipeline.py modules with Pipeline context managers

Design Documents

• docs/plans/2026-02-14-compositional-pipeline-api.md
• docs/plans/2026-02-15-path-free-engine-design.md
• docs/plans/2026-02-17-pipeline-unification-plan.md
• docs/plans/2026-02-17-pipeline-unification-cleanup.md

Closes #449

[Copilot]

Pull request overview

This PR replaces Pivot’s pipeline API and execution model with a compositional @stage/Pipeline interface and a path-free engine built around structured ArtifactIdentity(producer, key) identifiers, updating CLI/TUI, lock/state handling, and related utilities accordingly.

Changes:

• Introduces a compositional pipeline API (function composition via handles) and unifies pipeline implementations behind PipelineLike.
• Refactors engine/CLI/TUI to use identity keys/objects instead of filesystem paths (locks, skip detection, RPC, display).
• Removes YAML/DVC/matrix legacy pipeline machinery and associated command surface area.

Reviewed changes

Copilot reviewed 92 out of 245 changed files in this pull request and generated 9 comments.

Show a summary per file

File	Description
packages/pivot/src/pivot/storage/artifact_lock.py	Switches lock request expansion to identity-key based lock keys.
packages/pivot/src/pivot/status.py	Adapts status/explain plumbing to PipelineLike, identities, and workspace store resolution.
packages/pivot/src/pivot/stage_def.py	Removes large annotation-based stage-definition extraction machinery; updates docs/comments.
packages/pivot/src/pivot/skip.py	Changes dep/output comparisons and diffs to operate on ArtifactIdentity and supports “accessed hashes”.
packages/pivot/src/pivot/show/plots.py	Updates plot discovery and lock lookup to support ArtifactRef + identity-based hashes.
packages/pivot/src/pivot/show/metrics.py	Updates metrics discovery/head lookup to support ArtifactRef and identity-key lock access.
packages/pivot/src/pivot/show/data.py	Updates data output discovery/head lookup and adds loader-based format inference for ArtifactRef.
packages/pivot/src/pivot/show/common.py	Adds compatibility extraction of output hashes from lock entries using key/path/display.
packages/pivot/src/pivot/run_history.py	Updates run-history input hash computation to include identity fields instead of paths.
packages/pivot/src/pivot/remote/sync.py	Migrates remote sync target hashing logic to identity keys and identity-based lock maps.
packages/pivot/src/pivot/pipeline/init.py	Removes legacy pipeline package exports.
packages/pivot/src/pivot/outputs.py	Removes legacy Dep/PlaceholderDep markers; updates doc examples to match new API.
packages/pivot/src/pivot/names.py	Adds helpers for display/resolution of stage names with optional pipeline-prefix stripping.
packages/pivot/src/pivot/merkle.py	Adds merkle-id computation helper for identity-first hashing.
packages/pivot/src/pivot/matrix.py	Deletes legacy matrix expansion module.
packages/pivot/src/pivot/loaders.py	Adds format_extension() helper for mapping loader instances to default extensions.
packages/pivot/src/pivot/import_artifact.py	Makes import path resolution/logging more robust with lock entries using key/display/path.
packages/pivot/src/pivot/ignore.py	Updates protected config filenames for pipeline discovery (pipeline.py).
packages/pivot/src/pivot/fingerprint.py	Updates guidance text to match new API (“function parameter” instead of Dep(...)).
packages/pivot/src/pivot/explain.py	Migrates explain to identity-keyed deps/outs and to Store-based hashing when available.
packages/pivot/src/pivot/executor/core.py	Migrates executor plumbing to PipelineLike and store specs; adjusts worker stage info shape.
packages/pivot/src/pivot/executor/commit.py	Refactors commit pipeline to identity-based hashing and Store-based output hashing.
packages/pivot/src/pivot/exceptions.py	Updates pipeline-not-found messaging and introduces PipelineConfigError.
packages/pivot/src/pivot/engine/watch.py	Migrates graph queries to identity parsing for producer/consumer lookup.
packages/pivot/src/pivot/engine/types.py	Updates code/config change docstring to remove pivot.yaml mention.
packages/pivot/src/pivot/engine/sources.py	Removes pivot.yaml/yml from config file watch list.
packages/pivot/src/pivot/engine/agent_rpc.py	Updates RPC to return structured identity JSON objects for deps/outs and uses PipelineLike.
packages/pivot/src/pivot/discovery.py	Removes YAML discovery and validates discovered pipelines against PipelineLike.
packages/pivot/src/pivot/cli/verify.py	Migrates verify to identity-keyed lock lookups with optional workspace display-path resolution.
packages/pivot/src/pivot/cli/track.py	Updates overlap detection to use identity keys for stage outputs.
packages/pivot/src/pivot/cli/targets.py	Adds identity-aware CLI target parsing/resolution and bare-stage-name resolution helpers.
packages/pivot/src/pivot/cli/status.py	Integrates orphaned lock detection and passes pipeline into status/explain queries.
packages/pivot/src/pivot/cli/repro.py	Updates DAG validation/watch path selection for identity-first model and removes YAML references.
packages/pivot/src/pivot/cli/remote.py	Improves CLI target normalization to accept identity targets (stage:key).
packages/pivot/src/pivot/cli/list.py	Updates list output to show deps/outs as identity keys and removes YAML messaging.
packages/pivot/src/pivot/cli/init.py	Updates init message to instruct pipeline.py creation only.
packages/pivot/src/pivot/cli/helpers.py	Replaces registry access with PipelineLike accessors and adds workspace store helper.
packages/pivot/src/pivot/cli/doctor.py	Removes pivot.yaml checks; validates pipeline.py existence only.
packages/pivot/src/pivot/cli/decorators.py	Stores/retrieves PipelineLike from click context.
packages/pivot/src/pivot/cli/data.py	Adds identity-aware CLI target resolution for data diffs via workspace store mapping.
packages/pivot/src/pivot/cli/console.py	Updates change rendering to support ArtifactIdentity display and dep-change field rename.
packages/pivot/src/pivot/cli/completion.py	Adds bare-name completion and identity (stage:key) completion by loading pipeline on demand.
packages/pivot/src/pivot/cli/checkout.py	Makes checkout use identity-based stage outputs via workspace store resolution.
packages/pivot/src/pivot/cli/_run_common.py	Updates discovery typing to PipelineLike.
packages/pivot/src/pivot/cli/init.py	Removes export/import-dvc/schema commands from CLI surface.
packages/pivot/src/pivot/cli/AGENTS.md	Updates CLI docs to remove export reference.
packages/pivot/src/pivot/init.py	Bumps version and switches public API exports to composition API and merkle helper.
packages/pivot/pyproject.toml	Bumps package version to 0.2.0a1.
packages/pivot-tui/tests/test_watch.py	Updates test pipeline.py to new compose API and uses fully-qualified stage names.
packages/pivot-tui/tests/test_tui_force_rerun.py	Updates stage names to fully-qualified prefixed form.
packages/pivot-tui/tests/test_run.py	Migrates helper stage signatures away from Dep(...) and updates output snapshot identity typing.
packages/pivot-tui/tests/test_rpc_contract.py	Updates RPC contract tests for structured identity payloads and compose pipeline creation.
packages/pivot-tui/tests/test_rpc_client_impl.py	Updates stage_info parsing expectations and adds structured identity test.
packages/pivot-tui/tests/test_fake_server.py	Updates fake server stage_info deps/outs to structured identity JSON.
packages/pivot-tui/tests/test_diff_panels.py	Updates diff panel tests to use ArtifactIdentity and identity-key indexing.
packages/pivot-tui/tests/test_client_protocol.py	Updates protocol types to use ArtifactIdentity lists for stage_info results.
packages/pivot-tui/tests/helpers.py	Refactors test pipeline/stage registration helpers to build compose pipelines.
packages/pivot-tui/tests/conftest.py	Switches fixtures to compose Pipeline / PipelineLike.
packages/pivot-tui/src/pivot_tui/testing/fake_server.py	Updates stage_info response shape (structured identities).
packages/pivot-tui/src/pivot_tui/run.py	Parses output summary “path” into ArtifactIdentity instead of string.
packages/pivot-tui/src/pivot_tui/rpc_client_impl.py	Decodes stage_info deps/outs from structured identity JSON into ArtifactIdentity.
packages/pivot-tui/src/pivot_tui/diff_panels.py	Updates diff indexing and rendering to use identity keys derived from ArtifactIdentity.
packages/pivot-tui/src/pivot_tui/client.py	Updates StageInfoResult typing to structured identity objects.
docs/task9-cli-tui-update-locations.md	Adds implementation notes on identity display changes across CLI/TUI.
docs/research/textual-rich-formatting-patterns.md	Adds research notes on Rich/Textual formatting for identity display.
docs/plans/2026-02-17-pipeline-unification-cleanup.md	Adds plan doc for pipeline unification and dead code removal.
docs/plans/2026-02-16-single-field-typeddict-bug.md	Adds plan doc for single-field TypedDict output key preservation.
docs/plans/2026-02-15-path-free-engine*.md	Adds design/plan docs for identity-first engine architecture.
docs/gen_ref_pages.py	Updates API reference generation to point to pivot.compose instead of legacy pipeline module.

[Copilot]

what_if_changed() currently ignores the provided paths and returns all stages unconditionally. This breaks the CLI expectation of narrowing to affected stages. If path-based resolution is no longer available in an identity-first graph, consider re-implementing this by mapping user-provided paths/identity targets to ArtifactIdentity (via WorkspaceStore/presentation reverse index) and then querying the graph consumers; otherwise, remove/disable this command path with a clear error until implemented.

[Copilot]

watch.py is parsing filesystem paths (from the watcher) as artifact identity strings. For typical watch events, str(path) will be a workspace path like data/train.csv, not an identity key like stage:key, so graph lookups will miss and watch-mode change detection will silently degrade. Recommended fix: resolve filesystem paths to ArtifactIdentity via a store/presentation reverse mapping (path→identity), or keep watch graph queries path-based until a reliable path↔identity index exists.

[Copilot]

watch.py is parsing filesystem paths (from the watcher) as artifact identity strings. For typical watch events, str(path) will be a workspace path like data/train.csv, not an identity key like stage:key, so graph lookups will miss and watch-mode change detection will silently degrade. Recommended fix: resolve filesystem paths to ArtifactIdentity via a store/presentation reverse mapping (path→identity), or keep watch graph queries path-based until a reliable path↔identity index exists.

[Copilot]

watch.py is parsing filesystem paths (from the watcher) as artifact identity strings. For typical watch events, str(path) will be a workspace path like data/train.csv, not an identity key like stage:key, so graph lookups will miss and watch-mode change detection will silently degrade. Recommended fix: resolve filesystem paths to ArtifactIdentity via a store/presentation reverse mapping (path→identity), or keep watch graph queries path-based until a reliable path↔identity index exists.

[Copilot]

For ArtifactRef metrics, _metric_read_path() treats identity_key(out.identity) as a filesystem path. Identity keys like producer:key are not guaranteed to be valid paths (and may never correspond to a real file location), so this can cause metrics discovery to fail or point to non-existent files. Suggestion: resolve metric read paths via WorkspaceStore.resolve_display_path(out) (similar to show/data.py), and only fall back to identity_key for display when no store is available.

[Copilot]

config.get_checkout_mode_order() is called but its return value is ignored, which reads like a leftover from the previous cache-write implementation. Either remove the call or thread the checkout mode order into the store/commit logic (e.g., when materializing cached artifacts) so the behavior is explicit.

[Copilot]

The commit path previously saved cached outputs into the local cache (CAS) for later checkout/push; now it only hashes via store.hash_artifact(out) without an explicit cache write step. If hash_artifact() does not also persist artifacts into cache_dir, pivot commit will no longer guarantee that cache-backed outputs are actually present in cache. Recommended fix: ensure commit calls a store API that both hashes and stores (or add a dedicated store.cache_artifact() / store.write_to_cache() step for non-metric outputs) and only treat metric-tagged refs as non-cached.

Suggested change

	hash_info = store.hash_artifact(out)
	# Metrics are not cached; all other outputs must be written to cache.
	if out.tag is ArtifactTag.METRIC:
	hash_info = store.hash_artifact(out)
	else:
	hash_info = store.cache_artifact(out)

[Copilot]

When out is an outputs.Plot, the x/y/template metadata is currently discarded (set to None), which is a functional regression for plot rendering/diff behavior that relied on these fields. If outputs.Plot remains supported, preserve out.x, out.y, and out.template for that branch (and keep None only for ArtifactRef where the metadata is unavailable).

Suggested change

	x=None,
	y=None,
	template=None,
	x=out.x,
	y=out.y,
	template=out.template,

[Copilot]

find_orphaned_lock_files() defaults state_dir to project_root / '.pivot', but other code paths use config.get_state_dir() (which may be customized). This can cause orphan lock detection to silently scan the wrong directory. Suggest defaulting to config.get_state_dir() (or reusing the same state-dir source as the status/explain caller) for consistency.

Suggested change

	state_dir = project.get_project_root() / ".pivot"
	state_dir = config.get_state_dir()