Dynamic Pipeline: Tuple artefact with substitutions breaks #4646
Description
Contact Details [Optional]
System Information
❯ uv run zenml info -a -s
~/jb/dynamic-pipeline/.venv/lib/python3.11/site-packages/requests/__init__.py:113: RequestsDependencyWarning: urllib3 (2.6.3) or chardet (7.3.0)/charset_normalizer (3.4.6) doesn't match a supported version!
warnings.warn(
ZENML_LOCAL_VERSION: 0.94.0
ZENML_SERVER_VERSION: 0.94.0
ZENML_SERVER_DATABASE: mysql
ZENML_SERVER_DEPLOYMENT_TYPE: cloud
ZENML_CONFIG_DIR: ~/Library/Application Support/zenml
ZENML_LOCAL_STORE_DIR: ~/Library/Application Support/zenml/local_stores
ZENML_SERVER_URL: [redacted]
ZENML_ACTIVE_REPOSITORY_ROOT: ~/jb/dynamic-pipeline
PYTHON_VERSION: 3.11.10
ENVIRONMENT: native
SYSTEM_INFO: {'os': 'mac', 'mac_version': '15.7.4'}
ACTIVE_PROJECT: learn
ACTIVE_STACK: [name]
ACTIVE_USER: [email]
TELEMETRY_STATUS: enabled
ANALYTICS_CLIENT_ID: e88584f6-0d02-4cf9-9fce-fc13814a2370
ANALYTICS_USER_ID: b121f647-e615-49a5-adb8-e55fbde72011
ANALYTICS_SERVER_ID: f352ce88-644b-4d0f-8ef2-b147e605277f
INTEGRATIONS: ['kaniko', 'airflow', 'pillow', 'github', 'slack', 'kubernetes', 'wandb']
PACKAGES: {'cint': '1.0.0', 'kaitaistruct': '0.11', 'google-cloud-container': '2.63.0', 'docstring_parser': '0.17.0', 'pydantic_core': '2.33.2', 'oauthlib': '3.3.1', 'platformdirs': '4.4.0',
'cloudpickle': '3.1.2', 'requests': '2.32.5', 'aiohttp': '3.13.3', 'intervaltree': '3.2.1', 'multidict': '6.7.1', 'google-genai': '1.68.0', 'polyfile-weave': '0.5.9', 'setuptools': '82.0.1',
'pydantic': '2.11.9', 'certifi': '2026.2.25', 'diskcache-weave': '5.6.3.post1', 'kubernetes': '25.3.0', 'googleapis-common-protos': '1.73.0', 'jsonschema-specifications': '2025.9.1', 'gql': '4.0.0',
'websockets': '16.0', 'grpcio': '1.78.0', 'propcache': '0.4.1', 'google-cloud-core': '2.5.0', 'networkx': '3.6.1', 'google-auth': '2.49.1', 'weave': '0.52.35', 'websocket-client': '1.9.0',
'jsonschema': '4.26.0', 'proto-plus': '1.27.1', 'yarl': '1.23.0', 'google-auth-oauthlib': '1.3.0', 'idna': '3.11', 'pdfminer.six': '20260107', 'google-cloud-run': '0.15.0', 'kfp-server-api':
'2.16.0', 'cffi': '2.0.0', 'zenml': '0.94.0', 'opentelemetry-semantic-conventions': '0.59b0', 'rich': '14.3.3', 'tenacity': '9.1.4', 'pillow': '12.1.1', 'httpcore': '1.0.9', 'filelock': '3.25.2',
'decorator': '5.2.1', 'slack_sdk': '3.30.0', 'cachetools': '7.0.5', 'google-cloud-logging': '3.14.0', 'google-cloud-pipeline-components': '2.22.0', 'markdown-it-py': '4.0.0', 'fsspec': '2024.12.0',
'google-cloud-build': '3.35.0', 'google-resumable-media': '2.8.0', 'h11': '0.16.0', 'protobuf': '6.33.6', 'gitdb': '4.0.12', 'sniffio': '1.3.1', 'google-crc32c': '1.8.0', 'smmap': '5.0.3',
'pycparser': '3.0', 'jb_mlops': '0.0.36', 'importlib_metadata': '8.7.1', 'typeguard': '4.5.1', 'kfp': '2.16.0', 'mdurl': '0.1.2', 'opentelemetry-api': '1.38.0', 'python-dateutil': '2.9.0.post0',
'google-cloud-audit-log': '0.4.0', 'PyYAML': '6.0.3', 'jsonref': '1.1.0', 'sortedcontainers': '2.4.0', 'google-cloud-bigquery': '3.40.1', 'abnf': '2.2.0', 'urllib3': '2.6.3', 'frozenlist': '1.8.0',
'PyGithub': '2.9.0', 'typing_extensions': '4.15.0', 'aiosignal': '1.4.0', 'click': '8.2.1', 'google-cloud-appengine-logging': '1.8.0', 'Jinja2': '3.1.6', 'fickling': '0.1.10', 'referencing':
'0.37.0', 'tyro': '1.0.10', 'MarkupSafe': '3.0.3', 'google-cloud-artifact-registry': '1.20.0', 'PyJWT': '2.12.1', 'distro': '1.9.0', 'GitPython': '3.1.46', 'tabulate': '0.10.0', 'graphviz': '0.21',
'chardet': '7.3.0', 'Pygments': '2.19.2', 'PyNaCl': '1.6.2', 'attrs': '26.1.0', 'aiohappyeyeballs': '2.6.1', 'requests-toolbelt': '1.0.0', 'docker': '7.1.0', 'packaging': '26.0', 'httpx': '0.28.1',
'grpc-google-iam-v1': '0.14.3', 'sentry-sdk': '2.56.0', 'asgiref': '3.10.0', 'backoff': '2.2.1', 'pyasn1_modules': '0.4.2', 'opentelemetry-sdk': '1.38.0', 'psutil': '7.2.2', 'google-api-core':
'2.30.0', 'graphql-core': '3.2.8', 'typing-inspection': '0.4.2', 'zipp': '3.23.0', 'annotated-types': '0.7.0', 'google-cloud-secret-manager': '2.26.0', 'cryptography': '46.0.5',
'charset-normalizer': '3.4.6', 'rpds-py': '0.30.0', 'six': '1.17.0', 'requests-oauthlib': '2.0.0', 'google-cloud-resource-manager': '1.16.0', 'click-option-group': '0.5.7', 'kfp-pipeline-spec':
'2.16.0', 'wandb': '0.25.1', 'google-cloud-aiplatform': '1.142.0', 'gcsfs': '2024.12.0', 'grpcio-status': '1.78.0', 'google-cloud-storage': '3.10.1', 'anyio': '4.13.0', 'pyasn1': '0.6.3', 'tomli':
'2.4.0', 'more-itertools': '10.8.0', 'jaraco.text': '4.0.0', 'wheel': '0.46.3', 'autocommand': '2.2.2', 'backports.tarfile': '1.2.0', 'jaraco.functools': '4.4.0', 'jaraco.context': '6.1.0'}
CURRENT STACK
Name: [name]
ID: [uuid]
User: [email] / [uuid]
ALERTER: zenml-notifications
Name: zenml-notifications
ID: [uuid]
Type: alerter
Flavor: slack
Configuration: {'slack_channel_id': [channel-id], 'timeout': 300, 'slack_token': '********'}
User: [uuid] / [uuid]
ORCHESTRATOR: [name]
Name: [name]
ID: [uuid]
Type: orchestrator
Flavor: kubernetes
The attribute stream_step_logs of class KubernetesOrchestratorConfig is deprecated and will be removed in the future.
Configuration: {'synchronous': True, 'service_account_name': 'zenml-runner', 'step_pod_service_account_name': 'zenml-runner', 'pod_settings': KubernetesPodSettings(node_selectors={}, affinity={},
tolerations=[], resources={'requests': {'memory': '200Mi', 'cpu': '100m'}, 'limits': {'memory': '500Mi', 'cpu': '600m'}}, annotations={}, volumes=[], volume_mounts=[], host_ipc=False,
scheduler_name=None, image_pull_secrets=['space-registry-grazi', 'regcred'], labels={}, env=[{'name': 'ZENML_CONFIG_PATH', 'value': '/app/.config'}, {'name': 'ZENML_LOGGING_VERBOSITY', 'value':
'INFO'}], env_from=[], container_security_context={}, additional_pod_spec_args={}), 'orchestrator_pod_settings': KubernetesPodSettings(node_selectors={}, affinity={}, tolerations=[],
resources={'requests': {'memory': '200Mi', 'cpu': '100m'}, 'limits': {'memory': '500Mi', 'cpu': '600m'}}, annotations={}, volumes=[], volume_mounts=[], host_ipc=False, scheduler_name=None,
image_pull_secrets=['space-registry-grazi', 'regcred'], labels={}, env=[{'name': 'ZENML_CONFIG_PATH', 'value': '/app/.config'}, {'name': 'ZENML_LOGGING_VERBOSITY', 'value': 'INFO'}], env_from=[],
container_security_context={}, additional_pod_spec_args={}), 'max_parallelism': 50, 'ttl_seconds_after_finished': 30, 'orchestrator_job_backoff_limit': 3, 'fail_on_container_waiting_reasons':
['InvalidImageName', 'ErrImagePull', 'ImagePullBackOff', 'CreateContainerConfigError'], 'job_monitoring_interval': 25.0, 'interrupt_check_interval': 1.0, 'pod_stop_grace_period': 30,
'kubernetes_namespace': 'zenml-ai-for-code', 'parallel_step_startup_waiting_period': 1.0}
User: [email] / [uuid]
EXPERIMENT_TRACKER: [name]
Name: ai-for-code
ID: [uuid]
Type: experiment_tracker
Flavor: wandb
Configuration: {'settings': {'base_url': '[redacted]'}, 'api_key': '********', 'entity': [redacted], 'project_name': [redacted]}
User: [email] / [uuid]
ARTIFACT_STORE: [name]
Name: [name]
ID: [uuid]
Type: artifact_store
Flavor: gcp
Configuration: {'path': [gcp-path]}
User: [email] / [uuid]
IMAGE_BUILDER: subprocess-builder
Name: subprocess-builder
ID: [uuid]
Type: image_builder
Flavor: local
Configuration: {'use_subprocess_call': True}
User: [email] / [uuid]
CONTAINER_REGISTRY: [name]
Name: [name]
ID: [uuid]
Type: container_registry
Flavor: gcp
Configuration: {'uri': '[redacted]'}
User: [email] / [uuid]What happened?
Returning an Annotated[Tuple[...]] with substitutions breaks when switching to dynamic pipeline, works fine in conventional pipeline.
Also, I get a weird RuntimeError about log duplication afterward, at the bottom of the attached logs.
Reproduction steps
Un-comment the 2 commented lines to reproduce
from typing import Annotated, Tuple, List
from zenml import pipeline, step
from zenml.config import DockerSettings
@step(
substitutions={'number': 'i'},
runtime = 'isolated',
)
def worker(number: int) -> Tuple[
Annotated[int, 'original_{i}'],
Annotated[int, 'double_{i}']
]:
return number, number*2
@step(runtime='isolated')
def reducer(doubles: List[int]) -> int:
return sum(doubles)
@pipeline(
# dynamic = True,
enable_cache = False,
settings = dict(
docker = DockerSettings(
user = '1000:1000',
build_config=dict(
build_options={"platform": "linux/amd64"}
)
)
),
)
def main(n: int):
number_tuples = [
worker
.with_options(substitutions=dict(i=str(i)))
(number=i)
# .submit(number=i)
for i in range(n)
]
number_doubles = [t[1] for t in number_tuples]
return reducer(number_doubles)
if __name__ == "__main__":
main(3)Relevant log output
❯ uv run main.py
/Users/Aral.De.Moor/jb/dynamic-pipeline/.venv/lib/python3.11/site-packages/requests/__init__.py:113: RequestsDependencyWarning: urllib3 (2.6.3) or chardet (7.3.0)/charset_normalizer (3.4.6) doesn't match a supported version!
warnings.warn(
The attribute stream_step_logs of class KubernetesOrchestratorConfig is deprecated and will be removed in the future.
Initiating a new run for the pipeline: main.
The attribute stream_step_logs of class KubernetesOrchestratorSettings is deprecated and will be removed in the future.
Unable to find a build to reuse. A previous build can be reused when the following conditions are met:
* The existing build was created for the same stack, ZenML version and Python version
* The stack contains a container registry
* The Docker settings of the pipeline and all its steps are the same as for the existing build.
Building Docker image(s) for pipeline main.
Building Docker image europe-docker.pkg.dev/grazie-development/zenml-generated/zenml:main-orchestrator.
Detected requirements.txt or pyproject.toml files in the source root. In future versions of ZenML, these will be automatically picked up and installed in Docker images by default. To disable this behavior and keep the current behavior, set DockerSettings.disable_automatic_requirements_detection to True. If you want to enable this behavior right away, you can do so by setting DockerSettings.disable_automatic_requirements_detection to False.
- Including stack requirements: Jinja2, Pillow>=9.1.0, gcsfs!=2025.5.0,!=2025.5.0.post1,<=2024.12.0, google-cloud-aiplatform>=1.34.0, google-cloud-artifact-registry>=1.11.3, google-cloud-build>=3.11.0, google-cloud-container>=2.21.0, google-cloud-logging>=3.8.0, google-cloud-pipeline-components>=2.19.0, google-cloud-run>=0.10.0, google-cloud-secret-manager, google-cloud-storage>=2.9.0, grpcio!=1.68.*,!=1.69.*,!=1.70.*, kfp>=2.6.0, kubernetes, kubernetes>=21.7,<26, slack-sdk==3.30.0, urllib3<2.6.0, wandb>=0.12.12,<1.0.0, weave>=0.51.33,<1.0.0
[+] Building 0.5s (12/12) FINISHED docker:desktop-linux
=> CACHED [internal] load remote build context 0.0s
=> CACHED copy /context / 0.0s
=> [internal] load metadata for docker.io/zenmldocker/zenml:0.94.0-py3.11 0.4s
=> [1/8] FROM docker.io/zenmldocker/zenml:0.94.0-py3.11@sha256:dbdb1d6d06c36c09be95158ee42 0.0s
=> => resolve docker.io/zenmldocker/zenml:0.94.0-py3.11@sha256:dbdb1d6d06c36c09be95158ee42 0.0s
=> CACHED [2/8] WORKDIR /app 0.0s
=> CACHED [3/8] RUN pip install uv 0.0s
=> CACHED [4/8] COPY .zenml_stack_integration_requirements . 0.0s
=> CACHED [5/8] RUN uv pip install --no-cache-dir -r .zenml_stack_integration_requirements 0.0s
=> CACHED [6/8] COPY . . 0.0s
=> CACHED [7/8] RUN chmod -R a+rw . 0.0s
=> CACHED [8/8] RUN chown -R 1000:1000 . 0.0s
=> exporting to image 0.0s
=> => exporting layers 0.0s
=> => exporting manifest sha256:f04cae1bb913e56128333e9fad30ea68eb067cd1881d85ab43d1cf2839 0.0s
=> => exporting config sha256:f890c660bd64984e28fd1ecdfa53f16201e96dfbfe3fd7603b60f522daa6 0.0s
=> => exporting attestation manifest sha256:7fd82327dc5a22be1715d67a112c2fe02371ec1e827dd0 0.0s
=> => exporting manifest list sha256:d0d32d4716ab12c0d5d55a4665c12750dba55b6a7ff6d146c32af 0.0s
=> => naming to europe-docker.pkg.dev/grazie-development/zenml-generated/zenml:main-orches 0.0s
Login Succeeded
Pushing Docker image [docker-url].
Finished pushing Docker image.
Finished building Docker image(s).
Archiving pipeline code directory: ~/jb/dynamic-pipeline. If this is taking longer than you expected, make sure your source root is set correctly by running zenml init, and that it does not contain unnecessarily huge files.
Uploading code to [gs-url] (Size: 151.34 KiB).
Code upload finished.
Caching is disabled by default for main.
Using a build:
Image(s): [docker-uri]
Using user: [email]
Using stack: [stack-name]
alerter: zenml-notifications
orchestrator: [name]
experiment_tracker: [name]
artifact_store: [name]
image_builder: subprocess-builder
container_registry: [name]
Dashboard URL for Pipeline Run: [url]
Waiting for orchestrator job to finish...
/opt/venv/lib/python3.11/site-packages/requests/__init__.py:113: RequestsDependencyWarning: urllib3 (2.5.0) or chardet (7.3.0)/charset_normalizer (3.4.4) doesn't match a supported version!
warnings.warn(
Downloading code from artifact store path [gs-uri]
Code download finished.
The attribute stream_step_logs of class KubernetesOrchestratorConfig is deprecated and will be removed in the future.
Uploading external artifact to 'external_artifacts/external_e796d6da-5439-4c99-aa60-fb89c58a0cfe'.
Finished uploading external artifact 9f0d09e6-70a1-4c67-8067-f43a5f6cefed.
Uploading external artifact to 'external_artifacts/external_78419adb-0ab5-4331-89d3-851e45d5e3a4'.
Finished uploading external artifact 8aee22b1-faca-466f-b99c-da8db9786ded.
Uploading external artifact to 'external_artifacts/external_31d3664b-a854-4d3d-8ac1-77b0ac4eccca'.
Finished uploading external artifact be4005f1-30c4-4636-8362-5a00b2230db9.
Waiting for step(s) worker, worker_2, worker_3 to finish before executing step reducer.
Step worker has started.
Launching job for step worker.
The attribute stream_step_logs of class KubernetesOrchestratorSettings is deprecated and will be removed in the future.
Unable to find config template for step worker. Falling back to the pipeline image.
Step worker launched.
Step worker_2 has started.
Launching job for step worker_2.
Unable to find config template for step worker_2. Falling back to the pipeline image.
Step worker_2 launched.
Step worker_3 has started.
Launching job for step worker_3.
Unable to find config template for step worker_3. Falling back to the pipeline image.
Step worker_3 launched.
Step worker finished successfully in 36.000s.
Step worker_2 finished successfully in 32.000s.
Step worker_3 finished successfully in 36.000s.
╭───────────────────── Traceback (most recent call last) ──────────────────────╮
│ in _run_module_as_main:198 │
│ in _run_code:88 │
│ │
│ /opt/venv/lib/python3.11/site-packages/zenml/entrypoints/entrypoint.py:48 in │
│ <module> │
│ │
│ 45 │
│ 46 │
│ 47 if __name__ == "__main__": │
│ ❱ 48 │ main() │
│ 49 │
│ │
│ /opt/venv/lib/python3.11/site-packages/zenml/entrypoints/entrypoint.py:44 in │
│ main │
│ │
│ 41 │ │ ) │
│ 42 │ │ entrypoint_config = entrypoint_config_class(arguments=remaining │
│ 43 │ │ │
│ ❱ 44 │ │ entrypoint_config.run() │
│ 45 │
│ 46 │
│ 47 if __name__ == "__main__": │
│ │
│ /opt/venv/lib/python3.11/site-packages/zenml/pipelines/dynamic/entrypoint_co │
│ nfiguration.py:74 in run │
│ │
│ 71 │ │ │ run = Client().get_pipeline_run(UUID(run_id)) │
│ 72 │ │ │
│ 73 │ │ runner = DynamicPipelineRunner(snapshot=snapshot, run=run) │
│ ❱ 74 │ │ runner.run_pipeline() │
│ 75 │
│ │
│ /opt/venv/lib/python3.11/site-packages/zenml/execution/pipeline/dynamic/runn │
│ er.py:503 in run_pipeline │
│ │
│ 500 │ │ │ │ │ # (artifacts, json serializable, anything?) │
│ 501 │ │ │ │ │ # how do we show it in the UI? │
│ 502 │ │ │ │ │ params = self.pipeline.configuration.parameters o │
│ ❱ 503 │ │ │ │ │ self.pipeline._call_entrypoint(**params) │
│ 504 │ │ │ │ │ # The pipeline function finished successfully, bu │
│ 505 │ │ │ │ │ # steps might still be running. We now wait for a │
│ 506 │ │ │ │ │ # them and raise any exceptions that occurred. │
│ │
│ /opt/venv/lib/python3.11/site-packages/zenml/pipelines/pipeline_definition.p │
│ y:1629 in _call_entrypoint │
│ │
│ 1626 │ │ """ │
│ 1627 │ │ self._clear_state() │
│ 1628 │ │ self._parameters = self._validate_entrypoint_args(*args, **kw │
│ ❱ 1629 │ │ return self.entrypoint(**self._parameters) │
│ 1630 │ │
│ 1631 │ def _prepare_if_possible(self) -> None: │
│ 1632 │ │ """Prepares the pipeline if possible. │
│ │
│ /app/code/main.py:40 in main │
│ │
│ 37 │ │ for i in range(n) │
│ 38 │ ] │
│ 39 │ number_doubles = [t[1] for t in number_tuples] │
│ ❱ 40 │ return reducer(number_doubles) │
│ 41 │
│ 42 │
│ 43 if __name__ == "__main__": │
│ │
│ /opt/venv/lib/python3.11/site-packages/zenml/steps/base_step.py:651 in │
│ __call__ │
│ │
│ 648 │ │ │ │ ], │
│ 649 │ │ │ │ after, │
│ 650 │ │ │ ) │
│ ❱ 651 │ │ │ return run_context.runner.launch_step( │
│ 652 │ │ │ │ step=self, │
│ 653 │ │ │ │ id=id, │
│ 654 │ │ │ │ args=args, │
│ │
│ /opt/venv/lib/python3.11/site-packages/zenml/execution/pipeline/dynamic/runn │
│ er.py:621 in launch_step │
│ │
│ 618 │ │ │ │ invocation_id, │
│ 619 │ │ │ ) │
│ 620 │ │ │
│ ❱ 621 │ │ compiled_step = compile_dynamic_step_invocation( │
│ 622 │ │ │ snapshot=self._snapshot, │
│ 623 │ │ │ pipeline=self.pipeline, │
│ 624 │ │ │ step=step, │
│ │
│ /opt/venv/lib/python3.11/site-packages/zenml/execution/pipeline/dynamic/runn │
│ er.py:923 in compile_dynamic_step_invocation │
│ │
│ 920 │ │ │ │ │ future.result() │
│ 921 │ │ │ │ │ upstream_steps.add(future.invocation_id) │
│ 922 │ │
│ ❱ 923 │ inputs = await_step_inputs(inputs) │
│ 924 │ │
│ 925 │ for value in inputs.values(): │
│ 926 │ │ if isinstance(value, OutputArtifact): │
│ │
│ /opt/venv/lib/python3.11/site-packages/zenml/execution/pipeline/dynamic/runn │
│ er.py:1273 in await_step_inputs │
│ │
│ 1270 │ │ │ and value │
│ 1271 │ │ │ and all(isinstance(item, ArtifactFuture) for item in valu │
│ 1272 │ │ ): │
│ ❱ 1273 │ │ │ value = [item.result() for item in value] │
│ 1274 │ │ │
│ 1275 │ │ if isinstance(value, ArtifactFuture): │
│ 1276 │ │ │ value = value.result() │
│ │
│ /opt/venv/lib/python3.11/site-packages/zenml/execution/pipeline/dynamic/runn │
│ er.py:1273 in <listcomp> │
│ │
│ 1270 │ │ │ and value │
│ 1271 │ │ │ and all(isinstance(item, ArtifactFuture) for item in valu │
│ 1272 │ │ ): │
│ ❱ 1273 │ │ │ value = [item.result() for item in value] │
│ 1274 │ │ │
│ 1275 │ │ if isinstance(value, ArtifactFuture): │
│ 1276 │ │ │ value = value.result() │
│ │
│ /opt/venv/lib/python3.11/site-packages/zenml/execution/pipeline/dynamic/outp │
│ uts.py:255 in result │
│ │
│ 252 │ │ │ load_step_run_outputs, │
│ 253 │ │ ) │
│ 254 │ │ │
│ ❱ 255 │ │ result = load_step_run_outputs(step_run.id) │
│ 256 │ │ │
│ 257 │ │ if isinstance(result, OutputArtifact): │
│ 258 │ │ │ return result │
│ │
│ /opt/venv/lib/python3.11/site-packages/zenml/execution/pipeline/dynamic/util │
│ s.py:194 in load_step_run_outputs │
│ │
│ 191 │ else: │
│ 192 │ │ # Make sure we return them in the same order as they're define │
│ 193 │ │ # step configuration, as we don't enforce any ordering in the │
│ ❱ 194 │ │ return tuple( │
│ 195 │ │ │ _convert_output_artifact( │
│ 196 │ │ │ │ output_name=name, artifact=output_artifacts[name] │
│ 197 │ │ │ ) │
│ │
│ /opt/venv/lib/python3.11/site-packages/zenml/execution/pipeline/dynamic/util │
│ s.py:196 in <genexpr> │
│ │
│ 193 │ │ # step configuration, as we don't enforce any ordering in the │
│ 194 │ │ return tuple( │
│ 195 │ │ │ _convert_output_artifact( │
│ ❱ 196 │ │ │ │ output_name=name, artifact=output_artifacts[name] │
│ 197 │ │ │ ) │
│ 198 │ │ │ for name in step_run.config.outputs.keys() │
│ 199 │ │ ) │
╰──────────────────────────────────────────────────────────────────────────────╯
KeyError: 'original_{i}'
/opt/venv/lib/python3.11/site-packages/requests/__init__.py:113: RequestsDependencyWarning: urllib3 (2.5.0) or chardet (7.3.0)/charset_normalizer (3.4.4) doesn't match a supported version!
warnings.warn(
Downloading code from artifact store path [gs-uri].
Code download finished.
The attribute stream_step_logs of class KubernetesOrchestratorConfig is deprecated and will be removed in the future.
╭───────────────────── Traceback (most recent call last) ──────────────────────╮
│ in _run_module_as_main:198 │
│ in _run_code:88 │
│ │
│ /opt/venv/lib/python3.11/site-packages/zenml/entrypoints/entrypoint.py:48 in │
│ <module> │
│ │
│ 45 │
│ 46 │
│ 47 if __name__ == "__main__": │
│ ❱ 48 │ main() │
│ 49 │
│ │
│ /opt/venv/lib/python3.11/site-packages/zenml/entrypoints/entrypoint.py:44 in │
│ main │
│ │
│ 41 │ │ ) │
│ 42 │ │ entrypoint_config = entrypoint_config_class(arguments=remaining │
│ 43 │ │ │
│ ❱ 44 │ │ entrypoint_config.run() │
│ 45 │
│ 46 │
│ 47 if __name__ == "__main__": │
│ │
│ /opt/venv/lib/python3.11/site-packages/zenml/pipelines/dynamic/entrypoint_co │
│ nfiguration.py:74 in run │
│ │
│ 71 │ │ │ run = Client().get_pipeline_run(UUID(run_id)) │
│ 72 │ │ │
│ 73 │ │ runner = DynamicPipelineRunner(snapshot=snapshot, run=run) │
│ ❱ 74 │ │ runner.run_pipeline() │
│ 75 │
│ │
│ /opt/venv/lib/python3.11/site-packages/zenml/execution/pipeline/dynamic/runn │
│ er.py:459 in run_pipeline │
│ │
│ 456 │ │ """ │
│ 457 │ │ logs_context: ContextManager[Any] = nullcontext() │
│ 458 │ │ if is_pipeline_logging_enabled(self._snapshot.pipeline_config │
│ ❱ 459 │ │ │ logs_context = setup_logging_context( │
│ 460 │ │ │ │ source="orchestrator", pipeline_run=self._run │
│ 461 │ │ │ ) │
│ 462 │
│ │
│ /opt/venv/lib/python3.11/site-packages/zenml/utils/logging_utils.py:582 in │
│ setup_logging_context │
│ │
│ 579 │ │ if step_run is None: │
│ 580 │ │ │ logs_request.pipeline_run_id = pipeline_run.id │
│ 581 │ │
│ ❱ 582 │ logs_response = Client().zen_store.create_logs(logs_request) │
│ 583 │ │
│ 584 │ return LoggingContext( │
│ 585 │ │ name=str(logs_response.id), │
│ │
│ /opt/venv/lib/python3.11/site-packages/zenml/zen_stores/rest_zen_store.py:13 │
│ 87 in create_logs │
│ │
│ 1384 │ │ Returns: │
│ 1385 │ │ │ The created logs entry. │
│ 1386 │ │ """ │
│ ❱ 1387 │ │ return self._create_resource( │
│ 1388 │ │ │ resource=logs, │
│ 1389 │ │ │ route=LOGS, │
│ 1390 │ │ │ response_model=LogsResponse, │
│ │
│ /opt/venv/lib/python3.11/site-packages/zenml/zen_stores/rest_zen_store.py:48 │
│ 43 in _create_resource │
│ │
│ 4840 │ │ Returns: │
│ 4841 │ │ │ The created resource. │
│ 4842 │ │ """ │
│ ❱ 4843 │ │ response_body = self.post(f"{route}", body=resource, params=p │
│ 4844 │ │ │
│ 4845 │ │ return response_model.model_validate(response_body) │
│ 4846 │
│ │
│ /opt/venv/lib/python3.11/site-packages/zenml/zen_stores/rest_zen_store.py:47 │
│ 83 in post │
│ │
│ 4780 │ │ Returns: │
│ 4781 │ │ │ The response body. │
│ 4782 │ │ """ │
│ ❱ 4783 │ │ return self._request( │
│ 4784 │ │ │ "POST", │
│ 4785 │ │ │ self.url + API + VERSION_1 + path, │
│ 4786 │ │ │ json=body.model_dump(mode="json"), │
│ │
│ /opt/venv/lib/python3.11/site-packages/zenml/zen_stores/rest_zen_store.py:46 │
│ 30 in _request │
│ │
│ 4627 │ │ │ │ ) │
│ 4628 │ │ │ │ │
│ 4629 │ │ │ │ status_code = str(response.status_code) │
│ ❱ 4630 │ │ │ │ return self._handle_response(response) │
│ 4631 │ │ │ except CredentialsNotValid as e: │
│ 4632 │ │ │ │ # NOTE: CredentialsNotValid is raised only when the s │
│ 4633 │ │ │ │ # explicitly indicates that the credentials are not v │
│ │
│ /opt/venv/lib/python3.11/site-packages/zenml/zen_stores/rest_zen_store.py:45 │
│ 49 in _handle_response │
│ │
│ 4546 │ │ elif response.status_code >= 400: │
│ 4547 │ │ │ exc = exception_from_response(response) │
│ 4548 │ │ │ if exc is not None: │
│ ❱ 4549 │ │ │ │ raise exc │
│ 4550 │ │ │ else: │
│ 4551 │ │ │ │ raise RuntimeError( │
│ 4552 │ │ │ │ │ f"{response.status_code} HTTP Error received from │
╰──────────────────────────────────────────────────────────────────────────────╯
RuntimeError: (pymysql.err.IntegrityError) (1062, "Duplicate entry
'27f216d3-83ab-4a84-a904-7223dc7f0cf8-None-orchestrator' for key
'logs.unique_log_key'")
[SQL: INSERT INTO logs (id, created, updated, uri, source, log_key, project_id,
user_id, pipeline_run_id, step_run_id, artifact_store_id, log_store_id) VALUES
(%(id)s, %(created)s, %(updated)s, %(uri)s, %(source)s, %(log_key)s,
%(project_id)s, %(user_id)s, %(pipeline_run_id)s, %(step_run_id)s,
%(artifact_store_id)s, %(log_store_id)s)]
[parameters: {'id': [uuid], 'created':
datetime.datetime(2026, 3, 25, 23, 50, 8, 836149), 'updated':
datetime.datetime(2026, 3, 25, 23, 50, 8, 836180), 'uri':
[gs-uri], 'source':
'orchestrator', 'log_key':
'27f216d3-83ab-4a84-a904-7223dc7f0cf8-None-orchestrator', 'project_id':
'9854f8ad178d4fabb8275188eeae47aa', 'user_id':
'b121f647e61549a5adb8e55fbde72011', 'pipeline_run_id':
'27f216d383ab4a84a9047223dc7f0cf8', 'step_run_id': None, 'artifact_store_id':
'485ccd38eeaf4e39bba7aebc906c382b', 'log_store_id': None}]
(Background on this error at: https://sqlalche.me/e/20/gkpj)
/opt/venv/lib/python3.11/site-packages/requests/__init__.py:113: RequestsDependencyWarning: urllib3 (2.5.0) or chardet (7.3.0)/charset_normalizer (3.4.4) doesn't match a supported version!
warnings.warn(Code of Conduct
- I agree to follow this project's Code of Conduct