Add hints process directive for executor-specific scheduling hints#7034
Open
pditommaso wants to merge 15 commits intomasterfrom
Open
Add hints process directive for executor-specific scheduling hints#7034pditommaso wants to merge 15 commits intomasterfrom
pditommaso wants to merge 15 commits intomasterfrom
Conversation
Propose a generic 'hints' process directive with namespaced keys as the extension point for executor-specific scheduling hints. First use case: AWS Batch consumable resources for license-seat management. Evaluates three options: dedicated consumableResources directive, overloading resourceLabels, and a new hints directive. Recommends hints for extensibility and separation of concerns. Ref: #5917 Signed-off-by: Rob Syme <rob.syme@gmail.com>
Signed-off-by: Paolo Di Tommaso <paolo.ditommaso@gmail.com> Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> Signed-off-by: Paolo Di Tommaso <paolo.ditommaso@gmail.com>
Signed-off-by: Ben Sherman <bentshermann@gmail.com>
Introduce a new map-type `hints` process directive that provides a structured, extensible way for pipeline authors to pass executor-specific scheduling hints. Keys use `[executor/][scope.]hintName` format. Core features: - Multiple `hints` calls within a process body accumulate (merge) - Config overrides via withName:/withLabel: replace the entire map - Values support String, Integer, and Closure types - Two-tier validation: warnings for unknown unprefixed keys (global registry), errors for unknown executor-prefixed keys - Initial global catalog contains only `consumableResource` Seqera Platform integration: - `seqera/machineRequirement.*` hints map to MachineRequirementOpts fields (arch, provisioning, maxSpotAttempts, machineTypes, diskType, diskThroughputMiBps, diskIops, diskEncrypted, diskAllocation, diskMountPath, diskSize, capacityMode) - Hints override Seqera config scope values at the task level - Unknown `seqera/` keys produce an error Ref: #5917, #6960 Signed-off-by: Paolo Di Tommaso <paolo.ditommaso@gmail.com> Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> Signed-off-by: Paolo Di Tommaso <paolo.ditommaso@gmail.com>
✅ Deploy Preview for nextflow-docs-staging ready!
To edit notification comments on pull requests, go to your Netlify project configuration. |
6 tasks
Signed-off-by: Ben Sherman <bentshermann@gmail.com>
Signed-off-by: Ben Sherman <bentshermann@gmail.com>
TaskConfig.getHints() returns Map<String, String>, but HintHelper declared Map<String, Object>, causing @CompileStatic compile errors in SeqeraTaskHandler. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> Signed-off-by: Paolo Di Tommaso <paolo.ditommaso@gmail.com>
… fast] TaskConfig.getHints() always delivers Map<String, String>; tests now reflect that contract instead of relying on generics erasure. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> Signed-off-by: Paolo Di Tommaso <paolo.ditommaso@gmail.com>
- Enforce Map<String,String> shape; remove AWS-specific KNOWN_HINTS leak from core HintDefs. Core validates shape only; executors validate their own namespaces. - AWS Batch: harden consumableResources parser with clear errors, prefer awsbatch/-prefixed key, warn on unknown awsbatch/* keys. - Seqera: derive known hint keys from MachineRequirementOpts via @ConfigOption reflection; overlay preserves all declared fields. Accept unprefixed machineRequirement.* keys too — prefixed wins on collision. Drop dangling references to the removed `arch` field. - Replace Mock(TaskConfig) integration test in AWS Batch with a real ProcessBuilder → TaskConfig path; add compact SeqeraTaskHandler submit tests covering hint overlay and foreign-namespace passthrough. - Docs: spell out unprefixed-applies-to-any-executor semantics; list all Seqera-supported hints. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> Signed-off-by: Paolo Di Tommaso <paolo.ditommaso@gmail.com>
Member
Author
|
@bentsherman this should be ready to go |
Signed-off-by: Ben Sherman <bentshermann@gmail.com>
Signed-off-by: Ben Sherman <bentshermann@gmail.com>
bentsherman
reviewed
Apr 23, 2026
Signed-off-by: Ben Sherman <bentshermann@gmail.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
hintsprocess directive for executor-specific hintsMap<String, String>where keys are eithernameorexecutor/name/, string values) — executors validate the keys they recognizeconsumableResourceshint → BatchConsumableResourcePropertieson the job definition; unknownawsbatch/*keys trigger alog.warn1machineRequirement.*fields overlay the config-scopeseqera.executor.machineRequirement. Known keys are derived fromMachineRequirementOptsvia@ConfigOptionreflection — adding a field there is automatically picked up. Unknownseqera/-prefixed keys throw.Context
Follows the refined ADR proposal from #6960 (comment). Replaces fragile
clusterOptionsand overloadedext/resourceLabelsworkarounds.Ref: #5917, #6960
Example
Test plan
HintDefsTest— shape validation: string values only, null allowed, non-empty keys, at most one/ProcessBuilderTest— hints DSL accumulation, duplicate-key overwrite, non-string rejectionTaskConfigTest—getHints()retrieval, empty-map default, config override replaces the entire mapAwsBatchTaskHandlerTest— realProcessBuilder → TaskConfig → makeJobDefRequestintegration;awsbatch/-prefixed precedence over unprefixed; malformedname=qtyinputs fail with clear errors; whitespace toleranceHintHelperTest— prefixed + unprefixed resolution with precedence; allmachineRequirement.*fields overlay;KNOWN_KEYSstays in sync with declared fields; unknownseqera/*throwsSeqeraTaskHandlerTest— end-to-endsubmit()wiring: hint overrides config-scopemachineRequirement, foreign-namespaced hint is ignored, unknownseqera/*fails submissionprocess.mdhints directive,executor.mdSeqera supported-hints table🤖 Generated with Claude Code