Skip to content

Add hints process directive for executor-specific scheduling hints#7034

Open
pditommaso wants to merge 15 commits intomasterfrom
260414-hints-directive
Open

Add hints process directive for executor-specific scheduling hints#7034
pditommaso wants to merge 15 commits intomasterfrom
260414-hints-directive

Conversation

@pditommaso
Copy link
Copy Markdown
Member

@pditommaso pditommaso commented Apr 14, 2026

Summary

  • Introduce a new map-type hints process directive for executor-specific hints
  • Shape: Map<String, String> where keys are either name or executor/name
  • Unprefixed keys apply to any executor that recognizes them; prefixed keys restrict to that executor. When both are present, the prefixed form wins for its executor
  • Core only validates the map shape (non-empty keys, at most one /, string values) — executors validate the keys they recognize
  • AWS Batch: consumableResources hint → Batch ConsumableResourceProperties on the job definition; unknown awsbatch/* keys trigger a log.warn1
  • Seqera executor: 11 machineRequirement.* fields overlay the config-scope seqera.executor.machineRequirement. Known keys are derived from MachineRequirementOpts via @ConfigOption reflection — adding a field there is automatically picked up. Unknown seqera/-prefixed keys throw.

Context

Follows the refined ADR proposal from #6960 (comment). Replaces fragile clusterOptions and overloaded ext/resourceLabels workarounds.

Ref: #5917, #6960

Example

process hello {
    // applies to any executor that supports it
    hints consumableResources: 'my-license=1'

    // restricted to AWS Batch
    hints 'awsbatch/consumableResources': 'my-license=1'

    // overrides the seqera.executor.machineRequirement.provisioning config value
    hints 'seqera/machineRequirement.provisioning': 'spotFirst'
}

Test plan

  • HintDefsTest — shape validation: string values only, null allowed, non-empty keys, at most one /
  • ProcessBuilderTest — hints DSL accumulation, duplicate-key overwrite, non-string rejection
  • TaskConfigTestgetHints() retrieval, empty-map default, config override replaces the entire map
  • AwsBatchTaskHandlerTest — real ProcessBuilder → TaskConfig → makeJobDefRequest integration; awsbatch/-prefixed precedence over unprefixed; malformed name=qty inputs fail with clear errors; whitespace tolerance
  • HintHelperTest — prefixed + unprefixed resolution with precedence; all machineRequirement.* fields overlay; KNOWN_KEYS stays in sync with declared fields; unknown seqera/* throws
  • SeqeraTaskHandlerTest — end-to-end submit() wiring: hint overrides config-scope machineRequirement, foreign-namespaced hint is ignored, unknown seqera/* fails submission
  • Docs: process.md hints directive, executor.md Seqera supported-hints table

🤖 Generated with Claude Code

robsyme and others added 4 commits March 23, 2026 17:05
Propose a generic 'hints' process directive with namespaced keys as
the extension point for executor-specific scheduling hints. First use
case: AWS Batch consumable resources for license-seat management.

Evaluates three options: dedicated consumableResources directive,
overloading resourceLabels, and a new hints directive. Recommends
hints for extensibility and separation of concerns.

Ref: #5917
Signed-off-by: Rob Syme <rob.syme@gmail.com>
Signed-off-by: Paolo Di Tommaso <paolo.ditommaso@gmail.com>

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Signed-off-by: Paolo Di Tommaso <paolo.ditommaso@gmail.com>
Signed-off-by: Ben Sherman <bentshermann@gmail.com>
Introduce a new map-type `hints` process directive that provides a
structured, extensible way for pipeline authors to pass executor-specific
scheduling hints. Keys use `[executor/][scope.]hintName` format.

Core features:
- Multiple `hints` calls within a process body accumulate (merge)
- Config overrides via withName:/withLabel: replace the entire map
- Values support String, Integer, and Closure types
- Two-tier validation: warnings for unknown unprefixed keys (global
  registry), errors for unknown executor-prefixed keys
- Initial global catalog contains only `consumableResource`

Seqera Platform integration:
- `seqera/machineRequirement.*` hints map to MachineRequirementOpts
  fields (arch, provisioning, maxSpotAttempts, machineTypes, diskType,
  diskThroughputMiBps, diskIops, diskEncrypted, diskAllocation,
  diskMountPath, diskSize, capacityMode)
- Hints override Seqera config scope values at the task level
- Unknown `seqera/` keys produce an error

Ref: #5917, #6960

Signed-off-by: Paolo Di Tommaso <paolo.ditommaso@gmail.com>
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Signed-off-by: Paolo Di Tommaso <paolo.ditommaso@gmail.com>
@netlify
Copy link
Copy Markdown

netlify Bot commented Apr 14, 2026

Deploy Preview for nextflow-docs-staging ready!

Name Link
🔨 Latest commit 43d69fc
🔍 Latest deploy log https://app.netlify.com/projects/nextflow-docs-staging/deploys/69ebadfc828c3d00080fbd61
😎 Deploy Preview https://deploy-preview-7034--nextflow-docs-staging.netlify.app
📱 Preview on mobile
Toggle QR Code...

QR Code

Use your smartphone camera to open QR code link.

To edit notification comments on pull requests, go to your Netlify project configuration.

bentsherman and others added 3 commits April 16, 2026 14:57
Signed-off-by: Ben Sherman <bentshermann@gmail.com>
Signed-off-by: Ben Sherman <bentshermann@gmail.com>
TaskConfig.getHints() returns Map<String, String>, but HintHelper
declared Map<String, Object>, causing @CompileStatic compile errors
in SeqeraTaskHandler.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Signed-off-by: Paolo Di Tommaso <paolo.ditommaso@gmail.com>
@pditommaso pditommaso marked this pull request as ready for review April 19, 2026 19:42
@pditommaso pditommaso requested review from a team as code owners April 19, 2026 19:42
pditommaso and others added 4 commits April 19, 2026 22:21
… fast]

TaskConfig.getHints() always delivers Map<String, String>; tests now
reflect that contract instead of relying on generics erasure.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Signed-off-by: Paolo Di Tommaso <paolo.ditommaso@gmail.com>
- Enforce Map<String,String> shape; remove AWS-specific KNOWN_HINTS leak
  from core HintDefs. Core validates shape only; executors validate their
  own namespaces.
- AWS Batch: harden consumableResources parser with clear errors, prefer
  awsbatch/-prefixed key, warn on unknown awsbatch/* keys.
- Seqera: derive known hint keys from MachineRequirementOpts via @ConfigOption
  reflection; overlay preserves all declared fields. Accept unprefixed
  machineRequirement.* keys too — prefixed wins on collision. Drop dangling
  references to the removed `arch` field.
- Replace Mock(TaskConfig) integration test in AWS Batch with a real
  ProcessBuilder → TaskConfig path; add compact SeqeraTaskHandler submit
  tests covering hint overlay and foreign-namespace passthrough.
- Docs: spell out unprefixed-applies-to-any-executor semantics; list all
  Seqera-supported hints.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Signed-off-by: Paolo Di Tommaso <paolo.ditommaso@gmail.com>
@pditommaso
Copy link
Copy Markdown
Member Author

@bentsherman this should be ready to go

Signed-off-by: Ben Sherman <bentshermann@gmail.com>
Signed-off-by: Ben Sherman <bentshermann@gmail.com>
Comment thread adr/20260323-hints-process-directive.md Outdated
Signed-off-by: Ben Sherman <bentshermann@gmail.com>
@bentsherman bentsherman added this to the 26.04 milestone Apr 24, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants