Skip to content

Add rollbackVersion option and default releaseTrain to stable in Ev2 SDP pipelines - #1781

Open
suyadav1 wants to merge 1 commit into
ci_prodfrom
suyadav/ev2-sdp-rollback-version
Open

Add rollbackVersion option and default releaseTrain to stable in Ev2 SDP pipelines#1781
suyadav1 wants to merge 1 commit into
ci_prodfrom
suyadav/ev2-sdp-rollback-version

Conversation

@suyadav1

Copy link
Copy Markdown
Contributor

Summary

Adds a rollbackVersion parameter to the two central-artifacts Ev2 Managed SDP rollout pipelines, and changes the releaseTrain default from preview to stable.

  • .pipelines/azure-pipeline-aks-extension-managed-ev2-sdp.yml
  • .pipelines/azure-pipeline-arc-extension-managed-ev2-sdp.yml

1. rollbackVersion

These pipelines could roll a version forward but had no knob to roll a bad one back — that required an out-of-band edit to the version document in ClusterConfigArtifacts.

The new parameter is threaded into all five per-cloud ConfigurationOverrides blocks (Prod, Fairfax, Mooncake, USNat, USSec) in both files. Verified path through the central artifacts service:

Stage Artifact
Pipeline ConfigurationSpecification.settings.rollbackVersion
Ev2 scope CentralExtensionRolloutArtifacts/Configuration.ServiceScope.json (default "")
Ev2 binding ScopeBindings.json__ROLLBACK_VERSION__$config(rollbackVersion)
Rollout params ExtensionRegistrationParameters.json — env var ROLLBACK_VERSION
Script Scripts/register/register.sh — empty ⇒ "RollbackVersion": null

The default is a no-op. 'none' collapses to "", which register.sh writes as JSON null — byte-identical to the request body produced today. Existing runs are unaffected.

Design notes for reviewers

Why 'none' and not '' — Azure DevOps renders a parameter whose default is an empty string as a required field. The 'none' sentinel is the workaround used by the platform team's own reference pipeline, ClusterConfigurationAgent/.pipelines/test1pextension-ev2managedsdp.yml.

Why compile-time ${{ }} and not a runtime $() variable — an empty-valued macro variable is not reliably expanded by the agent. A leftover literal $(rollbackVersionValue) would be non-empty and would be registered verbatim as a bogus rollback target. Compile-time expansion emits a literal "rollbackVersion": "" into the YAML, which has no runtime failure mode.

Why the Arc expression appends before stripping — Arc versions are stored upstream with an -arc suffix (3.5.0-arc, 3.6.0-arc), and the pipeline already appends it for overrideExtensionVersion, so rollbackVersion follows the same convention. The order matters:

# AKS
"rollbackVersion": "${{ replace(parameters.rollbackVersion, 'none', '') }}"

# Arc — append '-arc' FIRST, strip sentinel SECOND
"rollbackVersion": "${{ replace(format('{0}-arc', parameters.rollbackVersion), 'none-arc', '') }}"

Stripping first would leave a bare "-arc" on the default path — non-empty, and therefore registered as a real rollback to a nonexistent version.

How to actually roll backRollbackVersion is stored on the version document being registered (the rollout PATCHes .../versions/<version>), so a rollback run sets both:

Parameter Value
overrideExtensionVersion the bad version currently in the field (roll back FROM)
rollbackVersion the last known-good version (roll back TO)

This is non-obvious, so it is documented in the parameter comments in both files.

2. releaseTrain default previewstable

preview did not reflect how these pipelines are actually run — production version documents are registered with "Releasetrains": ["stable"] (e.g. 3.8.0-ci-prod-09-03-2026-fd42f68c). preview remains selectable in the dropdown.

Important

This is the one genuine behavioural change in the PR: a manually queued run that leaves the dropdown untouched now targets stable (customer-facing) instead of preview. Intentional, but worth a conscious ack before merge.

3. Drive-by

Reworded a stale comment in the Arc pipeline that described overrideExtensionVersion as the mechanism "for rollback" — misleading once a dedicated rollbackVersion parameter sits directly below it.

Testing

No build/test target covers these files, so validation was done by rendering them:

  • Both files parse as YAML; 0 editor diagnostics.
  • Built a small evaluator for the ADO replace/format expression subset and ran it against the actual expression text extracted from the files, rendering all 10 ConfigurationOverrides blocks under both scenarios — 20/20 checks pass, every rendered block is valid JSON:
Pipeline Input Rendered register.sh emits
AKS none (default) "" RollbackVersion: null
AKS 3.7.0-ci-prod-08-28-2026-925ade3d same that version
Arc none (default) "" RollbackVersion: null
Arc 3.7.0-ci-prod-08-28-2026-925ade3d …-925ade3d-arc suffixed version
  • Diff contains no incidental reformatting: the only removed lines are the two default: 'preview' lines and the two reworded comment lines.

…SDP pipelines

Adds a `rollbackVersion` parameter to the AKS and Arc central-artifacts Ev2
Managed SDP rollout pipelines, and flips the `releaseTrain` default from
`preview` to `stable`.

rollbackVersion
---------------
The pipelines could roll a version forward but had no knob to roll a bad one
back; that required an out-of-band edit to the version document in
ClusterConfigArtifacts.

The new parameter is threaded into all five per-cloud ConfigurationOverrides
blocks (Prod, Fairfax, Mooncake, USNat, USSec) in both files. It maps to
settings.rollbackVersion -> $config(rollbackVersion) -> __ROLLBACK_VERSION__ ->
the ROLLBACK_VERSION env var consumed by the central artifacts register.sh,
which writes an empty value as JSON null. The default is therefore equivalent
to today's behaviour and existing runs are unaffected.

Default is the literal 'none' rather than an empty string because Azure DevOps
renders a parameter with an empty-string default as a required field. This
matches the platform team's reference pipeline
(ClusterConfigurationAgent/.pipelines/test1pextension-ev2managedsdp.yml).

Values are resolved with compile-time ${{ }} expressions rather than runtime
$() macros: an empty-valued macro variable is not reliably expanded, and a
leftover literal would be non-empty and registered as a bogus rollback target.

The Arc pipeline appends the '-arc' suffix, consistent with how it already
handles overrideExtensionVersion. The suffix is appended BEFORE the sentinel is
stripped, so 'none' becomes 'none-arc' and then collapses to an empty string.
Stripping first would leave a bare '-arc', which is non-empty and would be
registered as a real rollback target pointing at a nonexistent version.

Note that RollbackVersion is stored on the version document being registered
(the rollout PATCHes .../versions/<version>), so a rollback run sets both
overrideExtensionVersion (the version to roll back FROM) and rollbackVersion
(the version to roll back TO). This is documented in the parameter comments.

releaseTrain
------------
Defaulted to 'preview', which did not reflect how these pipelines are run:
production version documents are registered with Releasetrains ["stable"].
'preview' remains selectable in the dropdown.

Note this is a behavioural change: a manually queued run that leaves the
dropdown untouched now targets stable rather than preview.

Also rewords a stale comment in the Arc pipeline that described
overrideExtensionVersion as the rollback mechanism, which became misleading
once a dedicated rollbackVersion parameter existed.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 5ba4db91-9e37-443d-b1cf-a41725eae1a5
@suyadav1
suyadav1 requested a review from a team as a code owner September 11, 2026 18:19
@azure-pipelines

Copy link
Copy Markdown
Azure Pipelines:
There may be pipelines that require an authorized user to comment /azp run to run.

@suyadav1

Copy link
Copy Markdown
Contributor Author

/azp run

@azure-pipelines

Copy link
Copy Markdown
Azure Pipelines:
Successfully started running 1 pipeline(s).
2 pipeline(s) were filtered out due to trigger conditions.

"userAssignedIdentity": "/subscriptions/30c56c3a-54da-46ea-b004-06eb33432687/resourceGroups/containerinsightsprod/providers/Microsoft.ManagedIdentity/userAssignedIdentities/ev2-agent-release",
"version": "$(extensionVersionValue)",
"releaseTrain": "${{ parameters.releaseTrain }}",
"rollbackVersion": "${{ replace(parameters.rollbackVersion, 'none', '') }}",

@suyadav1 suyadav1 Sep 11, 2026

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Comment on lines +43 to +44
# being registered (the rollout PATCHes .../versions/<version>), so set BOTH:
# overrideExtensionVersion -> the bad version currently in the field (roll back FROM)

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

- preview
- stable

# Optional. 'none' (the default) means NO rollback: the run is a normal

@zanejohnson-azure zanejohnson-azure Sep 11, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

based on the comment, it seems the release & rollback process is as follows:

  1. we start a release e.g. version 4.1.0, without specifying rollback version.
  2. eng found something bad happening and want to rollout to 4.0.0.
  3. we start another release by specifying bad version 4.1.0 and rollback version 4.0.0.

what I would expect:

  1. every time we start a new release, we must specify a rollback version par. e.g. when we release 4.1.0, we specify in the release pipeline a rollback version of 4.0.0.
  2. if everything goes well, no rollback.
  3. if rollback is needed, we rollback for one or more regions by going through manual ev2 rollout portal.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

When a rollback version is provided, the system rolls back every cluster on the roll-forward version to the rollback version that's why we shouldn't set this when rolling out a new version. The guidance from extension team is to start another release to rollback the version.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

hmm, interesting. i think you were referring to this section https://msazure.visualstudio.com/One/_wiki/wikis/One.wiki/1149646/Extension-Type-Registration?anchor=rolling-back-a-released-version

while reading this, what I understood the ^ link meant
scenario: one release is done, e.g. 4.1.0. a few days, we found issue we need rollback. since it has been a while, ev2 closed the rollback window already on ev2 portal.
when this happens, we need trigger a new release by specifying both bad version of 4.1.0 and rollback version of 4.0.0.

The use case I was thinking:
cx are sitting on 4.0.0 now.
we start a new release with 4.1.0 and in the meantime specifying 4.0.0 as rollback version. what I understand is cx will get 4.1.0 deployed unless one of the 2 conditions happens:

  1. eng manually click rollback for one or more regions on ev2 pipeline during a rollout.
  2. pipeline healthcheck fail after deployment in a region and ev2 will kick off the rollback automatically for that particular region, and all following region deployment should stop.

^ use case worth testing as it saves us effort from doing a new release, slow and error prone.

@zanejohnson-azure zanejohnson-azure left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

approved. left one comment about one rollback scenario that is worth testing.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants