Add rollbackVersion option and default releaseTrain to stable in Ev2 SDP pipelines - #1781
Add rollbackVersion option and default releaseTrain to stable in Ev2 SDP pipelines#1781suyadav1 wants to merge 1 commit into
Conversation
…SDP pipelines
Adds a `rollbackVersion` parameter to the AKS and Arc central-artifacts Ev2
Managed SDP rollout pipelines, and flips the `releaseTrain` default from
`preview` to `stable`.
rollbackVersion
---------------
The pipelines could roll a version forward but had no knob to roll a bad one
back; that required an out-of-band edit to the version document in
ClusterConfigArtifacts.
The new parameter is threaded into all five per-cloud ConfigurationOverrides
blocks (Prod, Fairfax, Mooncake, USNat, USSec) in both files. It maps to
settings.rollbackVersion -> $config(rollbackVersion) -> __ROLLBACK_VERSION__ ->
the ROLLBACK_VERSION env var consumed by the central artifacts register.sh,
which writes an empty value as JSON null. The default is therefore equivalent
to today's behaviour and existing runs are unaffected.
Default is the literal 'none' rather than an empty string because Azure DevOps
renders a parameter with an empty-string default as a required field. This
matches the platform team's reference pipeline
(ClusterConfigurationAgent/.pipelines/test1pextension-ev2managedsdp.yml).
Values are resolved with compile-time ${{ }} expressions rather than runtime
$() macros: an empty-valued macro variable is not reliably expanded, and a
leftover literal would be non-empty and registered as a bogus rollback target.
The Arc pipeline appends the '-arc' suffix, consistent with how it already
handles overrideExtensionVersion. The suffix is appended BEFORE the sentinel is
stripped, so 'none' becomes 'none-arc' and then collapses to an empty string.
Stripping first would leave a bare '-arc', which is non-empty and would be
registered as a real rollback target pointing at a nonexistent version.
Note that RollbackVersion is stored on the version document being registered
(the rollout PATCHes .../versions/<version>), so a rollback run sets both
overrideExtensionVersion (the version to roll back FROM) and rollbackVersion
(the version to roll back TO). This is documented in the parameter comments.
releaseTrain
------------
Defaulted to 'preview', which did not reflect how these pipelines are run:
production version documents are registered with Releasetrains ["stable"].
'preview' remains selectable in the dropdown.
Note this is a behavioural change: a manually queued run that leaves the
dropdown untouched now targets stable rather than preview.
Also rewords a stale comment in the Arc pipeline that described
overrideExtensionVersion as the rollback mechanism, which became misleading
once a dedicated rollbackVersion parameter existed.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 5ba4db91-9e37-443d-b1cf-a41725eae1a5
|
Azure Pipelines: There may be pipelines that require an authorized user to comment /azp run to run. |
|
/azp run |
|
Azure Pipelines: Successfully started running 1 pipeline(s). 2 pipeline(s) were filtered out due to trigger conditions. |
| "userAssignedIdentity": "/subscriptions/30c56c3a-54da-46ea-b004-06eb33432687/resourceGroups/containerinsightsprod/providers/Microsoft.ManagedIdentity/userAssignedIdentities/ev2-agent-release", | ||
| "version": "$(extensionVersionValue)", | ||
| "releaseTrain": "${{ parameters.releaseTrain }}", | ||
| "rollbackVersion": "${{ replace(parameters.rollbackVersion, 'none', '') }}", |
There was a problem hiding this comment.
| # being registered (the rollout PATCHes .../versions/<version>), so set BOTH: | ||
| # overrideExtensionVersion -> the bad version currently in the field (roll back FROM) |
There was a problem hiding this comment.
Here are the rollback instructions from extension team: https://msazure.visualstudio.com/One/_wiki/wikis/One.wiki/1149646/Extension-Type-Registration?anchor=rolling-back-a-released-version
| - preview | ||
| - stable | ||
|
|
||
| # Optional. 'none' (the default) means NO rollback: the run is a normal |
There was a problem hiding this comment.
based on the comment, it seems the release & rollback process is as follows:
- we start a release e.g. version 4.1.0, without specifying rollback version.
- eng found something bad happening and want to rollout to 4.0.0.
- we start another release by specifying bad version 4.1.0 and rollback version 4.0.0.
what I would expect:
- every time we start a new release, we must specify a rollback version par. e.g. when we release 4.1.0, we specify in the release pipeline a rollback version of 4.0.0.
- if everything goes well, no rollback.
- if rollback is needed, we rollback for one or more regions by going through manual ev2 rollout portal.
There was a problem hiding this comment.
When a rollback version is provided, the system rolls back every cluster on the roll-forward version to the rollback version that's why we shouldn't set this when rolling out a new version. The guidance from extension team is to start another release to rollback the version.
There was a problem hiding this comment.
hmm, interesting. i think you were referring to this section https://msazure.visualstudio.com/One/_wiki/wikis/One.wiki/1149646/Extension-Type-Registration?anchor=rolling-back-a-released-version
while reading this, what I understood the ^ link meant
scenario: one release is done, e.g. 4.1.0. a few days, we found issue we need rollback. since it has been a while, ev2 closed the rollback window already on ev2 portal.
when this happens, we need trigger a new release by specifying both bad version of 4.1.0 and rollback version of 4.0.0.
The use case I was thinking:
cx are sitting on 4.0.0 now.
we start a new release with 4.1.0 and in the meantime specifying 4.0.0 as rollback version. what I understand is cx will get 4.1.0 deployed unless one of the 2 conditions happens:
- eng manually click rollback for one or more regions on ev2 pipeline during a rollout.
- pipeline healthcheck fail after deployment in a region and ev2 will kick off the rollback automatically for that particular region, and all following region deployment should stop.
^ use case worth testing as it saves us effort from doing a new release, slow and error prone.
zanejohnson-azure
left a comment
There was a problem hiding this comment.
approved. left one comment about one rollback scenario that is worth testing.
Summary
Adds a
rollbackVersionparameter to the two central-artifacts Ev2 Managed SDP rollout pipelines, and changes thereleaseTraindefault frompreviewtostable..pipelines/azure-pipeline-aks-extension-managed-ev2-sdp.yml.pipelines/azure-pipeline-arc-extension-managed-ev2-sdp.yml1.
rollbackVersionThese pipelines could roll a version forward but had no knob to roll a bad one back — that required an out-of-band edit to the version document in
ClusterConfigArtifacts.The new parameter is threaded into all five per-cloud
ConfigurationOverridesblocks (Prod, Fairfax, Mooncake, USNat, USSec) in both files. Verified path through the central artifacts service:ConfigurationSpecification.settings.rollbackVersionCentralExtensionRolloutArtifacts/Configuration.ServiceScope.json(default"")ScopeBindings.json—__ROLLBACK_VERSION__→$config(rollbackVersion)ExtensionRegistrationParameters.json— env varROLLBACK_VERSIONScripts/register/register.sh— empty ⇒"RollbackVersion": nullThe default is a no-op.
'none'collapses to"", whichregister.shwrites as JSONnull— byte-identical to the request body produced today. Existing runs are unaffected.Design notes for reviewers
Why
'none'and not''— Azure DevOps renders a parameter whose default is an empty string as a required field. The'none'sentinel is the workaround used by the platform team's own reference pipeline,ClusterConfigurationAgent/.pipelines/test1pextension-ev2managedsdp.yml.Why compile-time
${{ }}and not a runtime$()variable — an empty-valued macro variable is not reliably expanded by the agent. A leftover literal$(rollbackVersionValue)would be non-empty and would be registered verbatim as a bogus rollback target. Compile-time expansion emits a literal"rollbackVersion": ""into the YAML, which has no runtime failure mode.Why the Arc expression appends before stripping — Arc versions are stored upstream with an
-arcsuffix (3.5.0-arc,3.6.0-arc), and the pipeline already appends it foroverrideExtensionVersion, sorollbackVersionfollows the same convention. The order matters:Stripping first would leave a bare
"-arc"on the default path — non-empty, and therefore registered as a real rollback to a nonexistent version.How to actually roll back —
RollbackVersionis stored on the version document being registered (the rolloutPATCHes.../versions/<version>), so a rollback run sets both:overrideExtensionVersionrollbackVersionThis is non-obvious, so it is documented in the parameter comments in both files.
2.
releaseTraindefaultpreview→stablepreviewdid not reflect how these pipelines are actually run — production version documents are registered with"Releasetrains": ["stable"](e.g.3.8.0-ci-prod-09-03-2026-fd42f68c).previewremains selectable in the dropdown.Important
This is the one genuine behavioural change in the PR: a manually queued run that leaves the dropdown untouched now targets
stable(customer-facing) instead ofpreview. Intentional, but worth a conscious ack before merge.3. Drive-by
Reworded a stale comment in the Arc pipeline that described
overrideExtensionVersionas the mechanism "for rollback" — misleading once a dedicatedrollbackVersionparameter sits directly below it.Testing
No build/test target covers these files, so validation was done by rendering them:
replace/formatexpression subset and ran it against the actual expression text extracted from the files, rendering all 10ConfigurationOverridesblocks under both scenarios — 20/20 checks pass, every rendered block is valid JSON:register.shemitsnone(default)""RollbackVersion: null3.7.0-ci-prod-08-28-2026-925ade3dnone(default)""RollbackVersion: null3.7.0-ci-prod-08-28-2026-925ade3d…-925ade3d-arcdefault: 'preview'lines and the two reworded comment lines.