[CI] DotNetWatch CoreCLR Hot Reload crashes with Bad IL range

[dalexsoto]

Failure signature

Xamarin.Tests.DotNetWatchTest.DotNetWatch using CoreCLR fails after applying a Hot Reload edit:

dotnet watch 🔥 C# and Razor changes applied
Unhandled exception. System.BadImageFormatException: Bad IL range.
   at HotReloadTestApp.Program.DoSomething(Int32 i)

The app builds and launches successfully, receives the edit, then crashes before the watched variable changes. The test reports Timed out waiting for the variable to change.

Confirmed monitored occurrences

PR / head	Azure run / job	Tests	Worker
#26454 / b956034e64eae11ea91afcec93b11996be9adbe0	15086535 — dotnettests_ios	1	AcesShared 30
#26454 / same head	15086535 — dotnettests_maccatalyst	2 (sandbox off/on)	AcesShared 21
#26454 / 918e83b75ed8152cb415cc7627e51ce69bc3124a	15088820 — dotnettests_ios	1	AcesShared 27
#26454 / same head	15088820 — dotnettests_maccatalyst	2 (sandbox off/on)	AcesShared 9
#26454 / 3adbceca8e2a7ceb62aaf29c8d03633fd7d3fea2	15090225 — dotnettests_ios	1	AcesShared 6
#26454 / same head	15090225 — dotnettests_maccatalyst	2 (sandbox off/on)	AcesShared 14
#26356 / ac84fef62c4a0babd9079876486d13e12f888133	15090832 — dotnettests_ios	1	AcesShared 67

Total: 10 confirmed test occurrences across 2 distinct monitored PRs, 4 heads, and 4 Azure runs.

First terminal occurrence: 2026-08-25T09:11:52Z. Latest terminal occurrence: 2026-08-25T17:06:44Z.

Classification and impact

Deterministic test/source-runtime failure. The same post-edit exception reproduces on iOS and Mac Catalyst, with and without Mac Catalyst sandboxing, across multiple heads and two dependency-update PRs. Builds and app launches succeed; the failure begins after Hot Reload applies the update. Unchanged retries are not appropriate.

This blocks the affected dotnettests_ios / dotnettests_maccatalyst jobs and can keep dependency-update PR CI red.

Investigation performed

• Inspected Azure build metadata, timelines, failed tasks, test-summary artifacts, and raw TRX output.
• Verified the exact exception and post-edit phase on every occurrence.
• Confirmed Build Analysis has no known-issue match.
• Searched all open and closed dotnet/macios issues separately by exact exception, test name, Bad IL, and Hot Reload component terms, then inspected candidate bodies and non-ignored comments.
• Existing [CI] Flaky: DotNetWatchTest.DotNetWatch on iOS — timed out waiting for the app to start #25935 is a different pre-launch timeout. Closed dotnet watch fails when targeting sandboxed maccatalyst apps #25700 is a different sandbox/startup-hook load failure. Neither represents this signature.

Workaround

No validated workaround is known. The CoreCLR Hot Reload edit/runtime behavior requires investigation. Do not retry unchanged heads merely to see whether the deterministic crash clears.

Azure logs and artifacts are retention-limited.

[dalexsoto]

Two additional exact-signature occurrences were confirmed on PR #26356 @ ac84fef62c4a0babd9079876486d13e12f888133:

• Build 15090832 attempt 1 — dotnettests_maccatalyst / Run tests on AcesShared 40.
• Both DotNetWatch(MacCatalyst, useMonoRuntime:false, enableSandbox:false) and the sandbox-enabled variant built and launched successfully, applied the Hot Reload edit, then crashed with System.BadImageFormatException: Bad IL range.

The canonical total is now 12 confirmed test occurrences across 2 distinct monitored PRs, 4 heads, and 4 Azure runs.

Classification: deterministic source/runtime failure. No retry submitted: build 15090832 remains active and an unchanged rerun is not appropriate.

Issue-specific checklist: none; the issue body has no Additional info to collect section.

[dalexsoto]

External retry outcome for PR #26356 @ ac84fef62c4a0babd9079876486d13e12f888133:

• Build 15090832 attempt 2 completed failed at 2026-08-26T00:02:12Z. Azure did not expose the triggering actor.
• dotnettests_ios / Run tests on AcesShared 4 reproduced DotNetWatch(iOS, useMonoRuntime:false, enableSandbox:false).
• dotnettests_maccatalyst / Run tests on AcesShared 2 reproduced both sandbox-disabled and sandbox-enabled variants.
• In all three tests, Hot Reload applied the edit and the app then crashed with System.BadImageFormatException: Bad IL range.

The canonical total is now 15 confirmed test occurrences across 2 distinct monitored PRs, 4 heads, 4 Azure runs, and 5 run attempts.

Classification: deterministic source/runtime failure. The unchanged failed-jobs retry reproduced every Bad IL occurrence, so no further retry is appropriate. Retry accounting for #26356 is 1 PR-lifetime / 1 current-head request.

Issue-specific checklist: none; the issue body has no Additional info to collect section.

[dalexsoto]

Three additional exact-signature occurrences were confirmed on PR #26454 @ 43cd3fb1c41f4d28eee679804e3421cd3f10022a:

• Build 15097859 attempt 1 — dotnettests_ios / Run tests on AcesShared 28 reproduced DotNetWatch(iOS, useMonoRuntime:false, enableSandbox:false).
• dotnettests_maccatalyst / Run tests on AcesShared 57 reproduced both sandbox-disabled and sandbox-enabled variants.
• In all three tests, Hot Reload applied the edit and the app then crashed with System.BadImageFormatException: Bad IL range.

The canonical total is now 18 confirmed test occurrences across 2 distinct monitored PRs, 5 heads, 5 Azure runs, and 6 run attempts.

Classification: deterministic source/runtime failure. No retry submitted: the unchanged retry on #26356 already reproduced this exact signature, and build 15097859 remains active for unrelated validation.

The Catalyst job reached test execution without the previous _mono_helper_box_nullable undefined-symbol failure, so that separate fix is holding. The iOS/tvOS linker jobs also passed; those outcomes are not part of this Bad IL signature.

Issue-specific checklist: none; the issue body has no Additional info to collect section.

[dalexsoto]

Two external failed-job retries completed and reproduced the full Bad IL signature again:

• PR [net11.0] Update dependencies from dotnet/dotnet #26356 @ ac84fef62c4a0babd9079876486d13e12f888133: build 15090832 attempt 3 completed failed at 2026-08-26T05:55:03Z.
  • dotnettests_ios / Run tests on AcesShared 26 reproduced the iOS test.
  • dotnettests_maccatalyst / Run tests on AcesShared 25 reproduced both Catalyst sandbox variants.
• PR [release/11.0.1xx-rc.1] Update dependencies from dotnet/dotnet #26454 @ 43cd3fb1c41f4d28eee679804e3421cd3f10022a: build 15097859 attempt 2 completed failed at 2026-08-26T05:29:52Z.
  • dotnettests_ios on AcesShared 24 reproduced the iOS test.
  • dotnettests_maccatalyst on AcesShared 23 reproduced both Catalyst sandbox variants.

In all six tests, Hot Reload applied the edit and the app then crashed with System.BadImageFormatException: Bad IL range.

The canonical total is now 24 confirmed test occurrences across 2 distinct monitored PRs, 5 heads, 5 Azure runs, and 8 run attempts.

Classification: deterministic source/runtime failure. Multiple unchanged retries have reproduced every variant, so no further retry is appropriate. Retry usage is 2 lifetime / 2 current-head requests for #26356 and 1 lifetime / 1 current-head request for #26454. Azure did not expose either retry initiator.

Issue-specific checklist: none; the issue body has no Additional info to collect section.

[dalexsoto]

A new exact-signature occurrence is confirmed on PR #26454 @ c9902ed89a6f2305ba25e00803588040ae58b9c2:

• Build 15108836 attempt 1 — dotnettests_maccatalyst / Run tests on AcesShared 26.
• DotNetWatch(MacCatalyst, useMonoRuntime:false, enableSandbox:false) built and launched successfully. Hot Reload applied the edit in 428 ms, then CoreCLR threw System.BadImageFormatException: Bad IL range from HotReloadTestApp.Program.DoSomething, and the process exited 134. The test subsequently reported Timed out waiting for the variable to change.
• Evidence came from TestSummary-simulator_testsdotnettests_maccatalyst-1, the HTML/NUnit report, and the Azure task log. The remaining simulator run jobs were still active when observed.

The canonical total is now 25 confirmed test occurrences across 2 distinct monitored PRs, 6 heads, 6 Azure runs, and 9 run attempts.

Classification: deterministic source/runtime failure. No retry was submitted: unchanged retries have already reproduced this signature, and this current head has used 0 retries (PR lifetime usage remains 1, on the superseded head). Draft PR #26473 adds diagnostics but is not a validated source/runtime fix.

Issue-specific checklist: none; the issue body has no Additional info to collect section.

Azure logs and artifacts are retention-limited.

[rolfbjarne]

Copilot's analysis (https://gist.github.com/rolfbjarne/fc3b58a978c8ab763fa4fd69e525d8eb):

The "Bad IL range" error comes from CoreCLR's `PEAssembly::GetIL`, which means
the runtime looked up the updated method's IL in the *baseline* PE image using
the RVA from the metadata delta (that RVA points into the delta IL stream, so
it's out of range in the baseline image). In other words the runtime never
picked up the delta IL when re-jitting the updated method.

This looks like fallout from https://github.com/dotnet/runtime/pull/130159
("Use CodeVersioning for EnC"), which switched the delta IL to be stored as an
`ILCodeVersion`: the mobile CoreCLR builds don't have tiered compilation
enabled (unlike the desktop macOS build), and only the code-versioned JIT path
knows about the EnC IL. That matches what we're seeing: the macOS test cases
still pass, only the mobile ones fail.

[rolfbjarne]

Root-caused this, and filed it upstream: dotnet/runtime#132804

Short version: it's a regression from dotnet/runtime#130159 ("Use CodeVersioning for EnC"). That PR made the EnC delta IL be stored as an ILCodeVersion, which is only reachable through VersionedPrepareCodeConfig::GetILHeader(). The plain PrepareCodeConfig::GetILHeader() falls back to MethodDesc::GetILHeader() → Module::GetIL(GetRVA()), and after a delta that RVA is delta-relative, so reading it against the baseline image is out of range — which is exactly the Bad IL range we see.

Worth flagging one thing, since it's the obvious guess and it's wrong: this is not because code versioning is unavailable on mobile. IsEligibleForEnC() doesn't depend on tiered compilation, and FEATURE_CODE_VERSIONING is enabled there anyway via FEATURE_METADATA_UPDATER. The real difference is that FEATURE_DYNAMIC_CODE_COMPILED is off for Apple mobile, so there's no JIT and these apps run the CoreCLR interpreter — some interpreter-side path prepares the updated method with a non-versioned config. That also explains why only iOS and Mac Catalyst fail while desktop macOS is fine.

Nothing we can fix on our side, so I've disabled the affected test cases for now (the Mono and desktop macOS cases still run) — that's in the PR referencing this issue. We should re-enable them once the runtime fix flows in.

[dalexsoto]

Workaround validation outcome for PR #26454 @ 565dc421e66fc277e7bfb2c690276bb396b732fb:

• Simulator build 15111445 completed succeeded at 2026-08-27T00:55:51Z; all 53 visible checks succeeded (1 unrelated check was skipped).
• dotnettests_ios and dotnettests_maccatalyst both succeeded.
• The NUnit reports confirm the remaining Mono cases passed: DotNetWatch(iOS, useMonoRuntime:true, enableSandbox:false) and DotNetWatch(MacCatalyst, useMonoRuntime:true, enableSandbox:false). The affected CoreCLR mobile cases are no longer discovered, consistent with the documented temporary test disablement.

This validates the test-disable workaround only; it does not demonstrate that the upstream CoreCLR interpreter regression is fixed. The canonical occurrence count remains 25 across 2 monitored PRs, 6 failing heads, 6 failing Azure runs, and 9 failing run attempts. No retry was submitted.

Issue-specific checklist: none; the issue body has no Additional info to collect section.

Azure logs and artifacts are retention-limited.