fix(rule): prevent concurrent double start - #4076
Merged
Merged
Conversation
Signed-off-by: Jiayin Ng <ngjaying@gmail.com>
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## master #4076 +/- ##
==========================================
- Coverage 70.76% 70.75% -0.01%
==========================================
Files 462 462
Lines 54269 54275 +6
==========================================
+ Hits 38400 38401 +1
- Misses 12883 12887 +4
- Partials 2986 2987 +1 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
ngjaying
marked this pull request as ready for review
July 17, 2026 03:45
Yisaer
approved these changes
Jul 17, 2026
ngjaying
added a commit
that referenced
this pull request
Aug 5, 2026
## Summary - Prevent a stale rule start operation from launching a topology after another start path has already completed. - Cover both normal and scheduled start paths with a deterministic concurrency regression test. - Refresh the modified files' copyright years to 2026. ## Why - `RestartRule` eventually calls `State.Start`, while a concurrent `StartRule` calls `State.Bootstrap`. - `Start` changed the state to `Starting` before acquiring `ruleLock`. If `Bootstrap` acquired the lock and completed first, the waiting `Start` continued without checking the new state and launched a second `runTopo` goroutine for the same topology. - This is a logical race rather than an unsynchronized memory access, so the Go race detector does not report it. ## Changes In This PR - Recheck that the rule is still in `Starting` state after `Start` or `ScheduleStart` acquires `ruleLock`. - Return without planning or launching when another lifecycle operation has already advanced the state. - Add a deterministic test that holds `ruleLock`, lets the start path enter `Starting`, models `Bootstrap` completing first, and verifies the stale start performs no topology work. ## Notes - The change only affects concurrent lifecycle operations on the same rule. - Sequential start, restart, stop, recovery, and scheduled-rule behavior are unchanged. - Validated with: - `go test -race ./internal/topo/rule/... -count=1` - focused regression test under `-race` for 10 consecutive runs - focused REST rule-management tests under `-race` - `go vet ./internal/topo/rule/...` - `git diff --check` Signed-off-by: Jiayin Ng <ngjaying@gmail.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Why
RestartRuleeventually callsState.Start, while a concurrentStartRulecallsState.Bootstrap.Startchanged the state toStartingbefore acquiringruleLock. IfBootstrapacquired the lock and completed first, the waitingStartcontinued without checking the new state and launched a secondrunTopogoroutine for the same topology.Changes In This PR
Startingstate afterStartorScheduleStartacquiresruleLock.ruleLock, lets the start path enterStarting, modelsBootstrapcompleting first, and verifies the stale start performs no topology work.Notes
go test -race ./internal/topo/rule/... -count=1-racefor 10 consecutive runs-racego vet ./internal/topo/rule/...git diff --check