Skip to content

fix(rule): prevent concurrent double start - #4076

Merged
ngjaying merged 1 commit into
lf-edge:masterfrom
ngjaying:agent/fix-rule-double-start
Jul 17, 2026
Merged

fix(rule): prevent concurrent double start#4076
ngjaying merged 1 commit into
lf-edge:masterfrom
ngjaying:agent/fix-rule-double-start

Conversation

@ngjaying

Copy link
Copy Markdown
Collaborator

Summary

  • Prevent a stale rule start operation from launching a topology after another start path has already completed.
  • Cover both normal and scheduled start paths with a deterministic concurrency regression test.
  • Refresh the modified files' copyright years to 2026.

Why

  • RestartRule eventually calls State.Start, while a concurrent StartRule calls State.Bootstrap.
  • Start changed the state to Starting before acquiring ruleLock. If Bootstrap acquired the lock and completed first, the waiting Start continued without checking the new state and launched a second runTopo goroutine for the same topology.
  • This is a logical race rather than an unsynchronized memory access, so the Go race detector does not report it.

Changes In This PR

  • Recheck that the rule is still in Starting state after Start or ScheduleStart acquires ruleLock.
  • Return without planning or launching when another lifecycle operation has already advanced the state.
  • Add a deterministic test that holds ruleLock, lets the start path enter Starting, models Bootstrap completing first, and verifies the stale start performs no topology work.

Notes

  • The change only affects concurrent lifecycle operations on the same rule.
  • Sequential start, restart, stop, recovery, and scheduled-rule behavior are unchanged.
  • Validated with:
    • go test -race ./internal/topo/rule/... -count=1
    • focused regression test under -race for 10 consecutive runs
    • focused REST rule-management tests under -race
    • go vet ./internal/topo/rule/...
    • git diff --check

Signed-off-by: Jiayin Ng <ngjaying@gmail.com>
@codecov

codecov Bot commented Jul 17, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 70.75%. Comparing base (3cc4206) to head (f6f25de).

Additional details and impacted files
@@            Coverage Diff             @@
##           master    #4076      +/-   ##
==========================================
- Coverage   70.76%   70.75%   -0.01%     
==========================================
  Files         462      462              
  Lines       54269    54275       +6     
==========================================
+ Hits        38400    38401       +1     
- Misses      12883    12887       +4     
- Partials     2986     2987       +1     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@ngjaying
ngjaying marked this pull request as ready for review July 17, 2026 03:45
@ngjaying
ngjaying requested a review from Yisaer July 17, 2026 03:45
@ngjaying
ngjaying merged commit f536109 into lf-edge:master Jul 17, 2026
62 checks passed
@ngjaying
ngjaying deleted the agent/fix-rule-double-start branch July 17, 2026 06:47
ngjaying added a commit that referenced this pull request Aug 5, 2026
## Summary
- Prevent a stale rule start operation from launching a topology after
another start path has already completed.
- Cover both normal and scheduled start paths with a deterministic
concurrency regression test.
- Refresh the modified files' copyright years to 2026.

## Why
- `RestartRule` eventually calls `State.Start`, while a concurrent
`StartRule` calls `State.Bootstrap`.
- `Start` changed the state to `Starting` before acquiring `ruleLock`.
If `Bootstrap` acquired the lock and completed first, the waiting
`Start` continued without checking the new state and launched a second
`runTopo` goroutine for the same topology.
- This is a logical race rather than an unsynchronized memory access, so
the Go race detector does not report it.

## Changes In This PR
- Recheck that the rule is still in `Starting` state after `Start` or
`ScheduleStart` acquires `ruleLock`.
- Return without planning or launching when another lifecycle operation
has already advanced the state.
- Add a deterministic test that holds `ruleLock`, lets the start path
enter `Starting`, models `Bootstrap` completing first, and verifies the
stale start performs no topology work.

## Notes
- The change only affects concurrent lifecycle operations on the same
rule.
- Sequential start, restart, stop, recovery, and scheduled-rule behavior
are unchanged.
- Validated with:
  - `go test -race ./internal/topo/rule/... -count=1`
  - focused regression test under `-race` for 10 consecutive runs
  - focused REST rule-management tests under `-race`
  - `go vet ./internal/topo/rule/...`
  - `git diff --check`

Signed-off-by: Jiayin Ng <ngjaying@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants