feat(matching): add bucket owner that hands out token slots

[YaweiZhang-930]

What changed?
New package service/matching/semaphore, plus its tests. Nothing calls it yet.

• Identifier — names one bucket (domain, name, bucket number). Comparable, so it works as a map key.
• Bucket — the Matching-side owner of one bucket. Start reads the bucket's partition once and builds two in-memory indexes: the set of open slots, and an owner → slot reverse index. Grant draws an open slot at random and claims it with a conditional write. Stop gives the bucket up.
• Real-Cassandra tests under host/persistence/cassandra/.

Why?

• A caller needs to ask "is a slot open here?" without a read before every write.
• Holding the open-slot set in memory on the single host that owns the bucket turns the common acquire into one conditional write.
• That memory is only a cache — the conditional write decides. A stale cache can turn away a grant that would have worked, but it cannot hand one slot to two owners or lose a grant.
• No caller and no config in this PR, to keep the review small.

How did you test it?

Unit tests — no external dependencies:

go test -race -cover ./service/matching/semaphore/...

Potential risks

• Nothing wires this up, so merging changes no behavior.
• The partition is read once at Start and nothing reloads it. A slot dropped by a failed write stays out until the bucket changes hands or the host restarts. The safe direction — it hands out fewer slots, never more — and marked TODO.
• Releasing a slot is not implemented yet, so the open-slot set only shrinks. Also TODO.
• Retries are capped at 5 with no backoff. Only a stale cache causes one, and every miss corrects the cache.

Release notes
None

Documentation Changes

None

[gitar-bot]

A couple of minor suggestions to strengthen this PR:

1. Test command: The "How did you test it?" section just says "unit test." Consider adding the concrete command, e.g. go test ./service/matching/semaphore/... ./host/persistence/cassandra/..., so reviewers can reproduce it directly.
2. Issue link: This PR isn't linked to a tracking issue. If there's a related design doc or issue in cadence-workflow, consider referencing it (e.g. Related to #123) for better traceability, especially since this introduces a new subsystem that will be built on in follow-up PRs.

Everything else looks solid — the description clearly explains the motivation (avoiding read-before-write), the safety argument for the cache being advisory only, and the known TODOs (partition reload, slot release). Nice work keeping this PR scoped to the new package with nothing wired up yet.

[gitar-bot]

CI failed: Integration tests failed due to a timeout in TestArchival_ArchiverWorker and internal execution manager errors.

Overview

1 unique failure pattern and multiple logs analyzed, showing integration test execution timeouts and persistence/execution manager failures.

Failures

Integration Test Timeout (confidence: high)

• Type: test
• Affected jobs: 100775254236
• Related to change: unclear
• Root cause: TestArchival_ArchiverWorker exceeded its time limit, alongside context deadline exceedances during task dispatch.
• Suggested fix: Investigate performance bottlenecks in the archival worker or adjust the integration test timeout limits.

Cassandra Execution Manager Failure (confidence: high)

• Type: build
• Affected jobs: 100775254236
• Related to change: yes
• Root cause: Operation failed with internal error (GetCurrentExecution operation failed / Update workflow with invalid state).
• Suggested fix: Review recent persistence layer and bucket owner token budget changes to ensure valid state transitions.

Summary

• Change-related failures: 1 failure in Cassandra execution manager and persistence tests.
• Infrastructure/flaky failures: 1 integration test timeout (TestArchival_ArchiverWorker).
• Recommended action: Review recent changes affecting state management and persistence token budgets, and re-run integration tests to rule out resource contention.

Code Review ✅ Approved 1 resolved / 1 findings

New service/matching/semaphore package implements bucket-owner logic that caches open token slots in memory and grants them via conditional write, reducing the common case to a single write operation. Transient write errors that permanently erode the free-set have been addressed. Unit tests cover the implementation with no external dependencies.

✅ 1 resolved

✅ Edge Case: Transient write errors can permanently erode the free-set

📄 service/matching/semaphore/bucket.go:311-319 📄 service/matching/semaphore/bucket.go:354-360 📄 service/matching/semaphore/bucket.go:119-123
In Grant, both the write-error path (bucket.go:318) and the unexpected-outcome path (bucket.go:359) intentionally keep the reserved tokenID out of the free-set, and nothing ever reloads the partition. This is correct for a single ambiguous write, but under a sustained persistence outage every failing acquire strips one more slot, so a bucket can shrink toward zero free slots and stay effectively dead until it changes hands or the host restarts — even after Cassandra recovers and most of those writes turn out never to have landed. The behavior is acknowledged in the comments and PR as an accepted tradeoff pending the periodic-reload TODO (bucket.go:122-123); flagging it so the reload work is prioritized before a caller is wired up, since without it a downstream outage degrades semaphore capacity persistently rather than transiently.

Rules ⚠️ 4/8 requirements met

Repository Rules

❌ issue_linking_required: PR description contains no linked issue (e.g., #123 or cadence-workflow/... reference); consider linking a tracking issue for this feat PR with 418+ lines changed.

❌ issue_linking_required_chunk_12: PR title 'feat(matching): ...' does not match maintenance-commit skip prefixes, and diff exceeds 50 lines, so an issue link is expected but absent.

❌ issue_linking_required_chunk_18: Description lacks any issue link; recommend adding one referencing cadence-workflow org.

❌ issue_linking_required_chunk_6: No GitHub issue reference found in the PR description body.

✅ pr_description_quality: Description covers What/Why/How tested/Potential risks/Release notes/Documentation with substantive, why-focused content and concrete test command.

✅ pr_description_quality_chunk_1: PR template guidance applied; all required sections present with substantive content.

✅ pr_description_quality_chunk_15: No forbidden extraneous sections (e.g., 'Issues Found', 'Summary') were added to the description.

✅ pr_description_quality_chunk_8: Testing section provides a concrete, copyable command: `go test -race -cover ./service/matching/semaphore/...`.

_{12 rules not applicable. Show all rules by commenting gitar display:verbose.}

Tip

Comment Gitar fix CI or enable auto-apply: gitar auto-apply:on

Options

Auto-apply is off → Gitar will not commit updates to this branch.
Display: compact → Showing less information.

Comment with these commands to change the behavior for this request:

Auto-apply	Compact
gitar auto-apply:on	gitar display:verbose

_{Was this helpful? React with 👍 / 👎 | Gitar}

[shijiesheng]

Consider renaming it to SemaphoreManager to be consistent with tasklistmanager.

[shijiesheng]

As discussed offline, we also need rangeID (host ownership) to ensure SemaphoreManager doesn't exist on two different hosts during deployment or network partition.

[shijiesheng]

nit: SemaphoreMetadataStore

[shijiesheng]

nit: SemaphoreTokenStore