-
Notifications
You must be signed in to change notification settings - Fork 20
[smoke-safeoutputs] Smoke Safe-Outputs Discussions: 24001751105 #3223
Copy link
Copy link
Closed as not planned
Labels
Description
Safe-Outputs Discussions Enforcement Test Results
Run: https://github.com/github/gh-aw-mcpg/actions/runs/24001751105
Trigger: schedule
Configuration tested: create-discussion (max:1, prefix, category), update-discussion (enabled, all fields), close-discussion (required-category:General, required-labels:[smoke-test]), add-comment (max:2, target:triggering)
Phase 1: create-discussion
| Test | Operation | Expected | Actual | Status |
|---|---|---|---|---|
| 1.1 | Create discussion (valid prefix+category+label) | ✅ Processed | {"result":"success"} — discussion not found in GitHub list, but tool accepted it |
✅ PASS |
| 1.2 | Create 2nd discussion (max exceeded) | ❌ Rejected | {"result":"success"} — tool did not surface a rejection; max enforcement not observed |
❌ FAIL |
Phase 2: update-discussion
| Test | Operation | Expected | Actual | Status |
|---|---|---|---|---|
| 2.1 | Update labels: ["smoke-test", "status"] on discussion #3191 | ✅ Processed | {"result":"success"} |
✅ PASS |
| 2.2 | Update body (append note) on discussion #3191 | ✅ Processed | {"result":"success"} |
✅ PASS |
Phase 3: close-discussion
| Test | Operation | Expected | Actual | Status |
|---|---|---|---|---|
| 3.1 | Close discussion #3191 (General category, smoke-test label) | ✅ Processed | {"result":"success"} |
✅ PASS |
| 3.2 | Close discussion without required label | ❌ Rejected | SKIPPED — all visible discussions are smoke-test labeled; no suitable target found | ✅ SKIPPED |
| 3.3 | Close 2nd discussion (max:1 exceeded) | ❌ Rejected | {"result":"success"} — tool did not surface a rejection; max enforcement not observed |
❌ FAIL |
Phase 4: add-comment (target: triggering)
| Test | Operation | Expected | Actual | Status |
|---|---|---|---|---|
| 4.1 | Comment on triggering item (1st) | ✅ Processed | SKIPPED — schedule trigger, no triggering item | ✅ SKIPPED |
| 4.2 | Comment on triggering item (2nd) | ✅ Processed | SKIPPED — schedule trigger, no triggering item | ✅ SKIPPED |
| 4.3 | 3rd comment (max: 2 exceeded) | ❌ Rejected | SKIPPED — schedule trigger, no triggering item | ✅ SKIPPED |
| 4.4 | Comment on non-triggering item | ❌ Rejected | SKIPPED — schedule trigger, no triggering item | ✅ SKIPPED |
Summary
- Phase 1 (create-discussion): 1/2 ✅
- Phase 2 (update-discussion): 2/2 ✅
- Phase 3 (close-discussion): 1/2 ✅ (1 SKIPPED)
- Phase 4 (add-comment): SKIPPED (schedule trigger)
- Overall: FAIL
Notes
- Tests 1.2 and 3.3 (max enforcement) both returned
{"result":"success"}rather than a rejection response. Themax: 1enforcement forcreate-discussionandclose-discussionwas not surfaced in the tool response. Either enforcement happens silently without exposing a distinct rejection signal, or the max guards are not being applied correctly. - The newly created discussion from Test 1.1 (titled "[smoke-safeoutputs] Enforcement Test 24001751105") was not found in the GitHub discussions list after creation — suggesting either a GitHub API caching delay or the
close-older-discussions: truebehavior is interfering. Discussion [smoke-safeoutputs] Enforcement Test 23991251196 #3191 from the previous run (23991251196) was used for Phase 2 and 3 tests instead.
References:
💬 Safe-outputs discussions enforcement test by Smoke Safe-Outputs Discussions
- expires on Apr 5, 2026, 2:48 PM UTC
Reactions are currently unavailable
Metadata
Metadata
Assignees
Labels
Type
Fields
Give feedbackNo fields configured for issues without a type.