evaluation: add PromptIter regression loop

[cyl6]

What changed

本 PR 完成 #2003 要求的“评测 → 失败归因 → Prompt 优化 → 验证集回归 → 接受门禁 → 产物审计”自动闭环，并提供无需 API Key 即可复现的完整样例。

1. 新增可复用的回归闭环包

在 evaluation/workflow/promptiter/regression 新增以下能力：

• Pipeline：依次运行 baseline train/validation、训练失败归因、PromptIter 候选生成、候选 train/validation 重跑、逐 case delta、gate 决策和报告组装；所有候选轮次均保留，最终只选择通过 gate 且验证集得分最高的候选。
• LocalEvaluator：支持 deterministic fake model 与 strict trace replay；按 final response、tool trajectory、route、structured output、knowledge recall 和 LLM rubric 等信号加权评分。
• 失败归因：输出 final_response_mismatch、tool_call_error、tool_parameter_error、route_error、format_error、knowledge_retrieval_insufficient，每个失败 case 至少包含一个带证据与置信度的可解释原因；针对中文否定表达、错误/失败状态和互相矛盾的 trace 信号做了保守判定。
• PromptIter 接入：复用原生 Profile、PatchSet、SurfaceID 等领域契约，根据训练集归因生成确定性的 instruction patch；候选 Prompt、Profile override 和 PatchSet value 必须完全绑定。
• 逐 case delta：区分 new_pass、new_failure、improved、regressed、不变、缺失和意外 case，并给出各 metric delta；case/metric 覆盖不完整时 fail closed。
• 可配置 gate：验证集最低增益、最大新增失败、禁止新增 hard fail、关键 case/任意 case 最大退化、成本增幅、模型/工具调用数和延迟预算均可独立配置。训练集提升不能抵消验证集退化。
• 完整性保护：candidate 输出与 Prompt 语义 SHA-256 绑定；validation 禁止复用 baseline fallback；校验 EvalSet ID、case ID、critical 标记、metric 配置、usage 汇总和 delta 一致性。EvaluationSummary 保存实际 passThreshold，delta/gate 会重新推导 pass 状态并拒绝阈值漂移，防止伪造 Passed 绕过门禁。
• 审计报告：生成稳定的 optimization_report.json 和人类可读的 optimization_report.md，记录 baseline、每轮 candidate、逐 case delta、归因统计、gate 每项判定与拒绝原因、Prompt/哈希、seed、fake/trace 配置、模型配置、usage、cost 和 latency。

2. 新增完整可运行样例

新增 examples/evaluation/promptiter_regression_loop：

• 3 条训练 case + 3 条验证 case，train/validation ID 完全隔离；
• baseline、overfit、balanced 三组输出，共 18 份显式 case rerun；
• 同时覆盖可优化成功、优化无效、训练提升但验证退化三类场景；
• 包含 baseline Prompt、train/validation evalset、6 类 metrics、PromptIter/gate 配置、300–500 字设计说明、README、中文学习文档和 JSON/Markdown 示例报告；
• CLI 默认三分钟超时，运行时不会覆盖源 Prompt，只将可审计结果写入指定输出目录。

3. 改善 OpenAI-compatible structured output 兼容性

修复 PromptIter backward schema 在空 surface/predecessor 集合时生成 enum: [] 和 maxItems: 0 的问题。部分 OpenAI-compatible provider 会在 grammar 编译阶段拒绝这类 schema；现在空集合仍声明合法 item shape，但省略空 enum/零 maxItems，运行时 sanitizer 继续严格拒绝模型返回任何无效 surface 或 predecessor，因此兼容性提升不会放宽业务约束。

4. 文档与测试

• README 给出快速运行、输入文件、输出文件、fake/trace 模式和 gate 行为说明。
• DESIGN.md 用 300–500 字说明失败归因、接受策略、防过拟合、PromptIter 接入和审计方式。
• LEARNING_GUIDE.zh_CN.md 详细介绍 Evaluation、PromptIter、闭环架构与原理，并逐段走读核心实现、样例数据、报告和生产接入方式。
• 单元测试覆盖评分、中文相似度、失败归因与否定表达、严格 trace、Prompt/输出绑定、usage、逐 case delta、gate 边界与篡改防护、报告生成、完整 pipeline 和 CLI。

Why

现有 Evaluation Service 与 PromptIter 分别解决“如何评价”和“如何生成候选”，但缺少生产落地所需的验证集隔离、逐 case 回归、过拟合阻断、预算限制和审计证据。本实现将两者通过稳定接口组合起来，并以 fake/trace 模式提供离线确定性基准；生产环境可以替换 Evaluator/Optimizer adapter，同时复用同一套 delta、gate 和 report 逻辑。

Result

默认样例结果：

场景	Train delta	Validation delta	新增失败 / hard fail	决策
candidate-overfit	+0.275152	-0.194583	1 / 1	拒绝
candidate-balanced	+0.275152	+0.136667	0 / 0	接受

• Baseline：train 0.522041，validation 0.720000。
• 最终候选：train 0.797193，validation 0.856667。
• 关键退款 case 在 overfit 候选中退化，gate 因验证集负增益、新增失败、新增 hard fail、关键 case 和单 case 退化共同拒绝该候选。
• balanced 候选修复未知城市 case、保持关键 case、不突破预算，最终被接受。
• train/validation 各有一个 no-gain case 保持 unchanged_failure，证明系统不会把“改了 Prompt”误判为“所有 case 已修复”。
• 本机 warm-cache fake pipeline 实测约 0.59s，显著低于 3 分钟验收限制。

How to run

无需设置任何 API Key：

cd examples/evaluation/promptiter_regression_loop
go run . \
  -data-dir ./data \
  -config ./promptiter.json \
  -prompt ./baseline_prompt.txt \
  -output-dir ./output

输出：

• output/optimization_report.json
• output/optimization_report.md

也可以通过 -timeout 修改总超时，或在 promptiter.json 中切换 fake / trace 模式及调整 gate、seed、候选 patch 和预算。

Testing

已执行并通过：

cd evaluation
go test -count=1 ./...
go test -count=1 -race ./workflow/promptiter/backwarder ./workflow/promptiter/regression
go vet ./workflow/promptiter/...
go mod tidy -diff

cd ../examples/evaluation/promptiter_regression_loop
go test -count=1 -race .
go vet .
go mod tidy -diff
go run . -output-dir "$(mktemp -d)"

evaluation/workflow/promptiter/regression 当前 statement coverage 为 78.3%。同时使用真实 OpenAI-compatible endpoint 完成过 PromptIter backward 与一轮 train/validation 闭环 smoke test；凭据仅通过进程环境传入，未写入代码、配置、报告或 Git 历史。

Notes for reviewers

• 建议重点审查 pipeline.go 的数据隔离/候选选择、delta.go 的完整性检查、gate.go 的 fail-closed 策略、evaluator.go/attribution.go 的证据判定，以及 backward schema 空集合兼容性修改。
• fake output 是离线扩展字段，不宣称替代 Evaluation Service；生产 adapter 的字段保真要求已在中文学习文档中列出。
• 源 Prompt 不会自动回写。只有报告给出接受决策，是否更新生产 Prompt 仍由人工审核决定。

Fixes #2003

RELEASE NOTES: Added an API-key-free PromptIter regression loop example with explainable failure attribution, configurable safety gates, and JSON/Markdown audit reports.

[coderabbitai]

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Pro

Run ID: 03becdba-a7a0-43b4-af53-94a49933ede2

📥 Commits

Reviewing files that changed from the base of the PR and between 8fc1254 and 90385ed.

📒 Files selected for processing (1)

• evaluation/workflow/promptiter/regression/codecov_boundaries_test.go

---

📝 Walkthrough

English

Overview

• Adds an API-key-free, deterministic PromptIter regression “auto-replay + admission gate” loop under evaluation/workflow/promptiter/regression.
• Implements an end-to-end closed loop: baseline + failure attribution → deterministic candidate prompt proposal → rerun on isolated train/validation → compute per-case/per-metric deltas → run configurable fail-closed acceptance gate (coverage, validation gain, new failures/hard-fail/critical regressions, prompt-no-op, and budget limits) → write atomic JSON + Markdown audit reports.
• Introduces deterministic local evaluation with multi-signal scoring (final response match, tool trajectory, route accuracy, structured-output validity, knowledge recall, rubric scoring) plus strict/non-strict trace replay evidence auditing.
• Adds rubric-based failure attribution with robust negation-scope and contrastive topic handling (English/Chinese), deterministically sorted with evidence.
• Fixes PromptIter backward schema compatibility for empty allowed surface/predecessor collections: avoids generating schema-level enum: [] and maxItems: 0; runtime validation still rejects invalid outputs.
• Provides runnable, API-key-free example loop examples/evaluation/promptiter_regression_loop with baseline/candidate/overfit-vs-balanced scenarios and committed sample JSON/Markdown outputs.
• Includes additional production-readiness utilities: strict JSON/config loaders (UTF-8, duplicate-key rejection, strict unknown-field handling options), deterministic similarity scoring (custom ROUGE tokenizer), and atomic report write/rollback.

Public API and compatibility implications (exported Go surface)

This PR significantly expands the exported API in evaluation/workflow/promptiter/regression. Affected/introduced exported symbols include:

Core workflow interfaces/constructors/execution

• type Evaluator, type LocalEvaluator, NewLocalEvaluator(...), (*LocalEvaluator).Evaluate(...)
• type Optimizer, type DeterministicPromptIter, NewDeterministicPromptIter(...), (*DeterministicPromptIter).Propose(...)
• type Pipeline, NewPipeline(...), (*Pipeline).Run(...), type Clock

Data models/constants for evaluation, deltas, attribution, gating, config, and reports

• Failure/delta identifiers: type FailureCategory, type DeltaOutcome, constants like:
  • Failure categories: FailureFinalResponseMismatch, FailureToolCallError, FailureToolParameterError, FailureRouteError, FailureFormatError, FailureKnowledgeRetrievalInsufficient
  • Delta outcomes: DeltaNewPass, DeltaNewFailure, DeltaImproved, DeltaRegressed, DeltaUnchangedPass, DeltaUnchangedFailure, DeltaMissingCandidate, DeltaUnexpectedCase
• Trace/usage/accounting: TraceStep, Usage, UsageDelta, (Usage).AddChecked(...)
• Evaluation artifacts: Expectations, FakeOutput, EvalCase, EvalSet, MetricConfig, MetricResult, Attribution, CaseResult, EvaluationSummary
• Delta artifacts: MetricDelta, CaseDelta, DeltaSummary, plus ComputeDelta(...)
• Gate/admission: GatePolicy, GateCheck, GateDecision, GateInput, EvaluateGate(...)
• Configuration & persisted artifacts: Config, Candidate, PromptSnapshot, RoundReport, CostLatencySummary, Report, SurfaceConfig, CandidateConfig, FakeEngineConfig
• Serialization/rendering: JSONReportName, MarkdownReportName, MarshalReportJSON(...), RenderMarkdown(...), WriteReports(...)

Loaders/validators/utilities

• LoadEvalSet(...), LoadMetrics(...), LoadConfig(...), LoadPrompt(...)
• AttributeFailure(...)
• HashText(...) (semantic prompt binding input)
• Similarity scoring is internal to the package, with deterministic behavior used by metrics.

Compatibility questions to confirm before downstream adoption

• Export necessity/ownership: should all persisted artifact types (Config, Report, EvalSet, DeltaSummary, TraceStep, etc.) remain broadly public, or be narrowed behind workflow-specific interfaces to reduce contract burden?
• Naming/semantics stability:
  • HashText as a stable serialization contract (lowercase SHA-256 of prompt text) and how strict binding should evolve.
  • runtimeMode semantics (especially strict trace requirements) and extension behavior.
  • FailureCategory vocabulary as an identifier contract (versioning/migration strategy).
• Extensibility/versioning:
  • how strict loaders handle new fields and schema evolution (unknown-field strictness, version gates like schemaVersion).
  • compatibility strategy for JSON/Markdown report schema changes (e.g., how consumers validate/parse optimization_report.json).
• Strictness contract: which invariants callers must satisfy to avoid fail-closed rejections (deterministic variant marker format, rerun requirements, provenance/hash bindings, trace evidence completeness).

Behavioral and operational risks

• Fail-closed admission gate: missing/unexpected cases, delta inconsistencies, critical regressions, hard-fail introductions, or budget/latency/cost boundary violations can hard-reject candidates.
• Strict trace replay evidence requirements: in strict mode, missing or contradictory evidence across route/tool/retrieval/terminal response/usage/rubric can become hard-fail.
• Provenance/hash fragility: mismatched prompt semantic hashes and candidate/fallback provenance bindings can cause rejection, even if surface text “looks similar”.
• Attribution ambiguity risk: rubric parsing (especially negation/contrast) may still misclassify under ambiguous rubric phrasing; mitigated via deterministic sorting and guarded scopes but remains a semantic risk.
• Atomic report persistence correctness: staging/rename/rollback must be verified to avoid partial outputs or permission/mode drift—tests cover atomicity, but downstream reliance should be explicit.
• Schema compatibility for backwarder: the backward schema generation fix prevents schema-level incompatibilities for empty allowed sets; downstream code that relies on prior schema output may observe schema differences.

Recommended validation

• Run:
  • go test ./evaluation/.../promptiter/regression
  • go test ./examples/evaluation/promptiter_regression_loop
• Execute the sample loop without API keys:
  • default deterministic fake mode: confirm overfit candidate is rejected and balanced candidate is accepted;
  • trace replay mode: confirm strict trace replay gates and evidentiary audits behave as expected.
• Add/confirm targeted validations:
  • provenance tampering: candidate/fallback prompt binding and evaluation-summary provenance must be rejected;
  • strict trace evidence completeness: terminal mismatch and missing tool/route/retrieval evidence must hard-fail in strict mode;
  • loader strictness: duplicate JSON keys rejection, UTF-8 validation, and strict/unknown-field behavior;
  • atomicity: ensure no leftover temp/backup files on success/failure;
  • backwarder schema regression: empty allowed surface/predecessor collections should not emit enum: [] / maxItems: 0, while runtime still rejects truly invalid outputs.

---

中文

概览

• 在 evaluation/workflow/promptiter/regression 新增一个无需 API Key、离线可复现的 PromptIter 回归闭环：候选自动回放（baseline→失败归因→候选生成→重跑）+ 可配置的 fail-closed 接收门禁，并产出审计报告（JSON/Markdown）。
• 实现端到端流程：baseline + 失败归因 → 确定性候选 Prompt 提案 → 在隔离的训练/验证集分别重跑 → 计算逐 case/逐指标 delta → 通过门禁拒绝验证集退化/硬失败/关键 case 回退/预算与调用/延迟超限等 → 原子落盘 JSON + Markdown 审计报告。
• 引入确定性本地评测：多信号打分（final response、tool 轨迹、route 准确性、结构化输出有效性、知识召回、rubric）并支持 strict / non-strict 的 trace replay 证据审计。
• 增加基于 rubric 的失败归因：支持英文/中文否定范围（否定作用域）、对比性表述与主题归属，并对归因做确定性排序输出证据。
• 修复 PromptIter backwarder schema 在空的 allowed surface / predecessor 集合时的兼容性问题：不再生成 schema 层面的 enum: [] 与 maxItems: 0；运行时校验仍会拒绝无效输出。
• 提供可直接运行示例 examples/evaluation/promptiter_regression_loop：隔离训练/验证数据、使用 fake engine（无 API Key），覆盖 overfit 被拒绝与 balanced 通过门禁，并包含提交的样例 JSON/Markdown 输出。
• 进一步补齐工程能力：严格 JSON/配置加载（UTF-8、重复 key 拒绝、严格 unknown 字段策略）、确定性相似度（自定义 ROUGE tokenizer）、以及报告落盘的原子写入与回滚。

公共 API 与兼容性影响（导出 Go 表面）

本 PR 在 evaluation/workflow/promptiter/regression 显著新增/导出 API。影响/新增的主要导出符号包括：

核心工作流接口/构造/执行

• type Evaluator、type LocalEvaluator、NewLocalEvaluator(...)、(*LocalEvaluator).Evaluate(...)
• type Optimizer、type DeterministicPromptIter、NewDeterministicPromptIter(...)、(*DeterministicPromptIter).Propose(...)
• type Pipeline、NewPipeline(...)、(*Pipeline).Run(...)、type Clock

用于评测/归因/门禁/配置/报告的导出数据模型与常量

• 失败/差异标识：type FailureCategory、type DeltaOutcome，以及常量如：
  • 失败类别：FailureFinalResponseMismatch、FailureToolCallError、FailureToolParameterError、FailureRouteError、FailureFormatError、FailureKnowledgeRetrievalInsufficient
  • delta 状态：DeltaNewPass、DeltaNewFailure、DeltaImproved、DeltaRegressed、DeltaUnchangedPass、DeltaUnchangedFailure、DeltaMissingCandidate、DeltaUnexpectedCase
• trace/usage 记账：TraceStep、Usage、UsageDelta、(Usage).AddChecked(...)
• 评测工件：Expectations、FakeOutput、EvalCase、EvalSet、MetricConfig、MetricResult、Attribution、CaseResult、EvaluationSummary
• delta 工件：MetricDelta、CaseDelta、DeltaSummary、ComputeDelta(...)
• 接收门禁：GatePolicy、GateCheck、GateDecision、GateInput、EvaluateGate(...)
• 配置与持久化产物：Config、Candidate、PromptSnapshot、RoundReport、CostLatencySummary、Report、SurfaceConfig、CandidateConfig、FakeEngineConfig
• 序列化/渲染：JSONReportName、MarkdownReportName、MarshalReportJSON(...)、RenderMarkdown(...)、WriteReports(...)

加载/校验/工具

• LoadEvalSet(...)、LoadMetrics(...)、LoadConfig(...)、LoadPrompt(...)
• AttributeFailure(...)
• HashText(...)（语义 prompt 绑定所用输入）
• 相似度计算为内部实现，但会影响用于打分的度量结果。

下游采用前建议确认的兼容性问题

• 是否需要继续“全量导出”这些持久化模型类型（Config/Report/EvalSet/DeltaSummary/TraceStep/...）作为稳定契约，还是收敛到工作流内部/接口层以降低契约成本？
• 命名/语义契约稳定性：
  • HashText 作为稳定格式契约（prompt 文本的小写 SHA-256），以及未来严格绑定策略的演进方式；
  • runtimeMode 的语义，尤其 strict trace 的证据要求与扩展规则；
  • FailureCategory 词表作为标识契约时的版本化/迁移策略。
• 扩展与版本化：
  • 严格 loader 对新增字段/ schema 演进的处理方式（unknown 字段 strictness、schemaVersion 等 gate）；
  • JSON/Markdown 报告 schema 的向后兼容策略（尤其 optimization_report.json 的解析与校验）。
• 严格程度契约：调用方需要满足哪些不变式以避免 fail-closed（确定性 variant marker 格式、必须 rerun 的校验、provenance/hash 绑定、strict trace 证据完备性等）。

行为与运行风险

• fail-closed 接收门禁：缺失/意外 case、delta 不一致、引入 hard-fail、关键 case 回退、预算/调用/延迟/cost 边界违规均可能触发硬拒绝。
• strict trace replay 证据缺失/矛盾：在 strict 模式下，route/tool/retrieval/terminal response/usage/rubric 任一环节证据缺失或冲突都可能硬失败。
• provenance/hash 脆弱性：prompt 语义 hash 与候选/回退绑定不一致会导致拒绝（即使表面文本看似相近）。
• 归因语义风险：rubric 解析（尤其否定范围/对比性语句）在表述模糊时仍可能误归类；虽然通过确定性排序与防护策略降低风险，但语义误判仍需关注。
• 原子写入一致性风险：报告落盘 staging/rename/rollback 必须保证不会留下半成品或权限/模式漂移（仓内已有原子性测试，但下游依赖时应明确约束）。
• backwarder schema 兼容变更：空 allowed 集合的 schema 输出会不同于旧版本；依赖旧 schema 输出的下游需要评估。

建议验证

• 运行：
  • go test ./evaluation/.../promptiter/regression
  • go test ./examples/evaluation/promptiter_regression_loop
• 在无 API Key条件下执行示例循环：
  • 默认 deterministic fake 模式：确认 overfit 候选被拒绝、balanced 候选通过门禁；
  • trace replay 模式：确认 strict trace 证据审计与 gate 行为符合预期。
• 补充重点校验：
  • provenance 篡改用例：候选/回退 prompt 绑定与 evaluation-summary 证据链必须被拒绝；
  • strict trace 证据完备性：terminal mismatch、缺失 tool/route/retrieval 证据应硬失败；
  • loader 严格性：重复 JSON key 拒绝、UTF-8 校验、unknown 字段 strict/非 strict 行为；
  • 原子性：成功/失败场景均应无残留临时文件/备份文件；
  • backwarder schema 回归：空 allowed 集合不产生 enum: [] / maxItems: 0（schema 层），同时运行时仍拒绝真正无效输出。

Walkthrough

The pull request adds a deterministic PromptIter regression workflow with strict validation, local evaluation, multilingual failure attribution, candidate provenance checks, delta and gate evaluation, atomic JSON/Markdown reporting, runnable example data, and a backwarder schema compatibility fix.

Changes

PromptIter regression workflow

Layer / File(s)	Summary
Contracts and strict input validation evaluation/workflow/promptiter/regression/types.go, load.go, usage.go, evaluation/workflow/promptiter/backwarder/*	Adds regression data contracts, checked usage arithmetic, strict JSON/config/evaluation loading, and compatible empty-collection backwarder schemas.
Evaluation and failure attribution evaluation/workflow/promptiter/regression/evaluator.go, attribution.go, similarity.go, *_test.go	Adds deterministic local evaluation, trace replay validation, multilingual similarity scoring, metric validation, and structured/rubric-based attribution.
Candidate generation and orchestration evaluation/workflow/promptiter/regression/optimizer.go, pipeline.go, config.go, *_test.go	Adds deterministic candidates, semantic/hash provenance binding, profile and patch validation, fallback auditing, validation reruns, and candidate selection.
Delta computation and acceptance gates evaluation/workflow/promptiter/regression/delta.go, gate.go, *_test.go	Adds fail-closed case and metric deltas plus configurable score, failure, critical-case, prompt-change, and resource-budget checks.
Reports and runnable example evaluation/workflow/promptiter/regression/report.go, examples/evaluation/promptiter_regression_loop/*	Adds atomic JSON/Markdown reports and a deterministic example with datasets, configuration, CLI, documentation, tests, and sample outputs.

Estimated code review effort: 5 (Critical) | ~120 minutes

Possibly related PRs

• trpc-group/trpc-agent-go#1868: Updates related backwarder behavior for empty allowed ID collections.
• trpc-group/trpc-agent-go#2191: Adds a closely related PromptIter regression-loop example.
• trpc-group/trpc-agent-go#2201: Directly overlaps with the regression-loop implementation.

Suggested reviewers: sandyskies, winechord

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name	Status	Explanation	Resolution
Docstring Coverage	⚠️ Warning	Docstring coverage is 6.25% which is insufficient. The required threshold is 80.00%.	Write docstrings for the functions missing them to satisfy the coverage threshold.

✅ Passed checks (4 passed)

Check name	Status	Explanation
Title check	✅ Passed	The title accurately summarizes the main change: adding a PromptIter regression loop for evaluation and optimization.
Description check	✅ Passed	The description is detailed and directly matches the implemented regression loop, example workflow, docs, tests, and schema fix.
Linked Issues check	✅ Passed	The PR satisfies #2003 by adding baseline evaluation, failure attribution, candidate generation, validation deltas, gates, reports, and runnable examples.
Out of Scope Changes check	✅ Passed	The changes appear scoped to the regression loop, supporting docs/tests/examples, and the backward schema compatibility fix.

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[github-actions]

All contributors have signed the CLA ✍️ ✅
_{Posted by the CLA Assistant Lite bot.}

[cyl6]

I have read the CLA Document and I hereby sign the CLA

[cyl6]

recheck

[coderabbitai]

🧹 Nitpick comments (1)

evaluation/workflow/promptiter/backwarder/backwarder.go (1)

312-321: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick win

Dead empty-collection conditional in both backward schema helpers. Removing maxItems: 0 left the len(...) == 0 branch returning the exact same array schema as the fallthrough return, so the conditional no longer does anything in either helper.

• evaluation/workflow/promptiter/backwarder/backwarder.go#L312-L321: drop the if len(surfaceIDs) == 0 block and keep the single return in backwardGradientArraySchema.
• evaluation/workflow/promptiter/backwarder/backwarder.go#L362-L371: drop the if len(predecessorStepIDs) == 0 block and keep the single return in backwardPropagationArraySchema.

中文

两个 backward schema 辅助函数中的空集合条件已成死代码。 去掉 maxItems: 0 后，len(...) == 0 分支返回的数组 schema 与后续 return 完全相同，两个函数中的该条件判断均已无实际作用，建议直接删除并保留单个 return。

🤖 Prompt for AI Agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@evaluation/workflow/promptiter/backwarder/backwarder.go` around lines 312 -
321, Remove the redundant empty-collection conditional from
backwardGradientArraySchema at
evaluation/workflow/promptiter/backwarder/backwarder.go:312-321 and retain one
array-schema return. Apply the same cleanup to backwardPropagationArraySchema at
evaluation/workflow/promptiter/backwarder/backwarder.go:362-371, removing its
len(predecessorStepIDs) branch and preserving the shared return.

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Nitpick comments:
In `@evaluation/workflow/promptiter/backwarder/backwarder.go`:
- Around line 312-321: Remove the redundant empty-collection conditional from
backwardGradientArraySchema at
evaluation/workflow/promptiter/backwarder/backwarder.go:312-321 and retain one
array-schema return. Apply the same cleanup to backwardPropagationArraySchema at
evaluation/workflow/promptiter/backwarder/backwarder.go:362-371, removing its
len(predecessorStepIDs) branch and preserving the shared return.

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Pro

Run ID: 71d7452e-3be1-41bf-afa4-019a444ea724

📥 Commits

Reviewing files that changed from the base of the PR and between 0c77741 and 93a60b8.

📒 Files selected for processing (35)

• evaluation/go.mod
• evaluation/workflow/promptiter/backwarder/backwarder.go
• evaluation/workflow/promptiter/backwarder/backwarder_test.go
• evaluation/workflow/promptiter/regression/attribution.go
• evaluation/workflow/promptiter/regression/attribution_negation_test.go
• evaluation/workflow/promptiter/regression/binding_test.go
• evaluation/workflow/promptiter/regression/config.go
• evaluation/workflow/promptiter/regression/delta.go
• evaluation/workflow/promptiter/regression/delta_gate_test.go
• evaluation/workflow/promptiter/regression/evaluator.go
• evaluation/workflow/promptiter/regression/evaluator_attribution_test.go
• evaluation/workflow/promptiter/regression/gate.go
• evaluation/workflow/promptiter/regression/load.go
• evaluation/workflow/promptiter/regression/load_test.go
• evaluation/workflow/promptiter/regression/optimizer.go
• evaluation/workflow/promptiter/regression/pipeline.go
• evaluation/workflow/promptiter/regression/report.go
• evaluation/workflow/promptiter/regression/report_pipeline_test.go
• evaluation/workflow/promptiter/regression/similarity.go
• evaluation/workflow/promptiter/regression/similarity_test.go
• evaluation/workflow/promptiter/regression/strict_trace_test.go
• evaluation/workflow/promptiter/regression/types.go
• evaluation/workflow/promptiter/regression/usage.go
• examples/evaluation/promptiter_regression_loop/DESIGN.md
• examples/evaluation/promptiter_regression_loop/LEARNING_GUIDE.zh_CN.md
• examples/evaluation/promptiter_regression_loop/README.md
• examples/evaluation/promptiter_regression_loop/baseline_prompt.txt
• examples/evaluation/promptiter_regression_loop/data/metrics.json
• examples/evaluation/promptiter_regression_loop/data/train.evalset.json
• examples/evaluation/promptiter_regression_loop/data/validation.evalset.json
• examples/evaluation/promptiter_regression_loop/main.go
• examples/evaluation/promptiter_regression_loop/main_test.go
• examples/evaluation/promptiter_regression_loop/promptiter.json
• examples/evaluation/promptiter_regression_loop/sample_output/optimization_report.json
• examples/evaluation/promptiter_regression_loop/sample_output/optimization_report.md

[codecov]

Codecov Report

❌ Patch coverage is 85.86422% with 431 lines in your changes missing coverage. Please review.
✅ Project coverage is 89.80154%. Comparing base (d4f78f6) to head (90385ed).
⚠️ Report is 51 commits behind head on main.

Files with missing lines	Patch %	Lines
...luation/workflow/promptiter/regression/pipeline.go	68.41046%	83 Missing and 74 partials ⚠️
...uation/workflow/promptiter/regression/evaluator.go	81.36143%	62 Missing and 53 partials ⚠️
...valuation/workflow/promptiter/regression/report.go	68.51211%	57 Missing and 34 partials ⚠️
...tion/workflow/promptiter/regression/attribution.go	92.61261%	29 Missing and 12 partials ⚠️
evaluation/workflow/promptiter/regression/load.go	93.21429%	12 Missing and 7 partials ⚠️
evaluation/workflow/promptiter/regression/delta.go	98.49057%	3 Missing and 1 partial ⚠️
...ation/workflow/promptiter/regression/similarity.go	88.88889%	2 Missing and 2 partials ⚠️

Additional details and impacted files

@@                 Coverage Diff                 @@
##                main       #2278         +/-   ##
===================================================
- Coverage   89.86563%   89.80154%   -0.06409%     
===================================================
  Files           1144        1162         +18     
  Lines         198483      206590       +8107     
===================================================
+ Hits          178368      185521       +7153     
- Misses         12602       13167        +565     
- Partials        7513        7902        +389     

Flag	Coverage Δ
unittests	89.80154% <85.86422%> (-0.06409%)	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[cyl6]

recheck

[coderabbitai]

🧹 Nitpick comments (1)

evaluation/workflow/promptiter/regression/codecov_boundaries_test.go (1)

334-359: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick win

Gate test mutates shared pointer-backed summaries and relies on manual resets.

GateInput is copied by value, but BaselineValidation/CandidateValidation are pointers, so input.BaselineValidation.Usage.ModelCalls = 1 (Line 351) and input.CandidateValidation.Usage.ToolCalls = 1 (Line 356) mutate the shared baseline/candidate summaries in place. Correctness then hinges on the manual resets at Lines 354 and 359 holding, because the same baseline/candidate are reused downstream (e.g., ComputeDelta(baseline, regressed) at Line 373 and the overflow check at Lines 404-409). A future edit that reorders these sub-checks or forgets a reset would silently break isolation without an obvious failure. Consider cloning the summaries (or valid) per sub-check so each assertion runs against pristine state.

中文

门禁测试就地修改共享指针字段并依赖手动复位。

GateInput 按值拷贝，但 BaselineValidation/CandidateValidation 是指针，因此第 351、356 行通过 input 修改的其实是共享的 baseline/candidate。正确性依赖第 354、359 行的手动复位，而这两个对象在后续（第 373 行 ComputeDelta、第 404-409 行溢出用例）被复用。若日后重排子用例或漏掉复位，隔离会被悄悄破坏且不易察觉。建议为每个子检查克隆 summary（或 valid），使断言基于干净状态运行。

（依据 *_test.go 路径规范：优先保证测试对外部可观测行为的确定性与隔离性。）

🤖 Prompt for AI Agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@evaluation/workflow/promptiter/regression/codecov_boundaries_test.go` around
lines 334 - 359, Isolate each validation sub-check in the gate test from the
shared pointer-backed baseline and candidate summaries. Update the cases around
EvaluateGate, especially the baseline and candidate gate usage checks, to clone
valid or the relevant summaries before mutation, then remove reliance on manual
resets so later ComputeDelta and overflow checks always use pristine baseline
and candidate data.

Source: Path instructions

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Nitpick comments:
In `@evaluation/workflow/promptiter/regression/codecov_boundaries_test.go`:
- Around line 334-359: Isolate each validation sub-check in the gate test from
the shared pointer-backed baseline and candidate summaries. Update the cases
around EvaluateGate, especially the baseline and candidate gate usage checks, to
clone valid or the relevant summaries before mutation, then remove reliance on
manual resets so later ComputeDelta and overflow checks always use pristine
baseline and candidate data.

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Pro

Run ID: 7562c387-c810-47c4-afa2-67bb44a14ce8

📥 Commits

Reviewing files that changed from the base of the PR and between fce5473 and 8fc1254.

📒 Files selected for processing (1)

• evaluation/workflow/promptiter/regression/codecov_boundaries_test.go