Merge pull request #414 from Integration-Automation/dev

Release: test-robustness lane — failure signatures, run diff, flake clustering, step timeline (v191–v194)

Author: JE-Chen

WHATS_NEW.md

@@ -1,5 +1,29 @@
 # What's New — AutoControl
 
+## What's new (2026-06-25) — Per-Run Step Timeline (waterfall + bottleneck steps)
+
+Read why *this* run was slow — a step waterfall and its bottlenecks. Full reference: [`docs/source/Eng/doc/new_features/v194_features_doc.rst`](docs/source/Eng/doc/new_features/v194_features_doc.rst).
+
+- **`build_timeline` / `critical_steps`** (`AC_build_timeline`, `AC_critical_steps`): the action profiler aggregates timings by step *name* across runs — useless for "why was *this* run slow". This turns one run's ordered steps into a waterfall (each step's offset, duration, and `pct` share of the total) with the `bottleneck` step and a `parallelism` ratio (`> 1` when steps overlap via explicit `start` times); `critical_steps` ranks the dominant steps to optimise. A step is any `{name, duration, start?}` dict. Pure stdlib. No `PySide6`.
+
+## What's new (2026-06-25) — Flaky-Test Co-Failure Clustering
+
+Find the tests that flake *together* — and the shared root cause behind them. Full reference: [`docs/source/Eng/doc/new_features/v193_features_doc.rst`](docs/source/Eng/doc/new_features/v193_features_doc.rst).
+
+- **`cofailure_pairs` / `failure_clusters`** (`AC_cofailure_pairs`, `AC_failure_clusters`): flaky tests are rarely independent — a wobbly fixture or noisy dependency makes a *group* fail in the same runs (~75% of flaky tests cluster). Ranking tests one-by-one by flip rate misses that. This measures how often each pair of tests fails in the *same* runs (Jaccard over their failing-run sets) and groups tests above a threshold into connected clusters with a cohesion score — so you chase one root cause instead of N symptoms. Input is a list of runs, each the test names that failed in it. Pure stdlib. No `PySide6`.
+
+## What's new (2026-06-25) — Run-Trace Diff (what changed between two executions)
+
+See exactly what changed between a passing run and a failing one. Full reference: [`docs/source/Eng/doc/new_features/v192_features_doc.rst`](docs/source/Eng/doc/new_features/v192_features_doc.rst).
+
+- **`diff_runs` / `summarize_run_diff`** (`AC_diff_runs`): a run history says a run *failed* but not *what changed* from the run that passed. This aligns two step sequences with a longest-common-subsequence walk (so an inserted/removed step shifts the rest into place instead of mis-pairing everything) and classifies the differences: **added**/**removed** steps, **status_flips** (an aligned step that changed status — with the new failure's `failure_signature` when it carries an error), and **timing_regressions** (a step that got `regress_factor`× slower). `summarize_run_diff` renders a one-line summary. Pure stdlib over lists of `{name,status,duration,error}` step dicts. No `PySide6`.
+
+## What's new (2026-06-25) — Stable Failure Signatures
+
+Match the *same kind* of failure across runs, despite differing paths and ids. Full reference: [`docs/source/Eng/doc/new_features/v191_features_doc.rst`](docs/source/Eng/doc/new_features/v191_features_doc.rst).
+
+- **`normalize_error` / `failure_signature` / `group_failures`** (`AC_failure_signature`, `AC_group_failures`): two runs that failed the same way rarely have byte-identical error text — paths, line numbers, addresses, ids and timestamps differ every time — which defeats "is this the same failure?" and "which tests fail together?". This strips the variable parts of an error to a canonical form and hashes it (SHA-256), so the same kind of failure gets the same short signature across runs — the join key the rest of the test-robustness tools (run diffing, flake clustering) group on. `group_failures` buckets a list of errors by signature, most frequent first. Pure stdlib (`re` + `hashlib`). No `PySide6`.
+
 ## What's new (2026-06-24) — Visual Saliency (where to look — spectral-residual)
 
 Find the region that stands out, with no template / colour / text. Full reference: [`docs/source/Eng/doc/new_features/v190_features_doc.rst`](docs/source/Eng/doc/new_features/v190_features_doc.rst).

docs/source/Eng/doc/new_features/v191_features_doc.rst

@@ -0,0 +1,48 @@
+Stable Failure Signatures
+=========================
+
+Two runs that failed the *same way* almost never have byte-identical error text —
+paths, line numbers, memory addresses, ids and timestamps differ every time. That
+defeats any attempt to ask "is this the same failure as yesterday?" or "which
+tests fail *together*?". ``failure_signature`` strips the variable parts of an
+error to a canonical form and hashes it (SHA-256), so the same *kind* of failure
+gets the same short signature across runs — the join key the rest of the
+test-robustness tools (run diffing, flake clustering) group on.
+
+* :func:`normalize_error` — collapse paths / hex addresses / UUIDs / timestamps /
+  line numbers / bare integers to placeholders,
+* :func:`failure_signature` — a short stable SHA-256 of the normalised message,
+* :func:`group_failures` — group a list of errors by signature, most frequent
+  first.
+
+Pure standard library (``re`` + ``hashlib``); no device, no ``PySide6``.
+
+Headless API
+------------
+
+.. code-block:: python
+
+    from je_auto_control import (normalize_error, failure_signature,
+                                 group_failures)
+
+    a = r"Timeout at C:\app\run.py line 42 (0x7ffab12c) at 2026-06-24 11:03:21"
+    b = r"Timeout at C:\app\run.py line 88 (0x1234abcd) at 2026-06-25 09:15:00"
+    normalize_error(a)          # "Timeout at <path> line <n> (0x<addr>) at <ts>"
+    failure_signature(a) == failure_signature(b)        # True — same failure
+
+    group_failures([a, b, "Connection refused to /tmp/x.sock"])
+    # [{"signature": "...", "normalized": "...", "count": 2, "examples": [...]},
+    #  {"signature": "...", "count": 1, ...}]
+
+Windows and POSIX paths, ``0x`` addresses, UUIDs, ISO timestamps, ``line N`` and
+any leftover integers become placeholders; whitespace is squeezed.
+``group_failures`` keeps up to three distinct raw examples per group and skips
+empty / ``None`` messages.
+
+Executor commands
+-----------------
+
+``AC_failure_signature`` (``error`` / ``length``) returns ``{signature,
+normalized}``; ``AC_group_failures`` (``errors``) returns the grouped list. They
+are exposed as read-only ``ac_*`` MCP tools and as Script Builder commands under
+**Testing**.

docs/source/Eng/doc/new_features/v192_features_doc.rst

@@ -0,0 +1,48 @@
+Run-Trace Diff (what changed between two executions)
+====================================================
+
+A run history tells you a run *failed*, but not *what changed* from the run that
+passed: which step was added or dropped, which step flipped pass→fail, which step
+got slower. ``run_diff`` aligns the two step sequences with a longest-common-
+subsequence walk — so an inserted or removed step shifts the rest into place
+instead of mis-pairing everything — and classifies the differences:
+
+* **added** / **removed** — steps present in only one run,
+* **status_flips** — an aligned step whose status changed, with the new failure's
+  :func:`failure_signature` when it carries an ``error``,
+* **timing_regressions** — an aligned step that got ``regress_factor`` x slower.
+
+A step is any dict with a name key (default ``"name"``) and optional ``status`` /
+``duration`` / ``error``. Pure standard library; no device, no ``PySide6``.
+
+Headless API
+------------
+
+.. code-block:: python
+
+    from je_auto_control import diff_runs, summarize_run_diff
+
+    before = [{"name": "login", "status": "ok", "duration": 1.0},
+              {"name": "submit", "status": "ok", "duration": 1.0}]
+    after = [{"name": "login", "status": "ok", "duration": 1.1},
+             {"name": "accept_cookies", "status": "ok"},          # inserted
+             {"name": "submit", "status": "error", "error": "Timeout ..."}]
+
+    diff = diff_runs(before, after)
+    # {"added": [accept_cookies], "removed": [],
+    #  "status_flips": [{"name": "submit", "from": "ok", "to": "error",
+    #                    "signature": "..."}],
+    #  "timing_regressions": [], "aligned": 2, "identical": False}
+
+    summarize_run_diff(diff)        # "+1 added, 1 status flip(s)"
+
+``regress_factor`` (default ``1.5``) is the slowdown ratio that counts as a
+regression; ``key`` selects the field steps are aligned on. ``summarize_run_diff``
+renders a one-line summary (``"no change"`` when identical).
+
+Executor commands
+-----------------
+
+``AC_diff_runs`` (``before`` / ``after`` / ``key`` / ``regress_factor``) returns
+the diff plus a ``summary`` field. It is exposed as the read-only ``ac_diff_runs``
+MCP tool and as a Script Builder command under **Testing**.

docs/source/Eng/doc/new_features/v193_features_doc.rst

@@ -0,0 +1,46 @@
+Flaky-Test Co-Failure Clustering
+================================
+
+Flaky tests are rarely independent: a wobbly shared fixture, a slow dependency or
+a noisy environment makes a *group* of tests fail in the same runs (research finds
+~75% of flaky tests fall into co-failure clusters). Ranking tests one-by-one by
+flip rate misses that shared root cause. ``flake_cluster`` measures how often each
+pair of tests fails in the *same* runs — Jaccard similarity over the set of runs
+each failed in — and groups tests whose co-failure exceeds a threshold, so you can
+chase one root cause instead of N symptoms.
+
+* :func:`cofailure_pairs` — test pairs that fail together above a threshold,
+* :func:`failure_clusters` — connected clusters of co-failing tests with a
+  cohesion score (mean pairwise Jaccard).
+
+Input is a list of runs, each a collection of the test names that failed in that
+run. Pure standard library; no device, no ``PySide6``.
+
+Headless API
+------------
+
+.. code-block:: python
+
+    from je_auto_control import failure_clusters, cofailure_pairs
+
+    runs = [["test_a", "test_b"],            # both failed in this run
+            ["test_a", "test_b"],
+            ["test_c"],
+            ["test_a", "test_b", "test_c"]]
+
+    failure_clusters(runs, threshold=0.6)
+    # [{"tests": ["test_a", "test_b"], "size": 2, "cohesion": 1.0}]
+
+    cofailure_pairs(runs, threshold=0.6)
+    # [{"tests": ["test_a", "test_b"], "jaccard": 1.0, "co_failures": 3}]
+
+``threshold`` is the minimum co-failure Jaccard to link two tests; ``min_size``
+(default ``2``) drops singletons so only genuine clusters surface. Clusters come
+back largest / most cohesive first.
+
+Executor commands
+-----------------
+
+``AC_failure_clusters`` (``runs`` / ``threshold`` / ``min_size``) and
+``AC_cofailure_pairs`` (``runs`` / ``threshold``). They are exposed as read-only
+``ac_*`` MCP tools and as Script Builder commands under **Testing**.

docs/source/Eng/doc/new_features/v194_features_doc.rst

@@ -0,0 +1,51 @@
+Per-Run Step Timeline (waterfall + bottleneck steps)
+====================================================
+
+The action profiler aggregates timings by step *name* across many runs — great
+for "which action is slow on average", useless for "why was *this* run slow". A
+single run is an ordered timeline: step A ran, then B, then C, and one of them
+dominated. ``step_timeline`` turns one run's steps into a waterfall (each step's
+offset from the start, its duration and its share of the total) and ranks the
+bottleneck steps, so you can read a single slow run instead of an average.
+
+* :func:`build_timeline` — the waterfall + total / busy / bottleneck /
+  parallelism,
+* :func:`critical_steps` — the steps that dominate the run, longest first.
+
+A step is any dict with a name (default ``"name"``) and a ``duration``; an
+optional ``start`` places it on an absolute timeline (overlapping / parallel
+steps), else steps are laid out back-to-back. Pure standard library; no device,
+no ``PySide6``.
+
+Headless API
+------------
+
+.. code-block:: python
+
+    from je_auto_control import build_timeline, critical_steps
+
+    steps = [{"name": "login", "duration": 1.0},
+             {"name": "load_dashboard", "duration": 4.0},
+             {"name": "submit", "duration": 1.0}]
+
+    build_timeline(steps)
+    # {"steps": [{"name": "login", "offset": 0.0, "duration": 1.0, "pct": 16.7},
+    #            {"name": "load_dashboard", "offset": 1.0, ..., "pct": 66.7}, ...],
+    #  "total": 6.0, "busy": 6.0,
+    #  "bottleneck": {"name": "load_dashboard", "duration": 4.0},
+    #  "parallelism": 1.0}
+
+    critical_steps(steps, top=2)
+    # [{"name": "load_dashboard", "duration": 4.0, "pct": 66.7},
+    #  {"name": "login", "duration": 1.0, "pct": 16.7}]
+
+``total`` is the wall-clock span, ``busy`` the summed step time; ``parallelism`` =
+busy / total is ``1.0`` for a purely sequential run and ``> 1`` when steps overlap
+(supply ``start`` times). ``pct`` is each step's share of the total time.
+
+Executor commands
+-----------------
+
+``AC_build_timeline`` (``steps``) and ``AC_critical_steps`` (``steps`` / ``top``).
+They are exposed as read-only ``ac_*`` MCP tools and as Script Builder commands
+under **Testing**.

docs/source/Zh/doc/new_features/v191_features_doc.rst

@@ -0,0 +1,41 @@
+穩定的失敗簽章
+==============
+
+兩次以*相同方式*失敗的執行,幾乎不會有逐位元組相同的錯誤文字——路徑、行號、記憶體位址、id 與
+時間戳每次都不同。這使得「這和昨天是同一個失敗嗎?」或「哪些測試會*一起*失敗?」無從問起。
+``failure_signature`` 把錯誤的變動部分剝離成標準形式並雜湊(SHA-256),於是*相同類型*的失敗在
+不同執行間會得到相同的短簽章——即其餘 test-robustness 工具(執行比較、flaky 分群)所依據的
+join key。
+
+* :func:`normalize_error` ——把路徑 / 十六進位位址 / UUID / 時間戳 / 行號 / 裸整數收斂成佔位符,
+* :func:`failure_signature` ——正規化訊息的短而穩定的 SHA-256,
+* :func:`group_failures` ——把一組錯誤依簽章分組,最常見者在前。
+
+純標準庫(``re`` + ``hashlib``);不涉及裝置,不匯入 ``PySide6``。
+
+無頭 API
+--------
+
+.. code-block:: python
+
+    from je_auto_control import (normalize_error, failure_signature,
+                                 group_failures)
+
+    a = r"Timeout at C:\app\run.py line 42 (0x7ffab12c) at 2026-06-24 11:03:21"
+    b = r"Timeout at C:\app\run.py line 88 (0x1234abcd) at 2026-06-25 09:15:00"
+    normalize_error(a)          # "Timeout at <path> line <n> (0x<addr>) at <ts>"
+    failure_signature(a) == failure_signature(b)        # True——同一個失敗
+
+    group_failures([a, b, "Connection refused to /tmp/x.sock"])
+    # [{"signature": "...", "normalized": "...", "count": 2, "examples": [...]},
+    #  {"signature": "...", "count": 1, ...}]
+
+Windows 與 POSIX 路徑、``0x`` 位址、UUID、ISO 時間戳、``line N`` 與任何殘留整數都會變成佔位符;
+空白會被壓縮。``group_failures`` 每組最多保留三個不同的原始範例,並略過空 / ``None`` 訊息。
+
+執行器指令
+----------
+
+``AC_failure_signature``(``error`` / ``length``)回傳 ``{signature, normalized}``;
+``AC_group_failures``(``errors``)回傳分組清單。皆以唯讀 ``ac_*`` MCP 工具及 Script Builder
+指令(位於 **Testing** 分類下)形式提供。

docs/source/Zh/doc/new_features/v192_features_doc.rst

@@ -0,0 +1,45 @@
+執行軌跡比較(兩次執行之間改變了什麼)
+======================================
+
+執行歷史告訴你某次執行*失敗*了,卻不告訴你相較於通過的那次*改變了什麼*:哪個步驟被加入或移除、
+哪個步驟由通過翻轉成失敗、哪個步驟變慢了。``run_diff`` 以最長共同子序列(LCS)走訪對齊兩個步驟
+序列——這樣插入或移除一個步驟會把其餘步驟順移到位,而非整個錯位配對——並將差異分類:
+
+* **added** / **removed** ——只存在於其中一次執行的步驟,
+* **status_flips** ——某個已對齊步驟的狀態改變,若帶有 ``error`` 則附上新失敗的
+  :func:`failure_signature`,
+* **timing_regressions** ——某個已對齊步驟變慢了 ``regress_factor`` 倍。
+
+步驟可為任何帶有名稱鍵(預設 ``"name"``)與選填 ``status`` / ``duration`` / ``error`` 的字典。
+純標準庫;不涉及裝置,不匯入 ``PySide6``。
+
+無頭 API
+--------
+
+.. code-block:: python
+
+    from je_auto_control import diff_runs, summarize_run_diff
+
+    before = [{"name": "login", "status": "ok", "duration": 1.0},
+              {"name": "submit", "status": "ok", "duration": 1.0}]
+    after = [{"name": "login", "status": "ok", "duration": 1.1},
+             {"name": "accept_cookies", "status": "ok"},          # 插入
+             {"name": "submit", "status": "error", "error": "Timeout ..."}]
+
+    diff = diff_runs(before, after)
+    # {"added": [accept_cookies], "removed": [],
+    #  "status_flips": [{"name": "submit", "from": "ok", "to": "error",
+    #                    "signature": "..."}],
+    #  "timing_regressions": [], "aligned": 2, "identical": False}
+
+    summarize_run_diff(diff)        # "+1 added, 1 status flip(s)"
+
+``regress_factor``(預設 ``1.5``)是算作退化的變慢比率;``key`` 選擇步驟對齊所依據的欄位。
+``summarize_run_diff`` 產生一行摘要(相同時為 ``"no change"``)。
+
+執行器指令
+----------
+
+``AC_diff_runs``(``before`` / ``after`` / ``key`` / ``regress_factor``)回傳該差異並附帶
+``summary`` 欄位。以唯讀 ``ac_diff_runs`` MCP 工具及 Script Builder 指令(位於 **Testing**
+分類下)形式提供。

docs/source/Zh/doc/new_features/v193_features_doc.rst

@@ -0,0 +1,41 @@
+不穩定測試的共同失敗分群
+========================
+
+不穩定(flaky)測試很少是獨立的:搖晃的共用 fixture、緩慢的相依、或吵雜的環境,會讓*一群*測試在
+相同的執行中一起失敗(研究發現約 75% 的 flaky 測試落在共同失敗的群集裡)。逐一以翻轉率排名測試
+會錯過這個共同根因。``flake_cluster`` 量測每對測試多常在*相同*執行中失敗——即各自失敗的執行集合
+之間的 Jaccard 相似度——並把共同失敗超過門檻的測試分群,讓你能追一個根因,而非 N 個症狀。
+
+* :func:`cofailure_pairs` ——共同失敗超過門檻的測試對,
+* :func:`failure_clusters` ——共同失敗測試的連通群集,附凝聚度分數(群內平均成對 Jaccard)。
+
+輸入是一份執行清單,每個元素為該次執行中失敗的測試名稱集合。純標準庫;不涉及裝置,不匯入
+``PySide6``。
+
+無頭 API
+--------
+
+.. code-block:: python
+
+    from je_auto_control import failure_clusters, cofailure_pairs
+
+    runs = [["test_a", "test_b"],            # 這次執行兩者皆失敗
+            ["test_a", "test_b"],
+            ["test_c"],
+            ["test_a", "test_b", "test_c"]]
+
+    failure_clusters(runs, threshold=0.6)
+    # [{"tests": ["test_a", "test_b"], "size": 2, "cohesion": 1.0}]
+
+    cofailure_pairs(runs, threshold=0.6)
+    # [{"tests": ["test_a", "test_b"], "jaccard": 1.0, "co_failures": 3}]
+
+``threshold`` 是連結兩測試所需的最小共同失敗 Jaccard;``min_size``(預設 ``2``)會丟棄單例,
+讓只有真正的群集浮現。群集以最大 / 最凝聚者在前回傳。
+
+執行器指令
+----------
+
+``AC_failure_clusters``(``runs`` / ``threshold`` / ``min_size``)與
+``AC_cofailure_pairs``(``runs`` / ``threshold``)。皆以唯讀 ``ac_*`` MCP 工具及 Script Builder
+指令(位於 **Testing** 分類下)形式提供。