Feat/report ti tabs lists#1879
Open
otavionvidia wants to merge 15 commits into
Open
Conversation
Add a Technique/Intent (CAS taxonomy) view to the garak HTML report: - Technique × Intent DEFCON-severity heatmap - "By technique" and "By intent" breakdown charts - Drill-down side panel with risk severity, evaluation counts, and the contributing detectors/probes for a selected cell or bar Renders only when a report carries technique/intent digest sections and degrades to a clear empty state otherwise, so it is safe ahead of the Python-side TI digest emission. Per-attempt hit evidence is intentionally out of scope here (it belongs to the interactive garak-ui). Rebuilds the bundled garak/analyze/ui/index.html artifact. Signed-off-by: Otavio PAdovani <opadovani@nvidia.com>
- recolour the risk ramp with KUI-native tokens (red → orange → gold → light-blue → deep-blue); avoids Tailwind/KUI palette clashes - keep heatmap cell labels legible: fixed per-cell contrast pinned in both normal and emphasis states, so they never inherit the fill or flash - drive the coordinated row/column highlight imperatively via the ECharts instance instead of rebuilding the option (removes the hover flicker) - make breakdown-bar category labels click and hover exactly like their bars - move the detail-level toggle to a panel control row mirroring the Modules tab's filter bar, so its global scope reads correctly - add the rollup util, breakdown-bar options hook, and their tests (needed by the already-committed components) Co-authored-by: Claude <noreply@anthropic.com> Signed-off-by: Otavio PAdovani <opadovani@nvidia.com>
Remove the technique × intent heatmap, detail drawer, and breakdown bar chart in favour of an objective summary (critical / at-risk / clean pairing counts) plus a conditional "notable pairings" callout for genuine interactions. Techniques and intents now render as Modules-style tabbed accordion lists: expand a row for a worst-first pass-rate bar chart, then click a bar or its axis label to open the pairing detail (severity, concrete pass/fail counts, pooled pairs, and grouped/humanised detectors) in a side panel. Move the detail-level toggle beside the sort control, default to the all-leaves view, and drop the redundant section header. Co-authored-by: Claude Signed-off-by: Otavio PAdovani <opadovani@nvidia.com>
Replace the technique×intent summary cards with a single "Pairing severity" card: one stacked bar split into the five DEFCON levels in their own colours, over a count-only legend (the bar carries the proportions, so per-band percentages are dropped). Counts are computed per DEFCON level so the bands fill the full width. Rename the global "Security Status" heading to "Module Security Status" so its unit (modules) is explicit and distinct from the tab's pairing-level summary. Co-authored-by: Claude Signed-off-by: Otavio PAdovani <opadovani@nvidia.com>
Regenerate garak/analyze/ui/index.html so the shipped report bundle matches the current technique×intent UI source (DEFCON severity bar and Module Security Status rename included). Co-authored-by: Claude Signed-off-by: Otavio PAdovani <opadovani@nvidia.com>
Add component tests for the technique×intent UI so the suite meets the 85% coverage gate: TaxonomyCellChart click handling (bar and axis-label), TaxonomyAxisList + FlatTaxonomyList render paths (chart, single-child, pooled-leaf detail, clean state, filter/sort/empty states), and the TechniqueIntentPanel matrix vs marginal-only flows with its filter, sort, level and tab controls. Co-authored-by: Claude Signed-off-by: Otavio PAdovani <opadovani@nvidia.com>
The previous bundle was built with a stale local vite 6.4.1; CI builds with yarn (vite 7.3.5 per yarn.lock) and rejects the mismatch. Rebuild via `yarn install && yarn build` so the committed bundle matches CI (vite 7.3.5, identical chunk hashes). Co-authored-by: Claude Signed-off-by: Otavio PAdovani <opadovani@nvidia.com>
otavionvidia
force-pushed
the
feat/report-ti-tabs-lists
branch
from
c1644bb to
40734f0
Compare
patriciapampanelli
moved this from In Progress
to In Review
in garak / Context Aware Scanning
patriciapampanelli
requested review from
jmartin-tech,
leondz and
patriciapampanelli
jmartin-tech
requested changes
Jun 24, 2026
jmartin-tech
left a comment
jmartin-tech
left a comment
Collaborator
There was a problem hiding this comment.
I suspect this was based on an in progress form of the digest entry.
Testing base on a report created using 0daa46d, the current tip of the target branch, fails to display the new tab. The latest format of the digest should contain all data needed however the current processing looks like it will need to be reworked.
A sample of the digest with faked testing data can be obtained using the following:
python -m garak -t test
Co-authored-by: Jeffrey Martin <jmartin@Op3n4M3.dev> Signed-off-by: Otavio Padovani <opadovani@nvidia.com>
Co-authored-by: Jeffrey Martin <jmartin@Op3n4M3.dev> Signed-off-by: Otavio Padovani <opadovani@nvidia.com>
The Techniques & Intents tab assumed a digest shape that `_compute_technique_intent_matrix` never emits (top-level `technique`/ `intent` marginal maps, `n_evaluations`, and `detectors_used` name arrays), so the tab failed to render on reports from the target branch. Source the view exclusively from `technique_intent_matrix`: - model the real cell (`score` nullable, `passed`, `total_evaluated`, `nones`, `n_detectors`) and the reserved per-row `_summary`; - skip `_summary` rows and drop unevaluated (null-score / zero-eval) pairings so they never reach the severity bands or worst-first order; - pool real passed/undetermined counts and surface concrete pass/fail stats plus a detector count in the detail panel (the digest carries no detector names at this granularity); - drop the marginal-only `FlatTaxonomyList` path that had no backing data. Co-authored-by: Claude <noreply@anthropic.com> Signed-off-by: Otavio PAdovani <opadovani@nvidia.com>
…eat/report-ti-tabs-lists Signed-off-by: Otavio PAdovani <opadovani@nvidia.com>
Act on review feedback for the Techniques & Intents tab: - Replace the DEFCON severity bar with a plain "Coverage" summary (techniques, intents, pairings evaluated, overall pass rate); severity stays on the per-pairing badges in the lists. - Consume the new digest naming: technique/intent cells now carry a human-readable `name`, and each technique a `_summary.name` and `description`. Labels, the technique×intent pairing rows, and a new per-technique description line use these instead of raw demon:* keys and intent codes. - Tighten the detail stats: drop the ambiguous standalone detector box and spell out that counts are evaluations (one per attempt × detector), so the figure no longer has to be inferred. Signed-off-by: Otavio PAdovani <opadovani@nvidia.com>
- Surface distinct prompt counts per technique×intent pairing: the evaluator now tracks unique attempt UUIDs per intent and the digest pools them across probes (max within a probe, sum across probes) so the figure reflects prompts rather than attempts × detectors. Carried through the frontend types and worst-first rollup. - Notable-pairing clicks now re-scroll to the open detail even when the same pairing is clicked again, via a per-click nonce; the scroll waits out the accordion open animation only when one is actually running. - Keep the single-file inlining plugin out of `vite dev` and enable polling watchers so HMR stops serving stale modules on macOS. Signed-off-by: Otavio PAdovani <opadovani@nvidia.com>
Address review feedback on the technique/intent report: - Normalize run.spec in the digest to the readable probespec (was rendering as an object in Setup), with a formatValue object fallback. - Name and describe grouped intent columns from a bundled trait-typology snapshot instead of showing raw codes like S006; guarded by a drift test against garak/data/cas/trait_typology.json. - Track distinct prompts per probe (evaluator n_attempts -> digest prompt_count) so the Modules view shows overall prompt counts.
jmartin-tech
reviewed
Jul 2, 2026
Collaborator
There was a problem hiding this comment.
Why is do we need to duplicate this file in the repository? A symlink or simply referencing the file already as garak/data/cas/trait_typology.json would remove the concern about drift.
jmartin-tech
reviewed
Jul 2, 2026
| "nones": nones, | ||
| "total_evaluated": outputs_evaluated, | ||
| "total_processed": outputs_processed, | ||
| "n_attempts": len(attempt_uuids), |
Collaborator
There was a problem hiding this comment.
I'm not sure this would be the right value, this entry is still for the probe/detector pair not the overall probe.
Collaborator
There was a problem hiding this comment.
I have confirmed this would not be the right value, I will offer a revision that provides the correct value somewhere in the digest on a per probe across all detectors summary.
…prompt-count fallback
Address further review on the technique/intent report:
- Intent columns now show the taxonomy code alongside the name
("C006 - Anthropomorphise") for both grouped families and leaves, so
the slug is unambiguous.
- Render a run.spec {include, exclude} object as one selector per line in
the Setup view (plugin paths verbatim, filters as key:value, excludes
prefixed with -), so reports whose digest predates spec rendering no
longer show raw JSON / [object Object].
- Fall back to the first detector's evaluation total for the probe
prompt-count badge when the digest carries no prompt_count, matching
the bar-chart tooltip; freshly run reports still show the exact count.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What this does
Screen.Recording.2026-06-23.at.4.50.46.PM.mov
Adds a Techniques & Intents view to the HTML report (
garak-report/), surfacinggarak's existing taxonomy digest data (
digest.technique,digest.intent,digest.technique_intent) that previously had no UI. This is a report/frontend-onlyfeature — no probe, detector, generator, or scoring behaviour changes.
Highlights:
DEFCON levels (own colour each, worst-first) over a count-only legend, plus a
pairings · techniques · intentscaption.style: each entry shows a worst-first horizontal bar chart of its cross-axis pairings,
with an inline detail panel (pass/fail evals, pooled leaves, grouped/prettified
detectors).
far worse than the technique or intent does on its own); silent for separable matrices.
reusing the shared
ReportFilterBar.explicit and distinct from the new pairing-level summary (both use the DC-3 threshold).
useSeverityColorhelper, auseTaxonomyCellChartOptionschart-options hook, andtechniqueIntentRollup/taxonomyLabelsutilities — all with unit tests.No tracked bug, so no issue number.
Unique branch
Submitted from
feat/report-ti-tabs-lists(fork) →mainupstream. No other open PRintroduces the technique×intent report UI.
AI assistance
AI assistance was used; the submitter reviewed every changed line.
Verification
Frontend feature under
garak-report/, so the JS toolchain applies (the generator-config/
garak -tflow isn't relevant here):cd garak-report && npm cinpm run check(tsc --noEmit) — cleannpm run lint(eslint .) — cleannpm test(vitest run --coverage) — incl.techniqueIntentRollup,useTaxonomyCellChartOptions,SummaryStatsCardnpm run build(regeneratesgarak/analyze/ui/index.html)python -m pytest tests/severity bar shows one coloured slice per populated DEFCON level; lists, charts, and
inline detail panels work; label and bar clicks both select
and no broken tab
pairing-level status card up top
No special hardware or proprietary software required.