Skip to content

Commit 8af8094

Browse files
tmatherntmathern
committed
fix: Workflow github
1 parent ebd7524 commit 8af8094

5 files changed

Lines changed: 178 additions & 80 deletions

File tree

‎.github/memory-benchmark.yml‎

Lines changed: 40 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,40 @@
1+
name: Memory benchmark
2+
3+
on:
4+
pull_request:
5+
types:
6+
- opened
7+
- reopened
8+
- synchronize
9+
- labeled
10+
11+
permissions:
12+
contents: read
13+
14+
jobs:
15+
memory-benchmark:
16+
name: Memory benchmark
17+
# Needs to match the arch the baseline was generated on.
18+
runs-on: ubuntu-24.04-arm
19+
if: |
20+
contains(github.event.pull_request.labels.*.name, 'check-memory-benchmark') &&
21+
(
22+
github.event.pull_request.author_association == 'COLLABORATOR' ||
23+
github.event.pull_request.author_association == 'MEMBER' ||
24+
github.event.pull_request.author_association == 'OWNER'
25+
)
26+
steps:
27+
- uses: actions/checkout@v4
28+
29+
# Uses the Dockerfile environment for repeatable runs.
30+
- name: Run memory benchmark
31+
run: make memory-use-bench
32+
33+
# Upload all three flamegraph views per scenario (peak/leaks/temporary).
34+
- name: Upload flamegraph reports
35+
if: always()
36+
uses: actions/upload-artifact@v4
37+
with:
38+
name: memray-flamegraphs
39+
path: tests/perf/reports/*.html
40+
if-no-files-found: warn

‎Makefile‎

Lines changed: 3 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -148,10 +148,12 @@ MEMRAY_ITERATIONS ?= 100
148148
MEMRAY_THRESHOLD ?= 1.1
149149
SCENARIO ?=
150150
SCENARIO_ARG := $(if $(SCENARIO),--scenario $(SCENARIO),)
151+
# In CI, use en vars to write the report to the job run
152+
GH_SUMMARY_MOUNT := $(if $(GITHUB_STEP_SUMMARY),-v $(GITHUB_STEP_SUMMARY):$(GITHUB_STEP_SUMMARY),)
151153
.PHONY: memory-use-bench
152154
memory-use-bench:
153155
docker build -f tests/perf/Dockerfiles/$(PERF_ENV)-perf-Dockerfile -t c2pa-memray-$(PERF_ENV) .
154-
docker run --rm -v $(PWD):/workspace -e PYTHONPATH=/workspace/src -e PERF_ENV=$(PERF_ENV) -e MEMRAY_ITERATIONS=$(MEMRAY_ITERATIONS) -e MEMRAY_THRESHOLD=$(MEMRAY_THRESHOLD) c2pa-memray-$(PERF_ENV) python -m tests.perf.run_profile $(SCENARIO_ARG) $(PERF_ARGS)
156+
docker run --rm -v $(PWD):/workspace $(GH_SUMMARY_MOUNT) -e PYTHONPATH=/workspace/src -e PERF_ENV=$(PERF_ENV) -e MEMRAY_ITERATIONS=$(MEMRAY_ITERATIONS) -e MEMRAY_THRESHOLD=$(MEMRAY_THRESHOLD) -e GITHUB_TOKEN -e GITHUB_STEP_SUMMARY c2pa-memray-$(PERF_ENV) python -m tests.perf.run_profile $(SCENARIO_ARG) $(PERF_ARGS)
155157
@echo ""
156158
@echo "Reports written to tests/perf/reports/"
157159
@echo "Open tests/perf/reports/<scenario>-{peak,leaks,temporary}.html in a browser"

‎tests/perf/README.md‎

Lines changed: 10 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -67,6 +67,16 @@ The trailing `VAR=value` arguments (e.g. `PERF_ENV=ubuntu-24.04`, `PERF_ARGS=--u
6767

6868
Reports are written to `tests/perf/reports/` on the local machine. Three HTML files per scenario, one per suffix (described below). Open any in a browser. After a run, the run also reports if the scenarios were or were not all within baseline threshold (baseline +10% memory use tolerance).
6969

70+
## Running in CI
71+
72+
The `.github/workflows/memory-benchmark.yml` workflow runs the Docker-based benchmarks on a PR, but only when the PR has the `check-memory-benchmark` label. This runs `make memory-use-bench`, so:
73+
74+
- A regression (peak or leaked > baseline +10%) makes the benchmark job exit non-zero.
75+
- A values report table is written to the job's Step Summary.
76+
- All three flamegraph HTML views per scenario are uploaded as the `memray-flamegraphs` artifact.
77+
78+
The gate only acts as regression test once a `tests/perf/baseline.json` is committed on the branch. Without one, `run_profile.py` treats the run as baseline creation (exits 0, no gating).
79+
7080
## Report views
7181

7282
Each scenario produces three [memray flamegraphs](https://bloomberg.github.io/memray/flamegraph.html). All three are flamegraphs of the same run. They differ only in which allocations they count.

‎tests/perf/baseline.json‎

Lines changed: 79 additions & 79 deletions
Original file line numberDiff line numberDiff line change
@@ -2,139 +2,139 @@
22
"_meta": {
33
"memray_version": "1.19.3",
44
"python_version": "3.12.13",
5-
"c2pa_native_version": "c2pa-v0.85.1",
5+
"c2pa_native_version": "c2pa-v0.86.1",
66
"iterations": 100,
77
"perf_env": "python-3.12-slim",
88
"arch": "aarch64"
99
},
1010
"reader_jpeg_legacy": {
11-
"peak_bytes": 3814421,
12-
"leaked_bytes": 3266116,
13-
"total_allocations": 698899
11+
"peak_bytes": 3790506,
12+
"leaked_bytes": 3337275,
13+
"total_allocations": 694247
1414
},
1515
"reader_jpeg_with_context": {
16-
"peak_bytes": 3822953,
17-
"leaked_bytes": 3257471,
18-
"total_allocations": 692953
16+
"peak_bytes": 3784932,
17+
"leaked_bytes": 3331065,
18+
"total_allocations": 688204
1919
},
2020
"reader_mp4": {
21-
"peak_bytes": 4876441,
22-
"leaked_bytes": 3257485,
23-
"total_allocations": 2112991
21+
"peak_bytes": 4192391,
22+
"leaked_bytes": 3331202,
23+
"total_allocations": 1856171
2424
},
2525
"reader_wav": {
26-
"peak_bytes": 5520266,
27-
"leaked_bytes": 3267427,
28-
"total_allocations": 400371
26+
"peak_bytes": 4440733,
27+
"leaked_bytes": 3281502,
28+
"total_allocations": 386809
2929
},
3030
"builder_sign_jpeg_legacy": {
31-
"peak_bytes": 7695310,
32-
"leaked_bytes": 3383623,
33-
"total_allocations": 522425
31+
"peak_bytes": 7739038,
32+
"leaked_bytes": 3438581,
33+
"total_allocations": 515197
3434
},
3535
"builder_sign_jpeg_with_context": {
36-
"peak_bytes": 7688236,
37-
"leaked_bytes": 3376293,
38-
"total_allocations": 516851
36+
"peak_bytes": 7732082,
37+
"leaked_bytes": 3431373,
38+
"total_allocations": 509328
3939
},
4040
"builder_sign_png_legacy": {
41-
"peak_bytes": 7932767,
42-
"leaked_bytes": 3383648,
43-
"total_allocations": 1694629
41+
"peak_bytes": 7939922,
42+
"leaked_bytes": 3401346,
43+
"total_allocations": 1399159
4444
},
4545
"builder_sign_png_with_context": {
46-
"peak_bytes": 7925490,
47-
"leaked_bytes": 3376452,
48-
"total_allocations": 1688908
46+
"peak_bytes": 7932212,
47+
"leaked_bytes": 3393558,
48+
"total_allocations": 1393313
4949
},
5050
"builder_sign_jpeg_parallel_split_pool": {
51-
"peak_bytes": 45764159,
52-
"leaked_bytes": 3818113,
53-
"total_allocations": 528785
51+
"peak_bytes": 45802848,
52+
"leaked_bytes": 3855293,
53+
"total_allocations": 520103
5454
},
5555
"builder_sign_jpeg_parallel_split_barrier": {
56-
"peak_bytes": 46225287,
57-
"leaked_bytes": 3809216,
58-
"total_allocations": 527412
56+
"peak_bytes": 45771974,
57+
"leaked_bytes": 3853543,
58+
"total_allocations": 519696
5959
},
6060
"builder_sign_png_parallel_split_pool": {
61-
"peak_bytes": 46002549,
62-
"leaked_bytes": 3817801,
63-
"total_allocations": 1700731
61+
"peak_bytes": 46511069,
62+
"leaked_bytes": 3831120,
63+
"total_allocations": 1405010
6464
},
6565
"builder_sign_png_parallel_split_barrier": {
66-
"peak_bytes": 45970433,
67-
"leaked_bytes": 3812044,
68-
"total_allocations": 1699396
66+
"peak_bytes": 45980602,
67+
"leaked_bytes": 3826012,
68+
"total_allocations": 1403879
6969
},
7070
"builder_sign_gif": {
71-
"peak_bytes": 14544515,
72-
"leaked_bytes": 3375865,
73-
"total_allocations": 7183237
71+
"peak_bytes": 14556418,
72+
"leaked_bytes": 3398315,
73+
"total_allocations": 5573733
7474
},
7575
"builder_sign_heic": {
76-
"peak_bytes": 4608484,
77-
"leaked_bytes": 3376030,
78-
"total_allocations": 771079
76+
"peak_bytes": 4624733,
77+
"leaked_bytes": 3402824,
78+
"total_allocations": 733409
7979
},
8080
"builder_sign_m4a": {
81-
"peak_bytes": 18849082,
82-
"leaked_bytes": 3376431,
83-
"total_allocations": 2273497
81+
"peak_bytes": 18864770,
82+
"leaked_bytes": 3402727,
83+
"total_allocations": 2126542
8484
},
8585
"builder_sign_webp": {
86-
"peak_bytes": 8900701,
87-
"leaked_bytes": 3376432,
88-
"total_allocations": 487683
86+
"peak_bytes": 8911369,
87+
"leaked_bytes": 3397659,
88+
"total_allocations": 473589
8989
},
9090
"builder_sign_avi": {
91-
"peak_bytes": 7040387,
92-
"leaked_bytes": 3376267,
93-
"total_allocations": 40315553
91+
"peak_bytes": 7357514,
92+
"leaked_bytes": 3610304,
93+
"total_allocations": 34863789
9494
},
9595
"builder_sign_mp4": {
96-
"peak_bytes": 6162851,
97-
"leaked_bytes": 3376431,
98-
"total_allocations": 1809672
96+
"peak_bytes": 6202809,
97+
"leaked_bytes": 3426948,
98+
"total_allocations": 1595487
9999
},
100100
"builder_sign_tiff": {
101-
"peak_bytes": 13124728,
102-
"leaked_bytes": 3376268,
103-
"total_allocations": 5139967
101+
"peak_bytes": 13430248,
102+
"leaked_bytes": 3661058,
103+
"total_allocations": 5484643
104104
},
105105
"builder_sign_jpeg_parent_of": {
106-
"peak_bytes": 14173992,
107-
"leaked_bytes": 3377656,
108-
"total_allocations": 1209933
106+
"peak_bytes": 14244270,
107+
"leaked_bytes": 3457332,
108+
"total_allocations": 1194872
109109
},
110110
"builder_sign_jpeg_component_of": {
111-
"peak_bytes": 14175518,
112-
"leaked_bytes": 3377891,
113-
"total_allocations": 1232336
111+
"peak_bytes": 14246032,
112+
"leaked_bytes": 3457636,
113+
"total_allocations": 1217308
114114
},
115115
"builder_sign_jpeg_parent_and_component": {
116-
"peak_bytes": 14530406,
117-
"leaked_bytes": 3474418,
118-
"total_allocations": 2160934
116+
"peak_bytes": 14602066,
117+
"leaked_bytes": 3560736,
118+
"total_allocations": 2137378
119119
},
120120
"builder_sign_jpeg_parent_and_component_mixed_mime": {
121-
"peak_bytes": 14476171,
122-
"leaked_bytes": 3378735,
123-
"total_allocations": 2451587
121+
"peak_bytes": 14553513,
122+
"leaked_bytes": 3464495,
123+
"total_allocations": 2148041
124124
},
125125
"builder_sign_jpeg_two_components_same_mime": {
126-
"peak_bytes": 14519270,
127-
"leaked_bytes": 3473673,
128-
"total_allocations": 2150782
126+
"peak_bytes": 14582525,
127+
"leaked_bytes": 3560177,
128+
"total_allocations": 2127097
129129
},
130130
"builder_sign_jpeg_two_components_mixed_mime": {
131-
"peak_bytes": 14473127,
132-
"leaked_bytes": 3377445,
133-
"total_allocations": 2441195
131+
"peak_bytes": 14549485,
132+
"leaked_bytes": 3462954,
133+
"total_allocations": 2137855
134134
},
135135
"builder_sign_jpeg_archive_roundtrip": {
136-
"peak_bytes": 14226832,
137-
"leaked_bytes": 3426491,
138-
"total_allocations": 1680290
136+
"peak_bytes": 14327685,
137+
"leaked_bytes": 3527329,
138+
"total_allocations": 1657777
139139
}
140140
}

‎tests/perf/run_profile.py‎

Lines changed: 46 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -169,6 +169,49 @@ def _fmt(n: int) -> str:
169169
return f"{n} B"
170170

171171

172+
def _delta_pct(current: int, base: int) -> str:
173+
"""Signed percentage change vs baseline, or '-' when no baseline."""
174+
if not base:
175+
return "-"
176+
return f"{(current - base) / base * 100:+.1f}%"
177+
178+
179+
def _write_github_summary(results: dict, baseline: dict) -> None:
180+
"""Append a values table to $GITHUB_STEP_SUMMARY when running in CI.
181+
"""
182+
summary_path = os.environ.get("GITHUB_STEP_SUMMARY")
183+
if not summary_path or not results:
184+
return
185+
186+
lines = [
187+
"## Memory benchmark (memray)",
188+
"",
189+
f"Iterations: {ITERATIONS} · threshold: +{(THRESHOLD - 1) * 100:.0f}%"
190+
f"{f' · env: {PERF_ENV}' if PERF_ENV else ''}",
191+
"",
192+
"| scenario | peak | leaked | allocs | peak Δ% | leaked Δ% | status |",
193+
"|----------|------|--------|--------|---------|-----------|--------|",
194+
]
195+
for name, m in results.items():
196+
b = baseline.get(name, {}) if baseline else {}
197+
peak_base = b.get("peak_bytes", 0)
198+
leaked_base = b.get("leaked_bytes", 0)
199+
regressed = (
200+
(peak_base and m["peak_bytes"] > peak_base * THRESHOLD)
201+
or (leaked_base and m["leaked_bytes"] > leaked_base * THRESHOLD)
202+
)
203+
status = "REGRESSED" if regressed else "ok"
204+
lines.append(
205+
f"| {name} | {_fmt(m['peak_bytes'])} | {_fmt(m['leaked_bytes'])} "
206+
f"| {m['total_allocations']} | {_delta_pct(m['peak_bytes'], peak_base)} "
207+
f"| {_delta_pct(m['leaked_bytes'], leaked_base)} | {status} |"
208+
)
209+
lines.append("")
210+
211+
with open(summary_path, "a", encoding="utf-8") as fh:
212+
fh.write("\n".join(lines) + "\n")
213+
214+
172215
def main() -> None:
173216
parser = argparse.ArgumentParser(description="c2pa-python memory profiler")
174217
parser.add_argument(
@@ -274,6 +317,9 @@ def main() -> None:
274317
verb = "Updated" if baseline else "Created"
275318
print(f"\n{verb} baseline: {BASELINE_FILE}")
276319

320+
# Emit the report table to the PR's Step Summary in CI.
321+
_write_github_summary(results, baseline)
322+
277323
if render_failures:
278324
print("\nFLAMEGRAPH RENDERS FAILED (capture + metrics still recorded):", file=sys.stderr)
279325
for r in render_failures:

0 commit comments

Comments
 (0)