This should be resolved in a separate PR. The CI sometimes takes over 6 hours in fixed-opcode-count mode, which is unexpected. In normal CI pipeline, running 1K opcode count benchmarks completes in ~20 minutes, while the fixed-opcode-count config takes ~3 min.
The root cause is in the collection/filtering logic. Tests without a code generator (i.e., those that provide blocks= or tx= directly) are only skipped after the test body has already executed. The flow is:
- The test body runs: pre-allocation, transaction construction, calldata generation, etc.
- The test calls
benchmark_test(blocks=...), which instantiates BenchmarkTest.
model_post_init checks for a code generator, if absent, the test is skipped via pytest.skip().
The expensive work in step 1 is wasted. For uncacheable precompile benchmarks, which construct blocks manually rather than using the code generator, this is especially costly: in fixed-opcode-count mode, gas_benchmark_value is set to 1 trillion gas. With Osaka's 16M transaction gas cap, the transaction-building loop iterates ~60,000 times per test: generating calldata, funding EOAs, and creating Transaction objecte, all before the inevitable skip.
The fix should move the skip earlier, either by filtering these tests at collection time in pytest_collection_modifyitems, or by skipping in the gas_benchmark_value fixture when it detects the test will not use a code generator.
Originally posted by @LouisTsai-Csie in #2600
Originally posted by @LouisTsai-Csie in #2600