QVAC-19368 infra: rebalance Android Device Farm shards + faster mobile CI for LLM - #2466
Merged
Conversation
…attern Android groupB (11 tests, 49 min) and groupImagesPerf (3 VLM tests, 69 min) were serialising heavy tests on a single device — hitting the 2h job timeout on Pixel. Mirror the iOS strategy: isolate each heavy test into its own group (heavy1–heavy10) and bundle fast tests into lightA/lightB (12 groups total). Longest single shard drops from ~69 min to ~23 min; pool recycles devices across groups dynamically. Co-authored-by: Cursor <cursoragent@cursor.com>
…ENCY The 12-group mirror of iOS overwhelmed the Device Farm account concurrency limit (24 total runs: 12 iOS + 12 Android). Groups queued up to 12.5 min on Android and 28 min on iOS waiting for a slot, making the monitor step slower than the original 3-group layout. Revised to 6 Android groups (18 total with iOS): - heavyA/heavyB: split the old groupB heavy tests into 2 balanced shards - imagePerfA/imagePerfB: split VLM tests 2+1 to avoid the 69-min single-group bottleneck - lightA/lightB: fast tests bundled Expected critical path: ~40-50 min (vs 69 min old, 87 min with 12 groups). Co-authored-by: Cursor <cursoragent@cursor.com>
With 6 Android groups × 3 devices each = 18 device-jobs, the serial log download took 52 min (each device-job ~3-7 min of API calls + artifact downloads). Process each run's logs in parallel (up to 4 concurrent), so the total is bounded by the slowest single run (~18 min) rather than the sum of all runs. Combined with the 6-group monitor improvement (57 min vs old 69 min), the estimated total Android job time drops to ~86 min — well within the 120 min timeout. Co-authored-by: Cursor <cursoragent@cursor.com>
Rename test groups to be self-documenting: - iOS: heavy1..heavy10 → finetuning, toolCalling, reasoning, etc. - Android: heavyA → heavyA-finetune-reason-ocr, imagePerfB → imagePerf-fruitPlate, etc. Add test-specs passthrough to the monitor step so it can print: - A "Run → tests" legend at the start (which tests are in each run) - Test names in the final results section next to each run link Now when a run fails you can immediately see which test(s) it contained without cross-referencing test-groups.json. Co-authored-by: Cursor <cursoragent@cursor.com>
Pass test-specs from upload-to-devicefarm through to the monitor step in all 12 addon integration workflows. Gives every addon the run-to-tests legend and test names in final results — not just LLM. Co-authored-by: Cursor <cursoragent@cursor.com>
Two issues from the Android shard split: 1. Image test instability: the fruit-plate test relied on elephant running first in the same group to warm up the VLM model. With split groups each image test cold-starts alone, causing crashes on Android. Extended the iosWarmupImage pre-warmup to all mobile platforms (isMobile) so fruit-plate gets the elephant pre-warmup on Android too. 2. Heavy group imbalance: heavyA (4 tests, ~44 min) and heavyB (3 tests, ~45 min) were both too slow. Split into 3 balanced groups of 2-3 tests each: - heavyA-finetune-reasoning (2 tests) - heavyB-toolCall-gemma (2 tests) - heavyC-ocr-sliding (3 tests) Android now has 7 groups (19 total with iOS 12). Co-authored-by: Cursor <cursoragent@cursor.com>
Two optimizations for Device Farm log collection: 1. Skip 'Setup Test' and 'Teardown Test' suites — they only contain framework bookkeeping (home screen screenshots, install logs), not test output. Saves 2 list-artifacts API calls + downloads per device-job (21 Android device-jobs × 2 = 42 fewer API round-trips). 2. Raise MAX_PARALLEL from 4 to 8 so all runs (up to 7 Android + 12 iOS) download simultaneously instead of in waves. AWS Device Farm API handles this fine — the bottleneck was I/O wait, not CPU. Target: Android log collection from 25 min → ~12-15 min. Co-authored-by: Cursor <cursoragent@cursor.com>
…rm-shard-split Co-authored-by: Cursor <cursoragent@cursor.com> # Conflicts: # packages/llm-llamacpp/test/mobile/test-groups.json
The 3-image VLM perf (gemma4 + qwen3-5) made the Android on-PR leg run too long. aurora is the heaviest image, so skip it when QVAC_PERF_RUNS is at the on-PR default (<=1); the benchmark (QVAC_PERF_RUNS>1) still runs all 3. On-PR now covers elephant + fruit-plate, keeping the Android run under ~1h. Co-authored-by: Cursor <cursoragent@cursor.com>
DmitryMalishev
previously approved these changes
Jun 8, 2026
Contributor
Tier-based Approval Status |
Per team agreement, remove the download + upload of the full Device Farm log tree (screenshots, XML, install logs, videos) — nobody uses it and it adds significant download time to the already-tight Android job. Only Customer_Artifacts.zip (bare_console.log, test-results.json, logcat_full, perf data) and Logcat files (C++ logs) are kept. The extracted console-logs and perf-report artifacts are unchanged. Raw Device Farm artifacts are still accessible via the AWS console links in the monitor output. Co-authored-by: Cursor <cursoragent@cursor.com>
…rm-shard-split Co-authored-by: Cursor <cursoragent@cursor.com> # Conflicts: # packages/llm-llamacpp/test/mobile/test-groups.json
…space in download filter Co-authored-by: Cursor <cursoragent@cursor.com>
The parallelized download_run_logs with export -f was crashing on both iOS and Android (exit code 1 before downloading any artifacts). Revert to main's proven sequential loop structure and add the name filter there instead. The filter still skips TCP dump (624MB), screenshots, XML, videos — only Customer Artifacts + Logcat are downloaded. Job-level artifacts restored too (iOS needs the job-level Customer_Artifacts.zip). Co-authored-by: Cursor <cursoragent@cursor.com>
Parse brittle's TAP output (ok N / not ok N lines) from logcat_full.txt (Android) and bare_console.log (iOS) to surface every individual test() case with its status (passed/failed/skipped) and timing per device. Produces test-case-details.json in the console-logs artifact with both runner-function-level and per-test() detail. The GitHub Step Summary gets a runner table + a collapsible per-test-case table so reviewers can see at a glance whether a newly added test() actually ran. Co-authored-by: Cursor <cursoragent@cursor.com>
…og-type suffix) Co-authored-by: Cursor <cursoragent@cursor.com>
…se summary Two fixes for the test-case-details step summary: 1. Normalize dynamic values in test names so the same logical test merges across devices — e.g. 'CacheTokens (53) > 0' and 'CacheTokens (55) > 0' both become 'CacheTokens (N) > 0'. This was inflating Android's count (1240) vs iOS (633) because each device produced slightly different token counts in assertion names. 2. Escape markdown-special characters (|, <, >, backtick) in test names before writing to the step summary table, so test descriptions containing these characters don't break the table layout. Co-authored-by: Cursor <cursoragent@cursor.com>
…ativeJS echo)
Android logcat echoes TAP lines twice: once from bare, once from
ReactNativeJS wrapped in single quotes ('ok 1 - name'). The trailing
quote made every test appear as a duplicate (e.g. 'All models available'
vs 'All models available' with trailing quote), inflating Android OCR
from 790 to 1461 test cases. Strip trailing quotes before deduplication.
Co-authored-by: Cursor <cursoragent@cursor.com>
… details
Two fixes for correct test case counts across platforms:
1. Deduplicate TAP results by normalized name (not num+name) so perf
iterations that reuse the same test name at different TAP numbers
don't inflate the count. NMT was 364/546 → now 182=182.
2. Truncate test names at assertion detail markers ('. Found ', ': "',
', got ') so variable model output embedded in assertion messages
doesn't create per-device duplicates. LLM elephant tests with
'Found keywords: elephant' in different phrasing now merge.
Verified across all addons with real data:
OCR: Android=669, iOS=669 (exact match)
NMT: Android=182, iOS=182 (exact match)
LLM: Android=622, iOS=608 (16 A-only = bitnet/Android-only tests,
2 I-only = Metal/iOS-only tests — all genuinely platform-specific)
Co-authored-by: Cursor <cursoragent@cursor.com>
olyasir
previously approved these changes
Jun 12, 2026
DmitryMalishev
requested changes
Jun 12, 2026
DmitryMalishev
left a comment
DmitryMalishev
left a comment
Contributor
There was a problem hiding this comment.
Requesting changes on the two critical points below. The CI plumbing in this PR (artifact slimming, shard rebalance, TAP visibility) is directionally good, but these two change what the perf pipeline measures and covers as silent side effects of a time-budget fix.
…ert pre-warmup Two fixes per Dima's review: 1. Aurora skip is now Android-scoped using the explicit QVAC_PERF_ONLY flag (already plumbed to the device via the testspec config) instead of proxying off PERF_RUNS. iOS + desktop always run aurora. The benchmark (QVAC_PERF_ONLY=true) runs all 3 images on all platforms, even with runs=1. 2. Revert the Android pre-warmup extension back to iOS-only. The change was silently altering what Android perf numbers measure (cold first- run vs warm steady-state) and doesn't fix the crash it targeted (the large buffer allocation still happens on the first real-image pass). Restores historical comparability of Android perf data. Co-authored-by: Cursor <cursoragent@cursor.com>
DmitryMalishev
approved these changes
Jun 12, 2026
DmitryMalishev
previously approved these changes
Jun 12, 2026
… func shards cacheStateMachine takes 30m on Pixel (hit the per-test Mocha timeout in funcShardB). Move it to a solo group (like toolCalling and gemma) and rebalance the remaining functional tests into 3 shards (~25-29m each on Pixel worst-case). Total Android groups: 8 (3 solo + 3 func + 2 vlmPerf). Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
DmitryMalishev
approved these changes
Jun 15, 2026
olyasir
approved these changes
Jun 15, 2026
Contributor
Author
|
/review |
Contributor
🧪 C++ Test Coverage ReportCoverage: 📊 Detailed Coverage |
Contributor
Mobile integration tests — @qvac/decoder-audio (Android)Result: passed
|
Contributor
Mobile integration tests — @qvac/decoder-audio (iOS)Result: passed
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What problem does this PR solve?
How does it solve it?
Android shard rebalance (data-driven)
VLM perf: skip aurora on normal PRs
QVAC_PERF_RUNS <= 1) so Android stays under ~1h. The benchmark (QVAC_PERF_RUNS > 1) still runs all 3 images. On-PR covers elephant + fruit-plate.Slimmed Device Farm artifact downloads
Test-list visibility for all addons
integration.auto.cjsrunners for non-grouped addons (NMT, TTS, etc.) so the monitor's "Run → tests" legend shows what runs, not just an empty "default →".Per-test() case results (TAP parsing)
ok N - name # time = Xms) from logcat_full.txt (Android) and bare_console.log (iOS) to surface every individual test() case with status + timing per device.test-case-details.jsonin the console-logs artifact + a collapsible summary table in the GitHub Step Summary.CacheTokens (53)→(N)) so the same logical test merges across devices.|,<,>) so test descriptions don't break the summary table.Other
How was it tested?