List similar tasks from previous pushes to identify regressions#1151
List similar tasks from previous pushes to identify regressions#1151vrigal wants to merge 12 commits intomozilla:masterfrom
Conversation
|
Uses HGMO to detect similar tasks on previous build. It is slower than with treeherder, so limited to 20 pushes (< 5 minutes of runtime). This changes a bit the behavior specified in #1145 (comment), as we will never get more than 20 pushes The changes in the regression command are only intended to simplify development and will be removed: |
| @property | ||
| def is_potential_regression(self): | ||
| # Check if the job is either tier 1 or 2 | ||
| if self.tier not in (1, 2): | ||
| return False | ||
|
|
||
| # Check if job result is busted or exception | ||
| if self.result not in ("busted", "exception"): | ||
| return False | ||
|
|
||
| # Check if extra.treeherder.jobKind is build | ||
| if self.job_kind != "build": | ||
| return False | ||
|
|
||
| return True |
There was a problem hiding this comment.
Same as #1148 (comment), we should move this in mozci/push.py
|
@vrigal will rebase his work on master instead of my own branch so he can iterate on your comments |
vrigal
force-pushed
the
regression-similar-jobs
branch
from
38facea to
57627b7
Compare
|
I removed most of the code. @marco-c Using
It does not count the actual push (hence the -1). But also excludes some tasks. I got 393 tasks in total, 29 tasks are returned by the actual I still have to merge build specific things with existing
|
|
The l10n- tasks never actually run, so it's OK that they are missing (the builds they depend on failed, so they were never scheduled to run). |
vrigal
force-pushed
the
regression-similar-jobs
branch
from
57627b7 to
5046214
Compare
|
I tried to implement a custom fetching of possible regression tasks (by label only) in
The above rules are from #1145. Maybe we could for now apply a simpler rule, like "Retry the build if previous_occurrences == 0"? Then look for the state of the requested build in another PR? |
vrigal
force-pushed
the
regression-similar-jobs
branch
from
5046214 to
41a2a6e
Compare
| return { | ||
| task.label: (0.0, summary) | ||
| for task in self.tasks | ||
| if self._check_build_task_regression(task) | ||
| and (summary := self.label_summaries.get(task.label)) | ||
| } |
There was a problem hiding this comment.
Question: if we don't add this, doesn't get_candidate_regressions already returns builds?
There was a problem hiding this comment.
Yes, by default get_candidate_regressions will return 61 results that include the 7 returned by _should_retrigger_build.
|
|
||
| return find("label") or find("config_group") | ||
|
|
||
| def _check_build_task_regression(self, task: Task): |
There was a problem hiding this comment.
| def _check_build_task_regression(self, task: Task): | |
| def _should_retrigger_build(self, task: Task): |
There was a problem hiding this comment.
I tried to separate the logic for "task is a build failure" (so I renamed this to _is_build_failure(self, task)) and rules for retriggering (in the regression command).
|
Here is the output of the actual command: Actually it only looks at pushes that have no child (so I have to cheat a bit to test with the above revision). I'm closing #1148 as most things happens here now. |
|
@marco-c I implemented most of the points we discussed this afternoon. I tried to play with MAX_DEPTH, but setting a value of 0 also prevents from going backwards (counting for previous occurrences) as the same setting is used for both directions. I'm not sure it is worth changing this anyway. I actually have the same output than the above (in ~2 minutes). |
|
OK, makes sense. We would have to split that into two values (backward_max_depth and forward_max_depth), but maybe it is not worth it, we can just filter by count == 0. |
|
@marco-c @Archaeopteryx Could you use the latest version of this command If that looks 🆗 @vrigal can integrate the build filter in |
|
78368f6e8144dff55f60ec291d25b0dd6d3c5985 backed out by cf16ff032ba42816626237cf3a1f6bb5ab63208e for Bp-nu failures - mozci detects no regression |
I did some investigating. It does not seem to be related to filtering by new regressions (previous occurrences == 0). We still have to figure out why nothing is returned for revision |
https://github.com/mozilla/mozci/blob/master/mozci/push.py#L1002 It seems that revision |
vrigal
force-pushed
the
regression-similar-jobs
branch
from
1fa78c9 to
0f34f76
Compare
|
As suggested by @marco-c, setting However, it will not work if the push is older than 1 day because of the |
vrigal
force-pushed
the
regression-similar-jobs
branch
from
0f34f76 to
9fa01b0
Compare
|
Closing as this has been merged with #1157 |
4 participants
References #1145
Based on #1148