ModernSearch and NonSpecificEnzymeSearch score 79.43% on mutation testing: 131 unpinned behaviours, listed

[trishorts]

Mutation testing over the two index-based search engines. Follows the pattern of mzLib #1236, which did the same for Deconvolution: the score is not the point, the enumerated list is.

Measured on master d93e20b4 with dotnet-stryker 4.16.0, scoped with -m "**/ModernSearchEngine.cs" -m "**/NonSpecificEnzymeSearchEngine.cs". 1h16m.

549 mutants tested
  Killed    479
  Survived   43
  Timeout    27
  NoCoverage 88     (not reached by any test in the suite)

final mutation score: 79.43 %

The score counts NoCoverage against, which is right — (479 + 27) / (479 + 27 + 43 + 88). 131 behaviours are unpinned: 43 that the tests execute and do not detect, and 88 they never reach.

The two engines are not in the same condition:

	Survived	NoCoverage	Timeout
NonSpecificEnzymeSearchEngine.cs	36	29	20
ModernSearchEngine.cs	7	59	7

Different problems, different fixes. NonSpecificEnzymeSearchEngine is exercised and not asserted. ModernSearchEngine is largely fine where it is tested, and has whole regions nothing enters.

The survivors that would be real bugs

Not a full list — the ones where the mutant is a defect a user would feel, rather than an equivalent mutant or a boundary nobody hits.

Comparisons that decide which match wins, inverted with nothing noticing. NonSpecificEnzymeSearchEngine.cs:

line 512:  if (majorPsm.Score > minorPsm.Score)              ->  <   , >=  , !(...)     all survive
line 530:  if (majorPsm.PsmFdrInfo.QValue > minorPsm...)     ->  <   , >=  , !(...)     all survive
line 504:  ...OrderByDescending(x => x.Score)                ->  OrderBy(...)           survives

Three mutants on line 512 and three on 530 all survive, and the ranking on 504 can be reversed. Whatever these are choosing between, no test distinguishes the better answer from the worse one.

The scoring table can be decremented, or not written at all.

line 189:  scoringTable[bin[pep]]++;   ->  scoringTable[bin[pep]]--   survives
                                       ->  ;   (statement removed)    survives

Bounds and loop limits.

line  99:  for (... bin <= highestBin; ...)                 ->  bin < highestBin              survives
line 297:  for (... fragmentBin <= compFragmentCeilingMass) ->  fragmentBin < ...             survives
line 284:  if (compFragmentCeilingMass >= FragmentIndex.Length) ->  >                         survives
line 292:  if (compFragmentFloorMass < 0)                   ->  <= 0                          survives
line 204:  if (fragmentBin >= 0)                            ->  > 0                           survives

ModernSearchEngine.cs has the same shape in two places:

line 266:  if (r - l < 2)                                   ->  r - l <= 2                    survives
line 352:  bool meetsScoreCutoff = thisScore >= ScoreCutoff; ->  thisScore > ScoreCutoff      survives

That last one is the score cutoff itself: a peptide scoring exactly at the threshold can be included or excluded and no test can tell.

The fragment-bin arithmetic. Present in both engines, and the multiplication can become a division:

NonSpecific 181 / Modern 119:
  (int)(Math.Round(masses[i].ToMass(1) / 1.0005079) * 1.0005079 * FragmentBinsPerDalton)
     ->  ... / FragmentBinsPerDalton      survives
     ->  masses[i].ToMass(1) * 1.0005079  survives

NonSpecific 202 / Modern 140:
  (int)Math.Round((scan.PrecursorMass + protonMassShift - masses[i]) / 1.0005079)
     ->  ... + masses[i]                  survives
     ->  scan.PrecursorMass - protonMassShift   survives

A peptide's full sequence is never asserted. NonSpecificEnzymeSearchEngine.cs:431 constructs a PeptideWithSetModifications, and replacing one of its strings with "Stryker was here!" survives.

The 88 nobody reaches

Clustered rather than scattered, which suggests whole paths rather than odd branches:

ModernSearchEngine.cs             33 lines in 8 regions
    lines 482-510  (10 mutants)
    lines 559-586  (12 mutants)
    lines 437-444, 537-541, 150-156, 61, 223, 593

NonSpecificEnzymeSearchEngine.cs  19 lines in 10 regions
    lines 286-293  (4)
    lines 248-251, 518-522  (3 each)
    lines 211-213, 553-556  (2 each)
    lines 56, 71, 363, 418, 531

The two large ModernSearchEngine regions — 482-510 and 559-586 — are 22 of the 59 between them and would be the place to start.

Notes

• 27 timeouts count as killed in the score, and mostly are: a mutated loop bound that never terminates is detected, just expensively. They are not a separate problem to fix.
• 1,465 mutants were dropped as compile errors before testing. That is Stryker's safe mode, not a signal about the code.
• The raw report is reproducible with the command above; happy to attach the JSON if useful.
• No claim here that all 131 are worth pinning. Some will be equivalent mutants. The comparison and bounds ones above are not.