Add F1 score, precision, and recall metrics as MultilabelSegmentation default metrics by FrenchKrab · Pull Request #1336 · pyannote/pyannote-audio

FrenchKrab · 2023-04-20T08:00:00Z

Currently the MultilabelSegmentation task has no default metric, this PR adds these 3 (torchmetrics) metrics as default.

codecov · 2023-04-20T08:05:55Z

Codecov Report

Patch coverage: 48.64% and project coverage change: +0.15 🎉

Comparison is base (bbe1395) 32.98% compared to head (3107105) 33.13%.

Additional details and impacted files

@@             Coverage Diff             @@
##           develop    #1336      +/-   ##
===========================================
+ Coverage    32.98%   33.13%   +0.15%     
===========================================
  Files           64       65       +1     
  Lines         4072     4134      +62     
===========================================
+ Hits          1343     1370      +27     
- Misses        2729     2764      +35

Impacted Files	Coverage Δ
pyannote/audio/cli/train.py	`0.00% <ø> (ø)`
pyannote/audio/pipelines/speaker_diarization.py	`0.00% <0.00%> (ø)`
pyannote/audio/pipelines/utils/oracle.py	`0.00% <0.00%> (ø)`
pyannote/audio/tasks/embedding/mixins.py	`26.51% <0.00%> (-0.41%)`	⬇️
.../tasks/segmentation/overlapped_speech_detection.py	`42.50% <0.00%> (ø)`
...dio/tasks/segmentation/voice_activity_detection.py	`40.54% <0.00%> (ø)`
pyannote/audio/utils/preview.py	`0.00% <0.00%> (ø)`
pyannote/audio/core/pipeline.py	`21.85% <14.28%> (-0.22%)`	⬇️
...te/audio/tasks/segmentation/speaker_diarization.py	`47.68% <20.00%> (+0.01%)`	⬆️
pyannote/audio/core/task.py	`81.48% <50.00%> (-0.40%)`	⬇️
... and 7 more

... and 1 file with indirect coverage changes

☔ View full report in Codecov by Sentry.
📢 Do you have feedback about the report comment? Let us know in this issue.

FrenchKrab · 2023-04-20T09:20:37Z

Before merging I think i need to reshape what's passed to the metric in the validation, so that it's compatible with more metrics (currently i think flat tensors are passed).

hbredin · 2023-04-20T13:29:10Z

        mask: torch.Tensor = y_true != -1
-        y_pred = y_pred[mask]
-        y_true = y_true[mask]
+        y_pred = y_pred[mask].reshape(shape)


This will break as soon as mask contains at least one -1 because the overall size of y_pred will then be smaller than shape.

Still support "global" metrics, BUT they have to be of a binary type.

FrenchKrab · 2023-04-20T15:14:22Z

Writing down some things before i forget them:

I added "per class" metrics, which probably need a better name since it's the metrics computed for each individual class (but the same metrics are used for each class)
I kept the "global" metric (computed for all outputs on all classes), BUT because we support missing targets, and torchmetrics does not (to my knowledge), without any assumptions, we can only treat the multilabel as a binary problem and are limited to these metrics.
We can still let the user set the global metric as "multilabel", but they have to make sure there arent missing targets.

(we can discuss this tomorrow !)

This reverts commit a890ef6.

(uses ignore_index for that, which ignores all targets of that index for computations, in pyannote's case : -1)

…annote-audio into multilabel_default_metrics

hbredin · 2023-05-10T20:36:16Z

Is this ready for review?

This reverts commit 5f51b50.

hbredin · 2023-09-15T15:24:52Z

Going over your PRs :) Is this mergeable?

FrenchKrab · 2023-09-18T07:51:11Z

I should test it before, but i'm done with the implementation (i dont know if it's ok for you though :) ).

In the end I didn't find how to do away with "metric_classwise", there's a torchmetric classwise wrapper but it doesn't do what we want.
Torchmetrics's wrapper is for metrics that return a multidim tensor, but for example the multilabel variant of Calibration Error doesn't exist, while the "binary" version exists for probably every metric.

stale · 2024-03-19T00:26:42Z