Sable paper figure question

[Chulabhaya]

Hi there! I was wondering how you guys generated this figure in the Sable paper?

I'd like to generate a similar "aggregated evaluator win rate" plot for SMAX where it's essentially just an average win rate across all of the SMAX tasks (instead of just a single task). Did you guys use Marl-eval to generate your aggregated plots? I can't seem to find a similar option and I'd like to be as similar as possible (hence why I haven't just put my own function together yet).

[OmaymaMahjoub]

Hello @Chulabhaya, the average is computed across all tasks per environment using the aggregation procedure described in https://arxiv.org/pdf/2209.10485, which is the approach used in MARL-Eval. But indeed, within MARL-Eval, the plots are generated per task, to produce aggregated plots over all tasks within an environment, we modified the create_matrices_for_rliable function [code] to support that.

We haven’t yet shared this modified version in the repo, but we are planning to do that soon.

[Chulabhaya]

Hello @Chulabhaya, the average is computed across all tasks per environment using the aggregation procedure described in https://arxiv.org/pdf/2209.10485, which is the approach used in MARL-Eval. But indeed, within MARL-Eval, the plots are generated per task, to produce aggregated plots over all tasks within an environment, we modified the create_matrices_for_rliable function [code] to support that.

We haven’t yet shared this modified version in the repo, but we are planning to do that soon.

Thanks for the reply @OmaymaMahjoub ! Is there any chance I could get access to that modified version of the create_matrices_for_rliable function in the near future? I'd love to make sure I'm using the same metric style for my own work.

[OmaymaMahjoub]

Hello @Chulabhaya sorry for the delays!
For now, the modified create_matrices_for_rliable is quite tightly coupled with our internal evaluation and plotting setup, so it’s not in a clean state to share yet, I’d prefer not to release something that’s overly nested or hard to reuse.
That said, I’ll keep this issue open and try to allocate some time to clean up the code and share it before the end of the month 🙂