How to get logs(SWE-bench evaluation artifacts dump) when using sb-cli

[sxthunder]

Hi,

I want to submit results to SWE-bench following the contribution guidelines, logs are required when submitting a PR. However, when using SB-CLI for submission, it's not clear how to generate or access these logs.

Questions:

• Does SB-CLI automatically generate the required logs?
• Or do I need to run swe-bench evaluation locally first to get the logs?

Thanks!

[sxthunder]

@klieret Hi, we found some instances failed when running locally but get resolved when using sb-cli, is it possible to get the logs required for leaderboard submission by sb-cli?

[shubhamrgandhi]

tagging similar (unresolved) issue - #17

[klieret]

Hi, good point. Let me tag @carlosejimenez here. sb-cli produces the same logs internally, they're just put in a S3 bucket, but they're not public. I agree it would be helpful to be able to download them. I don't think this is part of the API we expose yet

[klieret]

For the purpose of debugging the discrepancies between sb-cli and swebench local, could you upload the submissions (i.e., patches) for instances where you see discrepancies?

[sxthunder]

For the purpose of debugging the discrepancies between sb-cli and swebench local, could you upload the submissions (i.e., patches) for instances where you see discrepancies?

Hi @klieret, @carlosejimenez

Thanks for the clarification. We've observed that local environment differences cause many edge cases, particularly with instances involving the psf__requests repo.

Given these environmental inconsistencies, it would be ideal to access logs directly from sb-cli's unified evaluation environment rather than relying on local runs. This would help avoid environment-related mismatches for future researchers and ensure all submissions are evaluated consistently.

Since sb-cli already generates these logs internally in S3, exposing them through the API would be very valuable for the community.

Thanks!