Clarification Request for SRE Scenarios

[yihangyao]

Dear authors,

Thank you for sharing this impressive work! I’m currently exploring the SRE scenarios in ITBench, and would appreciate a bit of clarification on the tasks planned for future releases:

(1) In the sre folder of the repository, the open‑sourced incident scenarios are numbered 1, 3, 23, 26, 27, and 102. Does this imply that future versions may eventually include all scenarios up to at least 102 (at least 102 different scenarios)?

(2) Table 12 of the paper lists 21 unique SRE scenarios, while Table 4 notes that the full SRE set comprises 42 scenarios, 21 with traces and 21 without. Should I interpret the experimental results as being based on the 21 scenarios from Table 12, evaluated in both “with‑traces” and “without‑traces” modes?

(3) In Figure 14, do the scenario indices (e.g., 1 through 23) align directly with the scenario names: for instance, does index 1 correspond to CacheFailure, index 2 to HighCPU, and index 23 to MemoryResourceLimit?

Thank you very much for your time and for making these resources available! I look forward to your guidance!

Best wishes,

Yihang

[Red-GV]

Hello @yihangyao , sorry about the delay to getting to your question. We thank and appreciate you for your interest and the question.

(1) The incident number functions as more of an identification number versus an actual ordering. The SRE scenarios - currently - in this repository are samples from the team to show how to craft a scenario so that others may contribute theirs by way of a pull request. That being said, for the SRE use-case, the plan is to open source most (if not all) of the fault mechanisms which are leveraged to create a scenario. We like to think about a scenario as a recipe with the following ingredients:

• Environment (open sourced)
• Fault mechanism itself or any other applicable mechanisms to trigger the incident or a possible incident (mostly open sourced). If there are any fault mechanisms you would like to see as a part of the ITBench, please create an issue here or a pull request. We expect about another 20 fault mechanisms (thereby unlocking several scenarios) to be contributed over the next 45 days. The roadmap with them is to be made available soon.
• When and where the fault is injected (open sourced for limited scenarios until the scenario generator-runner being built reaches a certain level of maturity)

(2) Yes, that is correct. Table 4 has the combined average of the agent's performance on the same 21 scenarios with and without access to traces

(3) Figure 14 represents all the scenarios your agent would be evaluated against when you make a submission to the ITBench. For the scenarios that are be open sourced, please head here for additional information.

Please feel free to reach out if you have any additional questions.