- Covers repository level understanding
- QA pairs include both code snippets & natural language explanations
- Supports multiple programming languages (Python, Java, C etc)
- Evaluates both retrieval-based and generative models.
Structure of RepoQABench
Field Description Repository Name Name of the Repo Programming Language python, java, c, etc Question dev-level repo-wide query Golden Answer(Text & Code) natural lang explanation + extracted relevant code snippet Supporting COntext retrieved files, issue discussions License mit, apache, bsd, etc