FAME

An implementation for Faithful Question Answering with Monte-Carlo Planning.

Requirements

• Python 3.8
• Ubuntu 22.04
• Python Packages

conda create -n fame python=3.8
conda activate fame
pip install -r requirements.txt

Data

Download Data folder. The Data folder includes the EntailmentBank, EntailmentBankQA, training data for the controller, and training data for the verifier. See Data/Readme.md for details.

Evaluation

Download our trained models for direct reproduction, including Controller, Entailment Module, Retriever, and Verifier. Unzip the files and place them in exp/ folder. Run the following commands to reproduce the results.

EntailmentBankQA

sh scripts/eval_scripts/reason_EBQA.sh

The result will be saved in the save_dir.

EntailmentBank Task3

sh scripts/eval_scripts/reason_EB.sh

The result will be saved in the save_dir. Use the offical evaluation code of EntailmentBank to evaluate automatically.

Training

Entailment Module

Please refer to MetGen for the training of the single-step entialment module.

Controller

sh scripts/training_scripts/train_Controller_ddp.sh

Retriever

sh scripts/training_scripts/train_Retriever.sh

Verifier

sh scripts/training_scripts/train_StepScorer.sh

Citation

@inproceedings{hong2023fame,
  title={Faithful Question Answering with Monte-Carlo Planning},
  author={Ruixin Hong, Hongming Zhang, Hong Zhao, Dong Yu and Changshui Zhang},
  booktitle={The 61st Annual Meeting of the Association for Computational Linguistics ({ACL})},
  year={2023}
}