D-VQA
We provide the PyTorch implementation for Debiased Visual Question Answering from Feature and Sample Perspectives (NeurIPS 2021).
Dependencies
- Python 3.6
- PyTorch 1.1.0
- dependencies in requirements.txt
- We train and evaluate all of the models based on one TITAN Xp GPU
Getting Started
Installation
-
Clone this repository:
git clone https://github.com/Zhiquan-Wen/D-VQA.git cd D-VQA -
Install PyTorch and other dependencies:
pip install -r requirements.txt
Download and preprocess the data
cd data
bash download.sh
python preprocess_features.py --input_tsv_folder xxx.tsv --output_h5 xxx.h5
python feature_preprocess.py --input_h5 xxx.h5 --output_path trainval
python create_dictionary.py --dataroot vqacp2/
python preprocess_text.py --dataroot vqacp2/ --version v2
cd ..
Training
- Train our model
CUDA_VISIBLE_DEVICES=0 python main.py --dataroot data/vqacp2/ --img_root data/coco/trainval_features --output saved_models_cp2/ --self_loss_weight 3 --self_loss_q 0.7
- Train the model with 80% of the original training set
CUDA_VISIBLE_DEVICES=0 python main.py --dataroot data/vqacp2/ --img_root data/coco/trainval_features --output saved_models_cp2/ --self_loss_weight 3 --self_loss_q 0.7 --ratio 0.8
Evaluation
- A json file of results from the test set can be produced with:
CUDA_VISIBLE_DEVICES=0 python test.py --dataroot data/vqacp2/ --img_root data/coco/trainval_features --checkpoint_path saved_models_cp2/best_model.pth --output saved_models_cp2/result/
- Compute detailed accuracy for each answer type:
python comput_score.py --input saved_models_cp2/result/XX.json --dataroot data/vqacp2/
Pretrained model
A well-trained model can be found here. The test results file produced by it can be found here and its performance is as follows:
Overall score: 61.91
Yes/No: 88.93 Num: 52.32 other: 50.39
Reference
If you found this code is useful, please cite the following paper:
@inproceedings{D-VQA,
title = {Debiased Visual Question Answering from Feature and Sample Perspectives},
author = {Zhiquan Wen,
Guanghui Xu,
Mingkui Tan,
Qingyao Wu,
Qi Wu},
booktitle = {NeurIPS},
year = {2021}
}
Acknowledgements
This repository contains code modified from SSL-VQA, thank you very much!
Besides, we thank Yaofo Chen for providing MIO library to accelerate the data loading.
下面是我的代码
def plot_rect(image, boxes):
img = Image.fromarray(np.uint8(image))
draw = ImageDraw.Draw(img)
for k in range(2):
box = boxes[k,:]
print(box)
drawrect(draw, box, outline='green', width=3)
img = np.asarray(img)
return img
def drawrect(drawcontext, xy, outline=None, width=0):
x1, y1, x2, y2 = xy
points = (x1, y1), (x2, y1), (x2, y2), (x1, y2), (x1, y1)
drawcontext.line(points, fill=outline, width=width)
6 Jul 18, 2022
2 Dec 30, 2021
1.2k Dec 28, 2022
6 Nov 22, 2022
20 Jun 09, 2022
9 May 10, 2022
9 Dec 23, 2022
873 Jan 09, 2023
14 Dec 09, 2022
3 May 07, 2022
155 Jan 04, 2023
62 Dec 21, 2022
25 Jan 02, 2023
498 Dec 24, 2022
427 Dec 13, 2022
164 Nov 13, 2022
56 Dec 16, 2022
5 Dec 06, 2022
1 Apr 25, 2022
51 Nov 29, 2022