PyTorch implementation of "Debiased Visual Question Answering from Feature and Sample Perspectives" (NeurIPS 2021)

Last update: Dec 22, 2022

Related tags

Overview

D-VQA

We provide the PyTorch implementation for Debiased Visual Question Answering from Feature and Sample Perspectives (NeurIPS 2021).

Dependencies

Python 3.6
PyTorch 1.1.0
dependencies in requirements.txt
We train and evaluate all of the models based on one TITAN Xp GPU

Getting Started

Installation

Clone this repository:

 git clone https://github.com/Zhiquan-Wen/D-VQA.git
 cd D-VQA

Install PyTorch and other dependencies:
```
 pip install -r requirements.txt
```

Download and preprocess the data

cd data 
bash download.sh
python preprocess_features.py --input_tsv_folder xxx.tsv --output_h5 xxx.h5
python feature_preprocess.py --input_h5 xxx.h5 --output_path trainval 
python create_dictionary.py --dataroot vqacp2/
python preprocess_text.py --dataroot vqacp2/ --version v2
cd ..

Training

Train our model

CUDA_VISIBLE_DEVICES=0 python main.py --dataroot data/vqacp2/ --img_root data/coco/trainval_features --output saved_models_cp2/ --self_loss_weight 3 --self_loss_q 0.7

Train the model with 80% of the original training set

CUDA_VISIBLE_DEVICES=0 python main.py --dataroot data/vqacp2/ --img_root data/coco/trainval_features --output saved_models_cp2/ --self_loss_weight 3 --self_loss_q 0.7 --ratio 0.8

Evaluation

A json file of results from the test set can be produced with:

CUDA_VISIBLE_DEVICES=0 python test.py --dataroot data/vqacp2/ --img_root data/coco/trainval_features --checkpoint_path saved_models_cp2/best_model.pth --output saved_models_cp2/result/

Compute detailed accuracy for each answer type:

python comput_score.py --input saved_models_cp2/result/XX.json --dataroot data/vqacp2/

Pretrained model

A well-trained model can be found here. The test results file produced by it can be found here and its performance is as follows:

Overall score: 61.91
Yes/No: 88.93 Num: 52.32 other: 50.39

Reference

If you found this code is useful, please cite the following paper:

@inproceedings{D-VQA,
  title     = {Debiased Visual Question Answering from Feature and Sample Perspectives},
  author    = {Zhiquan Wen, 
               Guanghui Xu, 
               Mingkui Tan, 
               Qingyao Wu, 
               Qi Wu},
  booktitle = {NeurIPS},
  year = {2021}
}

Acknowledgements

This repository contains code modified from SSL-VQA, thank you very much!

Besides, we thank Yaofo Chen for providing MIO library to accelerate the data loading.

Comments

Questions about the code
Thank you very much for providing the code, but I still have two questions that I did not understand well.

A module, BDM, is used to capture negative bias, but this module only includes a multi-layer perceptron. Then how to ensure the features captured by this multi-layer perceptron are negative bias?

On the left of Figure 2 of the paper, there are no backward gradient of the question-to-answer and the vision-to-answer branches. Where did it reflect in the code?
opened by darwann 4
CVE-2007-4559 Patch

Patching CVE-2007-4559

Hi, we are security researchers from the Advanced Research Center at Trellix. We have began a campaign to patch a widespread bug named CVE-2007-4559. CVE-2007-4559 is a 15 year old bug in the Python tarfile package. By using extract() or extractall() on a tarfile object without sanitizing input, a maliciously crafted .tar file could perform a directory path traversal attack. We found at least one unsantized extractall() in your codebase and are providing a patch for you via pull request. The patch essentially checks to see if all tarfile members will be extracted safely and throws an exception otherwise. We encourage you to use this patch or your own solution to secure against CVE-2007-4559. Further technical information about the vulnerability can be found in this blog.

If you have further questions you may contact us through this projects lead researcher Kasimir Schulz.

opened by TrellixVulnTeam 0
LXMERT numbers

Hi, I wish to reproduce the LXMERT(LXMERT without D-VQA) numbers reported in the paper. It would be helpful if you could provide me with a way to do this using your code. I tried using the original LXMERT code, but I am not able to get the numbers reported in your paper on the VQA-CP2 dataset.

opened by Vaidehi99 0
Download trainval_36.zip error

Hi, thank you for your work on this.

I keep getting a download error when downloading the trainval_36.zip file. Is there another link I can use to download this?

Thanks in advance!

opened by chojw 0
关于box和image的对齐问题

您好，我将box的注释解开后，重新生成特征，然后将其绘制出来，但是明显感觉有偏差，不知道您是否可以提供一份绘图的代码。下面是我的代码 def plot_rect(image, boxes): img = Image.fromarray(np.uint8(image)) draw = ImageDraw.Draw(img) for k in range(2): box = boxes[k,:] print(box) drawrect(draw, box, outline='green', width=3) img = np.asarray(img) return img def drawrect(drawcontext, xy, outline=None, width=0): x1, y1, x2, y2 = xy points = (x1, y1), (x2, y1), (x2, y2), (x1, y2), (x1, y1) drawcontext.line(points, fill=outline, width=width)

opened by LemonQC 0

Releases(Results_LXMERT)

Results_LXMERT(Oct 26, 2021)

Source code(tar.gz)
Source code(zip)
test_best_epoch0.json(9.66 MB)
Models_LXMERT(Oct 26, 2021)

Source code(tar.gz)
Source code(zip)
BEST.pth(842.49 MB)
LXMERT_pretrained_model(Oct 25, 2021)

Source code(tar.gz)
Source code(zip)
Epoch20_LXRT.pth(870.06 MB)
Results(Oct 10, 2021)

Source code(tar.gz)
Source code(zip)
61.91_results.json(9.65 MB)
Models(Oct 10, 2021)

Source code(tar.gz)
Source code(zip)
61.91.pth(620.44 MB)

Owner

Zhiquan Wen

GitHub Repository

Locally Enhanced Self-Attention: Rethinking Self-Attention as Local and Context Terms

LESA Introduction This repository contains the official implementation of Locally Enhanced Self-Attention: Rethinking Self-Attention as Local and Cont

20 Dec 31, 2021

This is RFA-Toolbox, a simple and easy-to-use library that allows you to optimize your neural network architectures using receptive field analysis (RFA) and create graph visualizations of your architecture.

ReceptiveFieldAnalysisToolbox This is RFA-Toolbox, a simple and easy-to-use library that allows you to optimize your neural network architectures usin

84 Nov 23, 2022

Code for NeurIPS 2020 article "Contrastive learning of global and local features for medical image segmentation with limited annotations"

Contrastive learning of global and local features for medical image segmentation with limited annotations The code is for the article "Contrastive lea

152 Dec 22, 2022

End-to-end image segmentation kit based on PaddlePaddle.

English | 简体中文 PaddleSeg PaddleSeg has released the new version including the following features: Our team won the 6.2k Jan 02, 2023

A curated list of automated deep learning (including neural architecture search and hyper-parameter optimization) resources.

Awesome AutoDL A curated list of automated deep learning related resources. Inspired by awesome-deep-vision, awesome-adversarial-machine-learning, awe

2k Dec 30, 2022

[ArXiv 2021] Data-Efficient Instance Generation from Instance Discrimination

InsGen - Data-Efficient Instance Generation from Instance Discrimination Data-Efficient Instance Generation from Instance Discrimination Ceyuan Yang,

93 Dec 25, 2022

The Empirical Investigation of Representation Learning for Imitation (EIRLI)

31 Nov 06, 2022

Official implementation of the paper 'Details or Artifacts: A Locally Discriminative Learning Approach to Realistic Image Super-Resolution' in CVPR 2022

LDL Paper | Supplementary Material Details or Artifacts: A Locally Discriminative Learning Approach to Realistic Image Super-Resolution Jie Liang*, Hu

150 Dec 26, 2022

Make your first PR. A beginner friendly repository made specifically for open source beginners. Add any program under any language (it can be anything from a simple program to a complex data structure algorithm). Happy coding...

Hacktober Fest 2021 Upload Different Types of Programs in any Language Use this project to make your first contribution to an open source project on G

40 Oct 11, 2022

PyTorch implementation of "Debiased Visual Question Answering from Feature and Sample Perspectives" (NeurIPS 2021)

Related tags

Overview

D-VQA

Dependencies

Getting Started

Installation

Download and preprocess the data

Training

Evaluation

Pretrained model

Reference

Acknowledgements

Comments

Questions about the code

CVE-2007-4559 Patch

Patching CVE-2007-4559

LXMERT numbers

Download trainval_36.zip error

关于box和image的对齐问题

Releases(Results_LXMERT)

Results_LXMERT(Oct 26, 2021)

Models_LXMERT(Oct 26, 2021)

LXMERT_pretrained_model(Oct 25, 2021)

Results(Oct 10, 2021)

Models(Oct 10, 2021)

Owner

Zhiquan Wen

Locally Enhanced Self-Attention: Rethinking Self-Attention as Local and Context Terms

This is RFA-Toolbox, a simple and easy-to-use library that allows you to optimize your neural network architectures using receptive field analysis (RFA) and create graph visualizations of your architecture.

Code for NeurIPS 2020 article "Contrastive learning of global and local features for medical image segmentation with limited annotations"

End-to-end image segmentation kit based on PaddlePaddle.

A curated list of automated deep learning (including neural architecture search and hyper-parameter optimization) resources.

[ArXiv 2021] Data-Efficient Instance Generation from Instance Discrimination

The Empirical Investigation of Representation Learning for Imitation (EIRLI)

Official implementation of the paper 'Details or Artifacts: A Locally Discriminative Learning Approach to Realistic Image Super-Resolution' in CVPR 2022

A motion detection system with RaspberryPi, OpenCV, Python

Anomaly Transformer: Time Series Anomaly Detection with Association Discrepancy" (ICLR 2022 Spotlight)

🎁 3,000,000+ Unsplash images made available for research and machine learning

The official implementation of Autoregressive Image Generation using Residual Quantization (CVPR '22)

Anti-Adversarially Manipulated Attributions for Weakly and Semi-Supervised Semantic Segmentation (CVPR 2021)

ImVoxelNet: Image to Voxels Projection for Monocular and Multi-View General-Purpose 3D Object Detection

Training and Evaluation Code for Neural Volumes

The repository for the paper "When Do You Need Billions of Words of Pretraining Data?"

Algorithm to texture 3D reconstructions from multi-view stereo images

Graph Transformer Architecture. Source code for

ResNEsts and DenseNEsts: Block-based DNN Models with Improved Representation Guarantees

Make your first PR. A beginner friendly repository made specifically for open source beginners. Add any program under any language (it can be anything from a simple program to a complex data structure algorithm). Happy coding...