PyTorch implementation of "Debiased Visual Question Answering from Feature and Sample Perspectives" (NeurIPS 2021)

Last update: Dec 22, 2022

Related tags

Overview

D-VQA

We provide the PyTorch implementation for Debiased Visual Question Answering from Feature and Sample Perspectives (NeurIPS 2021).

Dependencies

Python 3.6
PyTorch 1.1.0
dependencies in requirements.txt
We train and evaluate all of the models based on one TITAN Xp GPU

Getting Started

Installation

Clone this repository:

 git clone https://github.com/Zhiquan-Wen/D-VQA.git
 cd D-VQA

Install PyTorch and other dependencies:
```
 pip install -r requirements.txt
```

Download and preprocess the data

cd data 
bash download.sh
python preprocess_features.py --input_tsv_folder xxx.tsv --output_h5 xxx.h5
python feature_preprocess.py --input_h5 xxx.h5 --output_path trainval 
python create_dictionary.py --dataroot vqacp2/
python preprocess_text.py --dataroot vqacp2/ --version v2
cd ..

Training

Train our model

CUDA_VISIBLE_DEVICES=0 python main.py --dataroot data/vqacp2/ --img_root data/coco/trainval_features --output saved_models_cp2/ --self_loss_weight 3 --self_loss_q 0.7

Train the model with 80% of the original training set

CUDA_VISIBLE_DEVICES=0 python main.py --dataroot data/vqacp2/ --img_root data/coco/trainval_features --output saved_models_cp2/ --self_loss_weight 3 --self_loss_q 0.7 --ratio 0.8

Evaluation

A json file of results from the test set can be produced with:

CUDA_VISIBLE_DEVICES=0 python test.py --dataroot data/vqacp2/ --img_root data/coco/trainval_features --checkpoint_path saved_models_cp2/best_model.pth --output saved_models_cp2/result/

Compute detailed accuracy for each answer type:

python comput_score.py --input saved_models_cp2/result/XX.json --dataroot data/vqacp2/

Pretrained model

A well-trained model can be found here. The test results file produced by it can be found here and its performance is as follows:

Overall score: 61.91
Yes/No: 88.93 Num: 52.32 other: 50.39

Reference

If you found this code is useful, please cite the following paper:

@inproceedings{D-VQA,
  title     = {Debiased Visual Question Answering from Feature and Sample Perspectives},
  author    = {Zhiquan Wen, 
               Guanghui Xu, 
               Mingkui Tan, 
               Qingyao Wu, 
               Qi Wu},
  booktitle = {NeurIPS},
  year = {2021}
}

Acknowledgements

This repository contains code modified from SSL-VQA, thank you very much!

Besides, we thank Yaofo Chen for providing MIO library to accelerate the data loading.

Comments

Questions about the code
Thank you very much for providing the code, but I still have two questions that I did not understand well.

A module, BDM, is used to capture negative bias, but this module only includes a multi-layer perceptron. Then how to ensure the features captured by this multi-layer perceptron are negative bias?

On the left of Figure 2 of the paper, there are no backward gradient of the question-to-answer and the vision-to-answer branches. Where did it reflect in the code?
opened by darwann 4
CVE-2007-4559 Patch

Patching CVE-2007-4559

Hi, we are security researchers from the Advanced Research Center at Trellix. We have began a campaign to patch a widespread bug named CVE-2007-4559. CVE-2007-4559 is a 15 year old bug in the Python tarfile package. By using extract() or extractall() on a tarfile object without sanitizing input, a maliciously crafted .tar file could perform a directory path traversal attack. We found at least one unsantized extractall() in your codebase and are providing a patch for you via pull request. The patch essentially checks to see if all tarfile members will be extracted safely and throws an exception otherwise. We encourage you to use this patch or your own solution to secure against CVE-2007-4559. Further technical information about the vulnerability can be found in this blog.

If you have further questions you may contact us through this projects lead researcher Kasimir Schulz.

opened by TrellixVulnTeam 0
LXMERT numbers

Hi, I wish to reproduce the LXMERT(LXMERT without D-VQA) numbers reported in the paper. It would be helpful if you could provide me with a way to do this using your code. I tried using the original LXMERT code, but I am not able to get the numbers reported in your paper on the VQA-CP2 dataset.

opened by Vaidehi99 0
Download trainval_36.zip error

Hi, thank you for your work on this.

I keep getting a download error when downloading the trainval_36.zip file. Is there another link I can use to download this?

Thanks in advance!

opened by chojw 0
关于box和image的对齐问题

您好，我将box的注释解开后，重新生成特征，然后将其绘制出来，但是明显感觉有偏差，不知道您是否可以提供一份绘图的代码。下面是我的代码 def plot_rect(image, boxes): img = Image.fromarray(np.uint8(image)) draw = ImageDraw.Draw(img) for k in range(2): box = boxes[k,:] print(box) drawrect(draw, box, outline='green', width=3) img = np.asarray(img) return img def drawrect(drawcontext, xy, outline=None, width=0): x1, y1, x2, y2 = xy points = (x1, y1), (x2, y1), (x2, y2), (x1, y2), (x1, y1) drawcontext.line(points, fill=outline, width=width)

opened by LemonQC 0

Releases(Results_LXMERT)

Results_LXMERT(Oct 26, 2021)

Source code(tar.gz)
Source code(zip)
test_best_epoch0.json(9.66 MB)
Models_LXMERT(Oct 26, 2021)

Source code(tar.gz)
Source code(zip)
BEST.pth(842.49 MB)
LXMERT_pretrained_model(Oct 25, 2021)

Source code(tar.gz)
Source code(zip)
Epoch20_LXRT.pth(870.06 MB)
Results(Oct 10, 2021)

Source code(tar.gz)
Source code(zip)
61.91_results.json(9.65 MB)
Models(Oct 10, 2021)

Source code(tar.gz)
Source code(zip)
61.91.pth(620.44 MB)

Owner

Zhiquan Wen

GitHub Repository

Implements a fake news detection program using classifiers.

Fake news detection Implements a fake news detection program using classifiers for Data Mining course at UoA. Description The project is the categoriz

1 Jan 09, 2022

Code for sound field predictions in domains with impedance boundaries. Used for generating results from the paper

11 Dec 17, 2022

To model the probability of a soccer coach leave his/her team during Campeonato Brasileiro for 10 chosen teams and considering years 2018, 2019 and 2020.

1 Jan 20, 2022

A Large-Scale Dataset for Spinal Vertebrae Segmentation in Computed Tomography

75 Dec 26, 2022

This is an open source python repository for various python tests

Welcome to Py-tests This is an open source python repository for various python tests. This is in response to the hacktoberfest2021 challenge. It is a

3 Oct 31, 2021

Visual Adversarial Imitation Learning using Variational Models (VMAIL)

Visual Adversarial Imitation Learning using Variational Models (VMAIL) This is the official implementation of the NeurIPS 2021 paper. Project website

14 Nov 18, 2022

Poplar implementation of "Bundle Adjustment on a Graph Processor" (CVPR 2020)

Poplar Implementation of Bundle Adjustment using Gaussian Belief Propagation on Graphcore's IPU Implementation of CVPR 2020 paper: Bundle Adjustment o

34 Dec 05, 2022

Bringing Characters to Life with Computer Brains in Unity

AI4Animation: Deep Learning for Character Control This project explores the opportunities of deep learning for character animation and control as part

5.5k Jan 04, 2023

MultiSiam: Self-supervised Multi-instance Siamese Representation Learning for Autonomous Driving

MultiSiam: Self-supervised Multi-instance Siamese Representation Learning for Autonomous Driving Code will be available soon. Motivation Architecture

24 Apr 19, 2022

Streamlit app demonstrating an image browser for the Udacity self-driving-car dataset with realtime object detection using YOLO.

Streamlit Demo: The Udacity Self-driving Car Image Browser This project demonstrates the Udacity self-driving-car dataset and YOLO object detection in

992 Jan 04, 2023

PyTorch implementation of "Debiased Visual Question Answering from Feature and Sample Perspectives" (NeurIPS 2021)

Related tags

Overview

D-VQA

Dependencies

Getting Started

Installation

Download and preprocess the data

Training

Evaluation

Pretrained model

Reference

Acknowledgements

Comments

Questions about the code

CVE-2007-4559 Patch

Patching CVE-2007-4559

LXMERT numbers

Download trainval_36.zip error

关于box和image的对齐问题

Releases(Results_LXMERT)

Results_LXMERT(Oct 26, 2021)

Models_LXMERT(Oct 26, 2021)

LXMERT_pretrained_model(Oct 25, 2021)

Results(Oct 10, 2021)

Models(Oct 10, 2021)

Owner

Zhiquan Wen

Implements a fake news detection program using classifiers.

Code for sound field predictions in domains with impedance boundaries. Used for generating results from the paper

To model the probability of a soccer coach leave his/her team during Campeonato Brasileiro for 10 chosen teams and considering years 2018, 2019 and 2020.

A Large-Scale Dataset for Spinal Vertebrae Segmentation in Computed Tomography

This is an open source python repository for various python tests

Visual Adversarial Imitation Learning using Variational Models (VMAIL)

Poplar implementation of "Bundle Adjustment on a Graph Processor" (CVPR 2020)

Bringing Characters to Life with Computer Brains in Unity

MultiSiam: Self-supervised Multi-instance Siamese Representation Learning for Autonomous Driving

Streamlit app demonstrating an image browser for the Udacity self-driving-car dataset with realtime object detection using YOLO.

AsymmetricGAN - Dual Generator Generative Adversarial Networks for Multi-Domain Image-to-Image Translation

Official implementation of particle-based models (GNS and DPI-Net) on the Physion dataset.

TF Image Segmentation: Image Segmentation framework

Finetune alexnet with tensorflow - Code for finetuning AlexNet in TensorFlow >= 1.2rc0

Swapping face using Face Mesh with TensorFlow Lite

COVINS -- A Framework for Collaborative Visual-Inertial SLAM and Multi-Agent 3D Mapping

Learning with Subset Stacking

U-Net: Convolutional Networks for Biomedical Image Segmentation

Quick program made to generate alpha and delta tables for Hidden Markov Models

(CVPR2021) ClassSR: A General Framework to Accelerate Super-Resolution Networks by Data Characteristic