Object-aware Contrastive Learning for Debiased Scene Representation

Last update: Dec 14, 2022

Overview

Object-aware Contrastive Learning

Official PyTorch implementation of "Object-aware Contrastive Learning for Debiased Scene Representation" by Sangwoo Mo*, Hyunwoo Kang*, Kihyuk Sohn, Chun-Liang Li, and Jinwoo Shin.

Installation

Install required libraries.

pip install -r requirements.txt

Download datasets in /data (e.g., /data/COCO).

Train models

Logs will be saved in logs/{dataset}_{model}_{arch}_b{global_batch_size} directory, where global_batch_size = num_nodes * gpus * batch_size (default batch size = 64 * 4 = 256).

Step 1. Train vanilla models

Train vanilla models (change dataset and ft_datasets as cub or in9).

python pretrain.py --dataset coco --model moco --arch resnet18\
    --ft_datasets coco --batch_size 64 --max_epochs 800

Step 2. Pre-compute CAM masks

Pre-compute bounding boxes for object-aware random crop.

python inference.py --mode save_box --model moco --arch resnet18\
    --ckpt_name coco_moco_r18_b256 --dataset coco\
    --expand_res 2 --cam_iters 10 --apply_crf\
    --save_path data/boxes/coco_cam-r18.txt

Pre-compute masks for background mixup.

python inference.py --mode save_mask --model moco --arch resnet18\
    --ckpt_name in9_moco_r18_256 --dataset in9\
    --expand_res 1 --cam_iters 1\
    --save_path data/masks/in9_cam-r18

Step 3. Re-train debiased models

Train contextual debiased model with object-aware random crop.

python pretrain.py --dataset coco-box-cam-r18 --model moco --arch resnet18\
     --ft_datasets coco --batch_size 64 --max_epochs 800

Train background debiased model with background mixup.

python pretrain.py --dataset in9-mask-cam-r18 --model moco_bgmix --arch resnet18\
    --ft_datasets in9 --batch_size 64 --max_epochs 800

Evaluate models

Linear evaluation

python inference.py --mode lineval --model moco --arch resnet18\
    --ckpt_name coco_moco_r18_b256 --dataset coco

Object localization

python inference.py --mode seg --model moco --arch resnet18\
    --ckpt_name cub200_moco_r18_b256 --dataset cub200\
    --expand_res 2 --cam_iters 10 --apply_crf

Detection & Segmentation (fine-tuning)

mv detection
python convert-pretrain-to-detectron2.py coco_moco_r50.pth coco_moco_r50.pkl
python train_net.py --config-file configs/coco_R_50_C4_2x_moco.yaml --num-gpus 8\
    MODEL.WEIGHTS weights/coco_moco_r18.pkl

Object-aware Contrastive Learning for Debiased Scene Representation

Related tags

Overview

Object-aware Contrastive Learning

Installation

Train models

Step 1. Train vanilla models

Step 2. Pre-compute CAM masks

Step 3. Re-train debiased models

Evaluate models

Linear evaluation

Object localization

Detection & Segmentation (fine-tuning)

Owner

Code and experiments for "Deep Neural Networks for Rank Consistent Ordinal Regression based on Conditional Probabilities"

Deep Markov Factor Analysis (NeurIPS2021)

Repo for FUZE project. I will also publish some Linux kernel LPE exploits for various real world kernel vulnerabilities here. the samples are uploaded for education purposes for red and blue teams.

You Only Hypothesize Once: Point Cloud Registration with Rotation-equivariant Descriptors

Code of paper "CDFI: Compression-Driven Network Design for Frame Interpolation", CVPR 2021

Deep learning with TensorFlow and earth observation data.

This is the repository for our paper SimpleTrack: Understanding and Rethinking 3D Multi-object Tracking

GPU Accelerated Non-rigid ICP for surface registration

Dilated Convolution for Semantic Image Segmentation

StellarGraph - Machine Learning on Graphs

Generate text captions for images from their CLIP embeddings. Includes PyTorch model code and example training script.

SimDeblur is a simple framework for image and video deblurring, implemented by PyTorch

Informer: Beyond Efficient Transformer for Long Sequence Time-Series Forecasting

1st Solution For NeurIPS 2021 Competition on ML4CO Dual Task

Stitch it in Time: GAN-Based Facial Editing of Real Videos

PyTorch implementation of the paper: "Preference-Adaptive Meta-Learning for Cold-Start Recommendation", IJCAI, 2021.

Code for CVPR2021 paper "Robust Reflection Removal with Reflection-free Flash-only Cues"

Code for Transformers Solve Limited Receptive Field for Monocular Depth Prediction

Provided is code that demonstrates the training and evaluation of the work presented in the paper: "On the Detection of Digital Face Manipulation" published in CVPR 2020.

On the adaptation of recurrent neural networks for system identification