PyTorch implementation of InstaGAN: Instance-aware Image-to-Image Translation

Last update: Dec 29, 2022

Overview

InstaGAN: Instance-aware Image-to-Image Translation

Warning: This repo contains a model which has potential ethical concerns. Remark that the task of jeans<->skirt was a bad application and should not be used in future research. See the twitter thread for the discussion.

PyTorch implementation of "InstaGAN: Instance-aware Image-to-Image Translation" (ICLR 2019). The implementation is based on the official CycleGAN code. Our major contributions are in ./models/insta_gan_model.py and ./models/networks.py.

Getting Started

Installation

Clone this repository

git clone https://github.com/sangwoomo/instagan

Install PyTorch 0.4+ and torchvision from http://pytorch.org and other dependencies (e.g., visdom and dominate). You can install all the dependencies by

pip install -r requirements.txt

For Conda users, you can use a script ./scripts/conda_deps.sh to install PyTorch and other libraries.
Acknowledgment: Installation scripts are from the official CycleGAN code.

Download base datasets

Download clothing-co-parsing (CCP) dataset:

git clone https://github.com/bearpaw/clothing-co-parsing ./datasets/clothing-co-parsing

Download multi-human parsing (MHP) dataset:

# Download "LV-MHP-v1" from the link and locate in ./datasets

Download MS COCO dataset:

./datasets/download_coco.sh

Generate two-domain datasets

Generate two-domain dataset for experiments:

python ./datasets/generate_ccp_dataset.py --save_root ./datasets/jeans2skirt_ccp --cat1 jeans --cat2 skirt
python ./datasets/generate_mhp_dataset.py --save_root ./datasets/pants2skirt_mhp --cat1 pants --cat2 skirt
python ./datasets/generate_coco_dataset.py --save_root ./datasets/shp2gir_coco --cat1 sheep --cat2 giraffe

Note: Generated dataset contains images and corresponding masks, which are located in image folders (e.g., 'trainA') and mask folders (e.g., 'trainA_seg'), respectively. For each image (e.g., '0001.png'), corresponding masks for each instance (e.g., '0001_0.png', '0001_1.png', ...) are provided.

Run experiments

Train a model:

python train.py --dataroot ./datasets/jeans2skirt_ccp --model insta_gan --name jeans2skirt_ccp_instagan --loadSizeH 330 --loadSizeW 220 --fineSizeH 300 --fineSizeW 200 --niter 400 --niter_decay 200
python train.py --dataroot ./datasets/pants2skirt_mhp --model insta_gan --name pants2skirt_mhp_instagan --loadSizeH 270 --loadSizeW 180 --fineSizeH 240 --fineSizeW 160
python train.py --dataroot ./datasets/shp2gir_coco --model insta_gan --name shp2gir_coco_instagan --loadSizeH 220 --loadSizeW 220 --fineSizeH 200 --fineSizeW 200

To view training results and loss plots, run python -m visdom.server and click the URL http://localhost:8097. To see more intermediate results, check out ./checkpoints/experiment_name/web/index.html.
For faster experiment, increase batch size and use more gpus:

python train.py --dataroot ./datasets/shp2gir_coco --model insta_gan --name shp2gir_coco_instagan --loadSizeH 220 --loadSizeW 220 --fineSizeH 200 --fineSizeW 200 --batch_size 4 --gpu_ids 0,1,2,3

Test the model:

python test.py --dataroot ./datasets/jeans2skirt_ccp --model insta_gan --name jeans2skirt_ccp_instagan --loadSizeH 300 --loadSizeW 200 --fineSizeH 300 --fineSizeW 200
python test.py --dataroot ./datasets/pants2skirt_mhp --model insta_gan --name pants2skirt_mhp_instagan --loadSizeH 240 --loadSizeW 160 --fineSizeH 240 --fineSizeW 160 --ins_per 2 --ins_max 20
python test.py --dataroot ./datasets/shp2gir_coco --model insta_gan --name shp2gir_coco_instagan --loadSizeH 200 --loadSizeW 200 --fineSizeH 200 --fineSizeW 200 --ins_per 2 --ins_max 20

The test results will be saved to a html file here: ./results/experiment_name/latest_test/index.html.

Apply a pre-trained model

You can download a pre-trained model (pants->skirt and/or sheep->giraffe) from the following Google drive link. Save the pretrained model in ./checkpoints/ directory.
We provide samples of two datasets (pants->skirt and sheep->giraffe) in this repository. To test the model:

python test.py --dataroot ./datasets/pants2skirt_mhp --model insta_gan --name pants2skirt_mhp_instagan --loadSizeH 240 --loadSizeW 160 --fineSizeH 240 --fineSizeW 160 --ins_per 2 --ins_max 20 --phase sample --epoch 200
python test.py --dataroot ./datasets/shp2gir_coco --model insta_gan --name shp2gir_coco_instagan --loadSizeH 200 --loadSizeW 200 --fineSizeH 200 --fineSizeW 200 --ins_per 2 --ins_max 20 --phase sample --epoch 200

Results

We provide some translation results of our model. See the link for more translation results.

1. Fashion dataset (pants->skirt)

2. COCO dataset (sheep->giraffe)

3. Results on Google-searched images (pants->skirt)

4. Results on YouTube-searched videos (pants->skirt)

Citation

If you use this code for your research, please cite our papers.

@inproceedings{
    mo2019instagan,
    title={InstaGAN: Instance-aware Image-to-Image Translation},
    author={Sangwoo Mo and Minsu Cho and Jinwoo Shin},
    booktitle={International Conference on Learning Representations},
    year={2019},
    url={https://openreview.net/forum?id=ryxwJhC9YX},
}

PyTorch implementation of InstaGAN: Instance-aware Image-to-Image Translation

Related tags

Overview

InstaGAN: Instance-aware Image-to-Image Translation

Getting Started

Installation

Download base datasets

Generate two-domain datasets

Run experiments

Apply a pre-trained model

Results

1. Fashion dataset (pants->skirt)

2. COCO dataset (sheep->giraffe)

3. Results on Google-searched images (pants->skirt)

4. Results on YouTube-searched videos (pants->skirt)

Citation

Owner

Sangwoo Mo

SustainBench: Benchmarks for Monitoring the Sustainable Development Goals with Machine Learning

LIMEcraft: Handcrafted superpixel selectionand inspection for Visual eXplanations

This GitHub repository contains code used for plots in NeurIPS 2021 paper 'Stochastic Multi-Armed Bandits with Control Variates.'

Python library for science observations from the James Webb Space Telescope

Individual Tree Crown classification on WorldView-2 Images using Autoencoder -- Group 9 Weak learners - Final Project (Machine Learning 2020 Course)

PyTorch implementation of the Quasi-Recurrent Neural Network - up to 16 times faster than NVIDIA's cuDNN LSTM

Code and datasets for the paper "KnowPrompt: Knowledge-aware Prompt-tuning with Synergistic Optimization for Relation Extraction"

PiCIE: Unsupervised Semantic Segmentation using Invariance and Equivariance in clustering (CVPR2021)

The official implementation of Equalization Loss for Long-Tailed Object Recognition (CVPR 2020) based on Detectron2

Code for "Share With Thy Neighbors: Single-View Reconstruction by Cross-Instance Consistency" paper

Towards Understanding Quality Challenges of the Federated Learning: A First Look from the Lens of Robustness

Augmenting Physical Models with Deep Networks for Complex Dynamics Forecasting

PyTorch implementation of paper “Unbiased Scene Graph Generation from Biased Training”

Create Data & AI apps in 20 lines of code with Shimoku

Benchmark datasets, data loaders, and evaluators for graph machine learning

Self-Supervised Learning for Domain Adaptation on Point-Clouds

Brax is a differentiable physics engine that simulates environments made up of rigid bodies, joints, and actuators

[CVPR 2020] Transform and Tell: Entity-Aware News Image Captioning

The code from the paper Character Transformations for Non-Autoregressive GEC Tagging

📚 Papermill is a tool for parameterizing, executing, and analyzing Jupyter Notebooks.