Curriculum Domain Adaptation for Semantic Segmentation of Urban Scenes, ICCV 2017

Overview

AdaptationSeg

This is the Python reference implementation of AdaptionSeg proposed in "Curriculum Domain Adaptation for Semantic Segmentation of Urban Scenes".

Curriculum Domain Adaptation for Semantic Segmentation of Urban Scenes
Yang Zhang; Philip David; Boqing Gong;
International Conference on Computer Vision, 2017
A Curriculum Domain Adaptation Approach to the Semantic Segmentation of Urban Scenes
Yang Zhang; Philip David;  Hassan Foroosh; Boqing Gong;
IEEE Transactions on Pattern Analysis and Machine Intelligence, 2019

[TPAMI paper] [ICCV paper] [ArXiv Extended paper] [Poster]

[New] Survey of domain adaptation for semantic segmentation

Check out our new survey of domain adaptation for semantic segmentation in our TPAMI paper.

Review

Overview

Qualitative Results

We introduced a set of constraints to domain-adapt an arbitrary segmentation convolutional neural network (CNN) trained on source domain (synthetic images) to target domain (real images) without accessing target domain annotations.

Overview

Prerequisites

  • Linux
  • A CUDA-enabled NVIDIA GPU; Recommend video memory >= 11GB

Getting Started

Installation

The code requires following dependencies:

  • Python 2/3
  • Theano (installation)
  • Keras>=2.0.5 (Lower version might encounter Conv2DTranspose problem with Theano backend) (installation; You might want to install though pip since conda only offers Keras<=2.0.2)
  • Pillow (installation)

Keras backend setup

Make sure your Keras's image_data_format is channels_first. It is recommended to use Theano as the backend. However Tensorflow should also be okay. Note that using Tensorflow will result in lower initial/baseline model performance because the baseline model was trained using Theano.

How do I check/switch them?

Download dataset

1, Download leftImg8bit_trainvaltest.zip and leftImg8bit_trainextra.zip in CityScape dataset here. (Require registration)

2, Download SYNTHIA-RAND-CITYSCAPES in SYNTHIA dataset here.

3, Download our auxiliary pre-inferred target domain properties (Including both superpixel landmark and label distribution described in the paper) & parsed annotation here.

4, Download the submodule cityscapesScripts for evaluation purpose.

5, Unzip and organize them in this way:

./
├── train_val_DA.py
├── ...
├── cityscapesScripts/
│   ├── ...
│   └── cityscapesscripts/
│       ├── ...
│       └── evaluation/...
└── data/
    ├── Image/
    │   ├── CityScape/           # Unzip from two CityScape zips
    │   │   ├── test/
    │   │   ├── train/
    │   │   ├── train_extra/
    │   │   └── val/
    │   └── SYNTHIA/             # Unzip from the SYNTHIA dataset
    │       └── train/
    ├── label_distribution/      # Unzip from our auxiliary dataset
    │   └── ...
    ├── segmentation_annotation/ # Unzip from our auxiliary dataset
    │   └── ...
    ├── SP_labels/               # Unzip from our auxiliary dataset
    │   └── ...
    └── SP_landmark/             # Unzip from our auxiliary dataset
        └── ...

(Hint: If you have already downloaded the datasets but do not want to move them around, you may want to create some symbolic links of exsiting local datasets)

Training

Run train_val_FCN_DA.py either in your favorite Python IDE or the terminal by typing:

python train_val_FCN_DA.py

This would train the model for six epochs and save the best model during the training. You can stop it and continue to the evaluation during training if you feel it takes too long, however, performance would not be guaranteed then.

Evaluation

After running train_val_FCN_DA.py for at least 500 steps, run test_FCN_DA.py either in your favorite Python IDE or the terminal by typing:

python test_FCN_DA.py

This would evaluate both pre-trained SYNTHIA-FCN and adapted FCN over CityScape dataset and print both mean IoU.

Note

The original framework was implemented in Keras 1 with a custom transposed convolution ops. The performance might be slightly different from the ones reported in the paper. Also, some new commits in TF/Theano optimizer implementation after the code release has broken the losses' numerical stability. I have changed code's optimizer to SGD despite the original paper used Adadelta. You are welcome to try Adadelta/Adam however it seems that they will result in a NaN loss right after training starts. If the NaN problem persists, try to remove the label distribution loss from the training.

Citation

Please cite our paper if this code benefits your reseaarch:

@InProceedings{Zhang_2017_ICCV,
author = {Zhang, Yang and David, Philip and Gong, Boqing},
title = {Curriculum Domain Adaptation for Semantic Segmentation of Urban Scenes},
booktitle={The IEEE International Conference on Computer Vision (ICCV)},
volume={2},
number={5},
pages={6},
month = {Oct},
year = {2017}
}

@ARTICLE{Zhang_2019_TPAMI,
author={Zhang, Yang and David, Philip and Foroosh, Hassan and Gong, Boqing},
journal={IEEE Transactions on Pattern Analysis and Machine Intelligence},
title={A Curriculum Domain Adaptation Approach to the Semantic Segmentation of Urban Scenes},
year={2019},
volume={},
number={},
pages={1-1},
doi={10.1109/TPAMI.2019.2903401},
ISSN={1939-3539},
month={},}
Owner
Yang Zhang
Perception @ Waymo
Yang Zhang
[ECE NTUA] 👁 Computer Vision - Lab Projects & Theoretical Problem Sets (2020-2021)

Computer Vision - NTUA (2020-2021) This repository hosts the lab projects and theoretical problem sets of the Computer Vision course held by ECE NTUA

Dimitris Dimos 6 Jul 21, 2022
Worktory is a python library created with the single purpose of simplifying the inventory management of network automation scripts.

Worktory is a python library created with the single purpose of simplifying the inventory management of network automation scripts.

Renato Almeida de Oliveira 18 Aug 31, 2022
This codebase is the official implementation of Test-Time Classifier Adjustment Module for Model-Agnostic Domain Generalization (NeurIPS2021, Spotlight)

Test-Time Classifier Adjustment Module for Model-Agnostic Domain Generalization This codebase is the official implementation of Test-Time Classifier A

47 Dec 28, 2022
OCRA (Object-Centric Recurrent Attention) source code

OCRA (Object-Centric Recurrent Attention) source code Hossein Adeli and Seoyoung Ahn Please cite this article if you find this repository useful: For

Hossein Adeli 2 Jun 18, 2022
Robustness via Cross-Domain Ensembles

Robustness via Cross-Domain Ensembles [ICCV 2021, Oral] This repository contains tools for training and evaluating: Pretrained models Demo code Traini

Visual Intelligence & Learning Lab, Swiss Federal Institute of Technology (EPFL) 27 Dec 23, 2022
3D-Transformer: Molecular Representation with Transformer in 3D Space

3D-Transformer: Molecular Representation with Transformer in 3D Space

55 Dec 19, 2022
A PyTorch implementation of QANet.

QANet-pytorch NOTICE I'm very busy these months. I'll return to this repo in about 10 days. Introduction An implementation of QANet with PyTorch. Any

H. Z. 343 Nov 03, 2022
Trax — Deep Learning with Clear Code and Speed

Trax — Deep Learning with Clear Code and Speed Trax is an end-to-end library for deep learning that focuses on clear code and speed. It is actively us

Google 7.3k Dec 26, 2022
Joint project of the duo Hacker Ninjas

Project Smoothie Společný projekt dua Hacker Ninjas. První pokus o hříčku po třech týdnech učení se programování. Jakub Kolář e:\

Jakub Kolář 2 Jan 07, 2022
A Simplied Framework of GAN Inversion

Framework of GAN Inversion Introcuction You can implement your own inversion idea using our repo. We offer a full range of tuning settings (in hparams

Kangneng Zhou 13 Sep 27, 2022
Adaptive FNO transformer - official Pytorch implementation

Adaptive Fourier Neural Operators: Efficient Token Mixers for Transformers This repository contains PyTorch implementation of the Adaptive Fourier Neu

NVIDIA Research Projects 77 Dec 29, 2022
Reliable probability face embeddings

ProbFace, arxiv This is a demo code of training and testing [ProbFace] using Tensorflow. ProbFace is a reliable Probabilistic Face Embeddging (PFE) me

Kaen Chan 34 Dec 31, 2022
A simple command line tool for text to image generation, using OpenAI's CLIP and a BigGAN.

Ryan Murdock has done it again, combining OpenAI's CLIP and the generator from a BigGAN! This repository wraps up his work so it is easily accessible to anyone who owns a GPU.

Phil Wang 2.3k Jan 09, 2023
ISNAS-DIP: Image Specific Neural Architecture Search for Deep Image Prior [CVPR 2022]

ISNAS-DIP: Image-Specific Neural Architecture Search for Deep Image Prior (CVPR 2022) Metin Ersin Arican*, Ozgur Kara*, Gustav Bredell, Ender Konukogl

Özgür Kara 24 Dec 18, 2022
A Large-Scale Dataset for Spinal Vertebrae Segmentation in Computed Tomography

A Large-Scale Dataset for Spinal Vertebrae Segmentation in Computed Tomography

ICT.MIRACLE lab 75 Dec 26, 2022
DeepVoxels is an object-specific, persistent 3D feature embedding.

DeepVoxels is an object-specific, persistent 3D feature embedding. It is found by globally optimizing over all available 2D observations of

Vincent Sitzmann 196 Dec 25, 2022
Source code for "Pack Together: Entity and Relation Extraction with Levitated Marker"

PL-Marker Source code for Pack Together: Entity and Relation Extraction with Levitated Marker. Quick links Overview Setup Install Dependencies Data Pr

THUNLP 173 Dec 30, 2022
Implicit Model Specialization through DAG-based Decentralized Federated Learning

Federated Learning DAG Experiments This repository contains software artifacts to reproduce the experiments presented in the Middleware '21 paper "Imp

Operating Systems and Middleware Group 5 Oct 16, 2022
implement of SwiftNet:Real-time Video Object Segmentation

SwiftNet The official PyTorch implementation of SwiftNet:Real-time Video Object Segmentation, which has been accepted by CVPR2021. Requirements Python

haochen wang 64 Dec 14, 2022
Structural Constraints on Information Content in Human Brain States

Structural Constraints on Information Content in Human Brain States Code accompanying the paper "The information content of brain states is explained

Leon Weninger 3 Sep 07, 2022