Semi-supervised semantic segmentation needs strong, varied perturbations

Overview

Semi-supervised semantic segmentation using CutMix and Colour Augmentation

Implementations of our papers:

Licensed under MIT license.

Colour augmentation

Please see our new paper for a full discussion, but a summary of our findings can be found in our [colour augmentation](Colour augmentation.ipynb) Jupyter notebook.

Requirements

We provide an environment.yml file that can be used to re-create a conda environment that provides the required packages:

conda env create -f environment.yml

Then activate with:

conda activate cutmix_semisup_seg

(note: this will not install the library needed to use the PSPNet architecture; see below)

In general we need:

  • Python >= 3.6
  • PyTorch >= 1.4
  • torchvision 0.5
  • OpenCV
  • Pillow
  • Scikit-image
  • Scikit-learn
  • click
  • tqdm
  • Jupyter notebook for the notebooks
  • numpy 1.18

Requirements for PSPNet

To use the PSPNet architecture (see Pyramid Scene Parsing Network by Zhao et al.), you will need to install the logits-from_models branch of https://github.com/Britefury/semantic-segmentation-pytorch:

pip install git+https://github.com/Britefury/[email protected]

Datasets

You need to:

  1. Download/acquire the datsets
  2. Write the config file semantic_segmentation.cfg giving their paths
  3. Convert them if necessary; the CamVid, Cityscapes and ISIC 2017 datasets must be converted to a ZIP-based format prior to use. You must run the provided conversion utilities to create these ZIP files.

Dataset preparation instructions can be found here.

Running the experiments

We provide four programs for running experiments:

  • train_seg_semisup_mask_mt.py: mask driven consistency loss (the main experiment)
  • train_seg_semisup_aug_mt.py: augmentation driven consistency loss; used to attempt to replicate the ISIC 2017 baselines of Li et al.
  • train_seg_semisup_ict.py: Interpolation Consistency Training; a baseline for contrast with our main approach
  • train_seg_semisup_vat_mt.py: Virtual Adversarial Training adapted for semantic segmentation

They can be configured via command line arguments that are described here.

Shell scripts

To replicate our results, we provide shell scripts to run our experiments.

Cityscapes
> sh run_cityscapes_experiments.sh <run> <split_rng_seed>

where <run> is the name of the run and <split_rng_seed> is an integer RNG seed used to select the supervised samples. Please see the comments at the top of run_cityscapes_experiments.sh for further explanation.

To re-create the 5 runs we used for our experiments:

> sh run_cityscapes_experiments.sh 01 12345
> sh run_cityscapes_experiments.sh 02 23456
> sh run_cityscapes_experiments.sh 03 34567
> sh run_cityscapes_experiments.sh 04 45678
> sh run_cityscapes_experiments.sh 05 56789
Pascal VOC 2012 (augmented)
> sh run_pascal_aug_experiments.sh <n_supervised> <n_supervised_txt>

where <n_supervised> is the number of supervised samples and <n_supervised_txt> is that number as text. Please see the comments at the top of run_pascal_aug_experiments.sh for further explanation.

We use the same data split as Mittal et al. It is stored in data/splits/pascal_aug/split_0.pkl that is included in the repo.

Pascal VOC 2012 (augmented) with DeepLab v3+
> sh run_pascal_aug_deeplab3plus_experiments.sh <n_supervised> <n_supervised_txt>
ISIC 2017 Segmentation
> sh run_isic2017_experiments.sh <run> <split_rng_seed>

where <run> is the name of the run and <split_rng_seed> is an integer RNG seed used to select the supervised samples. Please see the comments at the top of run_isic2017_experiments.sh for further explanation.

To re-create the 5 runs we used for our experiments:

> sh run_isic2017_experiments.sh 01 12345
> sh run_isic2017_experiments.sh 02 23456
> sh run_isic2017_experiments.sh 07 78901
> sh run_isic2017_experiments.sh 08 89012
> sh run_isic2017_experiments.sh 09 90123

In early experiments, we test 10 seeds and selected the middle 5 when ranked in terms of performance, hence the specific seed choice.

Exploring the input data distribution present in semantic segmentation problems

Cluster assumption

First we examine the input data distribution presented by semantic segmentation problems with a view to determining if the low density separation assumption holds, in the notebook Semantic segmentation input data distribution.ipynb This notebook also contains the code used to generate the images from Figure 1 in the paper.

Inter-class and intra-class variance

Secondly we examine the inter-class and intra-class distance (as a proxy for inter-class and intra-class variance) in the notebook Plot inter-class and intra-class distances from files.ipynb

Note that running the second notebook requires that you generate some data files using the intra_inter_class_patch_dist.py program.

Toy 2D experiments

The toy 2D experiments used to produce Figure 3 in the paper can be run using the toy2d_train.py program, which is documented here.

You can re-create the toy 2D experiments by running the run_toy2d_experiments.sh shell script:

> sh run_toy2d_experiments.sh <run>
ManipulaTHOR, a framework that facilitates visual manipulation of objects using a robotic arm

ManipulaTHOR: A Framework for Visual Object Manipulation Kiana Ehsani, Winson Han, Alvaro Herrasti, Eli VanderBilt, Luca Weihs, Eric Kolve, Aniruddha

AI2 65 Dec 30, 2022
This is my codes that can visualize the psnr image in testing videos.

CVPR2018-Baseline-PSNRplot This is my codes that can visualize the psnr image in testing videos. Future Frame Prediction for Anomaly Detection – A New

Wenhao Yang 12 May 29, 2021
Code for one-stage adaptive set-based HOI detector AS-Net.

AS-Net Code for one-stage adaptive set-based HOI detector AS-Net. Mingfei Chen*, Yue Liao*, Si Liu, Zhiyuan Chen, Fei Wang, Chen Qian. "Reformulating

Mingfei Chen 45 Dec 09, 2022
Pytorch implementation of U-Net, R2U-Net, Attention U-Net, and Attention R2U-Net.

pytorch Implementation of U-Net, R2U-Net, Attention U-Net, Attention R2U-Net U-Net: Convolutional Networks for Biomedical Image Segmentation https://a

leejunhyun 2k Jan 02, 2023
Faster RCNN with PyTorch

Faster RCNN with PyTorch Note: I re-implemented faster rcnn in this project when I started learning PyTorch. Then I use PyTorch in all of my projects.

Long Chen 1.6k Dec 23, 2022
Code accompanying the paper Say As You Wish: Fine-grained Control of Image Caption Generation with Abstract Scene Graphs (Chen et al., CVPR 2020, Oral).

Say As You Wish: Fine-grained Control of Image Caption Generation with Abstract Scene Graphs This repository contains PyTorch implementation of our pa

Shizhe Chen 178 Dec 29, 2022
Demo project for real time anomaly detection using kafka and python

kafkaml-anomaly-detection Project for real time anomaly detection using kafka and python It's assumed that zookeeper and kafka are running in the loca

Rodrigo Arenas 36 Dec 12, 2022
ACAV100M: Automatic Curation of Large-Scale Datasets for Audio-Visual Video Representation Learning. In ICCV, 2021.

ACAV100M: Automatic Curation of Large-Scale Datasets for Audio-Visual Video Representation Learning This repository contains the code for our ICCV 202

sangho.lee 28 Nov 08, 2022
"Moshpit SGD: Communication-Efficient Decentralized Training on Heterogeneous Unreliable Devices", official implementation

Moshpit SGD: Communication-Efficient Decentralized Training on Heterogeneous Unreliable Devices This repository contains the official PyTorch implemen

Yandex Research 21 Oct 18, 2022
It's A ML based Web Site build with python and Django to find the breed of the dog

ML-Based-Dog-Breed-Identifier This is a Django Based Web Site To Identify the Breed of which your DOG belogs All You Need To Do is to Follow These Ste

Sanskar Dwivedi 2 Oct 12, 2022
3D Multi-Person Pose Estimation by Integrating Top-Down and Bottom-Up Networks

3D Multi-Person Pose Estimation by Integrating Top-Down and Bottom-Up Networks Introduction This repository contains the code and models for the follo

124 Jan 06, 2023
Citation Intent Classification in scientific papers using the Scicite dataset an Pytorch

Citation Intent Classification Table of Contents About the Project Built With Installation Usage Acknowledgments About The Project Citation Intent Cla

Federico Nocentini 4 Mar 04, 2022
Bachelor's Thesis in Computer Science: Privacy-Preserving Federated Learning Applied to Decentralized Data

federated is the source code for the Bachelor's Thesis Privacy-Preserving Federated Learning Applied to Decentralized Data (Spring 2021, NTNU) Federat

Dilawar Mahmood 25 Nov 30, 2022
Airbus Ship Detection Challenge

Airbus Ship Detection Challenge This is an open solution to the Airbus Ship Detection Challenge. Our goals We are building entirely open solution to t

minerva.ml 55 Nov 29, 2022
Offline Multi-Agent Reinforcement Learning Implementations: Solving Overcooked Game with Data-Driven Method

Overcooked-AI We suppose to apply traditional offline reinforcement learning technique to multi-agent algorithm. In this repository, we implemented be

Baek In-Chang 14 Sep 16, 2022
Code to compute permutation and drop-column importances in Python scikit-learn models

Feature importances for scikit-learn machine learning models By Terence Parr and Kerem Turgutlu. See Explained.ai for more stuff. The scikit-learn Ran

Terence Parr 537 Dec 31, 2022
Face and Pose detector that emits MQTT events when a face or human body is detected and not detected.

Face Detect MQTT Face or Pose detector that emits MQTT events when a face or human body is detected and not detected. I built this as an alternative t

Jacob Morris 38 Oct 21, 2022
Code for EMNLP2020 long paper: BERT-Attack: Adversarial Attack Against BERT Using BERT

BERT-ATTACK Code for our EMNLP2020 long paper: BERT-ATTACK: Adversarial Attack Against BERT Using BERT Dependencies Python 3.7 PyTorch 1.4.0 transform

Linyang Li 142 Jan 04, 2023
The implementation of the algorithm in the paper "Safe Deep Semi-Supervised Learning for Unseen-Class Unlabeled Data" published in ICML 2020.

DS3L This is the code for paper "Safe Deep Semi-Supervised Learning for Unseen-Class Unlabeled Data" published in ICML 2020. Setups The code is implem

Guolz 36 Oct 19, 2022
Code for the CIKM 2019 paper "DSANet: Dual Self-Attention Network for Multivariate Time Series Forecasting".

Dual Self-Attention Network for Multivariate Time Series Forecasting 20.10.26 Update: Due to the difficulty of installation and code maintenance cause

Kyon Huang 223 Dec 16, 2022