Semi-supevised Semantic Segmentation with High- and Low-level Consistency
This Pytorch repository contains the code for our work Semi-supervised Semantic Segmentation with High- and Low-level Consistency. The approach uses two network branches that link semi-supervised classification with semi-supervised segmentation including self-training. The approach attains significant improvement over existing methods, especially when trained with very few labeled samples. On several standard benchmarks - PASCAL VOC 2012,PASCAL-Context, and Cityscapes - the approach achieves new state-of-the-art in semi-supervised learning.
We propose a two-branch approach to the task of semi-supervised semantic segmentation. The lower branch predicts pixel-wise class labels and is referred to as the Semi-Supervised Semantic Segmentation GAN(s4GAN). The upper branch performs image-level classification and is denoted as the Multi-Label Mean Teacher(MLMT).
Here, this repository contains the source code for the s4GAN branch. MLMT branch is adapted from Mean-Teacher work for semi-supervised classification. Instructions for setting up the MLMT branch are given below.
Package pre-requisites
The code runs on Python 3 and Pytorch 0.4 The following packages are required.
pip install scipy tqdm matplotlib numpy opencv-python
Dataset preparation
Download ImageNet pretrained Resnet-101(Link) and place it ./pretrained_models/
PASCAL VOC
Download the dataset(Link) and extract in ./data/voc_dataset/
PASCAL Context
Download the annotations(Link) and extract in ./data/pcontext_dataset/
Cityscapes
Download the dataset from the Cityscapes dataset server(Link). Download the files named 'gtFine_trainvaltest.zip', 'leftImg8bit_trainvaltest.zip' and extract in ./data/city_dataset/
Training and Validation on PASCAL-VOC Dataset
Results in the paper are averaged over 3 random splits. Same splits are used for reporting baseline performance for fair comparison.
Training fully-supervised Baseline (FSL)
python train_full.py --dataset pascal_voc \
--checkpoint-dir ./checkpoints/voc_full \
--ignore-label 255 \
--num-classes 21
Training semi-supervised s4GAN (SSL)
python train_s4GAN.py --dataset pascal_voc \
--checkpoint-dir ./checkpoints/voc_semi_0_125 \
--labeled-ratio 0.125 \
--ignore-label 255 \
--num-classes 21
Validation
python evaluate.py --dataset pascal_voc \
--num-classes 21 \
--restore-from ./checkpoints/voc_semi_0_125/VOC_30000.pth
Training MLMT Branch
python train_mlmt.py \
--batch-size-lab 16 \
--batch-size-unlab 80 \
--labeled-ratio 0.125 \
--exp-name voc_semi_0_125_MLMT \
--pkl-file ./checkpoints/voc_semi_0_125/train_voc_split.pkl
Final Evaluation S4GAN + MLMT
python evaluate.py --dataset pascal_voc \
--num-classes 21 \
--restore-from ./checkpoints/voc_semi_0_125/VOC_30000.pth \
--with-mlmt \
--mlmt-file ./mlmt_output/voc_semi_0_125_MLMT/output_ema_raw_100.txt
Training and Validation on PASCAL-Context Dataset
python train_full.py --dataset pascal_context \
--checkpoint-dir ./checkpoints/pc_full \
--ignore-label -1 \
--num-classes 60
python train_s4GAN.py --dataset pascal_context \
--checkpoint-dir ./checkpoints/pc_semi_0_125 \
--labeled-ratio 0.125 \
--ignore-label -1 \
--num-classes 60 \
--split-id ./splits/pc/split_0.pkl
--num-steps 60000
python evaluate.py --dataset pascal_context \
--num-classes 60 \
--restore-from ./checkpoints/pc_semi_0_125/VOC_40000.pth
Training and Validation on Cityscapes Dataset
python train_full.py --dataset cityscapes \
--checkpoint-dir ./checkpoints/city_full_0_125 \
--ignore-label 250 \
--num-classes 19 \
--input-size '256,512'
python train_s4GAN.py --dataset cityscapes \
--checkpoint-dir ./checkpoints/city_semi_0_125 \
--labeled-ratio 0.125 \
--ignore-label 250 \
--num-classes 19 \
--split-id ./splits/city/split_0.pkl \
--input-size '256,512' \
--threshold-st 0.7 \
--learning-rate-D 1e-5
python evaluate.py --dataset cityscapes \
--num-classes 19 \
--restore-from ./checkpoints/city_semi_0_125/VOC_30000.pth
Acknowledgement
Parts of the code have been adapted from: DeepLab-Resnet-Pytorch, AdvSemiSeg, PyTorch-Encoding
Citation
@ARTICLE{8935407,
author={S. {Mittal} and M. {Tatarchenko} and T. {Brox}},
journal={IEEE Transactions on Pattern Analysis and Machine Intelligence},
title={Semi-Supervised Semantic Segmentation With High- and Low-Level Consistency},
year={2021},
volume={43},
number={4},
pages={1369-1379},
doi={10.1109/TPAMI.2019.2960224}}
40 Dec 01, 2022
17 Dec 15, 2022
1 Feb 13, 2022
281 Dec 21, 2022
1.1k Dec 30, 2022
298 Dec 21, 2022
643 Dec 30, 2022
104 Dec 14, 2022
6.2k Jan 02, 2023
0 Dec 04, 2021
1 Nov 15, 2021
10 Nov 25, 2022
0 Oct 29, 2021
196 Dec 17, 2022
9 Sep 07, 2022
35 Nov 14, 2022
42 Jan 04, 2023
22 Oct 20, 2022
16 Jul 17, 2022
130 Dec 13, 2022