ICCV2021 - Mining Contextual Information Beyond Image for Semantic Segmentation

Related tags

Deep Learningmcibi
Overview

Introduction

The official repository for "Mining Contextual Information Beyond Image for Semantic Segmentation". Our full code has been merged into sssegmentation.

Abstract

This paper studies the context aggregation problem in semantic image segmentation. The existing researches focus on improving the pixel representations by aggregating the contextual information within individual images. Though impressive, these methods neglect the significance of the representations of the pixels of the corresponding class beyond the input image. To address this, this paper proposes to mine the contextual information beyond individual images to further augment the pixel representations. We first set up a feature memory module, which is updated dynamically during training, to store the dataset-level representations of various categories. Then, we learn class probability distribution of each pixel representation under the supervision of the ground-truth segmentation. At last, the representation of each pixel is augmented by aggregating the dataset-level representations based on the corresponding class probability distribution. Furthermore, by utilizing the stored dataset-level representations, we also propose a representation consistent learning strategy to make the classification head better address intra-class compactness and inter-class dispersion. The proposed method could be effortlessly incorporated into existing segmentation frameworks (e.g., FCN, PSPNet, OCRNet and DeepLabV3) and brings consistent performance improvements. Mining contextual information beyond image allows us to report state-of-the-art performance on various benchmarks: ADE20K, LIP, Cityscapes and COCO-Stuff.

Framework

img

Performance

COCOStuff-10k

Model Backbone Crop Size Schedule Train/Eval Set mIoU/mIoU (ms+flip) Download
DeepLabV3 R-50-D8 512x512 LR/POLICY/BS/EPOCH: 0.001/poly/16/110 train/test 38.84%/39.68% model | log
DeepLabV3 R-101-D8 512x512 LR/POLICY/BS/EPOCH: 0.001/poly/16/110 train/test 39.84%/41.49% model | log
DeepLabV3 S-101-D8 512x512 LR/POLICY/BS/EPOCH: 0.001/poly/32/150 train/test 41.18%/42.15% model | log
DeepLabV3 HRNetV2p-W48 512x512 LR/POLICY/BS/EPOCH: 0.001/poly/16/110 train/test 39.77%/41.35% model | log
DeepLabV3 ViT-Large 512x512 LR/POLICY/BS/EPOCH: 0.001/poly/16/110 train/test 44.01%/45.23% model | log

ADE20k

Model Backbone Crop Size Schedule Train/Eval Set mIoU/mIoU (ms+flip) Download
DeepLabV3 R-50-D8 512x512 LR/POLICY/BS/EPOCH: 0.01/poly/16/130 train/val 44.39%/45.95% model | log
DeepLabV3 R-101-D8 512x512 LR/POLICY/BS/EPOCH: 0.01/poly/16/130 train/val 45.66%/47.22% model | log
DeepLabV3 S-101-D8 512x512 LR/POLICY/BS/EPOCH: 0.004/poly/16/180 train/val 46.63%/47.36% model | log
DeepLabV3 HRNetV2p-W48 512x512 LR/POLICY/BS/EPOCH: 0.004/poly/16/180 train/val 45.79%/47.34% model | log
DeepLabV3 ViT-Large 512x512 LR/POLICY/BS/EPOCH: 0.01/poly/16/130 train/val 49.73%/50.99% model | log

CityScapes

Model Backbone Crop Size Schedule Train/Eval Set mIoU (ms+flip) Download
DeepLabV3 R-50-D8 512x1024 LR/POLICY/BS/EPOCH: 0.01/poly/16/440 trainval/test 79.90% model | log
DeepLabV3 R-101-D8 512x1024 LR/POLICY/BS/EPOCH: 0.01/poly/16/440 trainval/test 82.03% model | log
DeepLabV3 S-101-D8 512x1024 LR/POLICY/BS/EPOCH: 0.01/poly/16/500 trainval/test 81.59% model | log
DeepLabV3 HRNetV2p-W48 512x1024 LR/POLICY/BS/EPOCH: 0.01/poly/16/500 trainval/test 82.55% model | log

LIP

Model Backbone Crop Size Schedule Train/Eval Set mIoU/mIoU (flip) Download
DeepLabV3 R-50-D8 473x473 LR/POLICY/BS/EPOCH: 0.01/poly/32/150 train/val 53.73%/54.08% model | log
DeepLabV3 R-101-D8 473x473 LR/POLICY/BS/EPOCH: 0.01/poly/32/150 train/val 55.02%/55.42% model | log
DeepLabV3 S-101-D8 473x473 LR/POLICY/BS/EPOCH: 0.007/poly/40/150 train/val 56.21%/56.34% model | log
DeepLabV3 HRNetV2p-W48 473x473 LR/POLICY/BS/EPOCH: 0.007/poly/40/150 train/val 56.40%/56.99% model | log

Citation

If this code is useful for your research, please consider citing:

@article{jin2021mining,
  title={Mining Contextual Information Beyond Image for Semantic Segmentation},
  author={Jin, Zhenchao and Gong, Tao and Yu, Dongdong and Chu, Qi and Wang, Jian and Wang, Changhu and Shao, Jie},
  journal={arXiv preprint arXiv:2108.11819},
  year={2021}
}
Owner
student
This repository is an official implementation of the paper MOTR: End-to-End Multiple-Object Tracking with TRansformer.

MOTR: End-to-End Multiple-Object Tracking with TRansformer This repository is an official implementation of the paper MOTR: End-to-End Multiple-Object

348 Jan 07, 2023
Transfer Learning library for Deep Neural Networks.

Transfer and meta-learning in Python Each folder in this repository corresponds to a method or tool for transfer/meta-learning. xfer-ml is a standalon

Amazon 245 Dec 08, 2022
No-Reference Image Quality Assessment via Transformers, Relative Ranking, and Self-Consistency

This repository contains the implementation for the paper: No-Reference Image Quality Assessment via Transformers, Relative Ranking, and Self-Consiste

Alireza Golestaneh 75 Dec 30, 2022
Code for paper "Vocabulary Learning via Optimal Transport for Neural Machine Translation"

**Codebase and data are uploaded in progress. ** VOLT(-py) is a vocabulary learning codebase that allows researchers and developers to automaticaly ge

416 Jan 09, 2023
Multi-label classification of retinal disorders

Multi-label classification of retinal disorders This is a deep learning course project. The goal is to develop a solution, using computer vision techn

Sundeep Bhimireddy 1 Jan 29, 2022
Official implementation of the Implicit Behavioral Cloning (IBC) algorithm

Implicit Behavioral Cloning This codebase contains the official implementation of the Implicit Behavioral Cloning (IBC) algorithm from our paper: Impl

Google Research 210 Dec 09, 2022
Sign Language Translation with Transformers (COLING'2020, ECCV'20 SLRTP Workshop)

transformer-slt This repository gathers data and code supporting the experiments in the paper Better Sign Language Translation with STMC-Transformer.

Kayo Yin 107 Dec 27, 2022
Self-supervised Label Augmentation via Input Transformations (ICML 2020)

Self-supervised Label Augmentation via Input Transformations Authors: Hankook Lee, Sung Ju Hwang, Jinwoo Shin (KAIST) Accepted to ICML 2020 Install de

hankook 96 Dec 29, 2022
Pytorch implementation of Cut-Thumbnail in the paper Cut-Thumbnail:A Novel Data Augmentation for Convolutional Neural Network.

Cut-Thumbnail (Accepted at ACM MULTIMEDIA 2021) Tianshu Xie, Xuan Cheng, Xiaomin Wang, Minghui Liu, Jiali Deng, Tao Zhou, Ming Liu This is the officia

3 Apr 12, 2022
Simulating an AI playing 2048 using the Expectimax algorithm

2048-expectimax Simulating an AI playing 2048 using the Expectimax algorithm The base game engine uses code from here. The AI player is modeled as a m

Subha Ramesh 2 Jan 31, 2022
Convert weight file.pth to weight file.blob

CONVERT YOUR MODEL TO IR FORMAT INSTALLATION OpenVino Toolkit Download openvinotoolkit 2021.3 version : Link Instruction of installation : Link Pytorc

Tran Anh Tuan 3 Nov 18, 2021
Memory Efficient Attention (O(sqrt(n)) for Jax and PyTorch

Memory Efficient Attention This is unofficial implementation of Self-attention Does Not Need O(n^2) Memory for Jax and PyTorch. Implementation is almo

Amin Rezaei 126 Dec 27, 2022
Framework to build and train RL algorithms

RayLink RayLink is a RL framework used to build and train RL algorithms. RayLink was used to build a RL framework, and tested in a large-scale multi-a

Bytedance Inc. 32 Oct 07, 2022
code for CVPR paper Zero-shot Instance Segmentation

Code for CVPR2021 paper Zero-shot Instance Segmentation Code requirements python: python3.7 nvidia GPU pytorch1.1.0 GCC =5.4 NCCL 2 the other python

zhengye 86 Dec 13, 2022
Python periodic table module

elemenpy Hello! elements.py is a small Python periodic table module that is used for calling certain information about an element. Installation Instal

Eric Cheng 2 Dec 27, 2021
This project is a re-implementation of MASTER: Multi-Aspect Non-local Network for Scene Text Recognition by MMOCR

This project is a re-implementation of MASTER: Multi-Aspect Non-local Network for Scene Text Recognition by MMOCR,which is an open-source toolbox based on PyTorch. The overall architecture will be sh

Jianquan Ye 82 Nov 17, 2022
Membership Inference Attack against Graph Neural Networks

MIA GNN Project Starter If you meet the version mismatch error for Lasagne library, please use following command to upgrade Lasagne library. pip insta

6 Nov 09, 2022
Using LSTM write Tang poetry

本教程将通过一个示例对LSTM进行介绍。通过搭建训练LSTM网络,我们将训练一个模型来生成唐诗。本文将对该实现进行详尽的解释,并阐明此模型的工作方式和原因。并不需要过多专业知识,但是可能需要新手花一些时间来理解的模型训练的实际情况。为了节省时间,请尽量选择GPU进行训练。

56 Dec 15, 2022
A repository for the paper "Improved Adversarial Systems for 3D Object Generation and Reconstruction".

Improved Adversarial Systems for 3D Object Generation and Reconstruction: This is a repository for the paper "Improved Adversarial Systems for 3D Obje

Edward Smith 188 Dec 25, 2022
Learning to Initialize Neural Networks for Stable and Efficient Training

GradInit This repository hosts the code for experiments in the paper, GradInit: Learning to Initialize Neural Networks for Stable and Efficient Traini

Chen Zhu 124 Dec 30, 2022