RefineNet: Multi-Path Refinement Networks for High-Resolution Semantic Segmentation

Overview

Multipath RefineNet

A MATLAB based framework for semantic image segmentation and general dense prediction tasks on images.

This is the source code for the following paper and its extension:

  1. RefineNet: Multi-Path Refinement Networks for High-Resolution Semantic Segmentation; CVPR 2017
    https://arxiv.org/abs/1611.06612
  2. RefineNet extension in TPAMI 2019: DOI Link

Pytorch implementation

This codebase only provides MATLAB and MatConvNet based implementation.

Vladimir Nekrasov kindly provides a Pytorch implementation and a light-weight version of RefineNet at:
https://github.com/DrSleep/refinenet-pytorch

Update notes

  • 23 Dec 2016: We did a major update of our code.
  • (new!) 13 Feb 2018:
    1. Multi-scale prediction and evaluation code are added. We add demo files for multi-scale prediction, fusion and evaluation. Please refer to the Testing section below for more details.
    2. New models available: trained models using improved residual pooling. Available for these datasets: NYUDv2, Person_Parts, PASCAL_Context, SUNRGBD, ADE20k. These models will give better performance than the reported results in our CVPR paper.
    3. New models available: trained models using ResNet-152 for all 7 datasets. Apart from ResNet-101 based models, our ResNet-152 based models of all 7 datasets are now available for download.
    4. Updated trained model for VOC2012: this updated model is slightly better than the previous one. We previously uploaded a wrong model.
    5. All models are now available in Google Drive and Baidu Pan.
    6. More details are provided on testing, training and implementation. Please refer to Important notes in each section below.

Results

  • Results on the CityScapes Dataset (single scale prediction using ResNet-101 based RefineNet) RefineNet Results on the CityScapes Dataset

Trained models

  • (new!) Trained models for the following datasets are available for download.
  1. PASCAL VOC 2012
  2. Cityscapes
  3. NYUDv2
  4. Person_Parts
  5. PASCAL_Context
  6. SUNRGBD
  7. ADE20k
  • Downloads for the above datasets. Put the downloaded models in ./model_trained/
  • Important notes:
    • For the test set performance of our method on the dataset PASCAl VOC and Cityscapes, kindly note that we do not use any images in the validation set for training. Our models are trained only using the training set images.
    • The trained models of the the following datasets are using improved residual pooling: NYUDv2, Person_Parts, PASCAL_Context, SUNRGBD, ADE20k. These models will give better performance than the reported results in our CVPR paper. Please also refer to the Network architecture section below for more details about improved pooling.
    • The model for VOC2012 is updated. We previously uploaded a wrong model.

Network architecture and implementation

  • You can find the network graphs that illustrate our architecture in the folder net_graphs. Please refer to our paper for more details.
  • We include in this folder the details of improved residual pooling which improves the residual pooling block described in our CVPR paper.
  • Important notes:
    • In our up-sampling and fusion layer, we simply use down-sampling for gradient back-propagation. Please refer to the implementation of our fusion layer for details: My_sum_layer.m.
    • please refer to our training demo files for more details on implementation

Installation

  • Install MatConvNet and CuDNN. We have modified MatConvNet for our task. A modified copy of MatConvNet is provided in ./lib/. You need to compile the provided MatConvNet before running. Details of this modification and compiling can be found in main/my_matconvnet_resnet/README.md.

  • An example script for exporting lib paths is main/my_matlab.sh

  • Download the following ImageNet pre-trained models and place them in ./model_trained/:

    • imagenet-resnet-50-dag, imagenet-resnet-101-dag, imagenet-resnet-152-dag.

    They can be downloaded from: MatConvNet, we also have a copy in Google Drive, Baidu Pan.

Testing

1. Multi-scale prediction and evaluation (new!)

  • First download the trained models and put them in ./model_trained/. Please refer to the above section Trained Models.

  • Then refer to the below example scripts for prediction on your images:

    • demo_predict_mscale_[dataset name].m
    • e.g., demo_predict_mscale_voc.m, demo_predict_mscale_nyud, demo_predict_mscale_person_parts
  • You may need to carefully read through the comments in these demo scripts before using.

  • Important notes:

    • In the default setting, the example scripts will perform multi-scale prediction and fuse multi-scale results to generate final prediction.
    • The generated masks and scores maps will be saved in your disk. Note that the score maps are saved in the format of uint8 with values in [0 255]. You need to cast them into double and normalize into [0 1] if you want to use them.
    • The above demo files are able to perform multi-scale prediction and evaluation (e.g., in terms of IoU scores) in a single run. However, in the default setting, the performance evaluation part is disabled. Please refer to the comments in the demo files to turn on the performance evaluation.
    • Trained models using improved residual pooling will give better performance than the reported results in our CVPR paper. Please refer to the above section Trained models for more details.
    • For the images from NYUDv2 dataset, you may need to remove the white borders of the images before applying our models. More details and crop tools can be found in the NYUDv2 dataset webpage.

2. Single scale prediction and evaluation

  • Single scale prediction and evaluation can be done by changing the scale setting in the multi-scale prediction demo files. Please refer the the above section for multi-scale prediction.

  • We also provide simplified demo files for prediction with much less configurations. They are only for single scale prediction. Examples can be found at: demo_test_simple_voc.m and demo_test_simple_city.m.

3. Evaluation and fusion on saved results (score map files and mask files) (new!)

  • We provide an example script to perform multi-scale fusion on a number of predictions (score maps) saved in your disk:
    • demo_fuse_saved_prediction_voc.m : fuse multiple cached predictions to generate the final prediction
  • We provide an example script to evaluate the prediction masks saved in your disk:
    • demo_evaluate_saved_prediction_voc.m : evaluate the segmentation performance, e.g., in terms of IoU scores.

Training

  • The following demo files are provided for training a RefineNet on your own dataset. Please carefully read through the comments in the demo files before using this training code.
    • demo_refinenet_train.m
    • demo_refinenet_train_reduce_learning_rate.m
  • Important notes:
    • We use step-wise policy to reduce learning rate, and more importantly, you need to manually reduce the learning rate during the training stage. The setting of maximum training iteration just serves as a simple example and it should be adapted to your datasets. More details can be found in the comments of the training demo files.
    • We use the improved version of chained pooling in this training code, which may achieve better result than using the above provided models.

Citation

If you find the code useful, please cite our work as

@inproceedings{Lin:2017:RefineNet,
  title = {Refine{N}et: {M}ulti-Path Refinement Networks for High-Resolution Semantic Segmentation},
  shorttitle = {RefineNet: Multi-Path Refinement Networks},
  booktitle = {CVPR},
  author = {Lin, G. and Milan, A. and Shen, C. and Reid, I.},
  month = jul,
  year = {2017}
}

and

@article{lin2019refinenet,
  title={RefineNet: Multi-Path Refinement Networks for Dense Prediction},
  author={Lin, Guosheng and Liu, Fayao and Milan, Anton and Shen, Chunhua and Reid, Ian},
  journal={IEEE Transactions on Pattern Analysis and Machine Intelligence}, 
  year={2019},
  publisher={IEEE},
  doi={10.1109/TPAMI.2019.2893630}, 
}

License

For academic usage, the code is released under the permissive BSD license. For any commercial purpose, please contact the authors.

Owner
Guosheng Lin
Guosheng Lin
TensorFlow (Python) implementation of DeepTCN model for multivariate time series forecasting.

DeepTCN TensorFlow TensorFlow (Python) implementation of multivariate time series forecasting model introduced in Chen, Y., Kang, Y., Chen, Y., & Wang

Flavia Giammarino 21 Dec 19, 2022
This repo provides the official code for TransBTS: Multimodal Brain Tumor Segmentation Using Transformer (https://arxiv.org/pdf/2103.04430.pdf).

TransBTS: Multimodal Brain Tumor Segmentation Using Transformer This repo is the official implementation for TransBTS: Multimodal Brain Tumor Segmenta

Raymond 247 Dec 28, 2022
This repository contains all data used for writing a research paper Multiple Object Trackers in OpenCV: A Benchmark, presented in ISIE 2021 conference in Kyoto, Japan.

OpenCV-Multiple-Object-Tracking Python is version 3.6.7 to install opencv: pip uninstall opecv-python pip uninstall opencv-contrib-python pip install

6 Dec 19, 2021
RATE: Overcoming Noise and Sparsity of Textual Features in Real-Time Location Estimation (CIKM'17)

RATE: Overcoming Noise and Sparsity of Textual Features in Real-Time Location Estimation This is the implementation of RATE: Overcoming Noise and Spar

Yu Zhang 5 Feb 10, 2022
⚡️Optimizing einsum functions in NumPy, Tensorflow, Dask, and more with contraction order optimization.

Optimized Einsum Optimized Einsum: A tensor contraction order optimizer Optimized einsum can significantly reduce the overall execution time of einsum

Daniel Smith 653 Dec 30, 2022
The ICS Chat System project for NYU Shanghai Fall 2021

ICS_Chat_System [Catenger] This is the ICS Chat System project for NYU Shanghai Fall 2021 Creators: Shavarsh Melikyan, Skyler Chen and Arghya Sarkar,

1 Dec 20, 2021
Deep Learning Interviews book: Hundreds of fully solved job interview questions from a wide range of key topics in AI.

This book was written for you: an aspiring data scientist with a quantitative background, facing down the gauntlet of the interview process in an increasingly competitive field. For most of you, the

4.1k Dec 28, 2022
Align and Prompt: Video-and-Language Pre-training with Entity Prompts

ALPRO Align and Prompt: Video-and-Language Pre-training with Entity Prompts [Paper] Dongxu Li, Junnan Li, Hongdong Li, Juan Carlos Niebles, Steven C.H

Salesforce 127 Dec 21, 2022
The-Secret-Sharing-Schemes - This interactive script demonstrates the Secret Sharing Schemes algorithm

The-Secret-Sharing-Schemes This interactive script demonstrates the Secret Shari

Nishaant Goswamy 1 Jan 02, 2022
A Confidence-based Iterative Solver of Depths and Surface Normals for Deep Multi-view Stereo

idn-solver Paper | Project Page This repository contains the code release of our ICCV 2021 paper: A Confidence-based Iterative Solver of Depths and Su

zhaowang 43 Nov 17, 2022
an Evolutionary Algorithm assisted GAN

EvoGAN an Evolutionary Algorithm assisted GAN ckpts

3 Oct 09, 2022
JAXDL: JAX (Flax) Deep Learning Library

JAXDL: JAX (Flax) Deep Learning Library Simple and clean JAX/Flax deep learning algorithm implementations: Soft-Actor-Critic (arXiv:1812.05905) Transf

Patrick Hart 4 Nov 27, 2022
Code for the Higgs Boson Machine Learning Challenge organised by CERN & EPFL

A method to solve the Higgs boson challenge using Least Squares - Novae This project is the Project 1 of EPFL CS-433 Machine Learning. The project is

Giacomo Orsi 1 Nov 09, 2021
PyTorch implementation of Histogram Layers from DeepHist: Differentiable Joint and Color Histogram Layers for Image-to-Image Translation

deep-hist PyTorch implementation of Histogram Layers from DeepHist: Differentiable Joint and Color Histogram Layers for Image-to-Image Translation PyT

Winfried Lötzsch 10 Dec 06, 2022
Contour-guided image completion with perceptual grouping (BMVC 2021 publication)

Contour-guided Image Completion with Perceptual Grouping Authors Morteza Rezanejad*, Sidharth Gupta*, Chandra Gummaluru, Ryan Marten, John Wilder, Mic

Sid Gupta 6 Dec 27, 2022
Self-Supervised depth kalilia

Self-Supervised depth kalilia

24 Oct 15, 2022
Garbage Detection system which will detect objects based on whether it is plastic waste or plastics or just garbage.

Garbage Detection using Yolov5 on Jetson Nano 2gb Developer Kit. Garbage detection system which will detect objects based on whether it is plastic was

Rishikesh A. Bondade 2 May 13, 2022
Detecting and Tracking Small and Dense Moving Objects in Satellite Videos: A Benchmark

This dataset is a large-scale dataset for moving object detection and tracking in satellite videos, which consists of 40 satellite videos captured by Jilin-1 satellite platforms.

Qingyong 87 Dec 22, 2022
Solving reinforcement learning tasks which require language and vision

Multimodal Reinforcement Learning JAX implementations of the following multimodal reinforcement learning approaches. Dual-coding Episodic Memory from

Henry Prior 31 Feb 26, 2022
Framework for abstracting Amiga debuggers and access to AmigaOS libraries and devices.

Framework for abstracting Amiga debuggers. This project provides abstration to control an Amiga remotely using a debugger. The APIs are not yet stable

Roc Vallès 39 Nov 22, 2022