Official repository for the ISBI 2021 paper Transformer Assisted Convolutional Neural Network for Cell Instance Segmentation

Overview

SegPC-2021

This is the official repository for the ISBI 2021 paper Transformer Assisted Convolutional Neural Network for Cell Instance Segmentation by Deepanshu Pandey, Pradyumna Gupta, Sumit Bhattacharya, Aman Sinha, Rohit Agarwal.

About the Challenge

Website link: https://segpc-2021.grand-challenge.org/SegPC-2021/

Overview: In recent years, with the advancement of Deep Learning, there has been tremendous efforts in the application of image processing to build AI based models for cancer diagnosis. This challenge is also one such effort. It deals with the segmentation of plasma cell cancer, namely, Multiple Myeloma (MM), which is a type of blood cancer. We are provided with the stained color normalization imag of the cells.

Image distribution: The cells may have different structures because:

  • There is a varying amount of nucleus and cytoplasm from one cell to another.
  • The cells may appear in clusters or as isolated single cells.
  • The cells appearing in clusters may have three cases -
    • Cytoplasm of two cells touch each other,
    • The cytoplasm of one cell and nucleus of another touch each other,
    • nucleus of cells touch each other. Since the cytoplasm and nucleus have different colors, the segmentation of cells may pose challenges.
  • There may be multiple cells touching each other in the cluster.
  • There may be unstained cells, say a red blood cell underneath the cell of interest, changing its color and shade.
  • The cytoplasm of a cell may be close to the background of the whole image, making it difficult to identify the boundary of the cell and segment it.

Getting Started

We recommend using Python 3.7 for running the scripts in this repository. The necessary packages can be installed using requirements.txt in the respective folders. Since all of our work has been done on Google Colaboratory, the requirements.txt may have more packages/modules than is actually required and it might take quite long to install everything. Hence,for such a case, the folders of both the models also contain an essential-requirements.txt file which contains some essential packages that need to installed beforehand, while the other fundamental packages can be installed later as their need shows up as an error when running the given training and inference scripts.

To clone this repository:

$ git clone https://github.com/dsciitism/SegPC-2021

To run this repository, following the given steps using the sections mentioned in the subsequent sections:

  1. Prepare the data in COCO format
  2. Run the training script for Cascade Mask RCNN / DetectoRS
  3. Run the inference script for Cascade Mask RCNN / DetectoRS
  4. Run the ensemble script

Data Preparation

Note : This step is not required for inference.

All the models present in the paper require data in COCO format to train. Hence , to train the models the images and masks need to be resized and a json file in COCO format is required. The dataset_preparation.py script in the utils folder can be used to perform these tasks. The following flags need to be used for running the dataset_preparation.py script:

usage: dataset_preparation.py [-h] --img_root IMG_ROOT --mask_root MASK_ROOT --dest_root DEST_ROOT

arguments:
  -h, --help            show this help message and exit
  --img_root IMG_ROOT   path to the folder where the images are saved
  --mask_root MASK_ROOT
                        path to the folder where gt instances are saved
  --dest_root DEST_ROOT
                        path to the folder where the COCO format json file and resized masks and images will be saved

Cascade Mask RCNN

For installation of required packages:

$ cat Cascade_Mask_RCNN_X152/requirements.txt | xargs -n 1 pip3 install

Train

The following flags need to be used to run CMRCNN_X152_train.py:

usage: CMRCNN_X152_train.py [-h] --backbone {Original,Effb5,Transformer_Effb5} --train_data_root TRAIN_DATA_ROOT 
--training_json_path TRAINING_JSON_PATH --val_data_root VAL_DATA_ROOT --validation_json_path VALIDATION_JSON_PATH 
--work_dir WORK_DIR [--iterations ITERATIONS] [--batch_size BATCH_SIZE]

arguments:
  -h, --help            show this help message and exit
  --backbone {Original,Effb5,Transformer_Effb5}
                        The backbone to be used from the given choices
  --train_data_root TRAIN_DATA_ROOT
                        path to training data root folder
  --training_json_path TRAINING_JSON_PATH
                        path to the training json file in COCO format
  --val_data_root VAL_DATA_ROOT
                        path to validation data root folder
  --validation_json_path VALIDATION_JSON_PATH
                        path to validation json file in COCO format
  --work_dir WORK_DIR   path to the folder where models and logs will be saved
  --iterations ITERATIONS
  --batch_size BATCH_SIZE

Inference

The following flags need to be used while running CMRCNN_X152_inference.py:

usage: CMRCNN_X152_inference.py [-h] --backbone {Original,Effb5,Transformer_Effb5} 
--saved_model_path SAVED_MODEL_PATH --input_images_folder INPUT_IMAGES_FOLDER --save_path SAVE_PATH

arguments:
  -h, --help            show this help message and exit
  --backbone {Original,Effb5,Transformer_Effb5}
                        The backbone to be used from the given choices
  --saved_model_path SAVED_MODEL_PATH
                        path to the saved model which will be loaded
  --input_images_folder INPUT_IMAGES_FOLDER
                        path to the folder where images to inference on are
                        kept
  --save_path SAVE_PATH
                        path to the folder where the generated masks will be
                        saved

DetectoRS

Preparation script should be run with the following command before running any other file in the DetectoRS folder :

$ bash mmdetection_preparation.sh

For installation of required packages:

$ cat DetectoRS/requirements.txt | xargs -n 1 pip3 install

Train

The following flags need to be used while running DetectoRS_train.py:

usage: DetectoRS_train.py [-h] --backbone {Original,Effb5,Transformer_Effb5} --train_data_root TRAIN_DATA_ROOT 
--training_json_path TRAINING_JSON_PATH [--train_img_prefix TRAIN_IMG_PREFIX] [--train_seg_prefix TRAIN_SEG_PREFIX] 
--val_data_root VAL_DATA_ROOT --validation_json_path VALIDATION_JSON_PATH [--val_img_prefix VAL_IMG_PREFIX] 
[--val_seg_prefix VAL_SEG_PREFIX] --work_dir WORK_DIR [--epochs EPOCHS] [--batch_size BATCH_SIZE]

arguments:
  -h, --help            show this help message and exit
  --backbone {Original,Effb5,Transformer_Effb5}
                        The backbone to be used from the given choices
  --train_data_root TRAIN_DATA_ROOT
                        path to training data root folder
  --training_json_path TRAINING_JSON_PATH
                        path to the training json file in COCO format
  --train_img_prefix TRAIN_IMG_PREFIX
                        prefix path ,if any, to be added to the train_data_root path to access the input images
  --train_seg_prefix TRAIN_SEG_PREFIX
                        prefix path ,if any, to be added to the train_data_root path to access the semantic masks
  --val_data_root VAL_DATA_ROOT
                        path to validation data root folder
  --validation_json_path VALIDATION_JSON_PATH
                        path to validation json file in COCO format
  --val_img_prefix VAL_IMG_PREFIX
                        prefix path ,if any, to be added to the val_data_root path to access the input images
  --val_seg_prefix VAL_SEG_PREFIX
                        prefix path ,if any, to be added to the val_data_root path to access the semantic masks
  --work_dir WORK_DIR   path to the folder where models and logs will be saved
  --epochs EPOCHS
  --batch_size BATCH_SIZE

Note: DetectoRS requires semantic masks along with instance masks during training , hence the arguments - train_seg_prefix and val_seg_prefix

Inference

The following flags need to be used while running DetectoRS_inference.py:

usage: DetectoRS_inference.py [-h] --backbone {Original,Effb5,Transformer_Effb5} 
--saved_model_path SAVED_MODEL_PATH --input_images_folder INPUT_IMAGES_FOLDER --save_path SAVE_PATH

arguments:
  -h, --help            show this help message and exit
  --backbone {Original,Effb5,Transformer_Effb5}
                        The backbone to be used from the given choices
  --saved_model_path SAVED_MODEL_PATH
                        path to the saved model which will be loaded
  --input_images_folder INPUT_IMAGES_FOLDER
                        path to the folder where images to inference on are kept
  --save_path SAVE_PATH
                        path to the folder where the generated masks will be saved

Ensemble

Apart from the individual models, the paper also presents the scores of ensemble of any three models. The ensemble.py script in the utils folder can be used for making ensemble of the outputs of three models , using the following flags :

usage: ensemble.py [-h] --model1_predictions MODEL1_PREDICTIONS --model2_predictions MODEL2_PREDICTIONS --model3_predictions MODEL3_PREDICTIONS --final_predictions FINAL_PREDICTIONS

arguments:
  -h, --help            show this help message and exit
  --model1_predictions MODEL1_PREDICTIONS
                        path to the predictions of first model
  --model2_predictions MODEL2_PREDICTIONS
                        path to the predictions of second model
  --model3_predictions MODEL3_PREDICTIONS
                        path to the predictions of third model
  --final_predictions FINAL_PREDICTIONS
                        path where the ensembled outputs should be saved

Results and Models

Method Backbone mIoU Download
Cascade Mask R-CNN Original(ResNet) 0.9179 model
DetectoRS Original(ResNet) 0.9219 model
Cascade Mask R-CNN EfficientNet-b5 0.8793 model
DetectoRS EfficientNet-b5 0.9038 model
Cascade Mask R-CNN EfficientNet-b5+ViT 0.9281 model
DetectoRS EfficientNet-b5+ViT 0.9273 model
Owner
Datascience IIT-ISM
Datascience IIT-ISM
Code for Recurrent Mask Refinement for Few-Shot Medical Image Segmentation (ICCV 2021).

Recurrent Mask Refinement for Few-Shot Medical Image Segmentation Steps Install any missing packages using pip or conda Preprocess each dataset using

XIE LAB @ UCI 39 Dec 08, 2022
Code for paper "Do Language Models Have Beliefs? Methods for Detecting, Updating, and Visualizing Model Beliefs"

This is the codebase for the paper: Do Language Models Have Beliefs? Methods for Detecting, Updating, and Visualizing Model Beliefs Directory Structur

Peter Hase 19 Aug 21, 2022
使用yolov5训练自己数据集(详细过程)并通过flask部署

使用yolov5训练自己的数据集(详细过程)并通过flask部署 依赖库 torch torchvision numpy opencv-python lxml tqdm flask pillow tensorboard matplotlib pycocotools Windows,请使用 pycoc

HB.com 19 Dec 28, 2022
Parametric Contrastive Learning (ICCV2021)

Parametric-Contrastive-Learning This repository contains the implementation code for ICCV2021 paper: Parametric Contrastive Learning (https://arxiv.or

DV Lab 156 Dec 21, 2022
Generic U-Net Tensorflow implementation for image segmentation

Tensorflow Unet Warning This project is discontinued in favour of a Tensorflow 2 compatible reimplementation of this project found under https://githu

Joel Akeret 1.8k Dec 10, 2022
Qcover is an open source effort to help exploring combinatorial optimization problems in Noisy Intermediate-scale Quantum(NISQ) processor.

Qcover is an open source effort to help exploring combinatorial optimization problems in Noisy Intermediate-scale Quantum(NISQ) processor. It is devel

33 Nov 11, 2022
Deep Implicit Moving Least-Squares Functions for 3D Reconstruction

DeepMLS: Deep Implicit Moving Least-Squares Functions for 3D Reconstruction This repository contains the implementation of the paper: Deep Implicit Mo

103 Dec 22, 2022
(CVPR 2022 - oral) Multi-View Depth Estimation by Fusing Single-View Depth Probability with Multi-View Geometry

Multi-View Depth Estimation by Fusing Single-View Depth Probability with Multi-View Geometry Official implementation of the paper Multi-View Depth Est

Bae, Gwangbin 138 Dec 28, 2022
Code & Experiments for "LILA: Language-Informed Latent Actions" to be presented at the Conference on Robot Learning (CoRL) 2021.

LILA LILA: Language-Informed Latent Actions Code and Experiments for Language-Informed Latent Actions (LILA), for using natural language to guide assi

Sidd Karamcheti 11 Nov 25, 2022
Collection of in-progress libraries for entity neural networks.

ENN Incubator Collection of in-progress libraries for entity neural networks: Neural Network Architectures for Structured State Entity Gym: Abstractio

25 Dec 01, 2022
Deep deconfounded recommender (Deep-Deconf) for paper "Deep causal reasoning for recommendations"

Deep Causal Reasoning for Recommender Systems The codes are associated with the following paper: Deep Causal Reasoning for Recommendations, Yaochen Zh

Yaochen Zhu 22 Oct 15, 2022
Machine Learning in Asset Management (by @firmai)

Machine Learning in Asset Management If you like this type of content then visit ML Quant site below: https://www.ml-quant.com/ Part One Follow this l

Derek Snow 1.5k Jan 02, 2023
Implement of homography net by pytorch

HomographyNet Implement of homography net by pytorch Brief Introduction This project is based on the work Homography-Net: @article{detone2016deep, t

ronghao_CN 4 May 19, 2022
Offical implementation of Shunted Self-Attention via Multi-Scale Token Aggregation

Shunted Transformer This is the offical implementation of Shunted Self-Attention via Multi-Scale Token Aggregation by Sucheng Ren, Daquan Zhou, Shengf

156 Dec 27, 2022
CVPR '21: In the light of feature distributions: Moment matching for Neural Style Transfer

In the light of feature distributions: Moment matching for Neural Style Transfer (CVPR 2021) This repository provides code to recreate results present

Nikolai Kalischek 49 Oct 13, 2022
keyframes-CNN-RNN(action recognition)

keyframes-CNN-RNN(action recognition) Environment: python=3.7 pytorch=1.2 Datasets: Following the format of UCF101 action recognition. Run steps: Mo

4 Feb 09, 2022
Sequential model-based optimization with a `scipy.optimize` interface

Scikit-Optimize Scikit-Optimize, or skopt, is a simple and efficient library to minimize (very) expensive and noisy black-box functions. It implements

Scikit-Optimize 2.5k Jan 04, 2023
A Pytorch reproduction of Range Loss, which is proposed in paper 《Range Loss for Deep Face Recognition with Long-Tailed Training Data》

RangeLoss Pytorch This is a Pytorch reproduction of Range Loss, which is proposed in paper 《Range Loss for Deep Face Recognition with Long-Tailed Trai

Youzhi Gu 7 Nov 27, 2021
PyTorch reimplementation of minimal-hand (CVPR2020)

Minimal Hand Pytorch Unofficial PyTorch reimplementation of minimal-hand (CVPR2020). you can also find in youtube or bilibili bare hand youtube or bil

Hao Meng 228 Dec 29, 2022
A PyTorch Extension: Tools for easy mixed precision and distributed training in Pytorch

Introduction This is a Python package available on PyPI for NVIDIA-maintained utilities to streamline mixed precision and distributed training in Pyto

Artit 'Art' Wangperawong 5 Sep 29, 2021