DilatedNet in Keras for image segmentation

Overview

Keras implementation of DilatedNet for semantic segmentation

A native Keras implementation of semantic segmentation according to Multi-Scale Context Aggregation by Dilated Convolutions (2016). Optionally uses the pretrained weights by the authors'.

The code has been tested on Tensorflow 1.3, Keras 1.2, and Python 3.6.

Using the pretrained model

Download and extract the pretrained model:

curl -L https://github.com/nicolov/segmentation_keras/releases/download/model/nicolov_segmentation_model.tar.gz | tar xvf -

Install dependencies and run:

pip install -r requirements.txt
# For GPU support
pip install tensorflow-gpu==1.3.0

python predict.py --weights_path conversion/converted/dilation8_pascal_voc.npy

The output image will be under images/cat_seg.png.

Converting the original Caffe model

Follow the instructions in the conversion folder to convert the weights to the TensorFlow format that can be used by Keras.

Training

Download the Augmented Pascal VOC dataset here:

curl -L http://www.eecs.berkeley.edu/Research/Projects/CS/vision/grouping/semantic_contours/benchmark.tgz | tar -xvf -

This will create a benchmark_RELEASE directory in the root of the repo. Use the convert_masks.py script to convert the provided masks in .mat format to RGB pngs:

python convert_masks.py \
    --in-dir benchmark_RELEASE/dataset/cls \
    --out-dir benchmark_RELEASE/dataset/pngs

Start training:

python train.py --batch-size 2

Model checkpoints are saved under trained/, and can be used with the predict.py script for testing.

The training code is currently limited to the frontend module, and thus only outputs 16x16 segmentation maps. The augmentation pipeline does mirroring but not cropping or rotation.


Fisher Yu and Vladlen Koltun, Multi-Scale Context Aggregation by Dilated Convolutions, 2016

Comments
  • training and validation loss nan

    training and validation loss nan

    First of all I just want to thank you for the great work. I am having an issue during training, my loss and val_loss is nan, however I am still getting values for accuracy and val_acc. I am training on the PASCAL_VOC 2012 dataset with the segmentation class pngs. rsz_screenshot_from_2017-08-09_17-24-13

    keras 1.2.1 & 2.0.6 tensorflow-gpu 1.2.1 python 3.6.1

    opened by Barfknecht 9
  • Fine tuning ...

    Fine tuning ...

    Hello,

    You have provided the pre-trained model of VOC. I have a small dataset with 2 classes, which I annotated based on VOC and I want to fine-tune it. Would you please guide me through the process?

    opened by MyVanitar 8
  • Modifying number of class

    Modifying number of class

    Hi Nicolov,

    Thanks for the great work! I tried to train new dataset by generating my own set of jpg and png masks. However I realized it only works for pre-defined 20 classes. For example I wanted to re-train this network to segment screws from background, I wasn't able to find way to add new classes but to use a existed color 0x181818 which was originally trained for cats. After training it did segmented the screw. However I'm still wondering is there any way to change the number of classes and specify which color value are associated with certain class?

    opened by francisbitontistudio 7
  • Black image after segmentation

    Black image after segmentation

    Hi! I have val accuracy = 1, but when i am trying to predict mask on the image from train set it displays me black image. Does anybody know what is the reason of this behaviour?

    opened by dimaxano 7
  • docker running error

    docker running error

    Hi, @nicolov ,

    For the caffe weight conversion, I got the following error:

    (tf_1.0) [email protected]:/data/code/segmentation_keras/conversion# docker run -v $(pwd):/workspace -ti `docker build -q .`
    Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?
    "docker run" requires at least 1 argument(s).
    See 'docker run --help'.
    
    Usage:  docker run [OPTIONS] IMAGE [COMMAND] [ARG...]
    
    Run a command in a new container
    (tf_1.0) ro[email protected]:/data/code/segmentation_keras/conversion#
    
    

    It shows that docker daemon is not running. Any other command should I input before it?

    Thanks

    opened by amiltonwong 7
  • the way of loading the weight

    the way of loading the weight

    Hi nicolov,

    In the post, you explained how to do the weight conversion. Due to the development environment constraints, it is a little bit hard for me to follow exactly your step.

    In keras blog, author also show a way to load VGG16 weight from Keras directly. Do you think this weight can be used for your implementation? Do we have to use the converted caffe model weight for pascal_voc. The data set I will be using is of different domain with the data set published in the paper. Thanks for your advice.

    capture

    opened by wenouyang 5
  • Problems with CuDNN library

    Problems with CuDNN library

    While running train.py, this is the error message: Epoch 1/20 E tensorflow/stream_executor/cuda/cuda_dnn.cc:378] Loaded runtime CuDNN library: 6021 (compatibility version 6000) but source was compiled with 5105 (compatibility version 5100). If using a binary install, upgrade your CuDNN library to match. If building from sources, make sure the library loaded at runtime matches a compatible version specified during compile configuration.

    Since I don't have the root account, I can't install CuDNN v5. Do you know how I can fix this? Thanks!

    opened by Yuren-Zhong 4
  • IoU results

    IoU results

    Have you by any chance compared this to the original implementation with regards to the mean IoU? If so, what implementation of IoU did you use and what were your results?

    opened by Barfknecht 4
  • about the required pre-trained vgg model

    about the required pre-trained vgg model

    Hi, @nicolov ,

    According to this line, vgg_conv.npy is needed as pre-trained vgg model in training. Could you list the download location for corresponding caffemodel and prototxt file? And, is the conversion step the same as here?

    Thanks!

    opened by amiltonwong 4
  • regarding loading_weights

    regarding loading_weights

    Hi nicolov,

    In the train.py, you have included the function of load_weights(model, weights_path):. My understanding is that you are trying to load a pre-training vcg model. If I do not want to use this pretrained model because the problem I am working one may belong to a totally different domain, should I just skip calling this load_weights function? Or using a pre-trained model is always preferable, I am kind of confusing about this.

    In the notes, you mentioned that The training code is currently limited to the frontend module, and thus only outputs 16x16 segmentation maps. If I would like to leverage this code for my own data set, what are the modifications that I have to make? Do I still have to load the weights?

    Thank you very much!

    opened by wenouyang 4
  • Cannot locate Dockerfile: Dockerfile

    Cannot locate Dockerfile: Dockerfile

    Probably a rookie error but when I am trying to run the conversion step in conversion by running the docker I get the following error:

    $sudo docker run -v $(pwd):/workspace -ti `docker build -q .`
    time="2017-02-09T09:15:11-08:00" level=fatal msg="Cannot locate Dockerfile: Dockerfile" 
    docker: "run" requires a minimum of 1 argument. See 'docker run --help'.
    
    opened by mongoose54 4
  • Training freezes

    Training freezes

    On executing command: python train.py --batch-size 2 ,training freezes at last step of first epoch.

    All the libraries are according to the requirement.txt file

    opened by ghost 1
  • AtrousConvolution2D vs.Conv2DTranspose

    AtrousConvolution2D vs.Conv2DTranspose

    Hi @nicolov I was wondering whether in your model, you wouldn't need to have a Conv2DTranspose or Upsample layer to compensate for the maxpool and obtain predictions with the same size as your input image?

    opened by tinalegre 0
  • How to handle high resolution  images

    How to handle high resolution images

    Hello @nicolov ,

    let me first express my appreciation to your work in image segmentation its great (Y)

    small suggestion , i just want to notify you that there is a missing -- in input parsing . very minor change

    parser.add_argument('--input_path', nargs='?', default='images/cat.jpg',
                            help='Required path to input image') 
    

    I'm hoping you can help me in understanding how to handle high res images as 1028 and 4k ,

    also in the code i found you set input_width, input_height = 900, 900 and label_margin = 186 can you please illustrate what is the reason for this static number and how they effect on the output high and width

    output_height = input_height - 2 * label_margin
    output_width = input_width - 2 * label_margin
    
    opened by engahmed1190 2
  • Context module training implementation plans

    Context module training implementation plans

    Thanks for creating this implementation. Do you have any plans to implement training of the context module (to allow producing full resolution segmentation maps)?

    opened by OliverColeman 3
  • palette conversion not needed

    palette conversion not needed

    https://github.com/nicolov/segmentation_keras/blob/master/convert_masks.py isn't necessary.

    Just use Pillow and you can load the classes separately from the color palette, which means it will already be in the format you want!

    from https://github.com/aurora95/Keras-FCN/blob/master/utils/SegDataGenerator.py#L203

                    from PIL import Image
                    label = Image.open(label_filepath)
                    if self.save_to_dir and self.palette is None:
                        self.palette = label.palette
    

    cool right?

    opened by ahundt 6
Releases(caffemodel)
A Context-aware Visual Attention-based training pipeline for Object Detection from a Webpage screenshot!

CoVA: Context-aware Visual Attention for Webpage Information Extraction Abstract Webpage information extraction (WIE) is an important step to create k

Keval Morabia 41 Jan 01, 2023
DeepMind Alchemy task environment: a meta-reinforcement learning benchmark

The DeepMind Alchemy environment is a meta-reinforcement learning benchmark that presents tasks sampled from a task distribution with deep underlying structure.

DeepMind 188 Dec 25, 2022
Validated, scalable, community developed variant calling, RNA-seq and small RNA analysis

Validated, scalable, community developed variant calling, RNA-seq and small RNA analysis. You write a high level configuration file specifying your in

Blue Collar Bioinformatics 917 Jan 03, 2023
Transferable Unrestricted Attacks, which won 1st place in CVPR’21 Security AI Challenger: Unrestricted Adversarial Attacks on ImageNet.

Transferable Unrestricted Adversarial Examples This is the PyTorch implementation of the Arxiv paper: Towards Transferable Unrestricted Adversarial Ex

equation 16 Dec 29, 2022
Simultaneous Detection and Segmentation

Simultaneous Detection and Segmentation This is code for the ECCV Paper: Simultaneous Detection and Segmentation Bharath Hariharan, Pablo Arbelaez,

Bharath Hariharan 96 Jul 20, 2022
A new data augmentation method for extreme lighting conditions.

Random Shadows and Highlights This repo has the source code for the paper: Random Shadows and Highlights: A new data augmentation method for extreme l

Osama Mazhar 35 Nov 26, 2022
No-reference Image Quality Assessment(NIQA) Algorithms (BRISQUE, NIQE, PIQE, RankIQA, MetaIQA)

No-Reference Image Quality Assessment Algorithms No-reference Image Quality Assessment(NIQA) is a task of evaluating an image without a reference imag

Dae-Young Song 26 Jan 04, 2023
[CVPR'21] MonoRUn: Monocular 3D Object Detection by Reconstruction and Uncertainty Propagation

MonoRUn MonoRUn: Monocular 3D Object Detection by Reconstruction and Uncertainty Propagation. CVPR 2021. [paper] Hansheng Chen, Yuyao Huang, Wei Tian*

同济大学智能汽车研究所综合感知研究组 ( Comprehensive Perception Research Group under Institute of Intelligent Vehicles, School of Automotive Studies, Tongji University) 96 Dec 10, 2022
(EI 2022) Controllable Confidence-Based Image Denoising

Image Denoising with Control over Deep Network Hallucination Paper and arXiv preprint -- Our frequency-domain insights derive from SFM and the concept

Images and Visual Representation Laboratory (IVRL) at EPFL 5 Dec 18, 2022
Deep Implicit Moving Least-Squares Functions for 3D Reconstruction

DeepMLS: Deep Implicit Moving Least-Squares Functions for 3D Reconstruction This repository contains the implementation of the paper: Deep Implicit Mo

103 Dec 22, 2022
ScaleNet: A Shallow Architecture for Scale Estimation

ScaleNet: A Shallow Architecture for Scale Estimation Repository for the code of ScaleNet paper: "ScaleNet: A Shallow Architecture for Scale Estimatio

Axel Barroso 34 Nov 09, 2022
YOLOv4 / Scaled-YOLOv4 / YOLO - Neural Networks for Object Detection (Windows and Linux version of Darknet )

Yolo v4, v3 and v2 for Windows and Linux (neural networks for object detection) Paper YOLO v4: https://arxiv.org/abs/2004.10934 Paper Scaled YOLO v4:

Alexey 20.2k Jan 09, 2023
Dynamic wallpaper generator.

Wiki • About • Installation About This project is a dynamic wallpaper changer. It waits untill you turn on the music, downloads album cover if it's po

3 Sep 18, 2021
AdaFocus (ICCV 2021) Adaptive Focus for Efficient Video Recognition

AdaFocus (ICCV 2021) This repo contains the official code and pre-trained models for AdaFocus. Adaptive Focus for Efficient Video Recognition Referenc

Rainforest Wang 115 Dec 21, 2022
Scalable Graph Neural Networks for Heterogeneous Graphs

Neighbor Averaging over Relation Subgraphs (NARS) NARS is an algorithm for node classification on heterogeneous graphs, based on scalable neighbor ave

Facebook Research 67 Dec 03, 2022
Python implementation of "Single Image Haze Removal Using Dark Channel Prior"

##Dependencies pillow(~2.6.0) Numpy(~1.9.0) If the scripts throw AttributeError: __float__, make sure your pillow has jpeg support e.g. try: $ sudo ap

Joyee Cheung 73 Dec 20, 2022
[CVPR 2021] Rethinking Text Segmentation: A Novel Dataset and A Text-Specific Refinement Approach

Rethinking Text Segmentation: A Novel Dataset and A Text-Specific Refinement Approach This is the repo to host the dataset TextSeg and code for TexRNe

SHI Lab 174 Dec 19, 2022
Using pytorch to implement unet network for liver image segmentation.

Using pytorch to implement unet network for liver image segmentation.

zxq 1 Dec 17, 2021
Build and run Docker containers leveraging NVIDIA GPUs

NVIDIA Container Toolkit Introduction The NVIDIA Container Toolkit allows users to build and run GPU accelerated Docker containers. The toolkit includ

NVIDIA Corporation 15.6k Jan 01, 2023
TensorFlow (v2.7.0) benchmark results on an M1 Macbook Air 2020 laptop (macOS Monterey v12.1).

M1-tensorflow-benchmark TensorFlow (v2.7.0) benchmark results on an M1 Macbook Air 2020 laptop (macOS Monterey v12.1). I was initially testing if Tens

particle 2 Jan 05, 2022