Bird's-Eye-View Panoptic Segmentation Using Monocular Frontal View Images

This repository contains the PyTorch implementation of the PanopticBEV model proposed in our RA-L 2021 paper Bird's-Eye-View Panoptic Segmentation Using Monocular Frontal View Images.

Our approach, PanopticBEV, is the state-of-the-art approach for generating panoptic segmentation maps in the bird's eye view using only monocular frontal view images.

If you find this code useful for your research, please consider citing our paper:

@article{gosala2021bev,
  title={Bird's-Eye-View Panoptic Segmentation Using Monocular Frontal View Images},
  author={Gosala, Nikhil and Valada, Abhinav},
  journal={arXiv preprint arXiv:2108.03227},
  year={2021}
}

Relevant links

• Paper
• Video
• Project Webpage

System requirements

• Linux (Tested on Ubuntu 18.04)
• Python3 (Tested using Python 3.6.9)
• PyTorch (Tested using PyTorch 1.8.1)
• CUDA (Tested using CUDA 11.1)

Installation

a. Create a python virtual environment and activate it.

python3 -m venv panoptic_bev
source panoptic_bev/bin/activate

b. Update pip to the latest version.

python3 -m pip install --upgrade pip

c. Install the required python dependencies using the provided requirements.txt file.

pip3 install -r requirements.txt

d. Install the PanopticBEV code.

python3 setup.py develop

Obtaining the datasets

Please download the datasets from here and follow the instructions provided in the encapsulated readme file.

Code Execution

Configuration parameters

The configuration parameters of the model such as the learning rate, batch size, and dataloader options are stored in the experiments/config folder. If you intend to modify the model parameters, please do so here.

Training and Evaluation

The training and evaluation python codes along with the shell scripts to execute them are provided in the scripts folder. Before running the shell scripts, please fill in the missing parameters with your computer-specific data paths and parameters.

To train the model, execute the following command after replacing * with either kitti or nuscenes.

bash train_panoptic_bev_*.sh

To evaluate the model, execute the following command after replacing * with either kitti or nuscenes.

bash eval_panoptic_bev_*.sh 

Acknowledgements

This work was supported by the Federal Ministry of Education and Research (BMBF) of Germany under ISA 4.0 and by the Eva Mayr-Stihl Stiftung.

This project contains code adapted from other open-source projects. We especially thank the authors of:

• Mapillary Seamseg
• Yet-Another-EfficientDet-Pytorch

License

This code is released under the GPLv3 for academic usage. For commercial usage, please contact Nikhil Gosala.