Introduction

• We build GV-B(General Vision Benchmark) on Classification, Detection, Segmentation and Depth Estimation including 26 datasets for model evaluation.
• It is recommended to evaluate with low-data regime, using only 10% training data.
• The parameters of model backbone will be frozen during training, as known as 'linear probe'.
• Face Detection and Depth Estimation is not provided for now, you may evaluate via official repo if needed.
• Specifically, we use central_model.py in our repo to represent the implementation of Up-G models.

Task Supported

• Object Classification
• Object Detection (VOC Detection)
• Pedestrian Detection (CityPersons Detection)
• Semantic Segmentation (VOC Segmentation)
• Face Detection (WiderFace Detection)
• Depth Estimation (Kitti/NYU-v2 Depth Estimation)

Installation

Requirements

• Python 3.6+
• PyTorch 1.8+
• openmim==0.1.5
• MMCV>=1.3.16

Install Dependencies

a. Create a conda virtual environment and activate it.

conda create -n open-mmlab python=3.8 -y
conda activate open-mmlab

b. Install PyTorch and torchvision following the official instructions, e.g.：

conda install pytorch torchvision -c pytorch

Make sure that your compilation CUDA version and runtime CUDA version match.
You can check the supported CUDA version for precompiled packages on the
[PyTorch website](https://pytorch.org/).

c. Install openmm package via pip (mmcls, mmdet, mmseg)：

pip install mmcls
pip install mmdet
pip install mmsegmetation

Usage

This section provide basic tutorials about the usage of GV-B.

Prepare datasets

For each evaluation task, you can follow the official repo tutorial for data preparation.

mmclassification

mmdetection

mmsegmentation

Model evaluation

We use MIM to submit evaluation in GV-B.

a.If you run MMClassification on a cluster managed with slurm, you can use the script mim_slurm_train.sh. (This script also supports single machine training.)

sh tools/mim_slurm_train.sh $PARTITION $TASK $CONFIG $WORK_DIR

b.If you run on w/o slurm. (More details can be found in docs of openmim)

PYTHONPATH='.':$PYTHONPATH mim train $TASK $CONFIG $WORK_DIR

• PARTITION: The partition you are using
• WORK_DIR: The directory to save logs and checkpoints
• CONFIG: Config files corresponding to tasks.

Detailed Tutorials

Currently, we provide tutorials for users.

• add new modules
• add new dataset
• learn about config
• design data pipeline
• customize schedule
• customize runtime settings

Benchmark(with Hyperparameter searching)