TransTrack: Multiple Object Tracking with Transformer

• (22/02/2022) Multi-GPU testing is supported.
• (29/10/2021) Automatic Mixed Precision(AMP) training is supported.
• (28/04/2021) Higher performance is reported by training on mixture of CrowdHuman and MOT, instead of first CrowdHuman then MOT.
• (28/04/2021) Higher performance is reported by pre-training both detection and tracking on CrowdHuman, instead of only detection.
• (28/04/2021) Higher performance is reported by increasing the number of queries from 300 to 500.
• (08/04/2021) Refactoring the code.

If download link is invalid, models and logs are also available in Github Release and Baidu Drive by code m4iv.

• We observe about 1 MOTA noise.
• If the resulting MOTA of your self-trained model is not desired, playing around with the --track_thresh sometimes gives a better performance.
• The default track_thresh is 0.4, except for 0.5 in crowdhuman.
• The training time is on 8 NVIDIA V100 GPUs with batchsize 16.
• We use the models pre-trained on imagenet.
• (crowdhuman, mot17_half) is first training on crowdhuman, then fine-tuning on mot17_half.

The codebases are built on top of Deformable DETR and CenterTrack.

• Linux, CUDA>=9.2, GCC>=5.4
• Python>=3.7
• PyTorch ≥ 1.5 and torchvision that matches the PyTorch installation. You can install them together at pytorch.org to make sure of this
• OpenCV is optional and needed by demo and visualization

1. Install and build libs

git clone https://github.com/PeizeSun/TransTrack.git
cd TransTrack
cd models/ops
python setup.py build install
cd ../..
pip install -r requirements.txt

2. Prepare datasets and annotations

mkdir crowdhuman
cp -r /path_to_crowdhuman_dataset/CrowdHuman_train crowdhuman/CrowdHuman_train
cp -r /path_to_crowdhuman_dataset/CrowdHuman_val crowdhuman/CrowdHuman_val
mkdir mot
cp -r /path_to_mot_dataset/train mot/train
cp -r /path_to_mot_dataset/test mot/test

CrowdHuman dataset is available in CrowdHuman.

python3 track_tools/convert_crowdhuman_to_coco.py

MOT dataset is available in MOT.

python3 track_tools/convert_mot_to_coco.py

3. Pre-train on crowdhuman

sh track_exps/crowdhuman_train.sh
python3 track_tools/crowdhuman_model_to_mot.py

The pre-trained model is available crowdhuman_final.pth.

4. Train TransTrack

sh track_exps/crowdhuman_mot_trainhalf.sh

5. Evaluate TransTrack

sh track_exps/mot_val.sh
sh track_exps/mota.sh

6. Visualize TransTrack

python3 track_tools/txt2video.py

• Performance on test set is evaluated by MOT challenge.
• (crowdhuman + mot17) is training on mixture of crowdhuman and mot17.
• We won't release trained models for test test. Running as in Steps could reproduce them.

1. Train TransTrack

sh track_exps/crowdhuman_mot_train.sh

or

1. Mix crowdhuman and mot17

mkdir -p mix/annotations
cp mot/annotations/val_half.json mix/annotations/val_half.json
cp mot/annotations/test.json mix/annotations/test.json
cd mix
ln -s ../mot/train mot_train
ln -s ../crowdhuman/CrowdHuman_train crowdhuman_train
cd ..
python3 track_tools/mix_data.py

2. Train TransTrack

sh track_exps/crowdhuman_plus_mot_train.sh

TransTrack is released under MIT License.

If you use TransTrack in your research or wish to refer to the baseline results published here, please use the following BibTeX entries:

@article{transtrack,
  title   =  {TransTrack: Multiple-Object Tracking with Transformer},
  author  =  {Peize Sun and Jinkun Cao and Yi Jiang and Rufeng Zhang and Enze Xie and Zehuan Yuan and Changhu Wang and Ping Luo},
  journal =  {arXiv preprint arXiv: 2012.15460},
  year    =  {2020}
}