Traffic4D: Single View Reconstruction of Repetitious Activity Using Longitudinal Self-Supervision

Related tags

Deep LearningTraffic4D-Release
Overview

Traffic4D: Single View Reconstruction of Repetitious Activity Using Longitudinal Self-Supervision

Project | PDF | Poster
Fangyu Li, N. Dinesh Reddy, Xudong Chen and Srinivasa G. Narasimhan
Proceedings of IEEE Intelligent Vehicles Symposium (IV'21)
Best Paper Award

Following instructions below, the user will get keypoints, trajectory reconstruction and vehicular activity clustering results like

Set up

The set up process can be skipped if using docker. Please check "Docker" section.

Python

Python version 3.6.9 is used. Python packages are in requirements.txt .

git clone https://github.com/Emrys-Lee/Traffic4D-Release.git
sudo apt-get install python3.6
sudo apt-get install python3-pip
cd Traffic4D-Release
pip3 install -r requirements.txt

C++

Traffic4D uses C++ libraries ceres and pybind for efficient optimization. pybind needs clang compiler, so Traffic4D uses clang compiler.

Install clang compiler

sudo apt-get install clang++-6.0

Install prerequisites for ceres

# CMake
sudo apt-get install cmake
# google-glog + gflags
sudo apt-get install libgoogle-glog-dev libgflags-dev
# BLAS & LAPACK
sudo apt-get install libatlas-base-dev
# Eigen3
sudo apt-get install libeigen3-dev
# SuiteSparse and CXSparse (optional)
sudo apt-get install libsuitesparse-dev

Download and install ceres

wget https://github.com/ceres-solver/ceres-solver/archive/1.12.0.zip
unzip 1.12.0.zip
cd ceres-solver-1.12.0/
mkdir build
cd build
cmake ..
make
sudo make install

Download and install pybind

git clone https://github.com/pybind/pybind11
cd pybind11
cmake .
make
sudo make install

Build Traffic4D optimization library

cd Traffic4D-Release/src/ceres
make

ceres_reconstruct.so and ceres_spline.so are generated under path Traffic4D-Release/src/ceres/.

Dataset

Download dataset and pre-generated results from here, and put it under Traffic4D-Release/.

cd Traffic4D-Release
mv Data-Traffic4D.zip ./
unzip Data-Traffic4D.zip

The directory should be like

Traffic4D-Release/
    Data-Traffic4D/
    └───fifth_morewood/
        └───fifth_morewood_init.vd
        └───top_view.png
        └───images/
                00001.jpg
                00002.jpg
                ...
                06288.jpg
    └───arterial_kennedy/
        └───arterial_kennedy_init.vd
        └───top_view.png
        └───images/
                <put AI City Challenge frames here>
        ...

The input and output paths can be modified in config/*.yml.

Explanation

1. Input videos

Sample videos in Traffic4D are provided. Note arterial_kennedy and dodge_century are from Nvidia AI City Challenge City-Scale Multi-Camera Vehicle Tracking Challenge Track. Please request the access to the dataset here. Once get the data, run

ffmpeg -i <mtmc-dir>/train/S01/c001/vdo.avi Traffic4D-Release/Data-Traffic4D/arterial_kennedy/images/%05d.jpg
ffmpeg -i <mtmc-dir>/test/S02/c007/vdo.avi Traffic4D-Release/Data-Traffic4D/dodge_century/images/%05d.jpg

to extract frames into images/.

2. Pre-Generated 2D results

Detected 2D bounding boxes, keypoints and tracking IDs are stored in *_init.vd. Check Occlusionnet implementation for detecting keypoints; V-IOU for multi-object tracking.

3. Output folder

Folder Traffic4D-Release/Result/ will be created by default.

Experiments

Run python exp/traffic4d.py config/<intersection_name>.yml <action>. Here YML configuration files for multiple intersections are provided under config/ folder. <action> shoulbe be reconstruction or clustering to perform longitudinal reconstruction and activity clustering sequentially. For example, below runs Fifth and Morewood intersection.

cd Traffic4D-Release
python3 exp/traffic4d.py config/fifth_morewood.yml reconstruction
python3 exp/traffic4d.py config/fifth_morewood.yml clustering

Results

Find these results in the output folder:

  1. 2D keypoints: If 3D reconstruction is done, 2D reprojected keypoints will be plotted in Traffic4D-Release/Result/<intersection_name>_keypoints/.
  2. 3D reconstructed trajectories and clusters: The clustered 3D trajectories are plotted on the top view map as Traffic4D-Release/Result/<intersection_name>_top_view.jpg.

Docker

We provide docker image with dependencies already set up. The steps in "Set up" can be skipped if you use docker image. You still need to clone the repo and download the dataset and put it in under Traffic4D-Release/.

git clone https://github.com/Emrys-Lee/Traffic4D-Release.git

Pull Traffic4D docker image.

docker pull emrysli/traffic4d-release:latest

Then create a container and map the git repo into docker container to access the dataset. For example, if the cloned repo locates at host directory /home/xxx/Traffic4D-Release, <path_to_repo> should be /home/xxx. If <path_in_container> is /home/yyy, then /home/xxx/Traffic4D-Release will be mapped as /home/yyy/Traffic4D-Release inside the container.

docker run -it -v <path_to_repo>/Traffic4D-Release:<path_in_container>/Traffic4D-Release emrysli/traffic4d-release:latest /bin/bash

Inside container compile Traffic4D again.

# inside container
cd <path_in_container>/Traffic4D-Release/src/ceres
make

Run experiments.

cd <path_in_container>/Traffic4D-Release
python3 exp/traffic4d.py config/fifth_morewood.yml reconstruction
python3 exp/traffic4d.py config/fifth_morewood.yml clustering

Trouble Shooting

  1. tkinter module is missing
File "/usr/local/lib/python3.6/dist-packages/matplotlib/backends/_backend_tk.py", line 5, in <module>
    import tkinter as Tk
ModuleNotFoundError: No module named 'tkinter'

Solution: install tkinter.

sudo apt-get install python3-tk
  1. opencv import error such as
File "/usr/local/lib/python3.6/dist-packages/cv2/__init__.py", line 3, in <module>
    from .cv2 import *
ImportError: libSM.so.6: cannot open shared object file: No such file or directory

Solution: install the missing libraries.

sudo apt-get install libsm6 libxrender1 libfontconfig1 libxext6

Citation

Traffic4D

@conference{Li-2021-127410,
author = {Fangyu Li and N. Dinesh Reddy and Xudong Chen and Srinivasa G. Narasimhan},
title = {Traffic4D: Single View Reconstruction of Repetitious Activity Using Longitudinal Self-Supervision},
booktitle = {Proceedings of IEEE Intelligent Vehicles Symposium (IV '21)},
year = {2021},
month = {July},
publisher = {IEEE},
keywords = {Self-Supervision, vehicle Detection, 4D Reconstruction, 3D reconstuction, Pose Estimation.},
}

Occlusion-Net

@inproceedings{onet_cvpr19,
author = {Reddy, N. Dinesh and Vo, Minh and Narasimhan, Srinivasa G.},
title = {Occlusion-Net: 2D/3D Occluded Keypoint Localization Using Graph Networks},
booktitle = {The IEEE Conference on Computer Vision and Pattern Recognition (CVPR)},
pages = {7326--7335},
year = {2019}
}
Owner
GitHub Repository
FreeSOLO for unsupervised instance segmentation, CVPR 2022

FreeSOLO: Learning to Segment Objects without Annotations This project hosts the code for implementing the FreeSOLO algorithm for unsupervised instanc

NVIDIA Research Projects 253 Jan 02, 2023
A PyTorch implementation of SIN: Superpixel Interpolation Network

SIN: Superpixel Interpolation Network This is is a PyTorch implementation of the superpixel segmentation network introduced in our PRICAI-2021 paper:

6 Sep 28, 2022
Definition of a business problem according to Wilson Lower Bound Score and Time Based Average Rating

Wilson Lower Bound Score, Time Based Rating Average In this study I tried to calculate the product rating and sorting reviews more accurately. I have

3 Sep 30, 2021
PyTorch implementation of MuseMorphose, a Transformer-based model for music style transfer.

MuseMorphose This repository contains the official implementation of the following paper: Shih-Lun Wu, Yi-Hsuan Yang MuseMorphose: Full-Song and Fine-

Yating Music, Taiwan AI Labs 142 Jan 08, 2023
OptNet: Differentiable Optimization as a Layer in Neural Networks

OptNet: Differentiable Optimization as a Layer in Neural Networks This repository is by Brandon Amos and J. Zico Kolter and contains the PyTorch sourc

CMU Locus Lab 428 Dec 24, 2022
The all new way to turn your boring vector meshes into the new fad in town; Voxels!

Voxelator The all new way to turn your boring vector meshes into the new fad in town; Voxels! Notes: I have not tested this on a rotated mesh. With fu

6 Feb 03, 2022
2021:"Bridging Global Context Interactions for High-Fidelity Image Completion"

TFill arXiv | Project This repository implements the training, testing and editing tools for "Bridging Global Context Interactions for High-Fidelity I

Chuanxia Zheng 111 Jan 08, 2023
PyTorch implementation of "VRT: A Video Restoration Transformer"

VRT: A Video Restoration Transformer Jingyun Liang, Jiezhang Cao, Yuchen Fan, Kai Zhang, Rakesh Ranjan, Yawei Li, Radu Timofte, Luc Van Gool Computer

Jingyun Liang 837 Jan 09, 2023
StyleGAN2-ADA-training-jupyter - Training custom datasets in styleGAN2-ADA by NVIDIA using Jupyter

styleGAN2-ADA-training-jupyter Training custom datasets in styleGAN2-ADA on Jupyter Official StyleGAN2-ADA by NIVIDIA Paper Training Generative Advers

Mang Su Hyun 2 Feb 24, 2022
Chinese clinical named entity recognition using pre-trained BERT model

Chinese clinical named entity recognition (CNER) using pre-trained BERT model Introduction Code for paper Chinese clinical named entity recognition wi

Xiangyang Li 109 Dec 14, 2022
Image data augmentation scheduler for albumentations transforms

albu_scheduler Scheduler for albumentations transforms based on PyTorch schedulers interface Usage TransformMultiStepScheduler import albumentations a

19 Aug 04, 2021
Code and models for "Pano3D: A Holistic Benchmark and a Solid Baseline for 360 Depth Estimation", OmniCV Workshop @ CVPR21.

Pano3D A Holistic Benchmark and a Solid Baseline for 360o Depth Estimation Pano3D is a new benchmark for depth estimation from spherical panoramas. We

Visual Computing Lab, Information Technologies Institute, Centre for Reseach and Technology Hellas 50 Dec 29, 2022
Taking A Closer Look at Domain Shift: Category-level Adversaries for Semantics Consistent Domain Adaptation

Taking A Closer Look at Domain Shift: Category-level Adversaries for Semantics Consistent Domain Adaptation (CVPR2019) This is a pytorch implementatio

Yawei Luo 280 Jan 01, 2023
KIND: an Italian Multi-Domain Dataset for Named Entity Recognition

KIND (Kessler Italian Named-entities Dataset) KIND is an Italian dataset for Named-Entity Recognition. It contains more than one million tokens with t

Digital Humanities 5 Jun 21, 2022
Catch-all collection of generative art made using processing

Generative art with Processing.py Some art I have created for fun. Dependencies Processing for Python, see how to download/use here Packages contained

2 Mar 12, 2022
Code for "LASR: Learning Articulated Shape Reconstruction from a Monocular Video". CVPR 2021.

LASR Installation Build with conda conda env create -f lasr.yml conda activate lasr # install softras cd third_party/softras; python setup.py install;

Google 157 Dec 26, 2022
An SMPC companion library for Syft

SyMPC A library that extends PySyft with SMPC support SyMPC /ˈsɪmpəθi/ is a library which extends PySyft ≥0.3 with SMPC support. It allows computing o

Arturo Marquez Flores 0 Oct 13, 2021
State-Relabeling Adversarial Active Learning

State-Relabeling Adversarial Active Learning Code for SRAAL [2020 CVPR Oral] Requirements torch = 1.6.0 numpy = 1.19.1 tqdm = 4.31.1 AL Results The

10 Jul 14, 2022
Demo for Real-time RGBD-based Extended Body Pose Estimation paper

Real-time RGBD-based Extended Body Pose Estimation This repository is a real-time demo for our paper that was published at WACV 2021 conference The ou

Renat Bashirov 118 Dec 26, 2022
The repository forked from NVlabs uses our data. (Differentiable rasterization applied to 3D model simplification tasks)

nvdiffmodeling [origin_code] Differentiable rasterization applied to 3D model simplification tasks, as described in the paper: Appearance-Driven Autom

Qiujie (Jay) Dong 2 Oct 31, 2022