[arXiv] What-If Motion Prediction for Autonomous Driving ❓🚗💨

Last update: Dec 29, 2022

Overview

WIMP - What If Motion Predictor

Reference PyTorch Implementation for What If Motion Prediction [PDF] [Dynamic Visualizations]

Setup

Requirements

The WIMP reference implementation and setup procedure has been tested to work with Ubuntu 16.04+ and has the following requirements:

python >= 3.7
pytorch >= 1.5.0

Installing Dependencies

Install remaining required Python dependencies using pip.
```
pip install -r requirements.txt
```
Install the Argoverse API module into the local Python environment by following steps 1, 2, and 4 in the README.

Argoverse Data

In order to set up the Argoverse dataset for training and evaluation, follow the steps below:

Download the the Argoverse Motion Forecasting v1.1 dataset and extract the compressed data subsets such that the raw CSV files are stored in the following directory structure:

├── WIMP
│   ├── src
│   ├── scripts
│   ├── data
│   │   ├── argoverse_raw
│   │   │   ├── train
│   │   │   │   ├── *.csv
│   │   │   ├── val
│   │   │   │   ├── *.csv
│   │   │   ├── test
│   │   │   │   ├── *.csv

Pre-process the raw Argoverse data into a WIMP-compatible format by running the following script. It should be noted that the Argoverse dataset is quite large and this script may take a few hours to run on a multi-threaded machine.

python scripts/run_preprocess.py --dataroot ./data/argoverse_raw/ \
--mode val --save-dir ./data/argoverse_processed --social-features \
--map-features --xy-features --normalize --extra-map-features \
--compute-all --generate-candidate-centerlines 6

Usage

For a detailed description of all possible configuration arguments, please run scripts with the -h flag.

Training

To train WIMP from scratch using a configuration similar to that reported in the paper, run a variant of the following command:

python src/main.py --mode train --dataroot ./data/argoverse_processed --IFC \
--lr 0.0001 --weight-decay 0.0 --non-linearity relu  --use-centerline-features \
--segment-CL-Encoder-Prob --num-mixtures 6 --output-conv --output-prediction \
--gradient-clipping --hidden-key-generator --k-value-threshold 10 \
--scheduler-step-size 60 90 120 150 180  --distributed-backend ddp \
--experiment-name example --gpus 4 --batch-size 25

Citing

If you've found this code to be useful, please consider citing our paper!

@article{khandelwal2020if,
  title={What-If Motion Prediction for Autonomous Driving},
  author={Khandelwal, Siddhesh and Qi, William and Singh, Jagjeet and Hartnett, Andrew and Ramanan, Deva},
  journal={arXiv preprint arXiv:2008.10587},
  year={2020}
}

Questions

This repo is maintained by William Qi and Siddhesh Khandelwal - please feel free to reach out or open an issue if you have additional questions/concerns.

We plan to clean up the codebase and add some additional utilities (possibly NuScenes data loaders and inference/visualization tools) in the near future, but don't expect to make significant breaking changes.

Comments

Pandas Error runpreprocess.py

Hello! First of all, thank you for making your code available for the readers of your great paper. I am having an issue while running run_preprocess.py. I think while reading the csv something goes wrong since my error is a pandas error. When I try to run the script, it gives me: KeyError: 'CITY_NAME' When I go to the script and give "MIA" as the CITY_NAME, just to see what happens, I receive a similar error: KeyError: 'OBJECT_TYPE' I checked the paths for the data. It seems fine. What could be the reason? Thank you!

opened by ahmetgurhan 0

Loss dimensions

Hi, thank you so much for your fantastic work.

Which is the order, and the dimensions, in this function?

def l1_ewta_loss(prediction, target, k=6, eps=1e-7, mr=2.0):
    num_mixtures = prediction.shape[1]

    target = target.unsqueeze(1).expand(-1, num_mixtures, -1, -1)
    l1_loss = nn.functional.l1_loss(prediction, target, reduction='none').sum(dim=[2, 3])

    # Get loss from top-k mixtures for each timestep
    mixture_loss_sorted, mixture_ranks = torch.sort(l1_loss, descending=False)
    mixture_loss_topk = mixture_loss_sorted.narrow(1, 0, k)

    # Aggregate loss across timesteps and batch
    loss = mixture_loss_topk.sum()
    loss = loss / target.size(0)
    loss = loss / target.size(2)
    loss = loss / k
    return loss

I am not able to obtain good results compared to NLL. I have as inputs:

predictions: batch_size x num_modes x pred_len x data_dim (e.g. 1024 x 6 x 30 x 2) gt: batch_size x pred_len x data_dim (e.g. 1024 x 30 x 2)

Is this correct?

opened by Cram3r95 0

Reproducing the Map-Free and only Social-Context Results form the Ablation Study

Hey there,

I want to reproduce the results of your ablation study, where you only used Social-Context with EWTA-Loss.

However, I habe problems training the model only with social context. What are the correct flags I need to set for preprocessing (run_preprocess.py) and for training (main.py)?

Looking forward hearing from you soon!

Best regards

SchDevel

opened by SchDevel 2
Can I get your inference/visualization code?

Hi, first of all, thanks for your awesome work and sharing that to us.

I tried to make inference/visualization code by myself, unfortunately, there were some problems.

Maybe library's mismatching, my insufficient coding skills, or something else.

So, can i get your inference/visualization code or even skeleton base code?

opened by raspbe34 3
What is the method for incomplete trajectories?
Hi, thanks for sharing your great work~ I am wondering how you deal with the incomplete trajectories problem (agents have less then 2 seconds of history).

I notice that for the neighboring agent wrt focal agent, you discard all the agents (code) if their trajectories are not complete

how would you deal with those incomplete trajectories for the focal agent? Did you use interpolation or some techniques?

Thanks!
opened by XHwind 0

Releases(1.0)

1.0(Oct 8, 2020)

Initial code release for WIMP.
Source code(tar.gz)
Source code(zip)

Owner

William Qi

Prediction @argoai

GitHub Repository

The PASS dataset: pretrained models and how to get the data - PASS: Pictures without humAns for Self-Supervised Pretraining

249 Dec 22, 2022

[arXiv] What-If Motion Prediction for Autonomous Driving ❓🚗💨

Related tags

Overview

WIMP - What If Motion Predictor

Setup

Requirements

Installing Dependencies

Argoverse Data

Usage

Training

Citing

Questions

Comments

Pandas Error runpreprocess.py

Loss dimensions

Reproducing the Map-Free and only Social-Context Results form the Ablation Study

Can I get your inference/visualization code?

What is the method for incomplete trajectories?

Releases(1.0)

1.0(Oct 8, 2020)

Owner

William Qi

3DMV jointly combines RGB color and geometric information to perform 3D semantic segmentation of RGB-D scans.

[AI6101] Introduction to AI & AI Ethics is a core course of MSAI, SCSE, NTU, Singapore

How will electric vehicles affect traffic congestion and energy consumption: an integrated modelling approach

Utilities and information for the signals.numer.ai tournament

Code for ACL2021 paper Consistency Regularization for Cross-Lingual Fine-Tuning.

Code for Talking Face Generation by Adversarially Disentangled Audio-Visual Representation (AAAI 2019)

Train Scene Graph Generation for Visual Genome and GQA in PyTorch >= 1.2 with improved zero and few-shot generalization.

rastrainer is a QGIS plugin to training remote sensing semantic segmentation model based on PaddlePaddle.

This is the source code of the 1st place solution for segmentation task (with Dice 90.32%) in 2021 CCF BDCI challenge.

SEAN: Image Synthesis with Semantic Region-Adaptive Normalization (CVPR 2020, Oral)

《A-CNN: Annularly Convolutional Neural Networks on Point Clouds》(2019)

Relative Uncertainty Learning for Facial Expression Recognition

CDTrans: Cross-domain Transformer for Unsupervised Domain Adaptation

[CVPR 2021] Exemplar-Based Open-Set Panoptic Segmentation Network (EOPSN)

Source code for "Interactive All-Hex Meshing via Cuboid Decomposition [SIGGRAPH Asia 2021]".

Source code and notebooks to reproduce experiments and benchmarks on Bias Faces in the Wild (BFW).

Python scripts form performing stereo depth estimation using the CoEx model in ONNX.

Code for You Only Cut Once: Boosting Data Augmentation with a Single Cut

Dynamical Wasserstein Barycenters for Time Series Modeling

The PASS dataset: pretrained models and how to get the data - PASS: Pictures without humAns for Self-Supervised Pretraining