[arXiv] What-If Motion Prediction for Autonomous Driving ❓🚗💨

Last update: Dec 29, 2022

Overview

WIMP - What If Motion Predictor

Reference PyTorch Implementation for What If Motion Prediction [PDF] [Dynamic Visualizations]

Setup

Requirements

The WIMP reference implementation and setup procedure has been tested to work with Ubuntu 16.04+ and has the following requirements:

python >= 3.7
pytorch >= 1.5.0

Installing Dependencies

Install remaining required Python dependencies using pip.
```
pip install -r requirements.txt
```
Install the Argoverse API module into the local Python environment by following steps 1, 2, and 4 in the README.

Argoverse Data

In order to set up the Argoverse dataset for training and evaluation, follow the steps below:

Download the the Argoverse Motion Forecasting v1.1 dataset and extract the compressed data subsets such that the raw CSV files are stored in the following directory structure:

├── WIMP
│   ├── src
│   ├── scripts
│   ├── data
│   │   ├── argoverse_raw
│   │   │   ├── train
│   │   │   │   ├── *.csv
│   │   │   ├── val
│   │   │   │   ├── *.csv
│   │   │   ├── test
│   │   │   │   ├── *.csv

Pre-process the raw Argoverse data into a WIMP-compatible format by running the following script. It should be noted that the Argoverse dataset is quite large and this script may take a few hours to run on a multi-threaded machine.

python scripts/run_preprocess.py --dataroot ./data/argoverse_raw/ \
--mode val --save-dir ./data/argoverse_processed --social-features \
--map-features --xy-features --normalize --extra-map-features \
--compute-all --generate-candidate-centerlines 6

Usage

For a detailed description of all possible configuration arguments, please run scripts with the -h flag.

Training

To train WIMP from scratch using a configuration similar to that reported in the paper, run a variant of the following command:

python src/main.py --mode train --dataroot ./data/argoverse_processed --IFC \
--lr 0.0001 --weight-decay 0.0 --non-linearity relu  --use-centerline-features \
--segment-CL-Encoder-Prob --num-mixtures 6 --output-conv --output-prediction \
--gradient-clipping --hidden-key-generator --k-value-threshold 10 \
--scheduler-step-size 60 90 120 150 180  --distributed-backend ddp \
--experiment-name example --gpus 4 --batch-size 25

Citing

If you've found this code to be useful, please consider citing our paper!

@article{khandelwal2020if,
  title={What-If Motion Prediction for Autonomous Driving},
  author={Khandelwal, Siddhesh and Qi, William and Singh, Jagjeet and Hartnett, Andrew and Ramanan, Deva},
  journal={arXiv preprint arXiv:2008.10587},
  year={2020}
}

Questions

This repo is maintained by William Qi and Siddhesh Khandelwal - please feel free to reach out or open an issue if you have additional questions/concerns.

We plan to clean up the codebase and add some additional utilities (possibly NuScenes data loaders and inference/visualization tools) in the near future, but don't expect to make significant breaking changes.

Comments

Pandas Error runpreprocess.py

Hello! First of all, thank you for making your code available for the readers of your great paper. I am having an issue while running run_preprocess.py. I think while reading the csv something goes wrong since my error is a pandas error. When I try to run the script, it gives me: KeyError: 'CITY_NAME' When I go to the script and give "MIA" as the CITY_NAME, just to see what happens, I receive a similar error: KeyError: 'OBJECT_TYPE' I checked the paths for the data. It seems fine. What could be the reason? Thank you!

opened by ahmetgurhan 0

Loss dimensions

Hi, thank you so much for your fantastic work.

Which is the order, and the dimensions, in this function?

def l1_ewta_loss(prediction, target, k=6, eps=1e-7, mr=2.0):
    num_mixtures = prediction.shape[1]

    target = target.unsqueeze(1).expand(-1, num_mixtures, -1, -1)
    l1_loss = nn.functional.l1_loss(prediction, target, reduction='none').sum(dim=[2, 3])

    # Get loss from top-k mixtures for each timestep
    mixture_loss_sorted, mixture_ranks = torch.sort(l1_loss, descending=False)
    mixture_loss_topk = mixture_loss_sorted.narrow(1, 0, k)

    # Aggregate loss across timesteps and batch
    loss = mixture_loss_topk.sum()
    loss = loss / target.size(0)
    loss = loss / target.size(2)
    loss = loss / k
    return loss

I am not able to obtain good results compared to NLL. I have as inputs:

predictions: batch_size x num_modes x pred_len x data_dim (e.g. 1024 x 6 x 30 x 2) gt: batch_size x pred_len x data_dim (e.g. 1024 x 30 x 2)

Is this correct?

opened by Cram3r95 0

Reproducing the Map-Free and only Social-Context Results form the Ablation Study

Hey there,

I want to reproduce the results of your ablation study, where you only used Social-Context with EWTA-Loss.

However, I habe problems training the model only with social context. What are the correct flags I need to set for preprocessing (run_preprocess.py) and for training (main.py)?

Looking forward hearing from you soon!

Best regards

SchDevel

opened by SchDevel 2
Can I get your inference/visualization code?

Hi, first of all, thanks for your awesome work and sharing that to us.

I tried to make inference/visualization code by myself, unfortunately, there were some problems.

Maybe library's mismatching, my insufficient coding skills, or something else.

So, can i get your inference/visualization code or even skeleton base code?

opened by raspbe34 3
What is the method for incomplete trajectories?
Hi, thanks for sharing your great work~ I am wondering how you deal with the incomplete trajectories problem (agents have less then 2 seconds of history).

I notice that for the neighboring agent wrt focal agent, you discard all the agents (code) if their trajectories are not complete

how would you deal with those incomplete trajectories for the focal agent? Did you use interpolation or some techniques?

Thanks!
opened by XHwind 0

Releases(1.0)

1.0(Oct 8, 2020)

Initial code release for WIMP.
Source code(tar.gz)
Source code(zip)

Owner

William Qi

Prediction @argoai

GitHub Repository

KUIELAB-MDX-Net got the 2nd place on the Leaderboard A and the 3rd place on the Leaderboard B in the MDX-Challenge ISMIR 2021

74 Dec 28, 2022

Implementation of Vision Transformer, a simple way to achieve SOTA in vision classification with only a single transformer encoder, in Pytorch

12.6k Jan 09, 2023

Making self-supervised learning work on molecules by using their 3D geometry to pre-train GNNs. Implemented in DGL and Pytorch Geometric.

3D Infomax improves GNNs for Molecular Property Prediction Video | Paper We pre-train GNNs to understand the geometry of molecules given only their 2D

95 Dec 30, 2022

[arXiv] What-If Motion Prediction for Autonomous Driving ❓🚗💨

Related tags

Overview

WIMP - What If Motion Predictor

Setup

Requirements

Installing Dependencies

Argoverse Data

Usage

Training

Citing

Questions

Comments

Pandas Error runpreprocess.py

Loss dimensions

Reproducing the Map-Free and only Social-Context Results form the Ablation Study

Can I get your inference/visualization code?

What is the method for incomplete trajectories?

Releases(1.0)

1.0(Oct 8, 2020)

Owner

William Qi

KUIELAB-MDX-Net got the 2nd place on the Leaderboard A and the 3rd place on the Leaderboard B in the MDX-Challenge ISMIR 2021

Deep Learning Algorithms for Hedging with Frictions

Code for 2021 NeurIPS --- Towards Multi-Grained Explainability for Graph Neural Networks

Video-face-extractor - Video face extractor with Python

GDSC-ML Team Interview Task

Open source Python module for computer vision

Split Variational AutoEncoder

Network Compression via Central Filter

Code for "Adversarial Training for a Hybrid Approach to Aspect-Based Sentiment Analysis

Task Transformer Network for Joint MRI Reconstruction and Super-Resolution (MICCAI 2021)

Like a cowsay but without cows!

NPBG++: Accelerating Neural Point-Based Graphics

LightningFSL: Pytorch-Lightning implementations of Few-Shot Learning models.

Image-Adaptive YOLO for Object Detection in Adverse Weather Conditions

[ICCV 2021] Group-aware Contrastive Regression for Action Quality Assessment

Semi-Supervised Semantic Segmentation via Adaptive Equalization Learning, NeurIPS 2021 (Spotlight)

Codebase for Time-series Generative Adversarial Networks (TimeGAN)

TensorFlow Metal Backend on Apple Silicon Experiments (just for fun)

Implementation of Vision Transformer, a simple way to achieve SOTA in vision classification with only a single transformer encoder, in Pytorch

Making self-supervised learning work on molecules by using their 3D geometry to pre-train GNNs. Implemented in DGL and Pytorch Geometric.