Neural Residual Flow Fields for Efficient Video Representations

1. Download MPI sintel dataset

Download MPI sintel dataset from here

2. GMA optical flow estimator

To obtain optical flow estimations for pretraining, we are using GMA from here. Note that it dose not have to do with our identity.

3. Training

Training neural residual flow fields (NRFF)

# frame 0 - 6
python train_video_flow_midkey.py --use-estimator --lr 0.0005 --training-step 30000 --data-dir {sintel dataset training directory} --video-name alley_1 --start-frame 0 --num-frames 7 --jpeg-quality 98 --hidden-features 96 --use-estimator --tag start0_jq98_hf96
# frame 7 - 13
python train_video_flow_midkey.py --use-estimator --lr 0.0005 --training-step 30000 --data-dir {sintel dataset training directory} --video-name alley_1 --start-frame 7 --num-frames 7 --jpeg-quality 98 --hidden-features 96 --use-estimator --tag start7_jq98_hf96
# frame 14 - 20
python train_video_flow_midkey.py --use-estimator --lr 0.0005 --training-step 30000 --data-dir {sintel dataset training directory} --video-name alley_1 --start-frame 14 --num-frames 7 --jpeg-quality 98 --hidden-features 96 --use-estimator --tag start14_jq98_hf96
# frame 21 - 27
python train_video_flow_midkey.py --use-estimator --lr 0.0005 --training-step 30000 --data-dir {sintel dataset training directory} --video-name alley_1 --start-frame 21 --num-frames 7 --jpeg-quality 98 --hidden-features 96 --use-estimator --tag start21_jq98_hf96

Training baseline (SIREN)

python train_video.py --data-dir {sintel dataset training directory} --video-name alley_1 --hidden-features 256 --num-frames 28 --lr 0.001 --training-step 30000 --tag baseline_siren_hf256

4. Examples

alley_2.mp4

HoneyBee.mp4

Eff video representation - Efficient video representation through neural fields

Related tags

Overview

Neural Residual Flow Fields for Efficient Video Representations

1. Download MPI sintel dataset

2. GMA optical flow estimator

3. Training

4. Examples

Owner

Only a Matter of Style: Age Transformation Using a Style-Based Regression Model

Spatiotemporal resampling methods for mlr3

A standard framework for modelling Deep Learning Models for tabular data

Detectron2 for Document Layout Analysis

Collection of generative models, e.g. GAN, VAE in Pytorch and Tensorflow.

pcnaDeep integrates cutting-edge detection techniques with tracking and cell cycle resolving models.

Benchmark for Answering Existential First Order Queries with Single Free Variable

A Flexible Generative Framework for Graph-based Semi-supervised Learning (NeurIPS 2019)

Convnet transfer - Code for paper How transferable are features in deep neural networks?

Official PyTorch Implementation of paper EAN: Event Adaptive Network for Efficient Action Recognition

PyTorch reimplementation of REALM and ORQA

Dense Prediction Transformers

Paddle pit - Rethinking Spatial Dimensions of Vision Transformers

Anonymize BLM Protest Images

NeurIPS 2021 paper 'Representation Learning on Spatial Networks' code

A PyTorch Implementation of Single Shot MultiBox Detector

Code to go with the paper "Decentralized Bayesian Learning with Metropolis-Adjusted Hamiltonian Monte Carlo"

Self Governing Neural Networks (SGNN): the Projection Layer

[ICLR'21] Counterfactual Generative Networks

Tensorflow implementation of "Learning Deep Features for Discriminative Localization"