Neural Residual Flow Fields for Efficient Video Representations

1. Download MPI sintel dataset

Download MPI sintel dataset from here

2. GMA optical flow estimator

To obtain optical flow estimations for pretraining, we are using GMA from here. Note that it dose not have to do with our identity.

3. Training

Training neural residual flow fields (NRFF)

# frame 0 - 6
python train_video_flow_midkey.py --use-estimator --lr 0.0005 --training-step 30000 --data-dir {sintel dataset training directory} --video-name alley_1 --start-frame 0 --num-frames 7 --jpeg-quality 98 --hidden-features 96 --use-estimator --tag start0_jq98_hf96
# frame 7 - 13
python train_video_flow_midkey.py --use-estimator --lr 0.0005 --training-step 30000 --data-dir {sintel dataset training directory} --video-name alley_1 --start-frame 7 --num-frames 7 --jpeg-quality 98 --hidden-features 96 --use-estimator --tag start7_jq98_hf96
# frame 14 - 20
python train_video_flow_midkey.py --use-estimator --lr 0.0005 --training-step 30000 --data-dir {sintel dataset training directory} --video-name alley_1 --start-frame 14 --num-frames 7 --jpeg-quality 98 --hidden-features 96 --use-estimator --tag start14_jq98_hf96
# frame 21 - 27
python train_video_flow_midkey.py --use-estimator --lr 0.0005 --training-step 30000 --data-dir {sintel dataset training directory} --video-name alley_1 --start-frame 21 --num-frames 7 --jpeg-quality 98 --hidden-features 96 --use-estimator --tag start21_jq98_hf96

Training baseline (SIREN)

python train_video.py --data-dir {sintel dataset training directory} --video-name alley_1 --hidden-features 256 --num-frames 28 --lr 0.001 --training-step 30000 --tag baseline_siren_hf256

4. Examples

alley_2.mp4

HoneyBee.mp4

Eff video representation - Efficient video representation through neural fields

Related tags

Overview

Neural Residual Flow Fields for Efficient Video Representations

1. Download MPI sintel dataset

2. GMA optical flow estimator

3. Training

4. Examples

Owner

Mesh Graphormer is a new transformer-based method for human pose and mesh reconsruction from an input image

AITUS - An atomatic notr maker for CYTUS

Code for the Paper: Conditional Variational Capsule Network for Open Set Recognition

SMPLpix: Neural Avatars from 3D Human Models

Estimating and Exploiting the Aleatoric Uncertainty in Surface Normal Estimation

Single-step adversarial training (AT) has received wide attention as it proved to be both efficient and robust.

Contrastive Multi-View Representation Learning on Graphs

Face Alignment using python

A torch.Tensor-like DataFrame library supporting multiple execution runtimes and Arrow as a common memory format

Predicting Axillary Lymph Node Metastasis in Early Breast Cancer Using Deep Learning on Primary Tumor Biopsy Slides

Pytorch implementation of “Recursive Non-Autoregressive Graph-to-Graph Transformer for Dependency Parsing with Iterative Refinement”

HashNeRF-pytorch - Pure PyTorch Implementation of NVIDIA paper on Instant Training of Neural Graphics primitives

Implementation of Multistream Transformers in Pytorch

Enhancing Knowledge Tracing via Adversarial Training

Code Release for the paper "TriBERT: Full-body Human-centric Audio-visual Representation Learning for Visual Sound Separation"

Data, model training, and evaluation code for "PubTables-1M: Towards a universal dataset and metrics for training and evaluating table extraction models".

The official code for PRIMER: Pyramid-based Masked Sentence Pre-training for Multi-document Summarization

HAR-stacked-residual-bidir-LSTMs - Deep stacked residual bidirectional LSTMs for HAR

FMA: A Dataset For Music Analysis

Pytorch implementation of MLP-Mixer with loading pre-trained models.