AdaMML: Adaptive Multi-Modal Learning for Efficient Video Recognition [ArXiv] [Project Page]
This repository is the official implementation of AdaMML: Adaptive Multi-Modal Learning for Efficient Video Recognition.
Rameswar Panda*, Chun-Fu (Richard) Chen*, Quanfu Fan, Ximeng Sun, Kate Saenko, Aude Oliva, Rogerio Feris, "AdaMML: Adaptive Multi-Modal Learning for Efficient Video Recognition", ICCV 2021. (*: Equal Contribution)
If you use the codes and models from this repo, please cite our work. Thanks!
@inproceedings{panda2021adamml,
title={{AdaMML: Adaptive Multi-Modal Learning for Efficient Video Recognition}},
author={Panda, Rameswar and Chen, Chun-Fu and Fan, Quanfu and Sun, Ximeng and Saenko, Kate and Oliva, Aude and Feris, Rogerio},
booktitle={International Conference on Computer Vision (ICCV)},
year={2021}
}
Requirements
pip3 install torch torchvision librosa tqdm Pillow numpy
Data Preparation
The dataloader (utils/video_dataset.py) can load RGB frames stored in the following format:
-- dataset_dir
---- train.txt
---- val.txt
---- test.txt
---- videos
------ video_0_folder
-------- 00001.jpg
-------- 00002.jpg
-------- ...
------ video_1_folder
------ ...
Each line in train.txt and val.txt includes 4 elements and separated by a symbol, e.g. space ( ) or semicolon (;). Four elements (in order) include (1) relative paths to video_x_folder from dataset_dir, (2) starting frame number, usually 1, (3) ending frame number, (4) label id (a numeric number).
E.g., a video_x has 300 frames and belong to label 1.
path/to/video_x_folder 1 300 1
The difference for test.txt is that each line will only have 3 elements (no label information).
The same format is used for optical flow but each file (00001.jpg) need to be x_00001.jpg and y_00001.jpg.
On the other hand, for audio data, you need to change the first elements to the path of corresponding wav files, like
path/to/audio_x.wav 1 300 1
After that, you need to update the utils/data_config.py for the datasets accordingly.
We provide the scripts in the tools folder to extract RGB frames and audios from a video. To extract the optical flow, we use the docker image provided by TSN. Please see the help in the script.
Pretrained models
We provide the pretrained models on the Kinetics-Sounds dataset, including the unimodality models and our AdaMML models. You can find all the models here.
Training
After downloding the unimodality pretrained models, here is the command template to train AdaMML:
python3 train.py --multiprocessing-distributed --backbone_net adamml -d 50 \
--groups 8 --frames_per_group 4 -b 72 -j 96 --epochs 20 --warmup_epochs 5 --finetune_epochs 10 \
--modality MODALITY1 MODALITY2 --datadir /PATH/TO/MODALITY1 /PATH/TO/MODALITY2 --dataset DATASET --logdir LOGDIR \
--dense_sampling --fusion_point logits --unimodality_pretrained /PATH/TO/MODEL_MODALITY1 /PATH/TO/MODEL_MODALITY2 \
--learnable_lf_weights --num_segments 5 --cost_weights 1.0 0.005 --causality_modeling lstm --gammas 10.0 --sync-bn \
--lr 0.001 --p_lr 0.01 --lr_scheduler multisteps --lr_steps 10 15
The length of the following arguments depended on how many modalities you would like to include in AdaMML.
--modality: the modalities, other augments needs to follow this order--datadir: the data dir for each modality--unimodality_pretrained: the pretrained unimodality model
Note that, to use rgbdiff as a proxy, both rgbdiff and flow needs to be specified in --modality and their corresponding --datadir. However, you only need to provided flow pretrained model in the --unimodality_pretrained
Here are the examples to train AdaMML with different combinations.
RGB + Audio
python3 train.py --multiprocessing-distributed --backbone_net adamml -d 50 \
--groups 8 --frames_per_group 4 -b 72 -j 96 --epochs 20 --warmup_epochs 5 --finetune_epochs 10 \
--modality rgb sound --datadir /PATH/TO/RGB_DATA /PATH/TO/AUDIO_DATA --dataset DATASET --logdir LOGDIR \
--dense_sampling --fusion_point logits --unimodality_pretrained /PATH/TO/RGB_MODEL /PATH/TO/AUDIO_MODEL \
--learnable_lf_weights --num_segments 5 --cost_weights 1.0 0.05 --causality_modeling lstm --gammas 10.0 --sync-bn \
--lr 0.001 --p_lr 0.01 --lr_scheduler multisteps --lr_steps 10 15
RGB + Flow (with RGBDiff as Proxy)
python3 train.py --multiprocessing-distributed --backbone_net adamml -d 50 \
--groups 8 --frames_per_group 4 -b 72 -j 96 --epochs 20 --warmup_epochs 5 --finetune_epochs 10 \
--modality rgb flow rgbdiff --datadir /PATH/TO/RGB_DATA /PATH/TO/FLOW_DATA /PATH/TO/RGB_DATA --dataset DATASET --logdir LOGDIR \
--dense_sampling --fusion_point logits --unimodality_pretrained /PATH/TO/RGB_MODEL /PATH/TO/FLOW_MODEL \
--learnable_lf_weights --num_segments 5 --cost_weights 1.0 1.0 --causality_modeling lstm --gammas 10.0 --sync-bn \
--lr 0.001 --p_lr 0.01 --lr_scheduler multisteps --lr_steps 10 15
RGB + Audio + Flow (with RGBDiff as Proxy)
python3 train.py --multiprocessing-distributed --backbone_net adamml -d 50 \
--groups 8 --frames_per_group 4 -b 72 -j 96 --epochs 20 --warmup_epochs 5 --finetune_epochs 10 \
--modality rgb sound flow rgbdiff --datadir /PATH/TO/RGB_DATA /PATH/TO/AUDIO_DATA /PATH/TO/FLOW_DATA /PATH/TO/RGB_DATA --dataset DATASET --logdir LOGDIR \
--dense_sampling --fusion_point logits --unimodality_pretrained /PATH/TO/RGB_MODEL /PATH/TO/SOUND_MODEL /PATH/TO/FLOW_MODEL \
--learnable_lf_weights --num_segments 5 --cost_weights 0.5 0.05 0.8 --causality_modeling lstm --gammas 10.0 --sync-bn \
--lr 0.001 --p_lr 0.01 --lr_scheduler multisteps --lr_steps 10 15
Evaluation
Testing an AdaMML model is very straight-forward, you can simply use the training command with following modifications:
- add
-ein the command - use
--pretrained /PATH/TO/MODELto load the trained model - remove
--multiprocessing-distributedand--unimodality_pretrained - set
--val_num_clipsif you would like to test under different number of video segments (default is 10)
Here is command template:
python3 train.py -e --backbone_net adamml -d 50 \
--groups 8 --frames_per_group 4 -b 72 -j 96 \
--modality MODALITY1 MODALITY2 --datadir /PATH/TO/MODALITY1 /PATH/TO/MODALITY2 --dataset DATASET --logdir LOGDIR \
--dense_sampling --fusion_point logits --pretrained /PATH/TO/ADAMML_MODEL \
--learnable_lf_weights --num_segments 5 --causality_modeling lstm --sync-bn
26 Oct 13, 2022
![[CVPR'21] Multi-Modal Fusion Transformer for End-to-End Autonomous Driving](https://github.com/autonomousvision/transfuser/raw/main/transfuser/assets/teaser.png)
695 Jan 5, 2023
46 Dec 21, 2022
106 Dec 29, 2022
5 Dec 13, 2021
7 Nov 14, 2022
12 Dec 12, 2022
5 Nov 9, 2022
![[LREC] MMChat: Multi-Modal Chat Dataset on Social Media](https://github.com/silverriver/MMChat/raw/main/bin/sample.jpg)
47 Jan 3, 2023
11 Aug 26, 2022
18 Dec 26, 2022
178 Jan 07, 2023
34 Sep 10, 2022
106 Dec 28, 2022
2 Jan 20, 2022
3 Aug 26, 2022
46 Dec 12, 2022
37 Sep 15, 2022
47 Jul 26, 2022
76 Dec 21, 2022
48 Nov 16, 2022
50 Dec 29, 2022
50 Dec 03, 2022
1 Nov 23, 2021
15 Jul 09, 2022
43 Nov 14, 2022
26 Nov 21, 2022
59 Dec 06, 2022
70 Dec 27, 2022