GMR(Camera Motion Agnostic 3D Human Pose Estimation)

This repo provides the source code of our arXiv paper:
Seong Hyun Kim, Sunwon Jeong, Sungbum Park, and Ju Yong Chang, "Camera motion agnostic 3D human pose estimation," arXiv preprint arXiv:2112.00343, 2021.

Environment

• Python : 3.6
• Ubuntu : 18.04
• CUDA : 11.1
• cudnn : 8.0.5
• torch : 1.7.1
• torchvision : 0.8.2
• GPU : one Nvidia RTX3090

Installation

• First, you need to install python and other packages.

  pip install -r requirements.txt

• Then, you need to install torch and torchvision. We tested our code on torch1.7.1 and torchvision0.8.2. But our code can also work with torch version >= 1.5.0.

Quick Demo

• Download pretrained GMR model from [pretrained GMR] and make them look like this:

  ${GMR_ROOT}
   |-- results
       |-- GMR
           |-- final_model.pth
  

• Download other model files from [other model files] and make them look like this:

  ${GMR_ROOT}
   |-- data
       |-- gmr_data
           |-- J_regressor_extra.npy
           |-- J_regressor_h36m.npy
           |-- SMPL_NEUTRAL.pkl
           |-- gmm_08.pkl
           |-- smpl_mean_params.npz
           |-- spin_model_checkpoint.pth.tar
           |-- vibe_model_w_3dpw.pth.tar
           |-- vibe_model_wo_3dpw.pth.tar
  

• Finally, download demo videos from [demo videos] and make them look like this:

  ${GMR_ROOT}
  |-- configs
  |-- data
  |-- lib
  |-- results
  |-- scripts
  |-- demo.py
  |-- eval_3dpw.py
  |-- eval_synthetic.py
  |-- DEMO_VIDEO1.mp4
  |-- DEMO_VIDEO2.mp4
  |-- DEMO_VIDEO3.mp4
  |-- DEMO_VIDEO4.mp4
  |-- README.md
  |-- requirements.txt
  |-- run_eval_3dpw.sh
  |-- run_eval_synthetic.sh
  |-- run_train.sh
  |-- train.py
  

Demo code consists of (bounding box tracking) - (VIBE) - (GMR)

python demo.py --vid_file DEMO_VIDEO1.mp4 --vid_type mp4 --vid_fps 30 --view_type back --cfg configs/GMR_config.yaml --output_folder './'

python demo.py --vid_file DEMO_VIDEO2.mp4 --vid_type mp4 --vid_fps 30 --view_type front_large --cfg configs/GMR_config.yaml --output_folder './'

python demo.py --vid_file DEMO_VIDEO3.mp4 --vid_type mp4 --vid_fps 30 --view_type back --cfg configs/GMR_config.yaml --output_folder './'

python demo.py --vid_file DEMO_VIDEO4.mp4 --vid_type mp4 --vid_fps 30 --view_type back --cfg configs/GMR_config.yaml --output_folder './'

Data

You need to follow directory structure of the data as below.

${GMR_ROOT}
  |-- data
    |-- amass
      |-- ACCAD
      |-- BioMotionLab_NTroje
      |-- CMU
      |-- EKUT
      |-- Eyes_Japan_Dataset
      |-- HumanEva
      |-- KIT
      |-- MPI_HDM05
      |-- MPI_Limits
      |-- MPI_mosh
      |-- SFU
      |-- SSM_synced
      |-- TCD_handMocap
      |-- TotalCapture
      |-- Transitions_mocap
    |-- gmr_data
      |-- J_regressor_extra.npy
      |-- J_regressor_h36m.npy
      |-- SMPL_NEUTRAL.pkl
      |-- gmm_08.pkl
      |-- smpl_mean_params.npz
      |-- spin_model_checkpoint.pth.tar
      |-- vibe_model_w_3dpw.pth.tar
      |-- vibe_model_wo_3dpw.pth.tar
    |-- gmr_db
      |-- amass_train_db.pt
      |-- h36m_dsd_val_db.pt
      |-- 3dpw_test_db.pt
      |-- synthetic_camera_motion_off.pt
      |-- synthetic_camera_motion_on.pt

• Download AMASS dataset from this link and place them in data/amass. Then, you can obtain the training data through the following command. Also, you can download the training data from this link.

  source scripts/prepare_training_data.sh
  

• Download processed 3DPW data [data]
• Download processed Human3.6 data [data]
• Download synthetic dataset [data]

Train

Run the commands below to start training:

./run_train.sh

Evaluation

Run the commands below to start evaluation:

# Evaluation on 3DPW dataset
./run_eval_3dpw.sh

# Evaluation on synthetic dataset
./run_eval_synthetic.sh

References

We borrowed some scripts and models externally. Thanks to the authors for providing great resources.

• Pretrained VIBE and most of functions are borrowed from VIBE.
• Pretrained SPIN is borrowed from SPIN.
• SMPL model files are borrowed from SPIN and SMPLify.