Human Performance Capture from Monocular Video in the Wild
Paper | Video | Project Page
Official code release for 3DV 2021 paper Human Performance Capture from Monocular Video in the Wild. We propose a method capable of capturing the dynamic 3D human shape from a monocular video featuring challenging body poses, without any additional input.
If you find our code or paper useful, please cite as
@inproceedings{guo2021human,
title={Human Performance Capture from Monocular Video in the Wild},
author={Guo, Chen and Chen, Xu and Song, Jie and Hilliges, Otmar},
booktitle={2021 International Conference on 3D Vision (3DV)},
pages={889--898},
year={2021},
organization={IEEE}
}
Quick Start
CLone this repo:
git clone https://github.com/MoyGcc/hpcwild.git
cd hpcwild
conda env create -f environment.yml
conda activate hpcwild
Additional Dependencies:
- Kaolin 0.1.0 (https://github.com/NVIDIAGameWorks/kaolin)
- MPI mesh library (https://github.com/MPI-IS/mesh)
- torch-mesh-isect (https://github.com/vchoutas/torch-mesh-isect)
Download SMPL models (1.0.0 for Python 2.7 (10 shape PCs)) and move them to the corresponding places:
mkdir lib/smpl/smpl_model/
mv /path/to/smpl/models/basicModel_f_lbs_10_207_0_v1.0.0.pkl smpl_rendering/smpl_model/SMPL_FEMALE.pkl
mv /path/to/smpl/models/basicmodel_m_lbs_10_207_0_v1.0.0.pkl smpl_rendering/smpl_model/SMPL_MALE.pkl
Download checkpoints for external modules:
wget https://download.01.org/opencv/openvino_training_extensions/models/human_pose_estimation/checkpoint_iter_370000.pth
mv /path/to/checkpoint_iter_370000.pth external/lightweight-human-pose-estimation.pytorch/checkpoint_iter_370000.pth
wget https://dl.fbaipublicfiles.com/pifuhd/checkpoints/pifuhd.pt pifuhd.pt
mv /path/to/pifuhd.pt external/pifuhd/checkpoints/pifuhd.pt
Download IPNet weights: https://datasets.d2.mpi-inf.mpg.de/IPNet2020/IPNet_p5000_01_exp_id01.zip
unzip IPNet_p5000_01_exp_id01.zip
mv /path/to/IPNet_p5000_01_exp_id01 registration/experiments/IPNet_p5000_01_exp_id01
gdown --id 1mcr7ALciuAsHCpLnrtG_eop5-EYhbCmz -O modnet_photographic_portrait_matting.ckpt
mv /path/to/modnet_photographic_portrait_matting.ckpt external/MODNet/pretrained/modnet_photographic_portrait_matting.ckpt
Test on 3DPW dataset
Download 3DPW dataset
- modify the
dataset_pathintest.conf. - run
bash mesh_recon.shto obtain the rigid body shape. - run
bash registration.shto register a SMPL+D model to the rigid human body. - run
bash tracking.shto capture the human performance temporally.
Test on your own video
- run OpenPose to obtain the 2D keypoints.
- run LGD to acquire the initial 3D poses.
- run MODNet to extract sihouettes.
Acknowledgement
We use the code in PIFuHD for the rigid body construction and adapt IPNet for human model registration. We use off-the-shelf methods OpenPose and MODNet for the extraction of 2D keypoints and sihouettes. We sincerely thank these authors for their awesome work.
65 Jan 07, 2023
44 Dec 21, 2022
21 Dec 19, 2022
50 Jan 01, 2023
16 Dec 23, 2022
109 Dec 29, 2022
20 Nov 25, 2022
18 Nov 08, 2022
323 Jan 01, 2023
200 Jan 02, 2023
54 Dec 08, 2022
27 Oct 06, 2022
29 Oct 19, 2022
4 Dec 20, 2021
72 Dec 31, 2022
8 May 28, 2022
5 Nov 09, 2022
65 Dec 01, 2022
339 Dec 23, 2022
108 Dec 20, 2022