Official code release for 3DV 2021 paper Human Performance Capture from Monocular Video in the Wild.

Last update: Dec 24, 2022

Overview

Human Performance Capture from Monocular Video in the Wild

Paper | Video | Project Page

Official code release for 3DV 2021 paper Human Performance Capture from Monocular Video in the Wild. We propose a method capable of capturing the dynamic 3D human shape from a monocular video featuring challenging body poses, without any additional input.

If you find our code or paper useful, please cite as

@inproceedings{guo2021human,
  title={Human Performance Capture from Monocular Video in the Wild},
  author={Guo, Chen and Chen, Xu and Song, Jie and Hilliges, Otmar},
  booktitle={2021 International Conference on 3D Vision (3DV)},
  pages={889--898},
  year={2021},
  organization={IEEE}
}

Quick Start

CLone this repo:

git clone https://github.com/MoyGcc/hpcwild.git
cd  hpcwild
conda env create -f environment.yml
conda activate hpcwild

Additional Dependencies:

Kaolin 0.1.0 (https://github.com/NVIDIAGameWorks/kaolin)
MPI mesh library (https://github.com/MPI-IS/mesh)
torch-mesh-isect (https://github.com/vchoutas/torch-mesh-isect)

Download SMPL models (1.0.0 for Python 2.7 (10 shape PCs)) and move them to the corresponding places:

mkdir lib/smpl/smpl_model/
mv /path/to/smpl/models/basicModel_f_lbs_10_207_0_v1.0.0.pkl smpl_rendering/smpl_model/SMPL_FEMALE.pkl
mv /path/to/smpl/models/basicmodel_m_lbs_10_207_0_v1.0.0.pkl smpl_rendering/smpl_model/SMPL_MALE.pkl

Download checkpoints for external modules:

wget https://download.01.org/opencv/openvino_training_extensions/models/human_pose_estimation/checkpoint_iter_370000.pth
mv /path/to/checkpoint_iter_370000.pth external/lightweight-human-pose-estimation.pytorch/checkpoint_iter_370000.pth

wget https://dl.fbaipublicfiles.com/pifuhd/checkpoints/pifuhd.pt pifuhd.pt 
mv /path/to/pifuhd.pt external/pifuhd/checkpoints/pifuhd.pt

Download IPNet weights: https://datasets.d2.mpi-inf.mpg.de/IPNet2020/IPNet_p5000_01_exp_id01.zip
unzip IPNet_p5000_01_exp_id01.zip
mv /path/to/IPNet_p5000_01_exp_id01 registration/experiments/IPNet_p5000_01_exp_id01

gdown --id 1mcr7ALciuAsHCpLnrtG_eop5-EYhbCmz -O modnet_photographic_portrait_matting.ckpt
mv /path/to/modnet_photographic_portrait_matting.ckpt external/MODNet/pretrained/modnet_photographic_portrait_matting.ckpt

Test on 3DPW dataset

Download 3DPW dataset

modify the dataset_path in test.conf.
run bash mesh_recon.sh to obtain the rigid body shape.
run bash registration.sh to register a SMPL+D model to the rigid human body.
run bash tracking.sh to capture the human performance temporally.

Test on your own video

run OpenPose to obtain the 2D keypoints.
run LGD to acquire the initial 3D poses.
run MODNet to extract sihouettes.

Acknowledgement

We use the code in PIFuHD for the rigid body construction and adapt IPNet for human model registration. We use off-the-shelf methods OpenPose and MODNet for the extraction of 2D keypoints and sihouettes. We sincerely thank these authors for their awesome work.

Official code release for 3DV 2021 paper Human Performance Capture from Monocular Video in the Wild.

Related tags

Overview

Human Performance Capture from Monocular Video in the Wild

Paper | Video | Project Page

Quick Start

Test on 3DPW dataset

Test on your own video

Acknowledgement

Owner

Chen Guo

Put blind watermark into a text with python

Keras implementations of Generative Adversarial Networks.

IEEE Winter Conference on Applications of Computer Vision 2022 Accepted

This repository contains the official MATLAB implementation of the TDA method for reverse image filtering

Deep Learning & 3D Convolutional Neural Networks for Speaker Verification

Code for the RA-L (ICRA) 2021 paper "SeqNet: Learning Descriptors for Sequence-Based Hierarchical Place Recognition"

MiniHack the Planet: A Sandbox for Open-Ended Reinforcement Learning Research

Fully Convolutional DenseNets for semantic segmentation.

M2MRF: Many-to-Many Reassembly of Features for Tiny Lesion Segmentation in Fundus Images

Towhee is a flexible machine learning framework currently focused on computing deep learning embeddings over unstructured data.

Simple, efficient and flexible vision toolbox for mxnet framework.

This is the official implementation for the paper "Heterogeneous Multi-player Multi-armed Bandits: Closing the Gap and Generalization" in NeurIPS 2021.

EMNLP 2021 Adapting Language Models for Zero-shot Learning by Meta-tuning on Dataset and Prompt Collections

Calibrate your listeners! Robust communication-based training for pragmatic speakers. Findings of EMNLP 2021.

RLHive: a framework designed to facilitate research in reinforcement learning.

[Official] Exploring Temporal Coherence for More General Video Face Forgery Detection(ICCV 2021)

DECAF: Generating Fair Synthetic Data Using Causally-Aware Generative Networks

PyTorch implementation of "LayoutTransformer: Layout Generation and Completion with Self-attention"

Learning to Communicate with Deep Multi-Agent Reinforcement Learning in PyTorch

[AAAI 2022] Separate Contrastive Learning for Organs-at-Risk and Gross-Tumor-Volume Segmentation with Limited Annotation