Semi-Supervised 3D Hand-Object Poses Estimation with Interactions in Time

Overview

Semi Hand-Object

Semi-Supervised 3D Hand-Object Poses Estimation with Interactions in Time (CVPR 2021). report

Project Page with Videos Teaser

Installation

  • Clone this repository:
    git clone https://github.com/stevenlsw/Semi-Hand-Object.git
  • Install the dependencies by the following command:
    pip install -r requirements.txt

Quick Demo (update soon)

Training and Evaluation on HO3D Dataset

Preparation

  • Download the MANO model files (mano_v1_2.zip) from MANO website. Unzip and put mano/models/MANO_RIGHT.pkl into assets/mano_models.

  • Download the YCB-Objects used in HO3D dataset. Put unzipped folder object_models under assets.

  • The structure should look like this:

Semi-Hand-Object/
  assets/
    mano_models/
      MANO_RIGHT.pkl
    object_models/
      006_mustard_bottle/
        points.xyz
        textured_simple.obj
      ......
  • Download and unzip HO3D dataset to path you like, the unzipped path is referred as $HO3D_root.

Evaluation

The hand & object pose estimation performance on HO3D dataset. We evaluate hand pose results on the official CodaLab challenge. The hand metric below is mean joint/mesh error after procrustes alignment, the object metric is average object vertices error within 10% of object diameter (ADD-0.1D).

In our model, we use transformer architecture to perform hand-object contextual reasoning.

Please download the trained model and save to path you like, the model path is refered as $resume.

trained-model joint↓ mesh↓ cleanser↑ bottle↑ can↑ ave↑
link 0.99 0.95 92.2 80.4 55.7 76.1
  • Testing with trained model

   python traineval.py --evaluate --HO3D_root={path to the dataset} --resume={path to the model} --test_batch=24 --host_folder=exp_results

The testing results will be saved in the $host_folder, which contains the following files:

  • option.txt (saved options)
  • object_result.txt (object pose evaluation performance)
  • pred.json (zip -j pred.zip pred.json and submit to the offical challenge for hand evaluation)

Training

Please download the preprocessed files to train HO3D dataset. The downloaded files contains training list and labels generated from the original dataset to accelerate training. Please put the unzipped folder ho3d-process to current directory.

    python traineval.py --HO3D_root={path to the dataset} --train_batch=24 --host_folder=exp_results

The models will be automatically saved in $host_folder

Citation

@inproceedings{liu2021semi,
  title={Semi-Supervised 3D Hand-Object Poses Estimation with Interactions in Time},
  author={Liu, Shaowei and Jiang, Hanwen and Xu, Jiarui and Liu, Sifei and Wang, Xiaolong},
  booktitle={Proceedings of the IEEE conference on computer vision and pattern recognition},
  year={2021}
}

TODO

  • Google colab demo

Acknowledgments

We thank:

PaRT: Parallel Learning for Robust and Transparent AI

PaRT: Parallel Learning for Robust and Transparent AI This repository contains the code for PaRT, an algorithm for training a base network on multiple

Mahsa 0 May 02, 2022
Benchmark library for high-dimensional HPO of black-box models based on Weighted Lasso regression

LassoBench LassoBench is a library for high-dimensional hyperparameter optimization benchmarks based on Weighted Lasso regression. Note: LassoBench is

Kenan Šehić 5 Mar 15, 2022
TensorLight - A high-level framework for TensorFlow

TensorLight is a high-level framework for TensorFlow-based machine intelligence applications. It reduces boilerplate code and enables advanced feature

Benjamin Kan 10 Jul 31, 2022
The codes and related files to reproduce the results for Image Similarity Challenge Track 2.

ISC-Track2-Submission The codes and related files to reproduce the results for Image Similarity Challenge Track 2. Required dependencies To begin with

Wenhao Wang 89 Jan 02, 2023
Makes patches from huge resolution .svs slide files using openslide

openslide_patcher Makes patches from huge resolution .svs slide files using openslide Example collage I made from outputs:

2 Dec 23, 2021
Official PyTorch implementation of the ICRA 2021 paper: Adversarial Differentiable Data Augmentation for Autonomous Systems.

Adversarial Differentiable Data Augmentation This repository provides the official PyTorch implementation of the ICRA 2021 paper: Adversarial Differen

Manli 3 Oct 15, 2022
Object Detection and Multi-Object Tracking

Object Detection and Multi-Object Tracking

Bobby Chen 1.6k Jan 04, 2023
使用深度学习框架提取视频硬字幕;docker容器免安装深度学习库,使用本地api接口使得界面和后端识别分离;

extract-video-subtittle 使用深度学习框架提取视频硬字幕; 本地识别无需联网; CPU识别速度可观; 容器提供API接口; 运行环境 本项目运行环境非常好搭建,我做好了docker容器免安装各种深度学习包; 提供windows界面操作; 容器为CPU版本; 视频演示 https

歌者 16 Aug 06, 2022
Get 2D point positions (e.g., facial landmarks) projected on 3D mesh

points2d_projection_mesh Input 2D points (e.g. facial landmarks) on an image Camera parameters (extrinsic and intrinsic) of the image Aligned 3D mesh

5 Dec 08, 2022
TensorFlow implementation of original paper : https://github.com/hszhao/PSPNet

Keras implementation of PSPNet(caffe) Implemented Architecture of Pyramid Scene Parsing Network in Keras. For the best compability please use Python3.

VladKry 386 Dec 29, 2022
A Comprehensive Empirical Study of Vision-Language Pre-trained Model for Supervised Cross-Modal Retrieval

CLIP4CMR A Comprehensive Empirical Study of Vision-Language Pre-trained Model for Supervised Cross-Modal Retrieval The original data and pre-calculate

24 Dec 26, 2022
Citation Intent Classification in scientific papers using the Scicite dataset an Pytorch

Citation Intent Classification Table of Contents About the Project Built With Installation Usage Acknowledgments About The Project Citation Intent Cla

Federico Nocentini 4 Mar 04, 2022
Source code and notebooks to reproduce experiments and benchmarks on Bias Faces in the Wild (BFW).

Face Recognition: Too Bias, or Not Too Bias? Robinson, Joseph P., Gennady Livitz, Yann Henon, Can Qin, Yun Fu, and Samson Timoner. "Face recognition:

Joseph P. Robinson 41 Dec 12, 2022
Facilitates implementing deep neural-network backbones, data augmentations

Introduction Nowadays, the training of Deep Learning models is fragmented and unified. When AI engineers face up with one specific task, the common wa

40 Dec 29, 2022
Segmentation Training Pipeline

Segmentation Training Pipeline This package is a part of Musket ML framework. Reasons to use Segmentation Pipeline Segmentation Pipeline was developed

Musket ML 52 Dec 12, 2022
本项目是一个带有前端界面的垃圾分类项目,加载了训练好的模型参数,模型为efficientnetb4,暂时为40分类问题。

说明 本项目是一个带有前端界面的垃圾分类项目,加载了训练好的模型参数,模型为efficientnetb4,暂时为40分类问题。 python依赖 tf2.3 、cv2、numpy、pyqt5 pyqt5安装 pip install PyQt5 pip install PyQt5-tools 使用 程

4 May 04, 2022
The 2nd place solution of 2021 google landmark retrieval on kaggle.

Leaderboard, taxonomy, and curated list of few-shot object detection papers.

229 Dec 13, 2022
[TOG 2021] PyTorch implementation for the paper: SofGAN: A Portrait Image Generator with Dynamic Styling.

This repository contains the official PyTorch implementation for the paper: SofGAN: A Portrait Image Generator with Dynamic Styling. We propose a SofGAN image generator to decouple the latent space o

Anpei Chen 694 Dec 23, 2022
Contextual Attention Localization for Offline Handwritten Text Recognition

CALText This repository contains the source code for CALText model introduced in "CALText: Contextual Attention Localization for Offline Handwritten T

0 Feb 17, 2022
Disease Informed Neural Networks (DINNs) — neural networks capable of learning how diseases spread, forecasting their progression, and finding their unique parameters (e.g. death rate).

DINN We introduce Disease Informed Neural Networks (DINNs) — neural networks capable of learning how diseases spread, forecasting their progression, a

19 Dec 10, 2022