AIST++ API This repo contains starter code for using the AIST++ dataset.

Overview

AIST++ API

This repo contains starter code for using the AIST++ dataset. To download the dataset or explore details of this dataset, please go to our dataset website.

Installation

The code has been tested on python>=3.7. You can install the dependencies and this repo by:

pip install -r requirements.txt
python setup.py install

You also need to make sure ffmpeg is installed on your machine, if you would like to visualize the annotations using this api.

How to use

We provide demo code for loading and visualizing AIST++ annotations. Note AIST++ annotations and videos, as well as the SMPL model (for SMPL visualization only) are required to run the demo code.

The directory structure of the data is expected to be:


├── motions/
├── keypoints2d/
├── keypoints3d/
├── splits/
├── cameras/
└── ignore_list.txt


└── *.mp4


├── SMPL_MALE.pkl
└── SMPL_FEMALE.pkl

Visualize 2D keypoints annotation

The command below will plot 2D keypoints onto the raw video and save it to the directory ./visualization/.

python demos/run_vis.py \
  --anno_dir <ANNOTATIONS_DIR> \
  --video_dir <VIDEO_DIR> \
  --save_dir ./visualization/ \
  --video_name gWA_sFM_c01_d27_mWA2_ch21 \
  --mode 2D

Visualize 3D keypoints annotation

The command below will project 3D keypoints onto the raw video using camera parameters, and save it to the directory ./visualization/.

python demos/run_vis.py \
  --anno_dir <ANNOTATIONS_DIR> \
  --video_dir <VIDEO_DIR> \
  --save_dir ./visualization/ \
  --video_name gWA_sFM_c01_d27_mWA2_ch21 \
  --mode 3D

Visualize the SMPL joints annotation

The command below will first calculate the SMPL joint locations from our motion annotations (joint rotations and root trajectories), then project them onto the raw video and plot. The result will be saved into the directory ./visualization/.

python demos/run_vis.py \
  --anno_dir <ANNOTATIONS_DIR> \
  --video_dir <VIDEO_DIR> \ 
  --smpl_dir <SMPL_DIR> \
  --save_dir ./visualization/ \ 
  --video_name gWA_sFM_c01_d27_mWA2_ch21 \ 
  --mode SMPL

Multi-view 3D keypoints and motion reconstruction

This repo also provides code we used for constructing this dataset from the multi-view AIST Dance Video Database. The construction pipeline starts with frame-by-frame 2D keypoint detection and manual camera estimation. Then triangulation and bundle adjustment are applied to optimize the camera parameters as well as the 3D keypoints. Finally we sequentially fit the SMPL model to 3D keypoints to get a motion sequence represented using joint angles and a root trajectory. The following figure shows our pipeline overview.

AIST++ construction pipeline overview.

The annotations in AIST++ are in COCO-format for 2D & 3D keypoints, and SMPL-format for human motion annotations. It is designed to serve general research purposes. However, in some cases you might need the data in different format (e.g., Openpose / Alphapose keypoints format, or STAR human motion format). With the code we provide, it should be easy to construct your own version of AIST++, with your own keypoint detector or human model definition.

Step 1. Assume you have your own 2D keypoint detection results stored in , you can start by preprocessing the keypoints into the .pkl format that we support. The code we used at this step is as follows but you might need to modify the script run_preprocessing.py in order to be compatible with your own data.

python processing/run_preprocessing.py \
  --keypoints_dir <KEYPOINTS_DIR> \
  --save_dir <ANNOTATIONS_DIR>/keypoints2d/

Step 2. Then you can estimate the camera parameters using your 2D keypoints. This step is optional as you can still use our camera parameter estimates which are quite accurate. At this step, you will need the /cameras/mapping.txt file which stores the mapping from videos to different environment settings.

# If you would like to estimate your own camera parameters:
python processing/run_estimate_camera.py \
  --anno_dir <ANNOTATIONS_DIR> \
  --save_dir <ANNOTATIONS_DIR>/cameras/
# Or you can skip this step by just using our camera parameter estimates.

Step 3. Next step is to perform 3D keypoints reconstruction from multi-view 2D keypoints and camera parameters. You can just run:

python processing/run_estimate_keypoints.py \
  --anno_dir <ANNOTATIONS_DIR> \
  --save_dir <ANNOTATIONS_DIR>/keypoints3d/

Step 4. Finally we can estimate SMPL-format human motion data by fitting the 3D keypoints to the SMPL model. If you would like to use another human model such as STAR, you will need to do some modifications in the script run_estimate_smpl.py. The following command runs SMPL fitting.

python processing/run_estimate_smpl.py \
  --anno_dir <ANNOTATIONS_DIR> \
  --smpl_dir <SMPL_DIR> \
  --save_dir <ANNOTATIONS_DIR>/motions/

Note that this step will take several days to process the entire dataset if your machine has only one GPU. In practise, we run this step on a cluster, but are only able to provide the single-threaded version.

MISC.

  • COCO-format keypoint definition:
[
"nose", 
"left_eye", "right_eye", "left_ear", "right_ear", "left_shoulder","right_shoulder", 
"left_elbow", "right_elbow", "left_wrist", "right_wrist", "left_hip", "right_hip", 
"left_knee", "right_knee", "left_ankle", "right_ankle"
]
  • SMPL-format body joint definition:
[
"root", 
"left_hip", "left_knee", "left_foot", "left_toe", 
"right_hip", "right_knee", "right_foot", "right_toe",
"waist", "spine", "chest", "neck", "head", 
"left_in_shoulder", "left_shoulder", "left_elbow", "left_wrist",
"right_in_shoulder", "right_shoulder", "right_elbow", "right_wrist"
]
Owner
Google
Google ❤️ Open Source
Google
Runtime profiler for Streamlit, powered by pyinstrument

streamlit-profiler 🏄🏼 Runtime profiler for Streamlit, powered by pyinstrument. streamlit-profiler is a Streamlit component that helps you find out w

Johannes Rieke 23 Nov 30, 2022
This is a simple python script for checking A/L Examination results of srilankan students

AL-Result-Checker This is a simple python script for checking A/L Examination results of srilankan students INSTALLATION [Termux] [Linux] : apt-get up

Razor Kenway 8 Oct 24, 2022
A python script developed to process Windows memory images based on triage type.

Overview A python script developed to process Windows memory images based on triage type. Requirements Python3 Bulk Extractor Volatility2 with Communi

CrowdStrike 245 Nov 24, 2022
An kind of operating system portal to a variety of apps with pure python

pyos An kind of operating system portal to a variety of apps. Installation Run this on your terminal: git clone https://github.com/arjunj132/pyos.git

1 Jan 22, 2022
An open source server for Super Mario Bros. 35

SMB35 A custom server for Super Mario Bros. 35 This server is highly experimental. Do not expect it to work without flaws.

Yannik Marchand 162 Dec 07, 2022
Py4J enables Python programs to dynamically access arbitrary Java objects

Py4J Py4J enables Python programs running in a Python interpreter to dynamically access Java objects in a Java Virtual Machine. Methods are called as

Barthelemy Dagenais 1k Jan 02, 2023
Integration of CCURE access control system with automation HVAC of a commercial building

API-CCURE-Automation-Quantity-Floor Integration of CCURE access control system with automation HVAC of a commercial building CCURE is an access contro

Alexandre Edson Silva Pereira 1 Nov 24, 2021
A simple wrapper for joy library

Joy CodeGround A simple wrapper for joy library to render joy sketches in browser using vs code, (or in other words, for those who are allergic to Jup

rijfas 9 Sep 08, 2022
Explores the python bytecode, provides some tools to access it for fun and profit.

Pyasmtools - looking at the python bytecode for fun and profit. The pyasmtools library is made up of two parts A python bytecode disassembler . See Py

Michael Moser 299 Jan 04, 2023
Uproot - A script to bring deeply nested files or directories to the surface

UPROOT Bring deeply nested files or folders to the surface Uproot helps convert

Ted 2 Jan 15, 2022
Youtube Channel Website

Videos-By-Sanjeevi Youtube Channel Website YouTube Channel Website Features: Free Hosting using GitHub Pages and open-source code base in GitHub. It c

Sanjeevi Subramani 5 Mar 26, 2022
Minimal, super readable string pattern matching for python.

simplematch Minimal, super readable string pattern matching for python. import simplematch simplematch.match("He* {planet}!", "Hello World!") {"p

Thomas Feldmann 147 Dec 01, 2022
Airflow Operator for running Soda SQL scans

Airflow Operator for running Soda SQL scans

Todd de Quincey 7 Oct 18, 2022
A python program, imitating functionalities of a banking system

A python program, imitating functionalities of a banking system, in order for users to perform certain operations in a bank.

Moyosore Weke 1 Nov 26, 2021
Collaboration project to creating bank application maded by Anzhelica Sakun and Yuriy Konyukh

Collaboration project to creating bank application maded by Anzhelica Sakun and Yuriy Konyukh

Yuriy 1 Jan 08, 2022
A responsive package for Buttons, DropMenus and Combinations

A responsive package for Buttons, DropMenus and Combinations, This module makes the process a lot easier !

Skr Phoenix YT 0 Jan 30, 2022
Displays Christmas-themed ASCII art

Christmas Color Scripts Displays Christmas-themed ASCII art. This was mainly inspired by DistroTube's Shell Color Scripts Screenshots ASCII Shadow Tex

1 Aug 09, 2022
jonny is a stack based programming language

jonny-lang jonny is a stack based programming language also compiling jonny files currently doesnt work on windows you can probably compile jonny file

1 Nov 24, 2021
Make pack up python files easier.

python-easy-pack make pack up python files easier. 目前只提供了中文环境 如何使用? 将index.py复制到你的项目文件夹,或者把.py文件拷贝到这个文件夹。 打开你的cmd或者powershell 切换到程序所在目录,输入python index

2 Dec 15, 2021
A Way to Use Python, Easier.

PyTools A Way to Use Python, Easier. How to Install Just copy this code, then make a new file in your project directory called PyTools.py, then paste

Kamran 2 Aug 15, 2022