Human Action Controller - A human action controller running on different platforms.

Last update: Jul 20, 2022

Related tags

Overview

Human Action Controller (HAC)

Goal

A human action controller running on different platforms.

Fun	Easy-to-use
Accurate	Anywhere

Fun Examples

Mouse Control

Keyboard Control

Playing Game

Enhancing interaction

Solutions provided by HAC

Platform	Module	Progress
PC / Win10	Mouse Control	V
PC / Win10	Keyboard Control	V
PC / Ubuntu	Mouse Control
PC / Ubuntu	Keyboard Control

Getting started

Installation

$ pip install pyhac

Run the demo of mouse control

$ git clone https://github.com/dabit-lucas/hac.git
$ cd hac
$ python demo.py

Recording custom actions

$ python recording.py -d {action name} -k True

Press key "r" to start recording, the data will be saved into ./data

Training a custom module

Here is an example of a config file of action set,

{
    "actions": [
        "r_five",
        "r_zero",
        "l_five",
        "l_zero",
        "two_index_fingers_up",
        "two_index_fingers_down",
        "33",
        "55",
        "sit"
    ],
    "type": "gesture_only"
}

These actions form a model by running a training process:

$ python train.py --conf {path_of_action} --model_name {name_of_model}

The generated model will become a module. Take mouse control module as an exmaple, it can create mappings among actions and controls by the following code:

mouse_module = hac.add_module("mouse_control")
hac.set_init_module(mouse_module)

# create mapping between controls and actions
mouse_module.add_mouse_mapping("mouse_left_down", ["r_five", "r_zero"])
mouse_module.add_mouse_mapping("mouse_left_up", "r_five")
mouse_module.add_mouse_mapping("mouse_right_down", ["l_five", "l_zero"])
mouse_module.add_mouse_mapping("mouse_right_up", "l_five")
mouse_module.add_mouse_mapping("right_move_diff", ["r_five", "r_five"])
mouse_module.add_mouse_mapping("right_move_diff", ["r_zero", "r_zero"])
mouse_module.add_mouse_mapping("left_move_diff", ["l_five", "l_five"])
mouse_module.add_mouse_mapping("left_move_diff", ["l_zero", "l_zero"])
mouse_module.add_mouse_mapping("roll_up", "two_index_fingers_up")
mouse_module.add_mouse_mapping("roll_down", "two_index_fingers_down")

If the five gesture with a right hand shows in consecutive two frames ["r_five", "r_five"], then do control right_move_diff, which means moving the mouse cursor. The above description can be represented by the following code:

mouse_module.add_mouse_mapping("right_move_diff", ["r_five", "r_five"])

Development guideline

The structure of HAC

Community

Welcome to ask any question in issues.

Contributing

Any contribution is welcomed. Please fork this repo and summit a pull request.

Human Action Controller - A human action controller running on different platforms.

Related tags

Overview

Human Action Controller (HAC)

Goal

Fun Examples

Mouse Control

Keyboard Control

Playing Game

Enhancing interaction

Solutions provided by HAC

Getting started

Installation

Run the demo of mouse control

Recording custom actions

Training a custom module

Development guideline

Community

Contributing

Owner

Making a music video with Wav2CLIP and VQGAN-CLIP

hySLAM is a hybrid SLAM/SfM system designed for mapping

A curated list of awesome open source libraries to deploy, monitor, version and scale your machine learning

A curated list of awesome Deep Learning tutorials, projects and communities.

Data, model training, and evaluation code for "PubTables-1M: Towards a universal dataset and metrics for training and evaluating table extraction models".

masscan + nmap + Finger

Code for "NeuralRecon: Real-Time Coherent 3D Reconstruction from Monocular Video", CVPR 2021 oral

How to Train a GAN? Tips and tricks to make GANs work

PyTorch implementation of the ACL, 2021 paper Parameter-efficient Multi-task Fine-tuning for Transformers via Shared Hypernetworks.

Source code of our TTH paper: Targeted Trojan-Horse Attacks on Language-based Image Retrieval.

PyTorch Implementation of our paper Explain Me the Painting: Multi-Topic Knowledgeable Art Description Generation

Code for Graph-to-Tree Learning for Solving Math Word Problems (ACL 2020)

A novel pipeline framework for multi-hop complex KGQA task. About the paper title: Improving Multi-hop Embedded Knowledge Graph Question Answering by Introducing Relational Chain Reasoning

Transformers provides thousands of pretrained models to perform tasks on different modalities such as text, vision, and audio.

Official repo for our 3DV 2021 paper "Monocular 3D Reconstruction of Interacting Hands via Collision-Aware Factorized Refinements".

This repository contains the code for "Self-Diagnosis and Self-Debiasing: A Proposal for Reducing Corpus-Based Bias in NLP".

Red Team tool for exfiltrating files from a target's Google Drive that you have access to, via Google's API.

Learning cell communication from spatial graphs of cells

Repo for "Physion: Evaluating Physical Prediction from Vision in Humans and Machines" submission to NeurIPS 2021 (Datasets & Benchmarks track)

Probabilistic Gradient Boosting Machines