2021 Real Robot Challenge Phase2 attemp

Overview

Real_Robot_Challenge_Phase2_AE_attemp

We(team name:thriftysnipe) are the first place winner of Phase1 in 2021 Real Robot Challenge.
Please see this page for more details: https://real-robot-challenge.com/leaderboard
To see more details about out Phase1 works: https://github.com/wq13552463699/Real_Robot_challenge
We were granted the access to Phase 2.

I am sorry, the project is too complex with too much large files, It is too hard to upload them all on Github. I just attached a part of the core code here for you to take a quick lreview. If you think my attempts is approriate, you can go to this Google Drive to download the full project file(all codes, results, trained models, environmental files,.etc):
https://drive.google.com/file/d/14vjCrWU6vzMdXxVSR2FeskMvuQpgqWqM/view?usp=sharing

RRC phase2 task description:

Randomly place 25 dices with the size of 0.01x0.01x0.01m in the environment. Use own controller to drive the three-finger robot to rearrange the dice to a specific pattern. Unfortunately, due to the set task is too difficult, no team could complete the task on the actual robot, so all teams with record are awarded third place in this phase. But I think our attempt has a reference value, if later scholars conduct related research, our method may be useful.

Our considerations:

We consider using a reinforcement learning algorithm as the controller in this phase. However, in this phase, information that can play as observations, such as coordinates and orientation of the dices, cannot be obtained from the environment directly but they are crucial for RL to run.
The alternative observations we can use are the images of the three cameras set in 3 different angles in the environment and their segmentation masks. We picked segmentation masks rather than the raw images since the attendance of noise and redundancy in the raw images were too much. Please see the following segmentation mask example(RGB's 3 channels represent segmentation masks from 3 different angles).

The segmentation masks have the dimension of 270x270x3, if directly passing it to the RL agent, which would lead to computational explosion and hard to converge. Hence, we planned to use some means to extract the principal components that can play as observations from it. In addition, the observation value also includes readable read-robot data(joint angle of the robot arm, end effector position, end effector speed, etc.).

Segmentation mask dimensionality reduction

This is the most important part of this task. We tried different methods, such as GAN, VAE, AE, to extract the principal conponents from the images. The quality of data dimensionality reduction can be easily seem from the discripency of reconstructed and oringinal images or the loss curves. After many trials(adjusting hyperparameters, network structure, depth, etc.), we got different trained VAE, GAN and AE models. We conducted offline tests on the obtained model and compared the results, we were surprised to find that the AE performed the best. When the latent of AE is 384, the quality of the reconstructed image is the best. The result is shown in the figure below.

The loss function also converges to an acceptable range:

Build up observation and trian RL agent.

We use the best AE encoder to deal with the segmentation masks to generate the observation and stitch with the readable data. The structure of the overall obervation is shown as follow:
We fed the above observations to several current cutting-edge model based and model free reinforcement learning algorithms, including DDPG+HER, PPO, SLAC, PlaNet and Dreamer. We thought it would work and enable the agent to learn for somewhat anyway. But it is a pity that after many attempts, the model still didn't have any trend to converge. Due to time limited, our attempts were over here.

Some reasons might lead to fail

  1. We used AE as the observation model. Although the AE's dimensionality reduction capability were the best, the latent space of AE were disordered and didn't make sense to RL agent. The observations passed to the RL must be fixed and orderly. Continuous delivery of unfixed data caused a dimensional disaster. For example, the third number in the observation vector passed at t1 represents 'infos of the 1st dice', and the number on the same position at t2 represents the 'infos of the 3rd dice'. This disorderly change with time makes RL very confused.
  2. The extracted latent space from segmentation mask dominates the observations, making RL ignore the existence of robots. The latent space size is 384, but which for the robot data is 27. The two are far apart, and there is a big data bias.
  3. Robot arm blocked the dices, segmentation masks can only represent a part of the dice. This problem cannot be avoided and can only be solved by more powerful image processing technology. This is also a major challenge in the current Image-based RL industry

Contribution

Pull requests are welcome. For major changes, please open an issue first to discuss what you would like to change. Please make sure to update tests as appropriate.

Owner
Qiang Wang
PhD at UCD. Research interest: Reinforcement Learning; Computer vision&Touch; Representation learning
Qiang Wang
a library for using WS2812b leds (aka neopixels) with Raspberry Pi Pico

pico_ws2812b a library for using WS2812b leds (aka neopixels) with Raspberry Pi Pico You'll first need to save the ws2812b.py file to your device (for

76 Nov 25, 2022
Play a song with a 3D printer.

MIDI to GCODE Play a song with a FDM 3D printer. SLA printers don't have motors, so they cannot play music. Warning: Be ready to turn off the 3D print

Patrick 6 Apr 11, 2022
A custom mechanical keyboard inspired by the CFTKB Mysterium

Env-KB A custom mechanical keyboard inspired by the CFTKB Mysterium Build Guide and Parts List What is to do? Right now for the first 5 PCBs I have, i

EnviousData 203 Jan 04, 2023
FHEM Connector for FHT Heating devices

home-assistant-fht from: https://github.com/Rsclub22 FHEM Connector for FHT Heating devices (connected via FHEM) Requires FHEM to work You can find FH

5 Dec 01, 2022
A 3rd party Moonraker component to create timelapse of 3D prints.

A 3rd party Moonraker component to create timelapse of 3D prints.

Mainsail-Crew 166 Dec 26, 2022
Quasi-static control of the centroid of quadruped robot

Quasi-static control of quadruped robot   This is a demo of the quasi-static controller for the centroid of the quadruped robot. The Quadratic Program

Junwen Cui 21 Dec 12, 2022
A simple portable USB MIDI controller based on Raspberry-PI Pico and a 16-button keypad, written in Circuit Python

RPI-Pico-16-BTn-MIDI-Controller-using-CircuitPython A simple portable USB MIDI controller based on Raspberry-PI Pico, written in Circuit Python. Link

Rounak Dutta 3 Dec 04, 2022
Isaac Gym Environments for Legged Robots

Isaac Gym Environments for Legged Robots This repository provides the environment used to train ANYmal (and other robots) to walk on rough terrain usi

Robotic Systems Lab - Legged Robotics at ETH Zürich 372 Jan 08, 2023
BMP180 sensor driver for Home Assistant used in Raspberry Pi

BMP180 sensor driver for Home Assistant used in Raspberry Pi Custom component BMP180 sensor for Home Assistant. Copy the content of this directory to

747Developments 1 Dec 17, 2021
Port of Uxn to digital hardware in the Logisim simulator

Uxn-Logisim Implements the Uxn instruction set in digital hardware. Very WIP. Contents cpu.circ - The Logisim file microcode.mc - Microcode source fil

DeltaF1 11 Mar 27, 2022
Python script: Enphase Envoy mqtt json for Home Assistant

A Python script that takes a real time stream from Enphase Envoy and publishes to a mqtt broker. This can then be used within Home Assistant or for other applications. The data updates at least once

29 Dec 27, 2022
Micropython automatic watering

micropython-automatic-watering micropython automatic watering his code was developed to be used with nodemcu esp8266, but can be modified to work with

1 Nov 24, 2021
A LiteX project which builds a SoC with DRAM / HDIM output via the GPDI SYZYGY addon.

ButterStick GPDI LiteX demo A LiteX project which builds a SoC with DRAM / HDIM output via the GPDI SYZYGY addon. Getting started Connect GPDI board t

4 Nov 21, 2021
ArucoFollow - A script for Robot Operating System and it is a part of a project Robot

ArucoFollow ArucoFollow is a script for Robot Operating System and it is a part

5 Jan 25, 2022
🎃 Some spooky code samples to hack yourself a pumpkin 👻

🎃 Tech Or Treat 👻 It's spooky season for those who celebrate Halloween, and to get in the spirit (spirit - get it? 👻 ) we thought it would be fun t

Jim Bennett 5 Feb 07, 2022
Skykettle ha - Redmond SkyKettle integration for Home Assistant

Redmond SkyKettle integration for Home Assistant This integration allows to cont

Alexey 'Cluster' Avdyukhin 48 Jan 06, 2023
Micro Displays for Raspberry Pi

micro-displays Micro Displays for Raspberry Pi Why? I'm super bored in lockdown. Add a Raspberry Pi 400 and a few tiny displays... The top half of the

ig 291 Jul 06, 2022
Python Client for ESPHome native API. Used by Home Assistant.

aioesphomeapi aioesphomeapi allows you to interact with devices flashed with ESPHome. Installation The module is available from the Python Package Ind

ESPHome 76 Jan 04, 2023
Terkin is a flexible data logger application for MicroPython and CPython environments.

Terkin Data logging for humans, written in MicroPython. Documentation: https://terkin.org/ Source Code: https://github.com/hiveeyes/terkin-datalogger

hiveeyes 45 Dec 15, 2022
A set of postprocessing scripts and macro to accelerate the gyroid infill print speed with Klipper

A set of postprocessing scripts and macro to accelerate the gyroid infill print speed with Klipper

Jérôme W. 75 Jan 07, 2023