Code release for our paper, "SimNet: Enabling Robust Unknown Object Manipulation from Pure Synthetic Data via Stereo"

Last update: Dec 14, 2022

Related tags

Overview

SimNet: Enabling Robust Unknown Object Manipulation from Pure Synthetic Data via Stereo

Thomas Kollar, Michael Laskey, Kevin Stone, Brijen Thananjeyan, Mark Tjersland

This repo contains the code to train the SimNet architecture on procedurally generated simulation data from scratch (no transfer learning required). We also provide a small set of in-house manually labelled validation data containing 3d oriented bounding box labels.

Training the model

Requirements

You will need a Nvidia GPU with at least 12GB of RAM. All code was tested and developed on Ubuntu 20.04.

All commands are assumed to be run from the root of the simnet repo directory (represented by $SIMNET_REPO in commands below).

Setup

Python

Create a python 3.8 virtual environment and install requirements:

cd $SIMNET_REPO
conda create -y --prefix ./env python=3.8
./env/bin/python -m pip install --upgrade pip
./env/bin/python -m pip install -r frozen_requirements.txt

Docker

Make sure docker is installed and working without requiring sudo. If it is not installed, follow the official instructions for setting it up.

docker ps

Wandb

Launch wandb local server for logging training results (you do not need to do this if you already have a wandb account setup). This will launch a local webserver http://localhost:8080 using docker that you can use to visualize training progress and validation images. You will have to visit the http://localhost:8080/authorize page to get the local API access token (this can take a few minutes the first time). Once you get the key you can paste it into the terminal to continue.

cd $SIMNET_REPO
./env/bin/wandb local

Datasets

Download and untar train+val datasets simnet2021a.tar (18GB, md5 checksum:b8e1d3cb7200b44b1de223e87141f14b). This file contains all the training and validation you need to replicate our small objects results.

cd $SIMNET_REPO
wget https://tri-robotics-public.s3.amazonaws.com/github/simnet/datasets/simnet2021a.tar -P datasets
tar xf datasets/simnet2021a.tar -C datasets

Train and Validate

Overfit test:

./runner.sh net_train.py @config/net_config_overfit.txt

Full training run (requires 12GB GPU memory)

./runner.sh net_train.py @config/net_config.txt

Results

Check wandb (http://localhost:8080) to see training progress. On a Titan V, it takes about 48 hours for training to converge, but decent validation results can be seen around 24 hours.

Example validation image visualization:

Example 3D oriented bounding box mAP on validation dataset:

Licenses

The source code is released under the MIT license.

The datasets are released under the Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International License.

You might also like...

The code release of paper Low-Light Image Enhancement with Normalizing Flow

[AAAI 2022] Low-Light Image Enhancement with Normalizing Flow Paper | Project Page Low-Light Image Enhancement with Normalizing Flow Yufei Wang, Renji

176 Jan 6, 2023

PyTorch implementation of our Adam-NSCL algorithm from our CVPR2021 (oral) paper "Training Networks in Null Space for Continual Learning"

Adam-NSCL This is a PyTorch implementation of Adam-NSCL algorithm for continual learning from our CVPR2021 (oral) paper: Title: Training Networks in N

34 Dec 21, 2022

Code release for NeX: Real-time View Synthesis with Neural Basis Expansion

Comments

depth noise model

I was looking through the code and was curious about the depth noise model. I found this: https://github.com/ToyotaResearchInstitute/simnet/blob/main/simnet/lib/camera.py but I can't seem to find camera_noise. Is it in the repository?

opened by seann999 1
Pre-trained Models

Hi Kevin and the team,

Thanks for making the data and code available, really impressive work on the paper.

Is there any plans to make the pre-trained model available, especially the SimNet benchmarked in the paper.

Thanks,

opened by ppyht2 0

Code release for our paper, "SimNet: Enabling Robust Unknown Object Manipulation from Pure Synthetic Data via Stereo"

Related tags

Overview

SimNet: Enabling Robust Unknown Object Manipulation from Pure Synthetic Data via Stereo

Training the model

Requirements

Setup

Python

Docker

Wandb

Datasets

Train and Validate

Results

Licenses

You might also like...

The code release of paper Low-Light Image Enhancement with Normalizing Flow

PyTorch implementation of our Adam-NSCL algorithm from our CVPR2021 (oral) paper "Training Networks in Null Space for Continual Learning"

Code release for NeX: Real-time View Synthesis with Neural Basis Expansion

Code release for "Transferable Semantic Augmentation for Domain Adaptation" (CVPR 2021)

Code release for "COTR: Correspondence Transformer for Matching Across Images"

We will release the code of "ConTNet: Why not use convolution and transformer at the same time?" in this repo

This is the dataset and code release of the OpenRooms Dataset.

Code release for DS-NeRF (Depth-supervised Neural Radiance Fields)

Code release for BlockGAN: Learning 3D Object-aware Scene Representations from Unlabelled Images

Comments

depth noise model

Pre-trained Models

Releases(v0.0.1)

v0.0.1(Jul 19, 2021)

Owner

Enhancing Knowledge Tracing via Adversarial Training

Official Repository of NeurIPS2021 paper: PTR

The GitHub repository for the paper: “Time Series is a Special Sequence: Forecasting with Sample Convolution and Interaction“.

Pytorch implementation for "Open Compound Domain Adaptation" (CVPR 2020 ORAL)

Implementation of "Selection via Proxy: Efficient Data Selection for Deep Learning" from ICLR 2020.

Learning from Synthetic Data with Fine-grained Attributes for Person Re-Identification

This script runs neural style transfer against the provided content image.

An University Project of Quera Web Crawling.

Official implementation of NLOS-OT: Passive Non-Line-of-Sight Imaging Using Optimal Transport (IEEE TIP, accepted)

African language Speech Recognition - Speech-to-Text

Dcf-game-infrastructure-public - Contains all the components necessary to run a DC finals (attack-defense CTF) game from OOO

[WWW 2022] Zero-Shot Stance Detection via Contrastive Learning

CIFAR-10_train-test - training and testing codes for dataset CIFAR-10

This is a re-implementation of TransGAN: Two Pure Transformers Can Make One Strong GAN (CVPR 2021) in PyTorch.

Covid19-Forecasting - An interactive website that tracks, models and predicts COVID-19 Cases

Pytorch Performace Tuning, WandB, AMP, Multi-GPU, TensorRT, Triton

Moment-DETR code and QVHighlights dataset

An OpenAI-Gym Package for Training and Testing Reinforcement Learning algorithms with OpenSim Models

More than a hundred strange attractors

Official PyTorch implementation of the paper Image-Based CLIP-Guided Essence Transfer.