This package is for running the semantic SLAM algorithm using extracted planar surfaces from the received detection

Related tags

Deep Learningsemantic_slam
Overview

Semantic SLAM

This package can perform optimization of pose estimated from VO/VIO methods which tend to drift over time. It uses planar surfaces extracted from object detections in order to create a sparse semantic map of the environment, thus optimizing the drift of the VO/VIO algorithms.

In order to run this package you will need two additional modules

Currently it can extract planar surfaces and create a semantic map from from the following objects:

  • chair
  • tvmonitor
  • book
  • keyboard
  • laptop
  • bucket
  • car

Related Paper:

@ARTICLE{9045978,
  author={Bavle, Hriday and De La Puente, Paloma and How, Jonathan P. and Campoy, Pascual},
  journal={IEEE Access}, 
  title={VPS-SLAM: Visual Planar Semantic SLAM for Aerial Robotic Systems}, 
  year={2020},
  volume={8},
  number={},
  pages={60704-60718},
  doi={10.1109/ACCESS.2020.2983121}}

Video

Semantic SLAM

How do I set it up?

First install g2o following these instructions (Tested on Kinetic and Melodic Distributions):

- sudo apt-get install ros-$ROS_DISTRO-libg2o
- sudo cp -r /opt/ros/$ROS_DISTRO/lib/libg2o_* /usr/local/lib
- sudo cp -r /opt/ros/$ROS_DISTRO/include/g2o /usr/local/include

Install OctopMap server for map generation capabilities:

- sudo apt install ros-$ROS_DISTRO-octomap*

Try a simple example with pre-recorded VIO pose and a blue bucket detector:

Create a ros workspace and clone the following packages:

  • Download the rosbag:
    wget -P ~/Downloads/ https://www.dropbox.com/s/jnywuvcn2m9ubu2/entire_lab_3_rounds.bag
  • Create a workspace, clone the repo and compile:
    mkdir -p workspace/ros/semantic_slam_ws/src/ && cd workspace/ros/semantic_slam_ws/src/    
    git clone https://github.com/hridaybavle/semantic_slam && git clone https://bitbucket.org/hridaybavle/bucket_detector.git   
    cd .. && catkin build --cmake-args -DCMAKE_BUILD_TYPE=Release
  • Launch and visualize
    source devel/setup.bash
    roslaunch semantic_SLAM ps_slam_with_snap_pose_bucket_det_lab_data_with_octomap.launch bagfile:=${HOME}/Downloads/entire_lab_3_rounds.bag show_rviz:=true

test

Using Docker Image

If the code is giving problems with you local machine, you can try the docker image created with the repo and the required settings.

Download Docker from: Docker

Follow the commands to run the algorithm with the docker

  docker pull hridaybavle/semantic_slam:v1 	
  docker run --rm -it --net="host" -p 11311:11311 hridaybavle/semantic_slam:v1 bash
  cd ~/workspace/ros/semantic_slam_ws/
  source devel/setup.bash
  roslaunch semantic_SLAM ps_slam_with_snap_pose_bucket_det_lab_data_with_octomap.launch bagfile:=${HOME}/Downloads/entire_lab_3_rounds.bag show_rviz:=false

Open a new terminal and rviz in local machine

  cd ~/Downloads/ && wget https://raw.githubusercontent.com/hridaybavle/semantic_slam/master/rviz/graph_semantic_slam.rviz
  rviz -d graph_semantic_slam.rviz

Subsribed Topics

Published Topics

The configurations of the algorithms can be found inside the cfg folder in order to be changed accordingly.

Published TFs

  • map to odom transform: The transform published between the map frame and the odom frame after the corrections from the semantic SLAM.

  • base_link to odom transform: The transform published between the base_link (on the robot) frame and the odom frame as estimated by the VO/VIO algorithm.

You might also like...
Indoor Panorama Planar 3D Reconstruction via Divide and Conquer
Indoor Panorama Planar 3D Reconstruction via Divide and Conquer

HV-plane reconstruction from a single 360 image Code for our paper in CVPR 2021: Indoor Panorama Planar 3D Reconstruction via Divide and Conquer (pape

sunset 36 Jan 3, 2023
[ICCV 2021 (oral)] Planar Surface Reconstruction from Sparse Views
[ICCV 2021 (oral)] Planar Surface Reconstruction from Sparse Views

Planar Surface Reconstruction From Sparse Views Linyi Jin, Shengyi Qian, Andrew Owens, David F. Fouhey University of Michigan ICCV 2021 (Oral) This re

Linyi Jin 89 Jan 5, 2023
PyTorch implementation of HDN(Homography Decomposition Networks) for planar object tracking
PyTorch implementation of HDN(Homography Decomposition Networks) for planar object tracking

Homography Decomposition Networks for Planar Object Tracking This project is the offical PyTorch implementation of HDN(Homography Decomposition Networ

CaptainHook 48 Dec 15, 2022
Pytorch implementation of paper:
Pytorch implementation of paper: "NeurMiPs: Neural Mixture of Planar Experts for View Synthesis"

NeurMips: Neural Mixture of Planar Experts for View Synthesis This is the official repo for PyTorch implementation of paper "NeurMips: Neural Mixture

James Lin 101 Dec 13, 2022
Sequence lineage information extracted from RKI sequence data repo
Sequence lineage information extracted from RKI sequence data repo

Pango lineage information for German SARS-CoV-2 sequences This repository contains a join of the metadata and pango lineage tables of all German SARS-

Cornelius Roemer 24 Oct 26, 2022
Official page of Struct-MDC (RA-L'22 with IROS'22 option); Depth completion from Visual-SLAM using point & line features
Official page of Struct-MDC (RA-L'22 with IROS'22 option); Depth completion from Visual-SLAM using point & line features

Struct-MDC (click the above buttons for redirection!) Official page of "Struct-MDC: Mesh-Refined Unsupervised Depth Completion Leveraging Structural R

Urban Robotics Lab. @ KAIST 37 Dec 22, 2022
Code for
Code for "Diffusion is All You Need for Learning on Surfaces"

Source code for "Diffusion is All You Need for Learning on Surfaces", by Nicholas Sharp Souhaib Attaiki Keenan Crane Maks Ovsjanikov NOTE: the linked

Nick Sharp 247 Dec 28, 2022
Code for Iso-Points: Optimizing Neural Implicit Surfaces with Hybrid Representations
Code for Iso-Points: Optimizing Neural Implicit Surfaces with Hybrid Representations

Implementation for Iso-Points (CVPR 2021) Official code for paper Iso-Points: Optimizing Neural Implicit Surfaces with Hybrid Representations paper |

Yifan Wang 66 Nov 8, 2022
Neural-Pull: Learning Signed Distance Functions from Point Clouds by Learning to Pull Space onto Surfaces(ICML 2021)
Neural-Pull: Learning Signed Distance Functions from Point Clouds by Learning to Pull Space onto Surfaces(ICML 2021)

Neural-Pull: Learning Signed Distance Functions from Point Clouds by Learning to Pull Space onto Surfaces(ICML 2021) This repository contains the code

null 149 Dec 15, 2022
Comments
  • errors at last step

    errors at last step

    Hi, I have finished all the steps following the instructions and nothing goes wrong. But when I run

    roslaunch semantic_SLAM ps_slam_with_snap_pose_bucket_det_lab_data.launch bagfile:=${HOME}/Downloads/entire_lab_3_rounds.bag show_rviz:=true

    I get errors like this and it stucks for a while.

    # Using CSparse poseDim -1 landMarkDim -1 blockordering 0
done
keyframe_delta_trans 0.5
keyframe_delta_angle 0.5
keyframe_delta_time 1
use_const_inf_matrix: 1
const_stddev_x: 0.00667
const_stddev_q: 1e-05
Initialized mapping thread 
camera angle in radians: 0.59219
update keyframe every detection: 1
add first landmark: 0
[semantic_graph_slam_node-9] process has died [pid 23067, exit code -11, cmd /home/nrc/workspace/ros/semantic_slam_ws/devel/lib/semantic_SLAM/semantic_graph_SLAM_node __name:=semantic_graph_slam_node __log:=/home/nrc/.ros/log/ccaf4b14-a47a-11ea-b300-000c29c39525/semantic_graph_slam_node-9.log].
log file: /home/nrc/.ros/log/ccaf4b14-a47a-11ea-b300-000c29c39525/semantic_graph_slam_node-9*.log

    then I get this. It seems that the visualization program doesn't go right.

    [rosbag-2] process has finished cleanly
log file: /home/nrc/.ros/log/ccaf4b14-a47a-11ea-b300-000c29c39525/rosbag-2*.log

    Is there something I have missed? Thank you!

    opened by ZhengXinyue 8
  • [semantic_graph_slam_node-9] process has died

    [semantic_graph_slam_node-9] process has died

    Hi, I have finished all the steps following the instructions and nothing goes wrong. But when I run

    roslaunch semantic_SLAM ps_slam_with_snap_pose_bucket_det_lab_data_with_octomap.launch bagfile:=${HOME}/Downloads/entire_lab_3_rounds.bag show_rviz:=true

    I get errors like this.

    done
keyframe_delta_trans 0.5
keyframe_delta_angle 0.5
keyframe_delta_time 1
use_const_inf_matrix: 1
const_stddev_x: 0.00667
const_stddev_q: 1e-05
camera angle in radians: 0.59219
update keyframe every detection: 1
add first landmark: 0
[ INFO] [1591944956.099907360, 1661396775.076756992]: waitForService: Service [/depth_rectifier_manager/load_nodelet] is now available.
[ INFO] [1591944956.100243666, 1661396775.076756992]: waitForService: Service [/depth_manager/load_nodelet] is now available.
[ INFO] [1591944956.545617511, 1661396775.518832629]: Stereo is NOT SUPPORTED
[ INFO] [1591944956.545842654, 1661396775.518832629]: OpenGl version: 4.5 (GLSL 4.5).
[pcl::OrganizedNeighbor::radiusSearch] Input dataset is not from a projective device!
Residual (MSE) 0.000614, using 1248 valid points
[pcl::OrganizedNeighbor::radiusSearch] Input dataset is not from a projective device!
Residual (MSE) 0.000748, using 1444 valid points
[pcl::OrganizedNeighbor::radiusSearch] Input dataset is not from a projective device!
Residual (MSE) 0.001710, using 2303 valid points
[semantic_graph_slam_node-9] process has died [pid 27314, exit code -9, cmd /home/nrc/hd/workspace/ros/semantic_slam_ws/devel/lib/semantic_SLAM/semantic_graph_SLAM_node __name:=semantic_graph_slam_node __log:=/home/nrc/.ros/log/c2c4ddd8-ac79-11ea-96ed-8ca982ff1833/semantic_graph_slam_node-9.log].
log file: /home/nrc/.ros/log/c2c4ddd8-ac79-11ea-96ed-8ca982ff1833/semantic_graph_slam_node-9*.log

    When it occurs

    [pcl::OrganizedNeighbor::radiusSearch] Input dataset is not from a projective device!
Residual (MSE) 0.000614, using 1248 valid points

    the program is still mapping , so I think the problem is not caused by 'pcl'.

    I tried to run the launchfile seperately :

    ROS_NAMESPACE=camera/color rosrun image_proc image_proc 
roslaunch semantic_SLAM shape.launch  
rosrun semantic_SLAM  semantic_graph_SLAM_node

    But at the last step i got 'Segmentation fault :

    add first landmark: 0
Segmentation fault (core dumped)

    Do you have any idea about it? Thanks a lot !!!

    opened by He-Rong 6
  • Dataset download failure problem

    Dataset download failure problem

    Hello, when I run the sample code, I always encounter network interruptions or unknown errors at the last moment when downloading the dataset entire_lab_3_rounds.bag. Can you provide a new way to download the bag?

    opened by kycwx 2
  • Problemas de incompatibilidad de opencv en el bucket detector

    Problemas de incompatibilidad de opencv en el bucket detector

    Hola, he conseguido que ambos paquetes en conjunto (semantic slam y bucket detector) funciones bien en una distro de ubuntu virgen con ROS melodic, sin embargo, cuando migro al pc donde trabajo habitualmente y que tiene ya instaladas dependencias anteriores y demás me encuentro con estos errores referentes a opencv: Captura de pantalla de 2021-05-26 11-29-18 Imagino que se deben a incompatibilidades entre versiones de opencv, podrías confirmarme esto último? Sería posible trabajar con una versión de opencv diferente? Gracias, un saludo!

    opened by iandresolares 2
Releases(2.0.0)
Owner
Hriday Bavle
Postdoctoral Researcher at the University of Luxembourg. My research interests are VO/VIO, SLAM, Perception and Planning applied to Mobile Robots.
Hriday Bavle
GitHub Repository
Boundary-preserving Mask R-CNN (ECCV 2020)

BMaskR-CNN This code is developed on Detectron2 Boundary-preserving Mask R-CNN ECCV 2020 Tianheng Cheng, Xinggang Wang, Lichao Huang, Wenyu Liu Video

Hust Visual Learning Team 178 Nov 28, 2022
TransVTSpotter: End-to-end Video Text Spotter with Transformer

TransVTSpotter: End-to-end Video Text Spotter with Transformer Introduction A Multilingual, Open World Video Text Dataset and End-to-end Video Text Sp

weijiawu 66 Dec 26, 2022
🏅 Top 5% in 제2회 연구개발특구 인공지능 경진대회 AI SPARK 챌린지

AI_SPARK_CHALLENG_Object_Detection 제2회 연구개발특구 인공지능 경진대회 AI SPARK 챌린지 🏅 Top 5% in mAP(0.75) (443명 중 13등, mAP: 0.98116) 대회 설명 Edge 환경에서의 가축 Object Dete

3 Sep 19, 2022
Defocus Map Estimation and Deblurring from a Single Dual-Pixel Image

Defocus Map Estimation and Deblurring from a Single Dual-Pixel Image This repository is an implementation of the method described in the following pap

21 Dec 15, 2022
[ECE NTUA] 👁 Computer Vision - Lab Projects & Theoretical Problem Sets (2020-2021)

Computer Vision - NTUA (2020-2021) This repository hosts the lab projects and theoretical problem sets of the Computer Vision course held by ECE NTUA

Dimitris Dimos 6 Jul 21, 2022
Official PyTorch implementation of "AASIST: Audio Anti-Spoofing using Integrated Spectro-Temporal Graph Attention Networks"

AASIST This repository provides the overall framework for training and evaluating audio anti-spoofing systems proposed in 'AASIST: Audio Anti-Spoofing

Clova AI Research 56 Jan 02, 2023
Portfolio analytics for quants, written in Python

QuantStats: Portfolio analytics for quants QuantStats Python library that performs portfolio profiling, allowing quants and portfolio managers to unde

Ran Aroussi 2.7k Jan 08, 2023
Models, datasets and tools for Facial keypoints detection

Template for Data Science Project This repo aims to give a robust starting point to any Data Science related project. It contains readymade tools setu

girafe.ai 1 Feb 11, 2022
Code for our paper "MG-GAN: A Multi-Generator Model Preventing Out-of-Distribution Samples in Pedestrian Trajectory Prediction" published at ICCV 2021.

MG-GAN: A Multi-Generator Model Preventing Out-of-Distribution Samples in Pedestrian Trajectory Prediction This repository contains the code for the p

Sven 30 Jan 05, 2023
PyTorch code for ICPR 2020 paper Future Urban Scene Generation Through Vehicle Synthesis

Future urban scene generation through vehicle synthesis This repository contains Pytorch code for the ICPR2020 paper "Future Urban Scene Generation Th

Alessandro Simoni 4 Oct 11, 2021
Equivariant layers for RC-complement symmetry in DNA sequence data

Equi-RC Equivariant layers for RC-complement symmetry in DNA sequence data This is a repository that implements the layers as described in "Reverse-Co

7 May 19, 2022
An implementation of the Contrast Predictive Coding (CPC) method to train audio features in an unsupervised fashion.

CPC_audio This code implements the Contrast Predictive Coding algorithm on audio data, as described in the paper Unsupervised Pretraining Transfers we

Meta Research 283 Dec 30, 2022
Learning Representational Invariances for Data-Efficient Action Recognition

Learning Representational Invariances for Data-Efficient Action Recognition Official PyTorch implementation for Learning Representational Invariances

Virginia Tech Vision and Learning Lab 27 Nov 22, 2022
House_prices_kaggle - Predict sales prices and practice feature engineering, RFs, and gradient boosting

House Prices - Advanced Regression Techniques Predicting House Prices with Machine Learning This project is build to enhance my knowledge about machin

Gurpreet Singh 1 Jan 01, 2022
NeurIPS 2021 Datasets and Benchmarks Track

AP-10K: A Benchmark for Animal Pose Estimation in the Wild Introduction | Updates | Overview | Download | Training Code | Key Questions | License Intr

AP-10K 82 Dec 11, 2022
GDSC-ML Team Interview Task

GDSC-ML-Team---Interview-Task Task 1 : Clean or Messy room In this task we have to classify the given test images as clean or messy. - Link for datase

Aayush. 1 Jan 19, 2022
lightweight python wrapper for vowpal wabbit

vowpal_porpoise Lightweight python wrapper for vowpal_wabbit. Why: Scalable, blazingly fast machine learning. Install Install vowpal_wabbit. Clone and

Joseph Reisinger 163 Nov 24, 2022
NEG loss implemented in pytorch

Pytorch Negative Sampling Loss Negative Sampling Loss implemented in PyTorch. Usage neg_loss = NEG_loss(num_classes, embedding_size) optimizer =

Daniil Gavrilov 123 Sep 13, 2022
This is an official implementation for "Exploiting Temporal Contexts with Strided Transformer for 3D Human Pose Estimation".

Exploiting Temporal Contexts with Strided Transformer for 3D Human Pose Estimation This repo is the official implementation of Exploiting Temporal Con

Vegetabird 241 Jan 07, 2023
Deploy pytorch classification model using Flask and Streamlit

Deploy pytorch classification model using Flask and Streamlit

Ben Seo 1 Nov 17, 2021