GeoTransformer - Geometric Transformer for Fast and Robust Point Cloud Registration

Overview

Geometric Transformer for Fast and Robust Point Cloud Registration

PyTorch implementation of the paper:

Geometric Transformer for Fast and Robust Point Cloud Registration.

Zheng Qin, Hao Yu, Changjian Wang, Yulan Guo, Yuxing Peng, and Kai Xu

Introduction

We study the problem of extracting accurate correspondences for point cloud registration. Recent keypoint-free methods bypass the detection of repeatable keypoints which is difficult in low-overlap scenarios, showing great potential in registration. They seek correspondences over downsampled superpoints, which are then propagated to dense points. Superpoints are matched based on whether their neighboring patches overlap. Such sparse and loose matching requires contextual features capturing the geometric structure of the point clouds. We propose Geometric Transformer to learn geometric feature for robust superpoint matching. It encodes pair-wise distances and triplet-wise angles, making it robust in low-overlap cases and invariant to rigid transformation. The simplistic design attains surprisingly high matching accuracy such that no RANSAC is required in the estimation of alignment transformation, leading to $100$ times acceleration. Our method improves the inlier ratio by $17% \sim 30%$ and the registration recall by over $7%$ on the challenging 3DLoMatch benchmark. Code will be released for paper reproduction.

News

Installation

Please use the following command for installation.

# It is recommended to create a new environment
conda create -n geotransformer python==3.8
conda activate geotransformer

# [Optional] If you are using CUDA 11.0 or newer, please install `torch==1.7.1+cu110`
pip install torch==1.7.1+cu110 -f https://download.pytorch.org/whl/torch_stable.html

# Install packages and other dependencies
python setup.py build develop

# Compile c++ wrappers
cd geotransformer/cpp_wrappers
sh ./compile_wrappers.sh

Code has been tested with Ubuntu 20.04, GCC 9.3.0, Python 3.8, PyTorch 1.7.1, CUDA 11.1 and cuDNN 8.1.0.

Data preparation

We provide code for training and testing on 3DMatch.

The dataset can be download from PREDATOR. The data should be organized as follows:

--data--3DMatch--metadata
              |--data--train--7-scenes-chess--cloud_bin_0.pth
                    |      |               |--...
                    |      |--...
                    |--test--7-scenes-redkitchen--cloud_bin_0.pth
                          |                    |--...
                          |--...

Training

The code for GeoTransformer is in experiments/geotransformer.3dmatch. Use the following command for training.

CUDA_VISIBLE_DEVICES=0 python trainval.py
# use "--snapshot=path/to/snapshot" to resume training.

Testing

Use the following command for testing.

# 3DMatch
CUDA_VISIBLE_DEVICES=0 ./eval.sh EPOCH 3DMatch
# 3DLoMatch
CUDA_VISIBLE_DEVICES=0 ./eval.sh EPOCH 3DLoMatch

EPOCH is the epoch id.

We also provide pretrained weights in weights, use the following command to test the pretrained weights.

CUDA_VISIBLE_DEVICES=0 python test.py --snapshot=../../weights/geotransformer-3dmatch.pth.tar --benchmark=3DMatch
CUDA_VISIBLE_DEVICES=0 python eval.py --run_matching --run_registration --benchmark=3DMatch

Replace 3DMatch with 3DLoMatch to evaluate on 3DLoMatch.

Results

Benchmark FMR IR RR
3DMatch 97.7 70.3 91.5
3DLoMatch 88.1 43.3 74.0
Comments
  • Test and visualize on two given point clouds

    Test and visualize on two given point clouds

    Hi, thanks for sharing your work. Is it easy to give some guidance or do you have any tutorial to test/evaluate the pre-trained models on two individual given point clouds?

    opened by ttsesm 10
  • Question about kitti result reproduce.

    Question about kitti result reproduce.

    Hi ,thanks for your amazing work.

    However , I have met a problem of kitti dataset result reproduce. I followed your code to train and test on kitti dataset , but my result seems always lower than the result in your paper. I can achieve Registration recall of 99.8 and RTE 6.8 same as yours ,but my result of RRE is about 0.33 which is much higher than 0.24 in your paper.

    Can you give me some idea of how to reproduce the same result ?

    opened by vision507 9
  • How to draw the point cloud like this ?

    How to draw the point cloud like this ?

    Hi , thanks for your great work. And I would like to know how can I get this kind of visualization result in your paper. 屏幕截图 2022-08-04 081115 When I use open3d to draw the point clouds , it seems very ugly, so I'm wondering how can I get your visualization result. It would be very grateful.

    opened by vision507 9
  • How to group data sets after downloading

    How to group data sets after downloading

    Hello, author: I ran the python files in the Kitti dataset and ModelNet dataset folders according to your requirements, but they did not work and were not classified into the ones you described. Is there a specific method? thank you!

    opened by yangsihai97 9
  • Some questions about downsampling of point clouds

    Some questions about downsampling of point clouds

    Hi there,

    Really appreciate releasing your amazing work! I'm quite new to point clouds and I have some questions regarding downsampling of the point clouds.

    Take 3DMatch as an example. The original raw dataset downloaded from PREDATOR is quite dense, and is preprocessed so that each grid size is 2.5cm. In your work, you further subsample the point clouds using KPConv (the first level downsampled points). My first question is what's the purpose of subsampling? Would a denser point cloud be more informative for registration?

    Also, is there a specific reason you downsample the points further for dense point correspondences as I believe the preprocessed input data is already of density 2.5cm?

    Last, is there a quick way to visualize the registration result for the 3DMatch benchmark? By running eval.py and test.py it seems to only produce the quantitative metrics.

    Thanks for your patience and reply in advance!

    opened by jinhong-ni 5
  • 3DMatch dataset

    3DMatch dataset

    Hi,

    I have a question regarding the D.2. section from the Supplementary material.

    Could you please share which 3DMatch files you use for the evaluation?

    • Do you use those with overlap > 30%?
    • Do you use consecutive pairs of point cloud indices (the original 3DMatch filters the pairs (i,j) with i+1 < j)?

    You mention that you use the files from PREDATOR, but PREDATOR has more pairs than the original 3DMatch dataset. Can you please elaborate?

    Thank you in advance for you response. Hopefully we can clear all this confusion of the registration pairs used in the datasets. Every paper seems to do it a bit differently.

    opened by DavidBoja 4
  • why need gt transoform in demo.py

    why need gt transoform in demo.py

    in line 74 in experiments/geotransformer.3dmatch.stage4.gse.k3.max.oacl.stage2.sinkhorn/model.py , transform = data_dict['transform'].detach(), and in line 110 you use gt transform in get_node_correspondences. however If i don't know how the transform matrix of src to ref, how can I use demo.py

    opened by MaybeOjbk 4
  • The different RR in test.py and eval.py

    The different RR in test.py and eval.py

    Hello, thank you for you amazing work again. When I evaluated on 3DMatch, I found the RR reported by test.py and eval.py are very different, what makes this difference?

    opened by Hui-design 4
  • Question about radius search

    Question about radius search

    First of all, thanks for your excellent work! Radius search in KPConv is quiet slow for large scale point clouds and it usually takes seconds in my own experiments. Is there a more efficient way for radius search, such as a version that can run on GPU. Hope to get your answer!

    opened by Fleurrr 4
  • Questions about RANSAC

    Questions about RANSAC

    Dear authors,

    Thanks for open-sourcing your amazing work. I'm experimenting with RANSAC and I noticed some unexpected results compared to the paper.

    What I've done is first change the number of iterations in RANSAC to 50k. The evaluation runs extremely slowly (approx. 16 hrs in contrast to about 1 hr for local to global), and I only got performance similar to local to global. I also tried to change the number of points in RANSAC to a larger number (by default it is set to 3), such as 250, 500, 1000, etc. I did not achieve a significant gain in performance, even the recall collapses to zero if the number of points is set too big (like 5k). This is contradicting to what's reported in the paper, as the performance is reported to keep improving as the number of samples increases.

    I would be super grateful if you could guide me through what I've done wrong here. Thanks in advance for your patience and help!

    opened by jinhong-ni 4
  • RuntimeError: Error compiling objects for extension

    RuntimeError: Error compiling objects for extension

    Hello, sir. The question is about creating the code runtime environment. After reinstalling CUDA and downloading torch, I executed 'python setup.py build develop' and it still report the error, 'RuntimeError: Error compiling objects for extension'.

    Traceback (most recent call last): File "setup.py", line 5, in setup( File "F:\Anaconda3\envs\geotransformer\lib\site-packages\setuptools_init_.py", line 87, in setup return distutils.core.setup(**attrs) File "F:\Anaconda3\envs\geotransformer\lib\site-packages\setuptools_distutils\core.py", line 148, in setup return run_commands(dist) File "F:\Anaconda3\envs\geotransformer\lib\site-packages\setuptools_distutils\core.py", line 163, in run_commands dist.run_commands() File "F:\Anaconda3\envs\geotransformer\lib\site-packages\setuptools_distutils\dist.py", line 967, in run_commands self.run_command(cmd) File "F:\Anaconda3\envs\geotransformer\lib\site-packages\setuptools\dist.py", line 1214, in run_command super().run_command(command) File "F:\Anaconda3\envs\geotransformer\lib\site-packages\setuptools_distutils\dist.py", line 986, in run_command cmd_obj.run() File "F:\Anaconda3\envs\geotransformer\lib\site-packages\setuptools_distutils\command\build.py", line 135, in run self.run_command(cmd_name) File "F:\Anaconda3\envs\geotransformer\lib\site-packages\setuptools_distutils\cmd.py", line 313, in run_command self.distribution.run_command(command) File "F:\Anaconda3\envs\geotransformer\lib\site-packages\setuptools\dist.py", line 1214, in run_command super().run_command(command) File "F:\Anaconda3\envs\geotransformer\lib\site-packages\setuptools_distutils\dist.py", line 986, in run_command cmd_obj.run() File "F:\Anaconda3\envs\geotransformer\lib\site-packages\setuptools\command\build_ext.py", line 79, in run _build_ext.run(self) File "F:\Anaconda3\envs\geotransformer\lib\site-packages\setuptools_distutils\command\build_ext.py", line 339, in run self.build_extensions() File "F:\Anaconda3\envs\geotransformer\lib\site-packages\torch\utils\cpp_extension.py", line 670, in build_extensions build_ext.build_extensions(self) File "F:\Anaconda3\envs\geotransformer\lib\site-packages\setuptools_distutils\command\build_ext.py", line 448, in build_extensions self._build_extensions_serial() File "F:\Anaconda3\envs\geotransformer\lib\site-packages\setuptools_distutils\command\build_ext.py", line 473, in _build_extensions_serial self.build_extension(ext) File "F:\Anaconda3\envs\geotransformer\lib\site-packages\setuptools\command\build_ext.py", line 202, in build_extension _build_ext.build_extension(self, ext) File "F:\Anaconda3\envs\geotransformer\lib\site-packages\setuptools_distutils\command\build_ext.py", line 528, in build_extension objects = self.compiler.compile(sources, File "F:\Anaconda3\envs\geotransformer\lib\site-packages\torch\utils\cpp_extension.py", line 643, in win_wrap_ninja_compile _write_ninja_file_and_compile_objects( File "F:\Anaconda3\envs\geotransformer\lib\site-packages\torch\utils\cpp_extension.py", line 1250, in _write_ninja_file_and_compile_objects _run_ninja_build( File "F:\Anaconda3\envs\geotransformer\lib\site-packages\torch\utils\cpp_extension.py", line 1555, in _run_ninja_build raise RuntimeError(message) from e RuntimeError: Error compiling objects for extension

    Is this because I use win10? Or other issues caused the error?

    opened by ChaunceyQ 4
  • The training model is not effective

    The training model is not effective

    Thank you for your excellent work. When I use the LGR training model without changing the code, the result on 3DMatch (about 91.0%) was still quite different from the model you provided (92.5%)

    opened by awdx551 3
Owner
Zheng Qin
computer vision, deep learning
Zheng Qin
It's A ML based Web Site build with python and Django to find the breed of the dog

ML-Based-Dog-Breed-Identifier This is a Django Based Web Site To Identify the Breed of which your DOG belogs All You Need To Do is to Follow These Ste

Sanskar Dwivedi 2 Oct 12, 2022
Finding an Unsupervised Image Segmenter in each of your Deep Generative Models

Finding an Unsupervised Image Segmenter in each of your Deep Generative Models Description Recent research has shown that numerous human-interpretable

Luke Melas-Kyriazi 61 Oct 17, 2022
SoK: Vehicle Orientation Representations for Deep Rotation Estimation

SoK: Vehicle Orientation Representations for Deep Rotation Estimation Raymond H. Tu, Siyuan Peng, Valdimir Leung, Richard Gao, Jerry Lan This is the o

FIRE Capital One Machine Learning of the University of Maryland 12 Oct 07, 2022
DenseCLIP: Language-Guided Dense Prediction with Context-Aware Prompting

DenseCLIP: Language-Guided Dense Prediction with Context-Aware Prompting Created by Yongming Rao*, Wenliang Zhao*, Guangyi Chen, Yansong Tang, Zheng Z

Yongming Rao 321 Dec 27, 2022
Providing the solutions for high-frequency trading (HFT) strategies using data science approaches (Machine Learning) on Full Orderbook Tick Data.

Modeling High-Frequency Limit Order Book Dynamics Using Machine Learning Framework to capture the dynamics of high-frequency limit order books. Overvi

Chang-Shu Chung 1.3k Jan 07, 2023
code for our ECCV-2020 paper: Self-supervised Video Representation Learning by Pace Prediction

Video_Pace This repository contains the code for the following paper: Jiangliu Wang, Jianbo Jiao and Yunhui Liu, "Self-Supervised Video Representation

Jiangliu Wang 95 Dec 14, 2022
Official PyTorch Implementation of Unsupervised Learning of Scene Flow Estimation Fusing with Local Rigidity

UnRigidFlow This is the official PyTorch implementation of UnRigidFlow (IJCAI2019). Here are two sample results (~10MB gif for each) of our unsupervis

Liang Liu 28 Nov 16, 2022
A big endian Gentoo port developed on a Pine64.org RockPro64

Gentoo-aarch64_be A big endian Gentoo port developed on a Pine64.org RockPro64 The endian wars are over... little endian won. As a result, it is incre

Rory Bolt 6 Dec 07, 2022
Pytorch implementation of COIN, a framework for compression with implicit neural representations 🌸

COIN 🌟 This repo contains a Pytorch implementation of COIN: COmpression with Implicit Neural representations, including code to reproduce all experim

Emilien Dupont 104 Dec 14, 2022
Official PyTorch repo for JoJoGAN: One Shot Face Stylization

JoJoGAN: One Shot Face Stylization This is the PyTorch implementation of JoJoGAN: One Shot Face Stylization. Abstract: While there have been recent ad

1.3k Dec 29, 2022
Convolutional Neural Networks on Graphs with Fast Localized Spectral Filtering

Graph ConvNets in PyTorch October 15, 2017 Xavier Bresson http://www.ntu.edu.sg/home/xbresson https://github.com/xbresson https://twitter.com/xbresson

Xavier Bresson 287 Jan 04, 2023
Deal or No Deal? End-to-End Learning for Negotiation Dialogues

Introduction This is a PyTorch implementation of the following research papers: (1) Hierarchical Text Generation and Planning for Strategic Dialogue (

Facebook Research 1.4k Dec 29, 2022
DNA sequence classification by Deep Neural Network

DNA sequence classification by Deep Neural Network: Project Overview worked on the DNA sequence classification problem where the input is the DNA sequ

Mohammed Jawwadul Islam Fida 0 Aug 02, 2022
FedGS: A Federated Group Synchronization Framework Implemented by LEAF-MX.

FedGS: Data Heterogeneity-Robust Federated Learning via Group Client Selection in Industrial IoT Preparation For instructions on generating data, plea

Lizonghang 9 Dec 22, 2022
VLGrammar: Grounded Grammar Induction of Vision and Language

VLGrammar: Grounded Grammar Induction of Vision and Language

Yining Hong 27 Dec 23, 2022
The repository offers the official implementation of our paper in PyTorch.

Cloth Interactive Transformer (CIT) Cloth Interactive Transformer for Virtual Try-On Bin Ren1, Hao Tang1, Fanyang Meng2, Runwei Ding3, Ling Shao4, Phi

Bingoren 49 Dec 01, 2022
Automated Evidence Collection for Fake News Detection

Automated Evidence Collection for Fake News Detection This is the code repo for the Automated Evidence Collection for Fake News Detection paper accept

Mrinal Rawat 2 Apr 12, 2022
BEAS: Blockchain Enabled Asynchronous & Secure Federated Machine Learning

BEAS Blockchain Enabled Asynchronous and Secure Federated Machine Learning Default Network Configuration: The default application uses the HyperLedger

Harpreet Virk 11 Nov 20, 2022
SCAAML is a deep learning framwork dedicated to side-channel attacks run on top of TensorFlow 2.x.

SCAAML (Side Channel Attacks Assisted with Machine Learning) is a deep learning framwork dedicated to side-channel attacks. It is written in python and run on top of TensorFlow 2.x.

Google 69 Dec 21, 2022
NOD: Taking a Closer Look at Detection under Extreme Low-Light Conditions with Night Object Detection Dataset

NOD (Night Object Detection) Dataset NOD: Taking a Closer Look at Detection under Extreme Low-Light Conditions with Night Object Detection Dataset, BM

Igor Morawski 17 Nov 05, 2022