Pip-package for trajectory benchmarking from "Be your own Benchmark: No-Reference Trajectory Metric on Registered Point Clouds", ECMR'21

Related tags

Deep Learningmap-metrics
Overview

Map Metrics for Trajectory Quality

Map metrics toolkit provides a set of metrics to quantitatively evaluate trajectory quality via estimating consistency of the map aggregated from point clouds.

GPS or Motion Capture systems are not always available in perception systems, or their quality is not enough (GPS on small-scale distances) for use as ground truth trajectory. Thus, common full-reference trajectory metrics (APE, RPE, and their modifications) could not be applied to evaluate trajectory quality. When 3D sensing technologies (depth camera, LiDAR) are available on the perception system, one can alternatively assess trajectory quality --- estimate the consistency of the map from registered point clouds via the trajectory.

Documentation: https://map-metrics.readthedocs.io.

Documentation Status Updates

Features

Our toolkit provides implementation of the next metrics:

  • Mean Map Entropy (MME), Mean Plane Variance(MPV) [1] [2]
  • Mutually Orthogonal Metric (MOM) [3] -- has strong correlation with RPE

Citation

If you use this toolkit or MOM-metric results, please, cite our work:

@misc{kornilova2021benchmark,
    title={Be your own Benchmark: No-Reference Trajectory Metric on Registered Point Clouds},
    author={Anastasiia Kornilova and Gonzalo Ferrer},
    year={2021},
    eprint={2106.11351},
    archivePrefix={arXiv},
    primaryClass={cs.RO}
}

Credits

This package was created with Cookiecutter and the audreyr/cookiecutter-pypackage project template.

Links

[1] Droeschel, David, Jörg Stückler, and Sven Behnke. "Local multi-resolution representation for 6D motion estimation and mapping with a continuously rotating 3D laser scanner." 2014 IEEE International Conference on Robotics and Automation (ICRA). IEEE, 2014.
[2] Razlaw, Jan, et al. "Evaluation of registration methods for sparse 3D laser scans." 2015 European Conference on Mobile Robots (ECMR). IEEE, 2015.
[3] Kornilova, Anastasiia, and Gonzalo Ferrer. "Be your own Benchmark: No-Reference Trajectory Metric on Registered Point Clouds." arXiv preprint arXiv:2106.11351 (2021).
Comments
  • Cross-Platform Wheels

    Cross-Platform Wheels

    Linux and Windows32-64 wheels.

    MacOS is running into from map_metrics import map_metrics E ImportError: dlopen(/Users/runner/hostedtoolcache/Python/3.6.15/x64/lib/python3.6/site-packages/map_metrics/map_metrics.cpython-36m-darwin.so, 2): Library not loaded: @rpath/libboost_graph-mt-x64.dylib E Referenced from: /Users/runner/hostedtoolcache/Python/3.6.15/x64/lib/python3.6/site-packages/map_metrics/map_metrics.cpython-36m-darwin.so E Reason: image not found

    opened by achains 1
  • Speed up Github Actions workflow

    Speed up Github Actions workflow

    It takes an hour to build Open3D

    Possible solutions:

    • [ ] Add -j flag
    • [ ] Specify Open3D build options
    • [ ] Store Open3D files in Github Actions Cache
    enhancement 
    opened by achains 1
  • Python version interface improvements

    Python version interface improvements

    Tasks:

    • [x] Break down large functions
    • [x] Break functions into modules
    • [x] Build pip-package, configure CI
    • [x] Add more functions, e.g. I/O processing
    opened by achains 0
  • pip-package. MPV. MME

    pip-package. MPV. MME

    Overview

    Python package wheels with implemented basic metrics (MPV, MME) on C++

    Platform tags

    • manylinux2010_x86_64
    • macosx_10_14_x86_64
    • win32 / win_amd64

    Python version

    • Python >=3.6
    opened by achains 0
  • Cosmetic fixes

    Cosmetic fixes

    What was done:

    • Removed redundant comments
    • Implemented general interface for methods
    • Added missing headers
    • Unsigned long -> int
    • All functions have CamelCase now
    opened by achains 0
  • CMake project structure. Baseline MME and MPV

    CMake project structure. Baseline MME and MPV

    Progress:

    • Configured CMake files
    • Baseline of mme and mpv algorithms

    Notes (Tasks for next PR)

    • Algorithms need to be tested
    • CI should be configured
    opened by achains 0
  • A drawback of mom when dealing drafting walls in indoor envs.

    A drawback of mom when dealing drafting walls in indoor envs.

    • Map Metrics 0.0.1:
    • Python version 3.8:
    • Operating System win10: 1

    Description

    It tired the metirc on a small scale data, likely a room with differential chassis and a RS-16. When the point cloud map consists of multiple wall (actually one) caused by drifting, the mom metric may have a smaller value than a normal map. I think it is caused by the orthogonal walls (both the real one and drifted one) and floor.

    Specifically, Fig.1 is generated by LOAM (without drift), and Fig. 2 is gererated by Lio-sam (with draft).

    Fig.1

    Fig.2

    I also considered to downsample the two maps into the same scale with voxel downsampling. The results are listed below.

    1640311688(1)

    I think this kind of problem is very familiar for indoor, I have seen many drafting wall during daily usage of lidar based slam, without semantic labels, it is very hard for distinguishing them automatically.

    What I Did

    I will try the 0.0.2 version later.

    Paste the command(s) you ran and the output.
If there was a crash, please include the traceback here.
    opened by hahakid 2
  • Add C++ executable with console interface

    Add C++ executable with console interface

    It will be useful for quick testing and performance measurments if library is able to run from command-line interface.

    For example, ./map-metrics --pc="data/kitti_00" --tj="data/Tj_0" --metric="mme" [--"some common hyperparameters"]

    It's build should be disabled by default

    enhancement 
    opened by achains 0
  • Support 10.9 MacOS

    Support 10.9 MacOS

    • Map Metrics version: 0.0.3
    • Python version: >=3.6
    • Operating System: MacOS

    Description

    Pip-package supports MacOS >=10.14 due to C++17 features. We can downgrade to C++14, but a bit later :)

    enhancement 
    opened by achains 0
  • editoring error

    editoring error

    A \delta seems missing in Eq.6 of the paper. Have you tested on all the sequences and other dataset. Recommend to ref LIBRE: The Multiple 3D LiDAR Dataset. VLP64 may have a low precise for long distance measurement. clipboard

    opened by hahakid 1
Owner
Mobile Robotics Lab. at Skoltech
Mobile Robotics Lab. at Skoltech
GitHub Repository
Exe-to-xlsm - Simple script to create VBscript of exe and inject to xlsm

🎁 Exe To Office Executable file injection to Office documents: .xlsm, .docm, .p

3 Jan 25, 2022
Implementation of the method described in the Speech Resynthesis from Discrete Disentangled Self-Supervised Representations.

Speech Resynthesis from Discrete Disentangled Self-Supervised Representations Implementation of the method described in the Speech Resynthesis from Di

4 Mar 11, 2022
Style transfer, deep learning, feature transform

FastPhotoStyle License Copyright (C) 2018 NVIDIA Corporation. All rights reserved. Licensed under the CC BY-NC-SA 4.0 license (https://creativecommons

NVIDIA Corporation 10.9k Jan 02, 2023
Visual Tracking by TridenAlign and Context Embedding

Visual Tracking by TridentAlign and Context Embedding (TACT) Test code for "Visual Tracking by TridentAlign and Context Embedding" Janghoon Choi, Juns

Janghoon Choi 32 Aug 25, 2021
Contrastive Learning with Non-Semantic Negatives

Contrastive Learning with Non-Semantic Negatives This repository is the official implementation of Robust Contrastive Learning Using Negative Samples

39 Jul 31, 2022
Official code for 'Pixel-wise Energy-biased Abstention Learning for Anomaly Segmentationon Complex Urban Driving Scenes'

PEBAL This repo contains the Pytorch implementation of our paper: Pixel-wise Energy-biased Abstention Learning for Anomaly Segmentationon Complex Urba

Yu Tian 115 Dec 29, 2022
A Text Attention Network for Spatial Deformation Robust Scene Text Image Super-resolution (CVPR2022)

A Text Attention Network for Spatial Deformation Robust Scene Text Image Super-resolution (CVPR2022) https://arxiv.org/abs/2203.09388 Jianqi Ma, Zheto

MA Jianqi, shiki 104 Jan 05, 2023
Parameterized Explainer for Graph Neural Network

PGExplainer This is a Tensorflow implementation of the paper: Parameterized Explainer for Graph Neural Network https://arxiv.org/abs/2011.04573 NeurIP

Dongsheng Luo 89 Dec 12, 2022
Code for Generating Disentangled Arguments with Prompts: A Simple Event Extraction Framework that Works

GDAP Code for Generating Disentangled Arguments with Prompts: A Simple Event Extraction Framework that Works Environment Python (verified: v3.8) CUDA

45 Oct 29, 2022
Point Cloud Registration Network

PCRNet: Point Cloud Registration Network using PointNet Encoding Source Code Author: Vinit Sarode and Xueqian Li Paper | Website | Video | Pytorch Imp

ViNiT SaRoDe 59 Nov 19, 2022
Public repository of the 3DV 2021 paper "Generative Zero-Shot Learning for Semantic Segmentation of 3D Point Clouds"

Generative Zero-Shot Learning for Semantic Segmentation of 3D Point Clouds Björn Michele1), Alexandre Boulch1), Gilles Puy1), Maxime Bucher1) and Rena

valeo.ai 15 Dec 22, 2022
Code for "Solving Graph-based Public Good Games with Tree Search and Imitation Learning"

Code for "Solving Graph-based Public Good Games with Tree Search and Imitation Learning" This is the code for the paper Solving Graph-based Public Goo

Victor-Alexandru Darvariu 3 Dec 05, 2022
Official repository for the paper "Going Beyond Linear Transformers with Recurrent Fast Weight Programmers"

Recurrent Fast Weight Programmers This is the official repository containing the code we used to produce the experimental results reported in the pape

IDSIA 36 Nov 15, 2022
Automatic voice-synthetised summaries of latest research papers on arXiv

PaperWhisperer PaperWhisperer is a Python application that keeps you up-to-date with research papers. How? It retrieves the latest articles from arXiv

Valerio Velardo 124 Dec 20, 2022
Official respository for "Modeling Defocus-Disparity in Dual-Pixel Sensors", ICCP 2020

Official respository for "Modeling Defocus-Disparity in Dual-Pixel Sensors", ICCP 2020 BibTeX @INPROCEEDINGS{punnappurath2020modeling, author={Abhi

Abhijith Punnappurath 22 Oct 01, 2022
For encoding a text longer than 512 tokens, for example 800. Set max_pos to 800 during both preprocessing and training.

LongScientificFormer For encoding a text longer than 512 tokens, for example 800. Set max_pos to 800 during both preprocessing and training. Some code

Athar Sefid 6 Nov 02, 2022
Code basis for the paper "Camera Condition Monitoring and Readjustment by means of Noise and Blur" (2021)

Camera Condition Monitoring and Readjustment by means of Noise and Blur This repository contains the source code of the paper: Wischow, M., Gallego, G

7 Dec 22, 2022
bespoke tooling for offensive security's Windows Usermode Exploit Dev course (OSED)

osed-scripts bespoke tooling for offensive security's Windows Usermode Exploit Dev course (OSED) Table of Contents Standalone Scripts egghunter.py fin

epi 268 Jan 05, 2023
A simple and extensible library to create Bayesian Neural Network layers on PyTorch.

Blitz - Bayesian Layers in Torch Zoo BLiTZ is a simple and extensible library to create Bayesian Neural Network Layers (based on whats proposed in Wei

Pi Esposito 722 Jan 08, 2023
Towards Interpretable Deep Metric Learning with Structural Matching

DIML Created by Wenliang Zhao*, Yongming Rao*, Ziyi Wang, Jiwen Lu, Jie Zhou This repository contains PyTorch implementation for paper Towards Interpr

Wenliang Zhao 75 Nov 11, 2022