Pip-package for trajectory benchmarking from "Be your own Benchmark: No-Reference Trajectory Metric on Registered Point Clouds", ECMR'21

Related tags

Deep Learningmap-metrics
Overview

Map Metrics for Trajectory Quality

Map metrics toolkit provides a set of metrics to quantitatively evaluate trajectory quality via estimating consistency of the map aggregated from point clouds.

GPS or Motion Capture systems are not always available in perception systems, or their quality is not enough (GPS on small-scale distances) for use as ground truth trajectory. Thus, common full-reference trajectory metrics (APE, RPE, and their modifications) could not be applied to evaluate trajectory quality. When 3D sensing technologies (depth camera, LiDAR) are available on the perception system, one can alternatively assess trajectory quality --- estimate the consistency of the map from registered point clouds via the trajectory.

Documentation: https://map-metrics.readthedocs.io.

Documentation Status Updates

Features

Our toolkit provides implementation of the next metrics:

  • Mean Map Entropy (MME), Mean Plane Variance(MPV) [1] [2]
  • Mutually Orthogonal Metric (MOM) [3] -- has strong correlation with RPE

Citation

If you use this toolkit or MOM-metric results, please, cite our work:

@misc{kornilova2021benchmark,
    title={Be your own Benchmark: No-Reference Trajectory Metric on Registered Point Clouds},
    author={Anastasiia Kornilova and Gonzalo Ferrer},
    year={2021},
    eprint={2106.11351},
    archivePrefix={arXiv},
    primaryClass={cs.RO}
}

Credits

This package was created with Cookiecutter and the audreyr/cookiecutter-pypackage project template.

Links

[1] Droeschel, David, Jörg Stückler, and Sven Behnke. "Local multi-resolution representation for 6D motion estimation and mapping with a continuously rotating 3D laser scanner." 2014 IEEE International Conference on Robotics and Automation (ICRA). IEEE, 2014.
[2] Razlaw, Jan, et al. "Evaluation of registration methods for sparse 3D laser scans." 2015 European Conference on Mobile Robots (ECMR). IEEE, 2015.
[3] Kornilova, Anastasiia, and Gonzalo Ferrer. "Be your own Benchmark: No-Reference Trajectory Metric on Registered Point Clouds." arXiv preprint arXiv:2106.11351 (2021).
Comments
  • Cross-Platform Wheels

    Cross-Platform Wheels

    Linux and Windows32-64 wheels.

    MacOS is running into from map_metrics import map_metrics E ImportError: dlopen(/Users/runner/hostedtoolcache/Python/3.6.15/x64/lib/python3.6/site-packages/map_metrics/map_metrics.cpython-36m-darwin.so, 2): Library not loaded: @rpath/libboost_graph-mt-x64.dylib E Referenced from: /Users/runner/hostedtoolcache/Python/3.6.15/x64/lib/python3.6/site-packages/map_metrics/map_metrics.cpython-36m-darwin.so E Reason: image not found

    opened by achains 1
  • Speed up Github Actions workflow

    Speed up Github Actions workflow

    It takes an hour to build Open3D

    Possible solutions:

    • [ ] Add -j flag
    • [ ] Specify Open3D build options
    • [ ] Store Open3D files in Github Actions Cache
    enhancement 
    opened by achains 1
  • Python version interface improvements

    Python version interface improvements

    Tasks:

    • [x] Break down large functions
    • [x] Break functions into modules
    • [x] Build pip-package, configure CI
    • [x] Add more functions, e.g. I/O processing
    opened by achains 0
  • pip-package. MPV. MME

    pip-package. MPV. MME

    Overview

    Python package wheels with implemented basic metrics (MPV, MME) on C++

    Platform tags

    • manylinux2010_x86_64
    • macosx_10_14_x86_64
    • win32 / win_amd64

    Python version

    • Python >=3.6
    opened by achains 0
  • Cosmetic fixes

    Cosmetic fixes

    What was done:

    • Removed redundant comments
    • Implemented general interface for methods
    • Added missing headers
    • Unsigned long -> int
    • All functions have CamelCase now
    opened by achains 0
  • CMake project structure. Baseline MME and MPV

    CMake project structure. Baseline MME and MPV

    Progress:

    • Configured CMake files
    • Baseline of mme and mpv algorithms

    Notes (Tasks for next PR)

    • Algorithms need to be tested
    • CI should be configured
    opened by achains 0
  • A drawback of mom when dealing drafting walls in indoor envs.

    A drawback of mom when dealing drafting walls in indoor envs.

    • Map Metrics 0.0.1:
    • Python version 3.8:
    • Operating System win10: 1

    Description

    It tired the metirc on a small scale data, likely a room with differential chassis and a RS-16. When the point cloud map consists of multiple wall (actually one) caused by drifting, the mom metric may have a smaller value than a normal map. I think it is caused by the orthogonal walls (both the real one and drifted one) and floor.

    Specifically, Fig.1 is generated by LOAM (without drift), and Fig. 2 is gererated by Lio-sam (with draft).

    Fig.1

    Fig.2

    I also considered to downsample the two maps into the same scale with voxel downsampling. The results are listed below.

    1640311688(1)

    I think this kind of problem is very familiar for indoor, I have seen many drafting wall during daily usage of lidar based slam, without semantic labels, it is very hard for distinguishing them automatically.

    What I Did

    I will try the 0.0.2 version later.

    Paste the command(s) you ran and the output.
If there was a crash, please include the traceback here.
    opened by hahakid 2
  • Add C++ executable with console interface

    Add C++ executable with console interface

    It will be useful for quick testing and performance measurments if library is able to run from command-line interface.

    For example, ./map-metrics --pc="data/kitti_00" --tj="data/Tj_0" --metric="mme" [--"some common hyperparameters"]

    It's build should be disabled by default

    enhancement 
    opened by achains 0
  • Support 10.9 MacOS

    Support 10.9 MacOS

    • Map Metrics version: 0.0.3
    • Python version: >=3.6
    • Operating System: MacOS

    Description

    Pip-package supports MacOS >=10.14 due to C++17 features. We can downgrade to C++14, but a bit later :)

    enhancement 
    opened by achains 0
  • editoring error

    editoring error

    A \delta seems missing in Eq.6 of the paper. Have you tested on all the sequences and other dataset. Recommend to ref LIBRE: The Multiple 3D LiDAR Dataset. VLP64 may have a low precise for long distance measurement. clipboard

    opened by hahakid 1
Owner
Mobile Robotics Lab. at Skoltech
Mobile Robotics Lab. at Skoltech
GitHub Repository
Pytorch implementation of SELF-ATTENTIVE VAD, ICASSP 2021

SELF-ATTENTIVE VAD: CONTEXT-AWARE DETECTION OF VOICE FROM NOISE (ICASSP 2021) Pytorch implementation of SELF-ATTENTIVE VAD | Paper | Dataset Yong Rae

97 Dec 23, 2022
Learning to trade under the reinforcement learning framework

Trading Using Q-Learning In this project, I will present an adaptive learning model to trade a single stock under the reinforcement learning framework

Uirá Caiado 470 Nov 28, 2022
Code repository for the paper "Tracking People with 3D Representations"

Tracking People with 3D Representations Code repository for the paper "Tracking People with 3D Representations" (paper link) (project site). Jathushan

Jathushan Rajasegaran 77 Dec 03, 2022
Where-Got-Time - An NUS timetable generator which uses a genetic algorithm to optimise timetables to suit the needs of NUS students

Where Got Time(table)? A timetable optimsier which uses an evolutionary algorith

Nicholas Lee 3 Jan 09, 2022
Official Code Release for "TIP-Adapter: Training-free clIP-Adapter for Better Vision-Language Modeling"

Official Code Release for "TIP-Adapter: Training-free clIP-Adapter for Better Vision-Language Modeling" Pipeline of Tip-Adapter Tip-Adapter can provid

peng gao 187 Dec 28, 2022
MAGMA - a GPT-style multimodal model that can understand any combination of images and language

MAGMA -- Multimodal Augmentation of Generative Models through Adapter-based Finetuning Authors repo (alphabetical) Constantin (CoEich), Mayukh (Mayukh

Aleph Alpha GmbH 331 Jan 03, 2023
Tools for manipulating UVs in the Blender viewport.

UV Tool Suite for Blender A set of tools to make editing UVs easier in Blender. These tools can be accessed wither through the Kitfox - UV panel on th

35 Oct 29, 2022
Code for Referring Image Segmentation via Cross-Modal Progressive Comprehension, CVPR2020.

CMPC-Refseg Code of our CVPR 2020 paper Referring Image Segmentation via Cross-Modal Progressive Comprehension. Shaofei Huang*, Tianrui Hui*, Si Liu,

spyflying 55 Dec 01, 2022
Tensorflow Implementation of the paper "Spectral Normalization for Generative Adversarial Networks" (ICML 2017 workshop)

tf-SNDCGAN Tensorflow implementation of the paper "Spectral Normalization for Generative Adversarial Networks" (https://www.researchgate.net/publicati

Nhat M. Nguyen 248 Nov 25, 2022
Deep Learning and Reinforcement Learning Library for Scientists and Engineers 🔥

TensorLayer is a novel TensorFlow-based deep learning and reinforcement learning library designed for researchers and engineers. It provides an extens

TensorLayer Community 7.1k Dec 29, 2022
Implementation of the paper "Self-Promoted Prototype Refinement for Few-Shot Class-Incremental Learning"

Self-Promoted Prototype Refinement for Few-Shot Class-Incremental Learning This is the implementation of the paper "Self-Promoted Prototype Refinement

Kai Zhu 78 Dec 02, 2022
Implementation of paper "DeepTag: A General Framework for Fiducial Marker Design and Detection"

Implementation of paper DeepTag: A General Framework for Fiducial Marker Design and Detection. Project page: https://herohuyongtao.github.io/research/

Yongtao Hu 46 Dec 12, 2022
PyTorch code for 'Efficient Single Image Super-Resolution Using Dual Path Connections with Multiple Scale Learning'

Efficient Single Image Super-Resolution Using Dual Path Connections with Multiple Scale Learning This repository is for EMSRDPN introduced in the foll

7 Feb 10, 2022
RID-Noise: Towards Robust Inverse Design under Noisy Environments

This is code of RID-Noise. Reproduce RID-Noise Results Toy tasks Please refer to the notebook ridnoise.ipynb to view experiments on three toy tasks. B

Thyrix 2 Nov 23, 2022
Code and data for "TURL: Table Understanding through Representation Learning"

TURL This Repo contains code and data for "TURL: Table Understanding through Representation Learning". Environment and Setup Data Pretraining Finetuni

SunLab-OSU 63 Nov 23, 2022
Pose estimation for iOS and android using TensorFlow 2.0

💃 Mobile 2D Single Person (Or Your Own Object) Pose Estimation for TensorFlow 2.0 This repository is forked from edvardHua/PoseEstimationForMobile wh

tucan9389 165 Nov 16, 2022
Omniverse sample scripts - A guide for developing with Python scripts on NVIDIA Ominverse

Omniverse sample scripts ここでは、NVIDIA Omniverse ( https://www.nvidia.com/ja-jp/om

ft-lab (Yutaka Yoshisaka) 37 Nov 17, 2022
This GitHub repo consists of Code and Some results of project- Diabetes Treatment using Gold nanoparticles. These Consist of ML Models used for prediction Diabetes and further the basic theory and working of Gold nanoparticles.

GoldNanoparticles This GitHub repo consists of Code and Some results of project- Diabetes Treatment using Gold nanoparticles. These Consist of ML Mode

1 Jan 30, 2022
The code of "Dependency Learning for Legal Judgment Prediction with a Unified Text-to-Text Transformer".

Code data_preprocess.py: preprocess data for Dependent-T5. parameters.py: define parameters of Dependent-T5. train_tools.py: traning and evaluation co

1 Apr 21, 2022
Utilities to bridge Canvas-generated course rosters with GitLab's API.

gitlab-canvas-utils A collection of scripts originally written for CSE 13S. Oversees everything from GitLab course group creation, student repository

Eugene Chou 5 Jun 08, 2022