Code and models for "Pano3D: A Holistic Benchmark and a Solid Baseline for 360 Depth Estimation", OmniCV Workshop @ CVPR21.

Overview

Pano3D

A Holistic Benchmark and a Solid Baseline for 360o Depth Estimation

made-with-python Maintaner Maintaner

Streamlit Demo YouTube Video Views

Pano3D Intro

Pano3D is a new benchmark for depth estimation from spherical panoramas. We generate a dataset (using GibsonV2) and provide baselines for holistic performance assessment, offering:

  1. Primary and secondary traits metrics:
    • Direct depth performance:
      • (w)RMSE
      • (w)RMSLE
      • AbsRel
      • SqRel
      • (w)Relative accuracy (\delta) @ {1.05, 1.1, 1.25, 1.252, 1.253 }
    • Boundary discontinuity preservation:
      • Precision @ {0.25, 0.5, 1.0}m
      • Recall @ {0.25, 0.5, 1.0}m
      • Depth boundary errors of accuracy and completeness
    • Surface smoothness:
      • RMSEo
      • Relative accuracy (\alpha) @ {11.25o, 22.5o, 30o}
  2. Out-of-distribution & Zero-shot cross dataset transfer:
    • Different depth distribution test set
    • Varying scene context test set
    • Shifted camera domain test set

By disentangling generalization and assessing all depth properties, Pano3D aspires to drive progress benchmarking for 360o depth estimation.

Using Pano3D to search for a solid baseline results in an acknowledgement of exploiting complementary error terms, adding encoder-decoder skip connections and using photometric augmentations.

TODO

  • Web Demo
  • Data Download
  • Loader & Splits
  • Models Weights Download
  • Model Serve Code
  • Model Hub Code
  • Metrics Code

Demo

A publicly hosted demo of the baseline models can be found here. Using the web app, it is possible to upload a panorama and download a 3D reconstructed mesh of the scene using the derived depth map.

Note that due to the external host's caching issues, it might be necessary to refresh your browser's cache in between runs to update the 3D models.

Data

Download

To download the data, follow the instructions at vcl3d.github.io/Pano3D/download/.

Please note that getting access to the data download links is a two step process as the dataset is a derivative and compliance with the original dataset's terms and usage agreements is required. Therefore:

  1. You first need to fill in this Google Form.
  2. And, then, you need to perform an access request at each one of the Zenodo repositories (depending on which dataset partition you need):

After both these steps are completed, you will soon receive the download links for each dataset partition.

Loader

Splits

Models

Download

Inference

Serve

Metrics

Direct

Boundary

Smoothness

Results

Owner
Visual Computing Lab, Information Technologies Institute, Centre for Reseach and Technology Hellas
Computer Vision Lab in CERTH-ITI
Visual Computing Lab, Information Technologies Institute, Centre for Reseach and Technology Hellas
A Multi-modal Perception Tracker (MPT) for speaker tracking using both audio and visual modalities

MPT A Multi-modal Perception Tracker (MPT) for speaker tracking using both audio and visual modalities. Implementation for our AAAI 2022 paper: Multi-

yidiLi 4 May 08, 2022
A Closer Look at Reference Learning for Fourier Phase Retrieval

A Closer Look at Reference Learning for Fourier Phase Retrieval This repository contains code for our NeurIPS 2021 Workshop on Deep Learning and Inver

Tobias Uelwer 1 Oct 28, 2021
This project is for a Twitter bot that monitors a bird feeder in my backyard. Any detected birds are identified and posted to Twitter.

Backyard Birdbot Introduction This is a silly hobby project to use existing ML models to: Detect any birds sighted by a webcam Identify whic

Chi Young Moon 71 Dec 25, 2022
Minimalist Error collection Service compatible with Rollbar clients. Sentry or Rollbar alternative.

Minimalist Error collection Service Features Compatible with any Rollbar client(see https://docs.rollbar.com/docs). Just change the endpoint URL to yo

Haukur Rósinkranz 381 Nov 11, 2022
Learning to Self-Train for Semi-Supervised Few-Shot

Learning to Self-Train for Semi-Supervised Few-Shot Classification This repository contains the TensorFlow implementation for NeurIPS 2019 Paper "Lear

86 Dec 29, 2022
Yolov5+SlowFast: Realtime Action Detection Based on PytorchVideo

Yolov5+SlowFast: Realtime Action Detection A realtime action detection frame work based on PytorchVideo. Here are some details about our modification:

WuFan 181 Dec 30, 2022
Implementation of various Vision Transformers I found interesting

Implementation of various Vision Transformers I found interesting

Kim Seonghyeon 78 Dec 06, 2022
Code for our WACV 2022 paper "Hyper-Convolution Networks for Biomedical Image Segmentation"

Hyper-Convolution Networks for Biomedical Image Segmentation Code for our WACV 2022 paper "Hyper-Convolution Networks for Biomedical Image Segmentatio

Tianyu Ma 17 Nov 02, 2022
Official implementation of our paper "Learning to Bootstrap for Combating Label Noise"

Learning to Bootstrap for Combating Label Noise This repo is the official implementation of our paper "Learning to Bootstrap for Combating Label Noise

21 Apr 09, 2022
TensorFlow implementation of "TokenLearner: What Can 8 Learned Tokens Do for Images and Videos?"

TokenLearner: What Can 8 Learned Tokens Do for Images and Videos? Source: Improving Vision Transformer Efficiency and Accuracy by Learning to Tokenize

Aritra Roy Gosthipaty 23 Dec 24, 2022
AI that generate music

PianoGPT ai that generate music try it here https://share.streamlit.io/annasajkh/pianogpt/main/main.py or here https://huggingface.co/spaces/Annas/Pia

Annas 28 Nov 27, 2022
A deep learning model for style-specific music generation.

DeepJ: A model for style-specific music generation https://arxiv.org/abs/1801.00887 Abstract Recent advances in deep neural networks have enabled algo

Henry Mao 704 Nov 23, 2022
[SIGGRAPH Asia 2021] Pose with Style: Detail-Preserving Pose-Guided Image Synthesis with Conditional StyleGAN

Pose with Style: Detail-Preserving Pose-Guided Image Synthesis with Conditional StyleGAN [Paper] [Project Website] [Output resutls] Official Pytorch i

Badour AlBahar 215 Dec 17, 2022
Computer Vision and Pattern Recognition, NUS CS4243, 2022

CS4243_2022 Computer Vision and Pattern Recognition, NUS CS4243, 2022 Cloud Machine #1 : Google Colab (Free GPU) Follow this Notebook installation : h

Xavier Bresson 142 Dec 15, 2022
People movement type classifier with YOLOv4 detection and SORT tracking.

Movement classification The goal of this project would be movement classification of people, in other words, walking (normal and fast) and running. Yo

4 Sep 21, 2021
A python library for self-supervised learning on images.

Lightly is a computer vision framework for self-supervised learning. We, at Lightly, are passionate engineers who want to make deep learning more effi

Lightly 2k Jan 08, 2023
PyTorch EO aims to make Deep Learning for Earth Observation data easy and accessible to real-world cases and research alike.

Pytorch EO Deep Learning for Earth Observation applications and research. 🚧 This project is in early development, so bugs and breaking changes are ex

earthpulse 28 Aug 25, 2022
📚 A collection of Jupyter notebooks for learning and experimenting with OpenVINO 👓

A collection of ready-to-run Python* notebooks for learning and experimenting with OpenVINO developer tools. The notebooks are meant to provide an introduction to OpenVINO basics and teach developers

OpenVINO Toolkit 840 Jan 03, 2023
For visualizing the dair-v2x-i dataset

3D Detection & Tracking Viewer The project is based on hailanyi/3D-Detection-Tracking-Viewer and is modified, you can find the original version of the

34 Dec 29, 2022
Franka Emika Panda manipulator kinematics&dynamics simulation

pybullet_sim_panda Pybullet simulation environment for Franka Emika Panda Dependency pybullet, numpy, spatial_math_mini Simple example (please check s

0 Jan 20, 2022