Personal project about genus-0 meshes, spherical harmonics and a cow

Related tags

Deep Learningmesh2sh
Overview

How to transform a cow into spherical harmonics ?

Spot the cow, from Keenan Crane's blog

Spot

Context

In the field of Deep Learning, training on images or text has made enormous progress in recent years (with a lot of data available + CNN/Transformers). The results are not yet as good for other types of signals, such as videos or 3D models. For 3D models, some recent models use a graph-based approach to deal with 3D meshes, such as Polygen. However, these networks remain difficult to train. There are plenty of alternative representations that have been used to train a Deep network on 3D models: voxels, multiview, point clouds, each having their advantages and disadvantages. In this project, I wanted to try a new one. In topology, a 3D model is nothing more than a 2D surface (possibly colored) embedded into a 3D space. If the surface is closed, we can define an interior and an exterior, but that's it. It is not like a scalar field, which is defined throughout space. Since the data is 2D, it would be useful to be able to project this 3D representation in a 2D Euclidean space, on a uniform grid, like an image, to be able to use a 2D CNN to predict our 3D models.

Deep Learning models have proven effective in learning from mel-spectrograms of audio signals, combined with convolutions. How to exploit this idea for 3D models? All periodic signals can be approximated by Fourier series. We can therefore use a Fourier series to represent any periodic function in the complex plane. In geometry, the "drawing" of this function is a closed line, so it has the topology of a circle, in 2D space. I tried to generalize this idea by using meshes with a spherical topology, which I reprojected on the sphere using a conformal (angle preserving) parametrization, then for which I calculated the harmonics thanks to a single base, that of spherical harmonics.

The origin of this project is inspired by this video by 3blue1brown.

Spherical harmonics of a 3D mesh

We only use meshes that have the topology of a sphere, i.e. they must be manifold and genus 0. The main idea is to get a spherical parametrization of the mesh, to define where are the attributes of the mesh on the sphere. Then, the spherical harmonic coefficients that best fit these attributes are calculated.

The attributes that interest us to describe the structure of the mesh are:

  • Its geometric properties. We could directly give the XYZ coordinates, but thanks to the parametrization algorithm that is used, only the density of curvature is necessary. Consequently, we also need to know the area distortion, since our parametrization is not authalic (area preserving).
  • Its colors, in RGB format. For simplicity, here I use colors by vertices, and not with a UV texture, so it loses detail.
  • The vertex density of the mesh, which allows to put more vertices in areas that originally had a lot. This density is obtained using Von Mises-Fisher kernel density estimator.

Calculates the spherical parametrization of the mesh, then displays its various attributes

First step

The spherical harmonic coefficients can be represented as images, with the coefficients corresponding to m=0 on the diagonal. The low frequencies are at the top left.

Spherical harmonics coefficients amplitude as an image for each attribute

Spherical harmonic images

Reconstruction

We can reconstruct the model from the 6 sets of coefficients, which act as 6 functions on the sphere. We first make a spherical mesh inspired by what they made in "A Curvature and Density based Generative Representation of Shapes". Some points are sampled according to the vertex density function. We then construct an isotropic mesh with respect to a given density, using Centroidal Voronoi Tesselation. The colors are interpolated at each vertex.

Then the shape is obtained by reversing our spherical parametrization. The spherical parametrization uses a mean curvature flow, which is a simple spherical parametrizations. We use the conformal variant from Can Mean-Curvature Flow Be Made Non-Singular?.

Mean curvature flow equations. See Roberta Alessandroni's Introduction to mean curvature flow for more details on the notations MCF

Reconstruction of the mesh using only spherical harmonics coefficients First step

Remarks

This project is a proof of concept. It allows to represent a model which has the topology of a sphere in spherical harmonics form. The results could be more precise, first with an authalic (area-preserving) parametrization rather than a conformal (angle-preserving) one. Also, I did not try to train a neural network using this representation, because that requires too much investment. It takes some pre-processing on common 3D datasets to keep only the watertight genus-0 meshes, and then you have to do the training, which takes time. If anyone wants to try, I'd be happy to help.

I did it out of curiosity, and to gain experience, not to have an effective result. All algorithms used were coded in python/pytorch except for some solvers from SciPy and spherical harmonics functions from shtools. It makes it easier to read, but it could be faster using other libraries.

Demo

Check the demo in Google Colab : Open In Colab

To use the functions of this project you need the dependencies below. The versions indicated are those that I have used, and are only indicative.

  • python (3.9.10)
  • pytorch (1.9.1)
  • scipy (1.7.3)
  • scikit-sparse (0.4.6)
  • pyshtools (4.9.1)

To run the demo main.ipynb, you also need :

  • jupyterlab (3.2.9)
  • trimesh (3.10.0)
  • pyvista (0.33.2)
  • pythreejs (optional, 2.3.0)

You can run these lines to install everything on Linux using conda :

conda create --name mesh2sh
conda activate mesh2sh
conda install python=3.9
conda install scipy=1.7 -c anaconda
conda install pytorch=1.9 cudatoolkit=11 -c pytorch -c conda-forge
conda install gmt intel-openmp -c conda-forge
conda install pyshtools pyvista jupyterlab -c conda-forge
conda update pyshtools -c conda-forge
pip install scikit-sparse
pip install pythreejs
pip install trimesh

Then just run the demo :

jupyter notebook main.ipynb

Contribution

To run tests, you need pytest and flake8 :

pip install pytest
pip install flake8

You can check coding style using flake8 --max-line-length=120, and run tests using python -m pytest tests/ from the root folder. Also, run the demo again to check that the results are consistent

References

Code to run experiments in SLOE: A Faster Method for Statistical Inference in High-Dimensional Logistic Regression.

Code to run experiments in SLOE: A Faster Method for Statistical Inference in High-Dimensional Logistic Regression. Not an official Google product. Me

Google Research 27 Dec 12, 2022
How to Leverage Multimodal EHR Data for Better Medical Predictions?

How to Leverage Multimodal EHR Data for Better Medical Predictions? This repository contains the code of the paper: How to Leverage Multimodal EHR Dat

13 Dec 13, 2022
This project uses Template Matching technique for object detecting by detection of template image over base image.

Object Detection Project Using OpenCV This project uses Template Matching technique for object detecting by detection the template image over base ima

Pratham Bhatnagar 7 May 29, 2022
Object detection and instance segmentation toolkit based on PaddlePaddle.

Object detection and instance segmentation toolkit based on PaddlePaddle.

9.3k Jan 02, 2023
The PASS dataset: pretrained models and how to get the data - PASS: Pictures without humAns for Self-Supervised Pretraining

The PASS dataset: pretrained models and how to get the data - PASS: Pictures without humAns for Self-Supervised Pretraining

Yuki M. Asano 249 Dec 22, 2022
Implementation for the "Surface Reconstruction from 3D Line Segments" paper.

Surface Reconstruction from 3D Line Segments Surface reconstruction from 3d line segments. Langlois, P. A., Boulch, A., & Marlet, R. In 2019 Internati

85 Jan 04, 2023
Solving reinforcement learning tasks which require language and vision

Multimodal Reinforcement Learning JAX implementations of the following multimodal reinforcement learning approaches. Dual-coding Episodic Memory from

Henry Prior 31 Feb 26, 2022
Black-Box-Tuning - Black-Box Tuning for Language-Model-as-a-Service

Black-Box-Tuning Source code for paper "Black-Box Tuning for Language-Model-as-a

Tianxiang Sun 149 Jan 04, 2023
Main Results on ImageNet with Pretrained Models

This repository contains Pytorch evaluation code, training code and pretrained models for the following projects: SPACH (A Battle of Network Structure

Microsoft 151 Dec 14, 2022
HAR-stacked-residual-bidir-LSTMs - Deep stacked residual bidirectional LSTMs for HAR

HAR-stacked-residual-bidir-LSTM The project is based on this repository which is presented as a tutorial. It consists of Human Activity Recognition (H

Guillaume Chevalier 287 Dec 27, 2022
A rule learning algorithm for the deduction of syndrome definitions from time series data.

README This project provides a rule learning algorithm for the deduction of syndrome definitions from time series data. Large parts of the algorithm a

0 Sep 24, 2021
Gems & Holiday Package Prediction

Predictive_Modelling Gems & Holiday Package Prediction This project is based on 2 cases studies : Gems Price Prediction and Holiday Package prediction

Avnika Mehta 1 Jan 27, 2022
This is the dataset for testing the robustness of various VO/VIO methods

KAIST VIO dataset This is the dataset for testing the robustness of various VO/VIO methods You can download the whole dataset on KAIST VIO dataset Ind

1 Sep 01, 2022
Hierarchical Memory Matching Network for Video Object Segmentation (ICCV 2021)

Hierarchical Memory Matching Network for Video Object Segmentation Hongje Seong, Seoung Wug Oh, Joon-Young Lee, Seongwon Lee, Suhyeon Lee, Euntai Kim

Hongje Seong 72 Dec 14, 2022
AFL binary instrumentation

E9AFL --- Binary AFL E9AFL inserts American Fuzzy Lop (AFL) instrumentation into x86_64 Linux binaries. This allows binaries to be fuzzed without the

242 Dec 12, 2022
PyTorch implementation of 1712.06087 "Zero-Shot" Super-Resolution using Deep Internal Learning

Unofficial PyTorch implementation of "Zero-Shot" Super-Resolution using Deep Internal Learning Unofficial Implementation of 1712.06087 "Zero-Shot" Sup

Jacob Gildenblat 196 Nov 27, 2022
Parametric Contrastive Learning (ICCV2021)

Parametric-Contrastive-Learning This repository contains the implementation code for ICCV2021 paper: Parametric Contrastive Learning (https://arxiv.or

DV Lab 156 Dec 21, 2022
《Where am I looking at? Joint Location and Orientation Estimation by Cross-View Matching》(CVPR 2020)

This contains the codes for cross-view geo-localization method described in: Where am I looking at? Joint Location and Orientation Estimation by Cross-View Matching, CVPR2020.

41 Oct 27, 2022
Source code for PairNorm (ICLR 2020)

PairNorm Official pytorch source code for PairNorm paper (ICLR 2020) This code requires pytorch_geometric=1.3.2 usage For SGC, we use original PairNo

62 Dec 08, 2022
RRxIO - Robust Radar Visual/Thermal Inertial Odometry: Robust and accurate state estimation even in challenging visual conditions.

RRxIO - Robust Radar Visual/Thermal Inertial Odometry RRxIO offers robust and accurate state estimation even in challenging visual conditions. RRxIO c

Christopher Doer 64 Dec 29, 2022