CoSMA: Convolutional Semi-Regular Mesh Autoencoder. From Paper "Mesh Convolutional Autoencoder for Semi-Regular Meshes of Different Sizes"

Overview

Mesh Convolutional Autoencoder for Semi-Regular Meshes of Different Sizes

Implementation of CoSMA: Convolutional Semi-Regular Mesh Autoencoder

arXiv publication

Sara Hahner and Jochen Garcke
Fraunhofer Center for Machine Learning and SCAI, Sankt Augustin, Germany
Institut für Numerische Simulation, Universität Bonn, Germany

Contact [email protected] for questions about code and data.

1. Abstract

The analysis of deforming 3D surface meshes is accelerated by autoencoders since the low-dimensional embeddings can be used to visualize underlying dynamics. But, state-of-the-art mesh convolutional autoencoders require a fixed connectivity of all input meshes handled by the autoencoder. This is due to either the use of spectral convolutional layers or mesh dependent pooling operations. Therefore, the types of datasets that one can study are limited and the learned knowledge cannot be transferred to other datasets that exhibit similar behavior. To address this, we transform the discretization of the surfaces to semi-regular meshes that have a locally regular connectivity and whose meshing is hierarchical. This allows us to apply the same spatial convolutional filters to the local neighborhoods and to define a pooling operator that can be applied to every semi-regular mesh. We apply the same mesh autoencoder to different datasets and our reconstruction error is more than 50% lower than the error from state-of-the-art models, which have to be trained for every mesh separately. Additionally, we visualize the underlying dynamics of unseen mesh sequences with an autoencoder trained on different classes of meshes.

2. Python Packages

  • pytorch (1.7.1)
  • pytorch3d (0.3.0)
  • tqdm (4.56.0)
  • hexagdly [1] (no installation neccesary, scripts are already included in the directory hexagly)
  • igl python bindings (2.2.1) (conda install -c conda-forge igl)
  • argparse

3. Scripts and Code:

  • 01_data_preprocessing: For the given dataset and experiment name (which has to correspond to the name of the semi-regular base mesh in directory data/name of the dataset/preprocessed/name of the sample) calculate the parametrization for the meshes of the same connectivity and project this parametrization over time.
  • 02_create_input_patches: For the given dataset, experiment name and test split create the patches and calculate the padding, which considers global context. The result is saved in data/name of the dataset/train_patches_name of the experiment
  • 03_training: Train the autoencoder on all training samples of the given dataset. See Table 5 for the detailed network architecture.
  • 04_testing: Set the patches back together and calculate the errors as done for the paper.

4. Results

In the directory model you can find our trained models. Compare your results to the training errors in the txt-files in the directories model/name of the dataset/logs. These files are written by the training and testing scripts. For each dataset we provide the data and code to reproduce the training and testing of the autoencoder for semi-regular meshes of different sizes.

5. Datasets and Reproduction of the Results

The data (*.obj, *.ply, *.p) is tracked with Git Large File Storage (LFS). If you install git LFS, the data is automatically downloaded when cloning the repository.

git lfs install
git clone

File Structure in data:

  • name of the dataset (gallop, FAUST, car_TRUCK, car_YARIS)
    • raw: obj or ply files for each sample and version over time
      • versions: for the car datasets there is one directory for each simulations
      • samples: for every version there are the same samples. Every sample can have a different mesh (car-dataset: different components, gallop: different animals, FAUST: different persons)
      • version/samples: these directories contain the deformed meshes
      • the raw-directories also contain the template meshes for the different samples. The remeshing for each sample/class of meshes is based on this template mesh. We provide our remeshing results to semi-regular connectivity.
    • preprocessed: for every sample we provide the semi-regular base mesh
    • semiregular: for every sample we provide the semi-regular mesh, which has been refined to level three and has been fit to the shape of the irregular template mesh
    • train_patches: train patches which are inputted to the network. This directory is created during the preprocessing.

a) GALLOP

Sumner et al: 2004: Deformation transferfor triangle meshes Webpage

A dataset containing triangular meshes representing a motion sequence froma galloping horse, elephant, and camel. Each sequence has 48 timesteps. The three animals move in a similar way butthe meshes that represent the surfaces of the three animals are highly different in connectivity and in the number of vertices

python 01_data_preprocessing.py --dataset gallop --exp_name coarsentofinalselection
python 02_create_input_patches.py --dataset gallop --exp_name coarsentofinalselection --test_split elephant
python 03_training.py --dataset gallop --exp_name coarsentofinalselection --model_name gallop_training.seed1 --hid_rep 8 --seed 1 
python 04_testing.py  --dataset gallop --exp_name coarsentofinalselection --model_name gallop_training.seed1 --hid_rep 8 --seed 1 --test_split elephant

b) FAUST

Bogo et al, 2014: FAUST: Dataset and evaluation for 3Dmesh registration Webpage

We conduct two different experiments: at first we consider known poses of two unseen bodies in the testing set. Then we consider two unknown poses of all bodies in the testing set. In both cases, 20% of the data is included in the testing set.

python 01_data_preprocessing.py --dataset FAUST --exp_name coarsento110
known poses: only interpolation of poses to different bodies
python 02_create_input_patches.py --dataset FAUST --exp_name coarsento110_inter --test_split faust8 faust9 --test_ratio 0
python 03_training.py --dataset FAUST --exp_name coarsento110_inter --model_name FAUST_knownpose.1 --hid_rep 8 --seed 1
python 04_testing.py  --dataset FAUST --exp_name coarsento110_inter --model_name FAUST_knownpose.1 --hid_rep 8 --seed 1 --test_split faust8 faust9 --test_ratio 0
unknown poses: only interpolation of poses to different bodies
python 02_create_input_patches.py --dataset FAUST --exp_name coarsento110 --test_split none --test_ratio 0.25
python 03_training.py --dataset FAUST --exp_name coarsento110 --model_name FAUST_unknownpose.1 --hid_rep 8 --seed 1 
python 04_testing.py  --dataset FAUST --exp_name coarsento110 --model_name FAUST_unknownpose.1 --hid_rep 8 --seed 1 --test_ratio 0.25

c) TRUCK and YARIS

National Crash Analysis Center (NCAC). Finite Element Model Archive

  • TRUCK : 32 completed frontal crash simulations of a Chevrolet C2500 pick-up truck, 6 components, 30 equally distributed time steps
  • YARIS: 10 completed frontal crash simulations of a detailed model of the Toyota Yaris, 10 components, 26 equally distributed time steps

We provide the semi-regular template meshes for each component and its projection over time, because of the size of the raw data.

python 02_create_input_patches.py --dataset car_YARIS --exp_name meshlab --test_ratio 1    --rotation_augment 0
python 02_create_input_patches.py --dataset car_TRUCK --exp_name meshlab --test_ratio -0.3 --rotation_augment 0 --test_version sim_041 sim_049
python 03_training.py --dataset car_TRUCK --exp_name meshlab_norot --model_name car_TRUCK_b50.2 --hid_rep 8 --seed 2 --Niter 250 --batch_size 50
python 04_testing.py  --dataset car_TRUCK --exp_name meshlab_norot --model_name car_TRUCK_b50.2 --hid_rep 8 --seed 2 --test_version sim_041 sim_049 --test_ratio -0.3
cp model/car_TRUCK/model_meshlab_norot_car_TRUCK_b50.2.pt model/car_YARIS/model_meshlab_norot_car_TRUCK_b50.2.pt
python 04_testing.py  --dataset car_YARIS --exp_name meshlab_norot --model_name car_TRUCK_b50.2 --hid_rep 8 --test_ratio 1

6. Remeshing

There are many ways to create the semi-regular meshes, that describe the irregular template meshes.

  1. Create a coarse base mesh, for example using the implementation of the "Surface Simplification Using Quadric Error Metrics"-algorithm by Garland and Heckbert [2] in meshlab.
  2. Iteratively subdivide the faces of the coarse base mesh into four faces.
  3. Fit the newly created semi-regular mesh to the irregular template mesh.

For the second and third step you can use this jupyter notebook, provided by the authors of the Pytorch3D publication [3]: deform_source_mesh_to_target_mesh

Citation

@misc{Hahner2021,
      title={Mesh Convolutional Autoencoder for Semi-Regular Meshes of Different Sizes}, 
      author={Sara Hahner and Jochen Garcke},
      year={2021},
      eprint={2110.09401},
      archivePrefix={arXiv},
      primaryClass={cs.CV}
}

References

  • [1] Steppa, Constantin, and Tim L. Holch. "HexagDLy—Processing hexagonally sampled data with CNNs in PyTorch." SoftwareX 9 (2019): 193-198.
  • [2] Michael Garland and Paul S Heckbert. Surface simplification using quadric error metrics. In Proceedings of the 24th Annual Conference on Computer Graphics and Interactive Techniques, pages 209–216, 1997.
  • [3] Nikhila Ravi, Jeremy Reizenstein, David Novotny, Taylor Gordon, Wan-Yen Lo, Justin Johnson, and Georgia Gkioxari. Accelerating 3D Deep Learning with PyTorch3D. arXivpreprint arXiv:2007.08501, 2020.
Owner
Fraunhofer SCAI
Fraunhofer SCAI
基于Paddle框架的arcface复现

arcface-Paddle 基于Paddle框架的arcface复现 ArcFace-Paddle 本项目基于paddlepaddle框架复现ArcFace,并参加百度第三届论文复现赛,将在2021年5月15日比赛完后提供AIStudio链接~敬请期待 参考项目: InsightFace Padd

QuanHao Guo 16 Dec 15, 2022
Madanalysis5 - A package for event file analysis and recasting of LHC results

Welcome to MadAnalysis 5 Outline What is MadAnalysis 5? Requirements Downloading

MadAnalysis 15 Jan 01, 2023
Official pytorch implementation of the AAAI 2021 paper Semantic Grouping Network for Video Captioning

Semantic Grouping Network for Video Captioning Hobin Ryu, Sunghun Kang, Haeyong Kang, and Chang D. Yoo. AAAI 2021. [arxiv] Environment Ubuntu 16.04 CU

Hobin Ryu 43 Nov 25, 2022
Surrogate-Assisted Genetic Algorithm for Wrapper Feature Selection

SAGA Surrogate-Assisted Genetic Algorithm for Wrapper Feature Selection Please refer to the Jupyter notebook (Example.ipynb) for an example of using t

9 Dec 28, 2022
Ultra-Data-Efficient GAN Training: Drawing A Lottery Ticket First, Then Training It Toughly

Ultra-Data-Efficient GAN Training: Drawing A Lottery Ticket First, Then Training It Toughly Code for this paper Ultra-Data-Efficient GAN Tra

VITA 77 Oct 05, 2022
PyTorch code for our ECCV 2018 paper "Image Super-Resolution Using Very Deep Residual Channel Attention Networks"

PyTorch code for our ECCV 2018 paper "Image Super-Resolution Using Very Deep Residual Channel Attention Networks"

Yulun Zhang 1.2k Dec 26, 2022
Code repository for the paper Computer Vision User Entity Behavior Analytics

Computer Vision User Entity Behavior Analytics Code repository for "Computer Vision User Entity Behavior Analytics" Code Description dataset.csv As di

Sameer Khanna 2 Aug 20, 2022
City-Scale Multi-Camera Vehicle Tracking Guided by Crossroad Zones Code

City-Scale Multi-Camera Vehicle Tracking Guided by Crossroad Zones Requirements Python 3.8 or later with all requirements.txt dependencies installed,

88 Dec 12, 2022
Learning to Reach Goals via Iterated Supervised Learning

Vanilla GCSL This repository contains a vanilla implementation of "Learning to Reach Goals via Iterated Supervised Learning" proposed by Dibya Gosh et

Christoph Heindl 4 Aug 10, 2022
A little Python application to auto tag your photos with the power of machine learning.

Tag Machine A little Python application to auto tag your photos with the power of machine learning. Report a bug or request a feature Table of Content

Florian Torres 14 Dec 21, 2022
MVP Benchmark for Multi-View Partial Point Cloud Completion and Registration

MVP Benchmark: Multi-View Partial Point Clouds for Completion and Registration [NEWS] 2021-07-12 [NEW 🎉 ] The submission on Codalab starts! 2021-07-1

PL 93 Dec 21, 2022
The repo of the preprinting paper "Labels Are Not Perfect: Inferring Spatial Uncertainty in Object Detection"

Inferring Spatial Uncertainty in Object Detection A teaser version of the code for the paper Labels Are Not Perfect: Inferring Spatial Uncertainty in

ZINING WANG 21 Mar 03, 2022
Learning-based agent for Google Research Football

TiKick 1.Introduction Learning-based agent for Google Research Football Code accompanying the paper "TiKick: Towards Playing Multi-agent Football Full

Tsinghua AI Research Team for Reinforcement Learning 90 Dec 26, 2022
SoK: Vehicle Orientation Representations for Deep Rotation Estimation

SoK: Vehicle Orientation Representations for Deep Rotation Estimation Raymond H. Tu, Siyuan Peng, Valdimir Leung, Richard Gao, Jerry Lan This is the o

FIRE Capital One Machine Learning of the University of Maryland 12 Oct 07, 2022
ZSL-KG is a general-purpose zero-shot learning framework with a novel transformer graph convolutional network (TrGCN) to learn class representation from common sense knowledge graphs.

ZSL-KG is a general-purpose zero-shot learning framework with a novel transformer graph convolutional network (TrGCN) to learn class representa

Bats Research 94 Nov 21, 2022
'Aligned mixture of latent dynamical systems' (amLDS) for stimulus decoding probabilistic manifold alignment across animals. P. Herrero-Vidal et al. NeurIPS 2021 code.

Across-animal odor decoding by probabilistic manifold alignment (NeurIPS 2021) This repository is the official implementation of aligned mixture of la

Pedro Herrero-Vidal 3 Jul 12, 2022
Jittor implementation of PCT:Point Cloud Transformer

PCT: Point Cloud Transformer This is a Jittor implementation of PCT: Point Cloud Transformer.

MenghaoGuo 547 Jan 03, 2023
Convolutional Neural Networks

Darknet Darknet is an open source neural network framework written in C and CUDA. It is fast, easy to install, and supports CPU and GPU computation. D

Joseph Redmon 23.7k Jan 05, 2023
Self-Adaptable Point Processes with Nonparametric Time Decays

NPPDecay This is our implementation for the paper Self-Adaptable Point Processes with Nonparametric Time Decays, by Zhimeng Pan, Zheng Wang, Jeff M. P

zpan 2 Sep 24, 2022
Conformer: Local Features Coupling Global Representations for Visual Recognition

Conformer: Local Features Coupling Global Representations for Visual Recognition (arxiv) This repository is built upon DeiT and timm Usage First, inst

Zhiliang Peng 378 Jan 08, 2023