POCO: Point Convolution for Surface Reconstruction

Related tags

Deep LearningPOCO
Overview

POCO: Point Convolution for Surface Reconstruction

by: Alexandre Boulch and Renaud Marlet

Abstract

Implicit neural networks have been successfully used for surface reconstruction from point clouds. However, many of them face scalability issues as they encode the isosurface function of a whole object or scene into a single latent vector. To overcome this limitation, a few approaches infer latent vectors on a coarse regular 3D grid or on 3D patches, and interpolate them to answer occupancy queries. In doing so, they loose the direct connection with the input points sampled on the surface of objects, and they attach information uniformly in space rather than where it matters the most, i.e., near the surface. Besides, relying on fixed patch sizes may require discretization tuning. To address these issues, we propose to use point cloud convolutions and compute latent vectors at each input point. We then perform a learning-based interpolation on nearest neighbors using inferred weights. Experiments on both object and scene datasets show that our approach significantly outperforms other methods on most classical metrics, producing finer details and better reconstructing thinner volumes.

Citation

TODO

Dependencies

Our code rely on Pytorch Geometric, which should be installed. Then, run:

python setup.py build_ext --inplace

Data

ShapeNet (Occupancy Network pre-processing)

We use the ShapeNet dataset as pre-processed by Occupancy Networks. Please refer to original repository for downloading the data.

It should be placed in the folder data/ShapeNet.

SyntheticRooms (Occupancy Network pre-processing)

We use the SyntheticRooms dataset as created by Occupancy Networks. Please refer to original repository for downloading the data.

It should be placed in the folder data/synthetic_room_dataset.

ABC / RealWorld / Famous and Thingi10k

They should be placed in the folders: data/3d_shapes_abc, data/3d_shapes_abc_training, data/3d_shapes_famous, data/3d_shapes_real_world and data/3d_shapes_thingi10k.

SceneNet

Download the SceneNet dataset.

Watertight models

We use the code from https://github.com/hjwdzh/Manifold for watertight model creation. Please download and compile it.

To create the watertight models, create a destination directory data/SceneNet, copy-paste the python script datasets/scenenet_watertight.py into the directory and run it with python scenenet_watertight.py. You will need to modify the raw_data_dir and the manifold_code_dir to match your data and code locations.

Creation of the point clouds

You can use the script datasets/scenenet_sample.py. You will also need to modify the paths.

Precomputed meshes and point clouds

For easy use of the code, we provide precomputed meshes, and point clouds SceneNet20, SceneNet100, SceneNet500 and SceneNet1000.

Location

The watertight models should be in data/SceneNet.

The points are located in data/SceneNet20, data/SceneNet100, data/SceneNet1000 and data/SceneNet500.

Training

# train on ShapeNet with 3k points, noise and without normals 
python train.py --config configs/config_shapenet.yaml 

# train on ShapeNet with 10k points, no noise and normals
python train.py --config configs/config_shapenet.yaml --normals True --random_noise 0 --experiment_name Normals

# train on ABC with 3k points
python train.py --config configs/config_abc.yaml --experiment_name 3k

# train on ABC with 10k points
python train.py --config configs/config_abc.yaml --experiment_name 10k --manifold_points 10000 --training_batch_size 8

Generation

ShapeNet

python generate.py --config results/ShapeNet_None_FKAConv_InterpAttentionKHeadsNet_None/config.yaml --gen_resolution_global 128

SyntheticRooms

python generate.py --config results/SyntheticRooms_None_FKAConv_InterpAttentionKHeadsNet_None/config.yaml --gen_resolution_global 256 --num_mesh 1

ABC, RealWorld, Famous and Thingi10k

# Model trained with 3k points
python generate.py --config results/ABC_3k_FKAConv_InterpAttentionKHeadsNet_None/config.yaml --dataset_name DATASET_NAME --dataset_root data/DATASET_DIR --gen_resolution_global 256

python generate.py --config results/ABC_3k_FKAConv_InterpAttentionKHeadsNet_None/config.yaml --dataset_name DATASET_NAME --dataset_root data/DATASET_DIR --gen_resolution_global 256 --manifold_points -1 --gen_subsample_manifold 3000 --gen_subsample_manifold_iter 10 --gen_descriptor gen_sub3k_iter10

# Model trained with 10k points
python generate.py --config results/ABC_10k_FKAConv_InterpAttentionKHeadsNet_None/config.yaml --dataset_name DATASET_NAME --dataset_root data/DATASET_DIR --gen_resolution_global 256

python generate.py --config results/ABC_10k_FKAConv_InterpAttentionKHeadsNet_None/config.yaml --dataset_name DATASET_NAME --dataset_root data/DATASET_DIR --gen_resolution_global 256 --manifold_points -1 --noise 0.0 --gen_subsample_manifold 10000 --gen_subsample_manifold_iter 10 --gen_descriptor gen_sub3k_iter10

For faster generation, one would want to use a lower marching cubes parameter --gen_resolution_global, e.g. 128.

DATASET_NAME should be replaced by ABCTest, ABCTestNoiseFree, ABCTestExtraNoise, RealWorld, FamousTest, FamousTestNoiseFree, FamousTestExtraNoisy, FamousTestSparse, FamousTestDense, Thingi10kTest, Thingi10kTestNoiseFree, Thingi10kTestExtraNoisy, Thingi10kTestSparse, Thingi10kTestDense

DATASET_DIR should be replaced by 3d_shapes_abc, 3d_shapes_famous, 3d_shapes_real_world, 3d_shapes_thingi10k

SceneNet

python generate.py --config results/ShapeNet_Normals_FKAConv_InterpAttentionKHeadsNet_None/config.yaml --gen_autoscale True --gen_autoscale_target 0.01 --gen_resolution_metric 0.01 --dataset_name DATASET_NAME --dataset_root data/DATASET_NAME --manifold_points -1

DATASET_NAME should be replaced by SceneNet20, SceneNet100, SceneNet500, SceneNet1000.

The --gen_autoscale_target parameter is the average distance between a point and its nearest neighbor in the training set, 0.01 for ShapeNet train set with 3000 points.

Evaluation

ShapeNet

python eval_meshes.py --gendir results/ShapeNet_None_FKAConv_InterpAttentionKHeadsNet_None/gen_ShapeNet_test_3000/ --meshdir meshes --dataset ShapeNet --split test --gtdir data/ShapeNet

SyntheticRooms

python eval_meshes.py --gendir results/SyntheticRooms_None_FKAConv_InterpAttentionKHeadsNet_None/gen_SyntheticRooms_test_10000/ --meshdir meshes --dataset SyntheticRooms --split test --gtdir data/synthetic_room_dataset

SceneNet

python eval_scenenet.py --gendir results/ShapeNet_Normals_FKAConv_InterpAttentionKHeadsNet_None/gen_SceneNet20_test_allPts/ --meshdir meshes --gtdir data/SceneNet

ABC, Famous and Thingi10k

For evaluation on the ABCTest please run:

python eval_point2surf_datasets.py --gendir results/ABC_3k_FKAConv_InterpAttentionKHeadsNet_None/gen_ABCTest_test_3000 --meshdir meshes/04_pts/ --gtdir data/3d_shapes_abc/abc/

You can adapt the paths to evaluate on the different datasets and noise variants.

Pretrained models

We provide pre-trained models for FKAConv backbone.

ShapeNet 3k, noise, no normals

ShapeNet 3k, no noise, normals

SyntheticRooms 10k

ABC 3k

ABC 10k

Configuration

The code was used with Ubuntu, Python 3.7.10, Cuda 11.1 and Pytorch 1.8.1

Comments
  • Package Versions

    Package Versions

    Hi,

    I tried to reproduce your results, but I ran into a possible version mismatch between Pytorch and Pytorch_geometric.

    I created my environment with the following commands:

    conda create --name poco python=3.7.10
conda install pytorch==1.8.1 torchvision==0.9.1 torchaudio==0.8.1 cudatoolkit=11.1 -c pytorch -c conda-forge
conda install -c conda-forge cython
conda install -c conda-forge tqdm 
conda install -c conda-forge scikit-image 
conda install -c open3d-admin open3d 
conda install -c conda-forge scikit-learn 
conda install -c conda-forge pyyaml 
conda install -c conda-forge addict 
conda install -c conda-forge pandas 
conda install -c conda-forge plyfile 
conda install -c conda-forge pytorch_geometric

    Compilation with python setup.py build_ext --inplace seems to work but python generate.py --config results/ABC_10k_FKAConv_InterpAttentionKHeadsNet_None/config.yaml --dataset_name DATASET_NAME --dataset_root data/3d_shapes_abc/abc/ --gen_resolution_global 256 results in OSError: /home/perler/miniconda3/envs/poco/lib/python3.7/site-packages/torch_sparse/_version.so: undefined symbol: _ZN5torch3jit17parseSchemaOrNameERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEE

    Installed versions are:

    (poco) [email protected]:~/repos/poco$ conda list pytorch
# packages in environment at /home/perler/miniconda3/envs/poco:
#
# Name                    Version                   Build  Channel
pytorch                   1.8.1           py3.7_cuda11.1_cudnn8.0.5_0    pytorch
pytorch-cpu               1.1.0               py3.7_cpu_0    pytorch
pytorch_geometric         2.0.3              pyh6c4a22f_0    conda-forge
pytorch_sparse            0.6.4            py37hcae2be3_0    conda-forge

    Again, the CPU-version... but that's a different issue.

    AFAIK, they added sparse tensors only recently to Pytorch, so the installed Pytorch-geometric version might be too new. Which version of Pytorch-geometric do I need?

    Can you please create a requiremtents.txt and/or environment.yaml?

    opened by ErlerPhilipp 13
  • Cython related error when building extensions

    Cython related error when building extensions

    Hi, I completely followed the instructions, using cuda11.1 and Ubuntu, but still having this issue gcc: error: /usr/POCO/build/temp.linux-x86_64-3.7/eval/src/utils/libkdtree/pykdtree/kdtree.o: No such file or directory gcc: error: /usr/POCO/build/temp.linux-x86_64-3.7/eval/src/utils/libkdtree/pykdtree/_kdtree_core.o: No such file or directory It's asking to link .o files before it is generated. But looks like required Cython 0.27.3 is not compatible with Python 3.7.10, which is required by this project. I'm wondering does anyone have a working combination of the version of the packages? Thank you!

    opened by zhaoyuanyuan2011 5
  • Download ABC / RealWorld / Famous and Thingi10k data

    Download ABC / RealWorld / Famous and Thingi10k data

    Hello,

    Thanks for the excellent paper and for releasing the code.

    I'm wondering where I can download the ABC, RealWorld, Famous, and Thingi10k data?

    Best and have a nice day, Mulin

    opened by MulinYu 2
  • Training on my own dataset

    Training on my own dataset

    I met some problem when training on my own dataset, which is processed as ShapeNet, containing pointclouds with normals, SDF points and occ values. However, there is an error when I try to train on it.

    File "/media/yangxilab/DiskB/sanren/3Drecon/POCOnasa/networks/backbone/fkaconv_network.py", line 193, in forward x0 = self.activation(self.bn0(self.cv0(x, pos, pos, data["ids00"]))) KeyError: 'ids00'

    I'm confusing that what data["ids00"] in x0 = self.activation(self.bn0(self.cv0(x, pos, pos, data["ids00"]))) refers to? And similarily, there are many unkown data like data["ids10"]. How could I get this data? Thanks in advance!

    opened by Green89757 2
  • Failed to run

    Failed to run "python setup.py build_ext --inplace"

    Hello,

    After installing Pytorch and Pytorch Geometric, running python setup.py build_ext --inplace gives the following error:

    gcc: error: /my/computer/path/to/POCO/build/temp.linux-x86_64-3.7/eval/src/utils/libkdtree/pykdtree/_kdtree_core.o: No such file or directory
error: command 'gcc' failed with exit status 1

    Any suggestions would be appreciated.

    opened by TixiaoShan 2
  • Bug during generating

    Bug during generating

    Hello again,

    When I try to generate the shapenet results with the command:

    python generate.py --config results/ShapeNet_Normals_FKAConv_InterpAttentionKHeadsNet_None/config.yaml --gen_resolution_global 128

    I got this error:

    **loading CUDA OK INFO:root:Creating the network INFO:root:InterpNet - Simple - K=64 INFO:root:Network -- backbone -- 12783956 parameters INFO:root:Network -- projection -- 6498 parameters INFO:root:Network -- Number of parameters 12790454 INFO:root:Getting the dataset INFO:root:Normals as features INFO:root:Dataset - ShapeNet- None INFO:root:Dataset - len 8751 INFO:root:InterpNet - Simple - K=64 INFO:root:Network -- backbone -- 12783956 parameters INFO:root:Network -- projection -- 6498 parameters

    0%| | 0/8751 [00:00<?, ?it/s] 0%| | 0/8751 [00:04<?, ?it/s] Traceback (most recent call last): File "generate.py", line 594, in main(config) File "generate.py", line 539, in main step=step File "generate.py", line 170, in export_mesh_and_refine_vertices_region_growing_v2 mesh = o3d.geometry.TriangleMesh(o3d_verts, o3d_faces) TypeError: init(): incompatible constructor arguments. The following argument types are supported: 1. open3d.open3d.geometry.TriangleMesh() 2. open3d.open3d.geometry.TriangleMesh(arg0: open3d.open3d.geometry.TriangleMesh)

    Invoked with: std::vectorEigen::Vector3d with 133788 elements. Use numpy.asarray() to access data., std::vectorEigen::Vector3i with 267574 elements. Use numpy.asarray() to access data.

    Did you forget to #include <pybind11/stl.h>? Or <pybind11/complex.h>, <pybind11/functional.h>, <pybind11/chrono.h>, etc. Some automatic conversions are optional and require extra headers to be included when compiling your pybind11 module. **

    Thanks in advance and have a nice day. Best. Mulin

    opened by MulinYu 1
  • It takes one hour to generate the mesh from 16k point clouds in ScanNet dataset

    It takes one hour to generate the mesh from 16k point clouds in ScanNet dataset

    Dear author,

    Thanks for sharing your interesting work!

    I use the provided script to generate the mesh from the ScanNet dataset, and I found that it takes a long time (e.g, nearly 1 hour from 16K point clouds input). I am wondering if it is a normal situation. BTW, I just use the following script to generate mesh; python generate.py --config results/ShapeNet_Normals_FKAConv_InterpAttentionKHeadsNet_None/config.yaml --gen_autoscale True --gen_autoscale_target 0.01 --gen_resolution_metric 0.01 --dataset_name SceneNet20 --dataset_root data/SceneNet20 --manifold_points -1

    Looking forward to your reply. Many thanks in advance.

    Best, Runsong

    opened by Runsong123 0
  • RuntimeError: CUDA error: CUBLAS_STATUS_EXECUTION_FAILED when running the code

    RuntimeError: CUDA error: CUBLAS_STATUS_EXECUTION_FAILED when running the code

    Hi! Thanks for your interesting and enlightening work on point cloud reconstruction tasks, and we are trying to reproduce your work. However, we encounter an error when running your code: Capture I am wondering how to fix this bug? Thanks for your timely response.

    opened by CUHKWilliam 3
Owner
valeo.ai
We are an international team based in Paris, conducting AI research for Valeo automotive applications, in collaboration with world-class academics.
valeo.ai
GitHub Repository
Breast Cancer Classification Model is applied on a different dataset

Breast Cancer Classification Model is applied on a different dataset

1 Feb 04, 2022
A High-Quality Real Time Upscaler for Anime Video

Anime4K Anime4K is a set of open-source, high-quality real-time anime upscaling/denoising algorithms that can be implemented in any programming langua

15.7k Jan 06, 2023
Synthesizing Long-Term 3D Human Motion and Interaction in 3D in CVPR2021

Long-term-Motion-in-3D-Scenes This is an implementation of the CVPR'21 paper "Synthesizing Long-Term 3D Human Motion and Interaction in 3D". Please ch

Jiashun Wang 76 Dec 13, 2022
Unofficial PyTorch implementation of Attention Free Transformer (AFT) layers by Apple Inc.

aft-pytorch Unofficial PyTorch implementation of Attention Free Transformer's layers by Zhai, et al. [abs, pdf] from Apple Inc. Installation You can i

Rishabh Anand 184 Dec 12, 2022
Attention-based CNN-LSTM and XGBoost hybrid model for stock prediction

Attention-based CNN-LSTM and XGBoost hybrid model for stock prediction Requirements The code has been tested running under Python 3.7.4, with the foll

zshicode 84 Jan 01, 2023
Reinforcement Learning for Portfolio Management

qtrader Reinforcement Learning for Portfolio Management Why Reinforcement Learning? Learns the optimal action, rather than models the market. Adaptive

Angelos Filos 406 Jan 01, 2023
This repository contains source code for the Situated Interactive Language Grounding (SILG) benchmark

SILG This repository contains source code for the Situated Interactive Language Grounding (SILG) benchmark. If you find this work helpful, please cons

Victor Zhong 17 Nov 27, 2022
A deep learning network built with TensorFlow and Keras to classify gender and estimate age.

Convolutional Neural Network (CNN). This repository contains a source code of a deep learning network built with TensorFlow and Keras to classify gend

Pawel Dziemiach 1 Dec 19, 2021
Learning trajectory representations using self-supervision and programmatic supervision.

Trajectory Embedding for Behavior Analysis (TREBA) Implementation from the paper: Jennifer J. Sun, Ann Kennedy, Eric Zhan, David J. Anderson, Yisong Y

58 Jan 06, 2023
A scientific and useful toolbox, which contains practical and effective long-tail related tricks with extensive experimental results

Bag of tricks for long-tailed visual recognition with deep convolutional neural networks This repository is the official PyTorch implementation of AAA

Yong-Shun Zhang 181 Dec 28, 2022
Semantic graph parser based on Categorial grammars

Lambekseq "Everyone who failed Greek or Latin hates it." This package is for proving theorems in Categorial grammars (CG) and constructing semantic gr

10 Aug 19, 2022
Generative Adversarial Networks for High Energy Physics extended to a multi-layer calorimeter simulation

CaloGAN Simulating 3D High Energy Particle Showers in Multi-Layer Electromagnetic Calorimeters with Generative Adversarial Networks. This repository c

Deep Learning for HEP 101 Nov 13, 2022
The AugNet Python module contains functions for the fast computation of image similarity.

AugNet AugNet: End-to-End Unsupervised Visual Representation Learning with Image Augmentation arxiv link In our work, we propose AugNet, a new deep le

Ming 74 Dec 28, 2022
💡 Learnergy is a Python library for energy-based machine learning models.

Learnergy: Energy-based Machine Learners Welcome to Learnergy. Did you ever reach a bottleneck in your computational experiments? Are you tired of imp

Gustavo Rosa 57 Nov 17, 2022
PyTorch implementation of our ICCV paper DeFRCN: Decoupled Faster R-CNN for Few-Shot Object Detection.

Introduction This repo contains the official PyTorch implementation of our ICCV paper DeFRCN: Decoupled Faster R-CNN for Few-Shot Object Detection. Up

133 Dec 29, 2022
Anomaly detection related books, papers, videos, and toolboxes

Anomaly Detection Learning Resources Outlier Detection (also known as Anomaly Detection) is an exciting yet challenging field, which aims to identify

Yue Zhao 6.7k Dec 31, 2022
Official pytorch implementation of Active Learning for deep object detection via probabilistic modeling (ICCV 2021)

Active Learning for Deep Object Detection via Probabilistic Modeling This repository is the official PyTorch implementation of Active Learning for Dee

NVIDIA Research Projects 130 Jan 06, 2023
Used to record WKU's utility bills on a regular basis.

WKU水电费小助手 一个用于定期记录WKU水电费的脚本 Looking for English Readme? 背景 由于WKU校园内的水电账单系统时常存在扣费延迟的现象，而补扣的费用缺乏令人信服的证明。不少学生为费用摸不着头脑，但也没有申诉的依据。为了更好地掌握水电费使用情况，留下一手证据，我开源

2 Jul 21, 2022
[NeurIPS'21 Spotlight] PyTorch code for our paper "Aligned Structured Sparsity Learning for Efficient Image Super-Resolution"

ASSL This repository is for a new network pruning method (Aligned Structured Sparsity Learning, ASSL) for efficient single image super-resolution (SR)

Huan Wang 47 Nov 28, 2022
Json2Xml tool will help you convert from json COCO format to VOC xml format in Object Detection Problem.

JSON 2 XML All codes assume running from root directory. Please update the sys path at the beginning of the codes before running. Over View Json2Xml t

Nguyễn Trường Lâu 6 Aug 22, 2022