We evaluate our method on different datasets (including ShapeNet, CUB-200-2011, and Pascal3D+) and achieve state-of-the-art results, outperforming all the other supervised and unsupervised methods and 3D representations, all in terms of performance, accuracy, and training time.

Related tags

Deep Learningcomputer-visiondeep-learningrenderingcomputer-graphicsvoxelpoint-cloudpytorchmeshganneural-networksshapenet3d-reconstructionloss-functionsshapenet-datasetcub-datasetpascal3dpose-predic
Overview

An Effective Loss Function for Generating 3D Models from Single 2D Image without Rendering

Papers with code | Paper

Nikola Zubić   Pietro Lio  

University of Novi Sad   University of Cambridge

AIAI 2021

Citation

Besides AIAI 2021, our paper is in a Springer's book entitled "Artificial Intelligence Applications and Innovations": link

Please, cite our paper if you find this code useful for your research.

@article{zubic2021effective,
  title={An Effective Loss Function for Generating 3D Models from Single 2D Image without Rendering},
  author={Zubi{\'c}, Nikola and Li{\`o}, Pietro},
  journal={arXiv preprint arXiv:2103.03390},
  year={2021}
}

Prerequisites

  • Download code:
    Git clone the code with the following command:

    git clone https://github.com/NikolaZubic/2dimageto3dmodel.git

  • Open the project with Conda Environment (Python 3.7)

  • Install packages:

    conda install pytorch torchvision torchaudio cudatoolkit=11.0 -c pytorch

    Then git clone Kaolin library in the root (2dimageto3dmodel) folder with the following commit and run the following commands:

    cd kaolin
python setup.py install
pip install --no-dependencies nuscenes-devkit opencv-python-headless scikit-learn joblib pyquaternion cachetools
pip install packaging

Run the program

Run the following commands from the root/code/ (2dimageto3dmodel/code/) directory:

python main.py --dataset cub --batch_size 16 --weights pretrained_weights_cub --save_results

for the CUB Birds Dataset. 

python main.py --dataset p3d --batch_size 16 --weights pretrained_weights_p3d --save_results

for the Pascal 3D+ Dataset.

The results will be saved at 2dimageto3dmodel/code/results/ path.

Continue training

To continue the training process:
Run the following commands (without --save_results) from the root/code/ (2dimageto3dmodel/code/) directory:

python main.py --dataset cub --batch_size 16 --weights pretrained_weights_cub

for the CUB Birds Dataset. 

python main.py --dataset p3d --batch_size 16 --weights pretrained_weights_p3d

for the Pascal 3D+ Dataset.

License

MIT

Acknowledgment

This idea has been built based on the architecture of Insafutdinov & Dosovitskiy.
Poisson Surface Reconstruction was used for Point Cloud to 3D Mesh transformation.
The GAN architecture (used for texture mapping) is a mixture of Xian's TextureGAN and Li's GAN.

Comments
  • Where is cmr_data?

    Where is cmr_data?

    Keep running into this issue from cmr_data.p3d import P3dDataset and from cmr_data.p3d import CUBDataset

    but you do not have these files in your repo. I tried using cub_200_2011_dataset.py but it does not take in the same number of arguments as the CUBDataset class used in run_reconstruction.py.

    opened by achhabria7 6
  • ModuleNotFoundError: No module named 'kaolin.graphics'

    ModuleNotFoundError: No module named 'kaolin.graphics'

    Pascal 3D+ dataset with 4722 images is successfully loaded.

    Traceback (most recent call last): File "main.py", line 149, in <module> from rendering.renderer import Renderer File "/home/ujjawal/my_work/object_recon/2d3d/code/rendering/renderer.py", line 1, in <module> from kaolin.graphics.dib_renderer.rasterizer import linear_rasterizer ModuleNotFoundError: No module named kaolin.graphics

    I also downloaded the graphics folder from here https://github.com/NVIDIAGameWorks/kaolin/tree/e7e513173bd4159ae45be6b3e156a3ad156a3eb9 and tried to place in the graphics folder in the kaolin folder locally and here is the error Traceback (most recent call last): File "main.py", line 149, in <module> from rendering.renderer import Renderer File "/home/ujjawal/my_work/object_recon/2d3d/code/rendering/renderer.py", line 1, in <module> from kaolin.graphics.dib_renderer.rasterizer import linear_rasterizer File "/usr/local/lib/python3.6/dist-packages/kaolin-0.9.0-py3.6-linux-x86_64.egg/kaolin/graphics/__init__.py", line 2, in <module> File "/usr/local/lib/python3.6/dist-packages/kaolin-0.9.0-py3.6-linux-x86_64.egg/kaolin/graphics/nmr/__init__.py", line 1, in <module> File "/usr/local/lib/python3.6/dist-packages/kaolin-0.9.0-py3.6-linux-x86_64.egg/kaolin/graphics/nmr/rasterizer.py", line 30, in <module> ImportError: cannot import name rasterize_cuda

    opened by ujjawalcse 6
  • No module named 'models.reconstruction'

    No module named 'models.reconstruction'

    Dear NikolaZubic :
    Thanks for you updated the code recently. Did you put the reconstruction.py in the models folder？When I run “python run_reconstruction.py --name pretrained_reconstruction_cub --dataset cub --batch_size 10 --generate_pseudogt” it display
    No module named 'models.reconstruction.

    opened by lw0210 2
  • inference with single RGB pictures

    inference with single RGB pictures

    Hi, I am interested with your work, it is wonderful, and I want to use my own picture to test the model, could you provided the pretrained model and inference scripts.

    opened by 523997931 2
  • can't find the pseudogt_512*512\.npz file

    can't find the pseudogt_512*512\.npz file

    Dear NikolaZubic: I want to quote your paper, but I can't find the pseudogt_512512.npz file and can't reproduce it. Can you give me the pseudogt_512512.npz file and help me reproduce it? Thanks

    opened by Yangfuha 1
  • ValueError: Training a model requires the pseudo-ground-truth to be setup beforehand.

    ValueError: Training a model requires the pseudo-ground-truth to be setup beforehand.

    I recently read your paper and was very interested in it . I want to reproduce the code of this paper. When I followed your instructions, I found it difficult for me to run the commands（python main.py --dataset cub --batch_size 16 --weights pretrained_weights_cub and python main.py --dataset p3d --batch_size 16 --weights pretrained_weights_p3d.）.And the program displayed a value error that training a model requires the pseudo-ground-truth to be setup beforehand. And I don’t know how to solve the problem, so I turn to you for help.I'm sorry to bother you, but I'really eager to solve the problem. I hope to get your reply.Thank you!

    opened by lw0210 1
  • Added step: switch to the correct correct Kaolin branch

    Added step: switch to the correct correct Kaolin branch

    This step will help others to avoid the "ModuleNotFoundError: No module named kaolin.graphics" error.

    Fix to issue: https://github.com/NikolaZubic/2dimageto3dmodel/issues/2

    opened by ricklentz 1
  • Shapenet V2 not training

    Shapenet V2 not training

    Great work guys. I was able to run the code on CUB dataset. But when I tried to run training_test_shape_net.py on Shape Net v2 chair class I'm getting errors because of missing files, unmatched file names, etc.

    So it would be helpful if you provide Shapenet Dataset Folder structure and files(images, masks) description or a sample folder and clear instructions for training the model shapenet dataset. And also if possible give pre-trained weights for the Shape net dataset models

    Thank you

    opened by girishdhegde 0
  • Pretrained model

    Pretrained model

    Hi, I find it hard to understand how to train the model on ShapeNet. It would be very helpful if you can provide a pretrained model on ShapeNet planes (I need it to test the performance in my project). If the pretrained models are not available, it would also be helpful to introduce me of how to train the model on ShapeNet.

    opened by YYYYYHC 0
  • How can I train on the boat set of the Pascal 3D+ dataset

    How can I train on the boat set of the Pascal 3D+ dataset

    I find the data of trainning such as "python run_reconstruction.py --name pretrained_reconstruction_p3d --dataset p3d --optimize_z0 --batch_size 50 --tensorboard" using the data of car.mat in sfm and data folder.Even if I rename the .mat to boat.mat and using the boat imageNet in Pascal 3D+ dataset,I find the shape of the result is more like a car not a boat.So I am wondering how to train the boat set.

    opened by lisentao 0
  • Custom Dataset

    Custom Dataset

    Hi!

    Love the work you guys have done. I am currently conducting a research. Could you please tell me how I would train on a custom dataset and how I would infer an image or create a 3d model out an image with pretrained weights that you have provided?

    opened by mahnoor-fatima-saad 0
  • How do I make my own dataset？

    How do I make my own dataset？

    Dear NikolaZubic： I want to use my own data set to replace the cub or P3D data set for training. Do you have any attention or requirements for images when making data sets?

    opened by lw0210 0
Releases(metadata)
Owner
Nikola Zubić
Interested in Artificial intelligence, Visual Computing and Cognitive science. For future AI projects: @reinai
Nikola Zubić
GitHub Repository
License Plate Detection Application

LicensePlate_Project 🚗 🚙 [Project] 2021.02 ~ 2021.09 License Plate Detection Application Overview 1. 데이터 수집 및 라벨링 차량 번호판 이미지를 직접 수집하여 각 이미지에 대해 '번호판

4 Oct 10, 2022
[ICLR 2022] Pretraining Text Encoders with Adversarial Mixture of Training Signal Generators

AMOS This repository contains the scripts for fine-tuning AMOS pretrained models on GLUE and SQuAD 2.0 benchmarks. Paper: Pretraining Text Encoders wi

Microsoft 22 Sep 15, 2022
Code for Paper Predicting Osteoarthritis Progression via Unsupervised Adversarial Representation Learning

Predicting Osteoarthritis Progression via Unsupervised Adversarial Representation Learning (c) Tianyu Han and Daniel Truhn, RWTH Aachen University, 20

Tianyu Han 7 Nov 22, 2022
Dynamic Graph Event Detection

DyGED Dynamic Graph Event Detection Get Started pip install -r requirements.txt TODO Paper link to arxiv, and how to cite. Twitter Weather dataset tra

Mert Koşan 3 May 09, 2022
Sarus implementation of classical ML models. The models are implemented using the Keras API of tensorflow 2. Vizualization are implemented and can be seen in tensorboard.

Sarus published models Sarus implementation of classical ML models. The models are implemented using the Keras API of tensorflow 2. Vizualization are

Sarus Technologies 39 Aug 19, 2022
ESPNet: Efficient Spatial Pyramid of Dilated Convolutions for Semantic Segmentation

ESPNet: Efficient Spatial Pyramid of Dilated Convolutions for Semantic Segmentation This repository contains the source code of our paper, ESPNet (acc

Sachin Mehta 515 Dec 13, 2022
Fuzzy Overclustering (FOC)

Fuzzy Overclustering (FOC) In real-world datasets, we need consistent annotations between annotators to give a certain ground-truth label. However, in

2 Nov 08, 2022
A collection of Jupyter notebooks to play with NVIDIA's StyleGAN3 and OpenAI's CLIP for a text-based guided image generation.

StyleGAN3 CLIP-based guidance StyleGAN3 + CLIP StyleGAN3 + inversion + CLIP This repo is a collection of Jupyter notebooks made to easily play with St

Eugenio Herrera 176 Dec 30, 2022
SpinalNet: Deep Neural Network with Gradual Input

SpinalNet: Deep Neural Network with Gradual Input This repository contains scripts for training different variations of the SpinalNet and its counterp

H M Dipu Kabir 142 Dec 30, 2022
Think Big, Teach Small: Do Language Models Distil Occam’s Razor?

Think Big, Teach Small: Do Language Models Distil Occam’s Razor? Software related to the paper "Think Big, Teach Small: Do Language Models Distil Occa

0 Dec 07, 2021
CLDF dataset derived from Robbeets et al.'s "Triangulation Supports Agricultural Spread" from 2021

CLDF dataset derived from Robbeets et al.'s "Triangulation Supports Agricultural Spread" from 2021 How to cite If you use these data please cite the o

Digital Linguistics 2 Dec 20, 2021
StocksMA is a package to facilitate access to financial and economic data of Moroccan stocks.

Creating easier access to the Moroccan stock market data What is StocksMA ? StocksMA is a package to facilitate access to financial and economic data

Salah Eddine LABIAD 28 Jan 04, 2023
A curated list of awesome neural radiance fields papers

Awesome Neural Radiance Fields A curated list of awesome neural radiance fields papers, inspired by awesome-computer-vision. How to submit a pull requ

Yen-Chen Lin 3.9k Dec 27, 2022
A Nim frontend for pytorch, aiming to be mostly auto-generated and internally using ATen.

Master Release Pytorch - Py + Nim A Nim frontend for pytorch, aiming to be mostly auto-generated and internally using ATen. Because Nim compiles to C+

Giovanni Petrantoni 425 Dec 22, 2022
An implementation of "Optimal Textures: Fast and Robust Texture Synthesis and Style Transfer through Optimal Transport"

Optex An implementation of Optimal Textures: Fast and Robust Texture Synthesis and Style Transfer through Optimal Transport for TU Delft CS4240. You c

Hans Brouwer 33 Jan 05, 2023
[SIGGRAPH 2020] Attribute2Font: Creating Fonts You Want From Attributes

Attr2Font Introduction This is the official PyTorch implementation of the Attribute2Font: Creating Fonts You Want From Attributes. Paper: arXiv | Rese

Yue Gao 200 Dec 15, 2022
Compare neural networks by their feature similarity

PyTorch Model Compare A tiny package to compare two neural networks in PyTorch. There are many ways to compare two neural networks, but one robust and

Anand Krishnamoorthy 181 Jan 04, 2023
Neural Scene Flow Prior (NeurIPS 2021 spotlight)

Neural Scene Flow Prior Xueqian Li, Jhony Kaesemodel Pontes, Simon Lucey Will appear on Thirty-fifth Conference on Neural Information Processing Syste

Lilac Lee 85 Jan 03, 2023
Ratatoskr: Worcester Tech's conference scheduling system

Ratatoskr: Worcester Tech's conference scheduling system In Norse mythology, Ratatoskr is a squirrel who runs up and down the world tree Yggdrasil to

4 Dec 22, 2022
Tensorflow implementation of DeepLabv2

TF-deeplab This is a Tensorflow implementation of DeepLab, compatible with Tensorflow 1.2.1. Currently it supports both training and testing the ResNe

Chenxi Liu 21 Sep 27, 2022