Official code release for "GRAF: Generative Radiance Fields for 3D-Aware Image Synthesis"

Related tags

Deep Learninggraf
Overview

GRAF


This repository contains official code for the paper GRAF: Generative Radiance Fields for 3D-Aware Image Synthesis.

You can find detailed usage instructions for training your own models and using pre-trained models below.

If you find our code or paper useful, please consider citing

@inproceedings{Schwarz2020NEURIPS,
  title = {GRAF: Generative Radiance Fields for 3D-Aware Image Synthesis},
  author = {Schwarz, Katja and Liao, Yiyi and Niemeyer, Michael and Geiger, Andreas},
  booktitle = {Advances in Neural Information Processing Systems (NeurIPS)},
  year = {2020}
}

Installation

First you have to make sure that you have all dependencies in place. The simplest way to do so, is to use anaconda.

You can create an anaconda environment called graf using

conda env create -f environment.yml
conda activate graf

Next, for nerf-pytorch install torchsearchsorted. Note that this requires torch>=1.4.0 and CUDA >= v10.1. You can install torchsearchsorted via

cd submodules/nerf_pytorch
pip install -r requirements.txt
cd torchsearchsorted
pip install .
cd ../../../

Demo

You can now test our code via:

python eval.py configs/carla.yaml --pretrained --rotation_elevation

This script should create a folder results/carla_128_from_pretrained/eval/ where you can find generated videos varying camera pose for the Cars dataset.

Datasets

If you only want to generate images using our pretrained models you do not need to download the datasets. The datasets are only needed if you want to train a model from scratch.

Cars

To download the Cars dataset from the paper simply run

cd data
./download_carla.sh
cd ..

This creates a folder data/carla/ downloads the images as a zip file and extracts them to data/carla/. While we do not use camera poses in this project we provide them for completeness. Your can download them by running

cd data
./download_carla_poses.sh
cd ..

This downloads the camera intrinsics (single file, equal for all images) and extrinsics corresponding to each image.

Faces

Download celebA. Then replace data/celebA in configs/celebA.yaml with *PATH/TO/CELEBA*/Img/img_align_celebA.

Download celebA_hq. Then replace data/celebA_hq in configs/celebAHQ.yaml with *PATH/TO/CELEBA_HQ*.

Cats

Download the CatDataset. Run

cd data
python preprocess_cats.py PATH/TO/CATS/DATASET
cd ..

to preprocess the data and save it to data/cats. If successful this script should print: Preprocessed 9407 images.

Birds

Download CUB-200-2011 and the corresponding Segmentation Masks. Run

cd data
python preprocess_cub.py PATH/TO/CUB-200-2011 PATH/TO/SEGMENTATION/MASKS
cd ..

to preprocess the data and save it to data/cub. If successful this script should print: Preprocessed 8444 images.

Usage

When you have installed all dependencies, you are ready to run our pre-trained models for 3D-aware image synthesis.

Generate images using a pretrained model

To evaluate a pretrained model, run

python eval.py CONFIG.yaml --pretrained --fid_kid --rotation_elevation --shape_appearance

where you replace CONFIG.yaml with one of the config files in ./configs.

This script should create a folder results/EXPNAME/eval with FID and KID scores in fid_kid.csv, videos for rotation and elevation in the respective folders and an interpolation for shape and appearance, shape_appearance.png.

Note that some pretrained models are available for different image sizes which you can choose by setting data:imsize in the config file to one of the following values:

configs/carla.yaml: 
    data:imsize 64 or 128 or 256 or 512
configs/celebA.yaml:
    data:imsize 64 or 128
configs/celebAHQ.yaml:
    data:imsize 256 or 512

Train a model from scratch

To train a 3D-aware generative model from scratch run

python train.py CONFIG.yaml

where you replace CONFIG.yaml with your config file. The easiest way is to use one of the existing config files in the ./configs directory which correspond to the experiments presented in the paper. Note that this will train the model from scratch and will not resume training for a pretrained model.

You can monitor on http://localhost:6006 the training process using tensorboard:

cd OUTPUT_DIR
tensorboard --logdir ./monitoring --port 6006

where you replace OUTPUT_DIR with the respective output directory.

For available training options, please take a look at configs/default.yaml.

Evaluation of a new model

For evaluation of the models run

python eval.py CONFIG.yaml --fid_kid --rotation_elevation --shape_appearance

where you replace CONFIG.yaml with your config file.

Multi-View Consistency Check

You can evaluate the multi-view consistency of the generated images by running a Multi-View-Stereo (MVS) algorithm on the generated images. This evaluation uses COLMAP and make sure that you have COLMAP installed to run

python eval.py CONFIG.yaml --reconstruction

where you replace CONFIG.yaml with your config file. You can also evaluate our pretrained models via:

python eval.py configs/carla.yaml --pretrained --reconstruction

This script should create a folder results/EXPNAME/eval/reconstruction/ where you can find generated multi-view images in images/ and the corresponding 3D reconstructions in models/.

Further Information

GAN training

This repository uses Lars Mescheder's awesome framework for GAN training.

NeRF

We base our code for the Generator on this great Pytorch reimplementation of Neural Radiance Fields.

Security evaluation module with onnx, pytorch, and SecML.

🚀 🐼 🔥 PandaVision Integrate and automate security evaluations with onnx, pytorch, and SecML! Installation Starting the server without Docker If you

Maura Pintor 11 Apr 12, 2022
Roger Labbe 13k Dec 29, 2022
Ranking Models in Unlabeled New Environments (iccv21)

Ranking Models in Unlabeled New Environments Prerequisites This code uses the following libraries Python 3.7 NumPy PyTorch 1.7.0 + torchivision 0.8.1

14 Dec 17, 2021
Apache Spark - A unified analytics engine for large-scale data processing

Apache Spark Spark is a unified analytics engine for large-scale data processing. It provides high-level APIs in Scala, Java, Python, and R, and an op

The Apache Software Foundation 34.7k Jan 04, 2023
交互式标注软件,暂定名 iann

iann 交互式标注软件,暂定名iann。 安装 按照官网介绍安装paddle。 安装其他依赖 pip install -r requirements.txt 运行 git clone https://github.com/PaddleCV-SIG/iann/ cd iann python iann

294 Dec 30, 2022
Companion code for "Bayesian logistic regression for online recalibration and revision of risk prediction models with performance guarantees"

Companion code for "Bayesian logistic regression for online recalibration and revision of risk prediction models with performance guarantees" Installa

0 Oct 13, 2021
High performance distributed framework for training deep learning recommendation models based on PyTorch.

PERSIA (Parallel rEcommendation tRaining System with hybrId Acceleration) is developed by AI 340 Dec 30, 2022

Repository for Driving Style Recognition algorithms for Autonomous Vehicles

Driving Style Recognition Using Interval Type-2 Fuzzy Inference System and Multiple Experts Decision Making Created by Iago Pachêco Gomes at USP - ICM

Iago Gomes 9 Nov 28, 2022
This repo is customed for VisDrone.

Object Detection for VisDrone(无人机航拍图像目标检测) My environment 1、Windows10 (Linux available) 2、tensorflow = 1.12.0 3、python3.6 (anaconda) 4、cv2 5、ensemble

53 Jul 17, 2022
Source code and dataset for ACL2021 paper: "ERICA: Improving Entity and Relation Understanding for Pre-trained Language Models via Contrastive Learning".

ERICA Source code and dataset for ACL2021 paper: "ERICA: Improving Entity and Relation Understanding for Pre-trained Language Models via Contrastive L

THUNLP 75 Nov 02, 2022
Official code of ICCV2021 paper "Residual Attention: A Simple but Effective Method for Multi-Label Recognition"

CSRA This is the official code of ICCV 2021 paper: Residual Attention: A Simple But Effective Method for Multi-Label Recoginition Demo, Train and Vali

163 Dec 22, 2022
This repository contains the code for: RerrFact model for SciVer shared task

RerrFact This repository contains the code for: RerrFact model for SciVer shared task. Setup for Inference 1. Download SciFact database Download the S

Ashish Rana 1 May 22, 2022
Hierarchical Few-Shot Generative Models

Hierarchical Few-Shot Generative Models Giorgio Giannone, Ole Winther This repo contains code and experiments for the paper Hierarchical Few-Shot Gene

Giorgio Giannone 6 Dec 12, 2022
pytorch implementation of the ICCV'21 paper "MVTN: Multi-View Transformation Network for 3D Shape Recognition"

MVTN: Multi-View Transformation Network for 3D Shape Recognition (ICCV 2021) By Abdullah Hamdi, Silvio Giancola, Bernard Ghanem Paper | Video | Tutori

Abdullah Hamdi 64 Jan 03, 2023
Official PyTorch Implementation of Embedding Transfer with Label Relaxation for Improved Metric Learning, CVPR 2021

Embedding Transfer with Label Relaxation for Improved Metric Learning Official PyTorch implementation of CVPR 2021 paper Embedding Transfer with Label

Sungyeon Kim 37 Dec 06, 2022
RTSeg: Real-time Semantic Segmentation Comparative Study

Real-time Semantic Segmentation Comparative Study The repository contains the official TensorFlow code used in our papers: RTSEG: REAL-TIME SEMANTIC S

Mennatullah Siam 592 Nov 18, 2022
The code for replicating the experiments from the LFI in SSMs with Unknown Dynamics paper.

Likelihood-Free Inference in State-Space Models with Unknown Dynamics This package contains the codes required to run the experiments in the paper. Th

Alex Aushev 0 Dec 27, 2021
Physical Anomalous Trajectory or Motion (PHANTOM) Dataset

Physical Anomalous Trajectory or Motion (PHANTOM) Dataset Description This dataset contains the six different classes as described in our paper[]. The

0 Dec 16, 2021
The repository is for safe reinforcement learning baselines.

Safe-Reinforcement-Learning-Baseline The repository is for Safe Reinforcement Learning (RL) research, in which we investigate various safe RL baseline

172 Dec 19, 2022
Pgn2tex - Scripts to convert pgn files to latex document. Useful to build books or pdf from pgn studies

Pgn2Latex (WIP) A simple script to make pdf from pgn files and studies. It's sti

12 Jul 23, 2022