(Arxiv 2021) NeRF--: Neural Radiance Fields Without Known Camera Parameters

Overview

NeRF--: Neural Radiance Fields Without Known Camera Parameters

Project Page | Arxiv | Colab Notebook | Data

Zirui Wang¹, Shangzhe Wu², Weidi Xie², Min Chen³, Victor Adrian Prisacariu¹.

¹Active Vision Lab + ²Visual Geometry Group + ³e-Research Centre, University of Oxford.

Overview

We provide 3 training targets in this repository, under the tasks directory:

  1. task/nerfmm/train.py: This is our main training script for the NeRF-LLFF dataset, which estimates camera poses, focal lenghts and a NeRF jointly and monitors the absolute trajectory error (ATE) between our estimation of camera parameters and COLMAP estimation during training. This target can also start training from a COLMAP initialisation and refine the COLMAP camera parameters.
  2. task/refine_nerfmm/train.py: This is the training script that refines a pretrained nerfmm system.
  3. task/any_folder/train.py: This is a training script that takes a folder that contains forward-facing images and trains with our nerfmm system without making any comparison with COLMAP. It is similar to what we offer in our CoLab notebook and we treat this any_folder target as a playgraound, where users can try novel view synthesis by just providing an image folder and do not care how the camera parameter estimation compares with COLMAP.

For each target, we provide relevant utilities to evaluate our system. Specifically,

  • for the nerfmm target, we provide three utility files:
    • eval.py to evaluate image rendering quality on validation splits with PSNR, SSIM and LPIPS, i.e, results in Table 1.
    • spiral.py to render novel views using a spiral camera trajectory, i.e. results in Figure 1.
    • vis_learned_poses.py to visualise our camera parameter estimation with COLMAP estimation in 3D. It also computes ATE between them, i.e. E1 in Table 2.
  • for the refine_nerfmm target, all utilities in nerfmm target above are compatible with refine_nerfmm target, since it just refines a pretrained nerfmm system.
  • for the any_folder target, it has its own spiral.py and vis_learned_poses.py utilities, as it does not compare with COLMAP. It does not have a eval.py file as this target is treated as a playground and does not split images to train/validation sets. It only provides novel view synthesis results via the spiral.py file.

Table of Content

Environment

We provide a requirement.yml file to set up a conda environment:

git clone https://github.com/ActiveVisionLab/nerfmm.git
cd nerfmm
conda env create -f environment.yml

Generally, our code should be able to run with any pytorch >= 1.1 .

(Optional) Install open3d for visualisation. You might need a physical monitor to install this lib.

pip install open3d

Get Data

We use the NeRF-LLFF dataset with two small structural changes:

  1. We remove their image_4 and image_8 folder and downsample images to any desirable resolution during data loading dataloader/with_colmap.py, by calling PyTorch's interpolate function.
  2. We explicitly generate two txt files for train/val image ids. i.e. take every 8th image as the validation set, as in the official NeRF train/val split. The only difference is that we store them as txt files while NeRF split them during data loading. The file produces these two txt files is utils/split_dataset.py.

In addition to the NeRF-LLFF dataset, we provide two demo scenes to demonstrate how to use the any_folder target.

We pack the re-structured LLFF data and our data to a tar ball (~1.8G), to get it, run:

wget https://www.robots.ox.ac.uk/~ryan/nerfmm2021/nerfmm_release_data.tar.gz

Untar the data:

tar -xzvf path/to/the/tar.gz

Training

We show how to:

  1. train a nerfmm from scratch, i.e. initialise camera poses with identity matrices and focal lengths with image resolution:
    python tasks/nerf/train.py \
    --base_dir='path/to/nerfmm_release/data' \
    --scene_name='LLFF/fern'
  2. train a nerfmm from COLMAP initialisation:
    python tasks/nerf/train.py \
    --base_dir='path/to/nerfmm_release/data' \
    --scene_name='LLFF/fern' \
    --start_refine_pose_epoch=1000 \
    --start_refine_focal_epoch=1000
    This command initialises a nerfmm target with COLMAP parameters, trains with them for 1000 epochs, and starts refining those parameters after 1000 epochs.
  3. train a nerfmm from a pretrained nerfmm:
    python tasks/refine_nerfmm/train.py \
    --base_dir='path/to/nerfmm_release/data' \
    --scene_name='LLFF/fern' --start_refine_epoch=1000 \
    --ckpt_dir='path/to/a/dir/contains/nerfmm/ckpts'
    This command initialises a refine_nerfmm target with a set of pretrained nerfmm parameters, trains with them for 1000 epochs, and starts refining those parameters after 1000 epochs.
  4. train an any_folder from scratch given an image folder:
    python tasks/any_folder/train.py \
    --base_dir='path/to/nerfmm_release/data' \
    --scene_name='any_folder_demo/desk'
    This command trains an any_folder target using a provided demo scene desk.

(Optional) set a symlink to the downloaded data:

mkdir data_dir  # do it in this nerfmm repo
cd data_dir
ln -s /path/to/downloaded/data ./nerfmm_release_data
cd ..

this can simplify the above training commands, for example:

python tasks/nerfmm/train.py

Evaluation

Compute image quality metrics

Call eval.py in nerfmm target:

python tasks/nerfmm/eval.py \
--base_dir='path/to/nerfmm_release/data' \
--scene_name='LLFF/fern' \
--ckpt_dir='path/to/a/dir/contains/nerfmm/ckpts'

This file can be used to evaluate a checkpoint trained with refine_nerfmm target. For some scenes, you might need to tweak with --opt_eval_lr option to get the best results. Common values for opt_eval_lr are 0.01 / 0.005 / 0.001 / 0.0005 / 0.0001. The default value is 0.001. Overall, it finds validation poses that can produce highest PSNR on validation set while freezing NeRF and focal lengths. We do this because the learned camera pose space is different from the COLMAP estimated camera pose space.

Render novel views

Call spiral.py in each target. The spiral.py in nerfmm is compatible with refine_nerfmm target:

python spiral.py \
--base_dir='path/to/nerfmm_release/data' \
--scene_name='LLFF/fern' \
--ckpt_dir='path/to/a/dir/contains/nerfmm/ckpts'

Visualise estimated poses in 3D

Call vis_learned_poses.py in each target. The vis_learned_poses.py in nerfmm is compatible with refine_nerfmm target:

python spiral.py \
--base_dir='path/to/nerfmm_release/data' \
--scene_name='LLFF/fern' \
--ckpt_dir='path/to/a/dir/contains/nerfmm/ckpts'

Acknowledgement

Shangzhe Wu is supported by Facebook Research. Weidi Xie is supported by Visual AI (EP/T028572/1).

The authors would like to thank Tim Yuqing Tang for insightful discussions and proofreading.

During our NeRF implementation, we referenced several open sourced NeRF implementations, and we thank their contributions. Specifically, we referenced functions from nerf and nerf-pytorch, and borrowed/modified code from nerfplusplus and nerf_pl. We especially appreciate the detailed code comments and git issue answers in nerf_pl.

Citation

@article{wang2021nerfmm,
  title={Ne{RF}$--$: Neural Radiance Fields Without Known Camera Parameters},
  author={Zirui Wang and Shangzhe Wu and Weidi Xie and Min Chen and Victor Adrian Prisacariu},
  journal={arXiv preprint arXiv:2102.07064},
  year={2021}
}
Owner
Active Vision Laboratory
Active Vision Laboratory
JAX-based neural network library

Haiku: Sonnet for JAX Overview | Why Haiku? | Quickstart | Installation | Examples | User manual | Documentation | Citing Haiku What is Haiku? Haiku i

DeepMind 2.3k Jan 04, 2023
[CIKM 2021] Enhancing Aspect-Based Sentiment Analysis with Supervised Contrastive Learning

Enhancing Aspect-Based Sentiment Analysis with Supervised Contrastive Learning. This repo contains the PyTorch code and implementation for the paper E

Akuchi 18 Dec 22, 2022
Code that accompanies the paper Semi-supervised Deep Kernel Learning: Regression with Unlabeled Data by Minimizing Predictive Variance

Semi-supervised Deep Kernel Learning This is the code that accompanies the paper Semi-supervised Deep Kernel Learning: Regression with Unlabeled Data

58 Oct 26, 2022
Research on Tabular Deep Learning (Python package & papers)

Research on Tabular Deep Learning For paper implementations, see the section "Papers and projects". rtdl is a PyTorch-based package providing a user-f

Yura Gorishniy 510 Dec 30, 2022
existing and custom freqtrade strategies supporting the new hyperstrategy format.

freqtrade-strategies Description Existing and self-developed strategies, rewritten to support the new HyperStrategy format from the freqtrade-develop

39 Aug 20, 2021
A Multi-modal Model Chinese Spell Checker Released on ACL2021.

ReaLiSe ReaLiSe is a multi-modal Chinese spell checking model. This the office code for the paper Read, Listen, and See: Leveraging Multimodal Informa

DaDa 106 Dec 29, 2022
Toontown House CT Edition

Toontown House: Classic Toontown House Classic source that should just work. ❓ W

Open Source Toontown Servers 5 Jan 09, 2022
TAug :: Time Series Data Augmentation using Deep Generative Models

TAug :: Time Series Data Augmentation using Deep Generative Models Note!!! The package is under development so be careful for using in production! Fea

35 Dec 06, 2022
Benchmarks for Model-Based Optimization

Design-Bench Design-Bench is a benchmarking framework for solving automatic design problems that involve choosing an input that maximizes a black-box

Brandon Trabucco 43 Dec 20, 2022
3D-aware GANs based on NeRF (arXiv).

CIPS-3D This repository will contain the code of the paper, CIPS-3D: A 3D-Aware Generator of GANs Based on Conditionally-Independent Pixel Synthesis.

Peterou 563 Dec 31, 2022
Contrastive Learning with Non-Semantic Negatives

Contrastive Learning with Non-Semantic Negatives This repository is the official implementation of Robust Contrastive Learning Using Negative Samples

39 Jul 31, 2022
Get started learning C# with C# notebooks powered by .NET Interactive and VS Code.

.NET Interactive Notebooks for C# Welcome to the home of .NET interactive notebooks for C#! How to Install Download the .NET Coding Pack for VS Code f

.NET Platform 425 Dec 25, 2022
Fashion Recommender System With Python

Fashion-Recommender-System Thr growing e-commerce industry presents us with a la

Omkar Gawade 2 Feb 02, 2022
A tf.keras implementation of Facebook AI's MadGrad optimization algorithm

MADGRAD Optimization Algorithm For Tensorflow This package implements the MadGrad Algorithm proposed in Adaptivity without Compromise: A Momentumized,

20 Aug 18, 2022
Applying curriculum to meta-learning for few shot classification

Curriculum Meta-Learning for Few-shot Classification We propose an adaptation of the curriculum training framework, applicable to state-of-the-art met

Stergiadis Manos 3 Oct 25, 2022
Codes accompanying the paper "Believe What You See: Implicit Constraint Approach for Offline Multi-Agent Reinforcement Learning" (NeurIPS 2021 Spotlight

Implicit Constraint Q-Learning This is a pytorch implementation of ICQ on Datasets for Deep Data-Driven Reinforcement Learning (D4RL) and ICQ-MA on SM

42 Dec 23, 2022
Apply a perspective transformation to a raster image inside Inkscape (no need to use an external software such as GIMP or Krita).

Raster Perspective Apply a perspective transformation to bitmap image using the selected path as envelope, without the need to use an external softwar

s.ouchene 19 Dec 22, 2022
Safe Model-Based Reinforcement Learning using Robust Control Barrier Functions

README Repository containing the code for the paper "Safe Model-Based Reinforcement Learning using Robust Control Barrier Functions". Specifically, an

Yousef Emam 13 Nov 24, 2022
Time Series Forecasting with Temporal Fusion Transformer in Pytorch

Forecasting with the Temporal Fusion Transformer Multi-horizon forecasting often contains a complex mix of inputs – including static (i.e. time-invari

Nicolás Fornasari 6 Jan 24, 2022
CKD - Collaborative Knowledge Distillation for Heterogeneous Information Network Embedding

Collaborative Knowledge Distillation for Heterogeneous Information Network Embed

zhousheng 9 Dec 05, 2022