Rasterize with the least efforts for researchers.

Related tags

Deep Learningutils3d
Overview

utils3d

Rasterize and do image-based 3D transforms with the least efforts for researchers. Based on numpy and OpenGL.

It could be helpful when you want to:

  • rasterize a simple mesh but don't want get into OpenGL chores
  • warp an image as a 2D or 3D mesh (eg. optical-flow-based warping)
  • render a optical flow image

This tool sets could help you achieve them in a few lines.

It is NOT what you are looking for when you want:

  • a differentiable rasterization tool. You should turn to nvdiffrast, pytorch3d, SoftRas etc.
  • a real-time graphics application. Though as fast as it could be, the expected performance of util3d rasterization is to be around 20 ~ 100 ms. It is not expected to fully make use of GPU performance because of the overhead of buffering every time calling rasterzation. If the best performance withou any overhead is demanded, You will have to manage buffer objects like VBO, VAO and FBO. I personally recommand moderngl as an alternative python OpenGL library.

Install

The folder of repo is a package. Clone the repo.

git clone https://github.com/EasternJournalist/utils3d.git 

Install requirements

pip install numpy
pip install moderngl

Usage

At first, one step to initialize a OpenGL context. It depends on your platform and machine.

import utils3d

ctx = utils3d.Context(standalone=True)                 # Recommanded for a standalone python program. The machine must have a display device (virtual display like X11 is also okay)
ctx = utils3d.Context(standalone=False)                 # Recommanded for a nested python script running in a windowed opengl program to share the OpenGL context, eg. Blender.
ctx = utils3d.Context(standalone=True, backend='egl')   # Recommanded for a program running on a headless linux server (without any display device)

The functions the most probably you would like to use

  • ctx.rasterize(...): rasterize trianglular mesh with vertex attributes.
  • ctx.texture(uv, texture): sample texture by a UV image. Exactly the same as grid sample, but an OpenGL shader implementation.
  • ctx.rasterize_texture(...): rasterize trianglular mesh with texture

Some other functions that could be helpful for certain purposes

  • ctx.render_flow(...): render an optical flow image given source and target geometry
  • ctx.warp_image_3d(image, pixel_positions, transform_matrix)
  • ctx.warp_image_by_flow(image, flow, occlusion_mask)

Useful tool functions

  • image_uv(width, height) : return a numpy array of shape [height, width, 2], the image uv of each pixel.
  • image_mesh(width, height, mask=None) : return a quad mesh connecting all neighboring pixels as vertices. A boolean array of shape [height, width] or [height, width, 1] mask is optional. If a mask is provided, only pixels where mask value is True are involved in te mesh.
  • triangulate(faces) : convert a polygonal mesh into a triangular mesh (naively).
  • perspective_from_image()
  • perspective_from_fov_xy()
  • projection(vertices, model_matrix=None, view_matrix=None, projection_matrix=None): project 3D points to 2D screen space following the OpenGL convention (except for using row major matrix). This also gives a insight of how the projection works when you have confusion about the coordinate system.
  • compute_face_normal(vertices, faces)
  • compute_vertex_normal(vertices, faces)
Owner
Ruicheng Wang
Microsoft Research Asia Intern
Ruicheng Wang
SwinTrack: A Simple and Strong Baseline for Transformer Tracking

SwinTrack This is the official repo for SwinTrack. A Simple and Strong Baseline Prerequisites Environment conda (recommended) conda create -y -n SwinT

LitingLin 196 Jan 04, 2023
Edge-oriented Convolution Block for Real-time Super Resolution on Mobile Devices, ACM Multimedia 2021

Codes for ECBSR Edge-oriented Convolution Block for Real-time Super Resolution on Mobile Devices Xindong Zhang, Hui Zeng, Lei Zhang ACM Multimedia 202

xindong zhang 236 Dec 26, 2022
Rl-quickstart - Reinforcement Learning Quickstart

Reinforcement Learning Quickstart To get setup with the repository, git clone ht

UCLA DataRes 3 Jun 16, 2022
Serve TensorFlow ML models with TF-Serving and then create a Streamlit UI to use them

TensorFlow Serving + Streamlit! ✨ 🖼️ Serve TensorFlow ML models with TF-Serving and then create a Streamlit UI to use them! This is a pretty simple S

Álvaro Bartolomé 18 Jan 07, 2023
Picasso: a methods for embedding points in 2D in a way that respects distances while fitting a user-specified shape.

Picasso Code to generate Picasso embeddings of any input matrix. Picasso maps the points of an input matrix to user-defined, n-dimensional shape coord

Pachter Lab 45 Dec 23, 2022
The ICS Chat System project for NYU Shanghai Fall 2021

ICS_Chat_System [Catenger] This is the ICS Chat System project for NYU Shanghai Fall 2021 Creators: Shavarsh Melikyan, Skyler Chen and Arghya Sarkar,

1 Dec 20, 2021
A Closer Look at Structured Pruning for Neural Network Compression

A Closer Look at Structured Pruning for Neural Network Compression Code used to reproduce experiments in https://arxiv.org/abs/1810.04622. To prune, w

Bayesian and Neural Systems Group 140 Dec 05, 2022
HyperLib: Deep learning in the Hyperbolic space

HyperLib: Deep learning in the Hyperbolic space Background This library implements common Neural Network components in the hypberbolic space (using th

105 Dec 25, 2022
Implementation of "StrengthNet: Deep Learning-based Emotion Strength Assessment for Emotional Speech Synthesis"

StrengthNet Implementation of "StrengthNet: Deep Learning-based Emotion Strength Assessment for Emotional Speech Synthesis" https://arxiv.org/abs/2110

RuiLiu 65 Dec 20, 2022
Implementation of "With a Little Help from my Temporal Context: Multimodal Egocentric Action Recognition, BMVC, 2021" in PyTorch

Multimodal Temporal Context Network (MTCN) This repository implements the model proposed in the paper: Evangelos Kazakos, Jaesung Huh, Arsha Nagrani,

Evangelos Kazakos 13 Nov 24, 2022
Exploit Camera Raw Data for Video Super-Resolution via Hidden Markov Model Inference

RawVSR This repo contains the official codes for our paper: Exploit Camera Raw Data for Video Super-Resolution via Hidden Markov Model Inference Xiaoh

Xiaohong Liu 23 Oct 08, 2022
3D Multi-Person Pose Estimation by Integrating Top-Down and Bottom-Up Networks

3D Multi-Person Pose Estimation by Integrating Top-Down and Bottom-Up Networks Introduction This repository contains the code and models for the follo

124 Jan 06, 2023
Text-to-Music Retrieval using Pre-defined/Data-driven Emotion Embeddings

Text2Music Emotion Embedding Text-to-Music Retrieval using Pre-defined/Data-driven Emotion Embeddings Reference Emotion Embedding Spaces for Matching

Minz Won 50 Dec 05, 2022
Invasive Plant Species Identification

Invasive_Plant_Species_Identification Used LiDAR Odometry and Mapping (LOAM) to create a 3D point cloud map which can be used to identify invasive pla

2 May 12, 2022
Code repository for "Free View Synthesis", ECCV 2020.

Free View Synthesis Code repository for "Free View Synthesis", ECCV 2020. Setup Install the following Python packages in your Python environment - num

Intelligent Systems Lab Org 253 Dec 07, 2022
Multi-View Consistent Generative Adversarial Networks for 3D-aware Image Synthesis (CVPR2022)

Multi-View Consistent Generative Adversarial Networks for 3D-aware Image Synthesis Multi-View Consistent Generative Adversarial Networks for 3D-aware

Xuanmeng Zhang 78 Dec 10, 2022
Based on the given clinical dataset, Predict whether the patient having Heart Disease or Not having Heart Disease

Heart_Disease_Classification Based on the given clinical dataset, Predict whether the patient having Heart Disease or Not having Heart Disease Dataset

Ashish 1 Jan 30, 2022
This is the repository of our article published on MDPI Entropy "Feature Selection for Recommender Systems with Quantum Computing".

Collaborative-driven Quantum Feature Selection This repository was developed by Riccardo Nembrini, PhD student at Politecnico di Milano. See the websi

Quantum Computing Lab @ Politecnico di Milano 10 Apr 21, 2022
Weighted QMIX: Expanding Monotonic Value Function Factorisation

This repo contains the cleaned-up code that was used in "Weighted QMIX: Expanding Monotonic Value Function Factorisation"

whirl 82 Dec 29, 2022
Trajectory Variational Autoencder baseline for Multi-Agent Behavior challenge 2022

MABe_2022_TVAE: a Trajectory Variational Autoencoder baseline for the 2022 Multi-Agent Behavior challenge This repository contains jupyter notebooks t

Andrew Ulmer 15 Nov 08, 2022