An executor that wraps 3D mesh models and encodes 3D content documents to d-dimension vector.

Related tags

Image Processingexecutor-3d-encoder
Overview

3D Mesh Encoder

An Executor that receives Documents containing point sets data in its blob attribute, with shape (N, 3) and encodes it to embeddings of shape (D,). Now, the following pretrained models are ready to be used to create embeddings:

  • PointConv-Shapenet-d512: A PointConv model resulted in 512 dimension of embeddings, which is finetuned based on ShapeNet dataset.
  • PointConv-Shapenet-d1024: A PointConv model resulted in 1024 dimension of embeddings, which is finetuned based on ShapeNet dataset.

Usage

via Docker image (recommended)

from jina import Flow

f = Flow().add(uses='jinahub+docker://3DMeshEncoder', \
               uses_with={'pretrained_model': 'PointConv-Shapenet-d512'})

via source code

from jina import Flow

f = Flow().add(uses='jinahub://3DMeshEncoder', \
               uses_with={'pretrained_model': 'PointConv-Shapenet-d512'})

This Executor offers a GPU tag to speed up encoding. For more information on how to run the executor on GPU, check out the documentation.

How to finetune pretrained-model?

install finetuner

$ pip install finetuner

prepare dataset

TBD...

finetuning model with labeled dataset

$ python finetune.py --help

$ python finetune.py --model_name pointconv \
    --train_dataset /path/to/train.bin \
    --eval_dataset /path/to/eval.bin \
    --batch_size 128 \
    --epochs 50

finetuning model with unlabeled dataset

$ python finetune.py --model_name pointconv \
    --train_dataset /path/to/unlabeled_data.bin \
    --interactive

References

  • PointNet: Deep Learning on Point Sets for 3D Classification and Segmentation
  • PointConv: Deep Convolutional Networks on 3D Point Clouds
Comments
  • which version should the `finetuner` module be?

    which version should the `finetuner` module be?

    I was preparing the environment of this repo. However, I found it confused that the python import code in finetune.py gives error message:

    from finetuner.tuner.callback import BestModelCheckpoint
from finetuner.tuner.pytorch.losses import TripletLoss
from finetuner.tuner.pytorch.miner import TripletEasyHardMiner

    It seems that there's no finetuner.tuner in finetuner module but finetuner.finetuner. Besides, callback/pytorch is not in finetuner.finetuner now.

    Now my finetuner module installed by running pip install -U finetuner is 0.5.0. Is there any solution? Maybe you can freeze the version in requirements.txt.

    opened by elenacliu 4
  • fix: fix previous structure

    fix: fix previous structure

    I got pretrain and finetune process mixed up, and previously I pretrained the models using finetune model structure, which might lead to worse results. The fix code is in process and may be finished in a few days.

    opened by elenacliu 2
  • Add dataset, preprocess and pointnet2

    Add dataset, preprocess and pointnet2

    Here is the modification of the structure. There are still some bugs in my pointnet2 codes. I will solve the possible problems after you comment.

    @numb3r3 @jemmyshin

    opened by elenacliu 1
  • In `farthest_point_sample` function, the first centroid sampled is not random.

    In `farthest_point_sample` function, the first centroid sampled is not random.

    I'm reading your codes.

    https://github.com/jina-ai/executor-3d-encoder/blob/main/executor/models/modules.py#L75

    The randint line is commented, but in the original implementation and the algorithm itself, it should be random? Maybe we can provide a fixed seed if we want to get the same result?

    opened by elenacliu 1
  • Provide diffusion encoder

    Provide diffusion encoder

    https://github.com/nmwsharp/diffusion-net

    I'm unfamiliar with the jina ecosystem. So this is my first issue.

    Final goal is to have dalle generate 3d digital human meshes like https://github.com/jina-ai/dalle-flow

    opened by fire 2
Releases(v0.3.0)
Owner
Jina AI
A Neural Search Company. We provide the cloud-native neural search solution powered by state-of-the-art AI technology.
Jina AI
GitHub Repository
Automatic picture transmission(APT) protocol decoder for NOAA weather satellites

Automatic picture transmission(APT) decoder Automatic picture transmission protocol is used by NOAA satellites. They constantly send a frequency modul

Jayachandra Kasarla 25 Aug 05, 2022
A python program to generate ANSI art from images and videos

ANSI Art Generator A python program that creates ASCII art (with true color support if enabled) from images and videos Dependencies The program runs u

Pratyush Kumar 12 Nov 08, 2022
GPU-accelerated image processing using cupy and CUDA

napari-cupy-image-processing GPU-accelerated image processing using cupy and CUDA This napari plugin was generated with Cookiecutter using with @napar

Robert Haase 16 Oct 26, 2022
An API that renders HTML/CSS content to PNG using Chromium

html_png An API that renders HTML/CSS content to PNG using Chromium Disclaimer I am not responsible if you happen to make your own instance of this AP

10 Aug 08, 2022
An API which would colorize a black and white image

Image Colorization API Machine Learning Model used- https://github.com/richzhang/colorization/tree/caffe Paper - https://arxiv.org/abs/1603.08511 Step

Neelesh Ranjan Jha 4 Nov 23, 2021
CropImage is a simple toolkit for image cropping, detecting and cropping main body from pictures.

CropImage is a simple toolkit for image cropping, detecting and cropping main body from pictures. Support face and saliency detection.

Haofan Wang 15 Dec 22, 2022
Kimimaro: Skeletonize Densely Labeled Images

Kimimaro: Skeletonize Densely Labeled Images # Produce SWC files from volumetric images. kimimaro forge labels.npy --progress # writes to ./kimimaro_o

92 Dec 17, 2022
Pyconvert is a python script that you can use to convert image files to another image format! (eg. PNG to ICO)

Pyconvert is a python script that you can use to convert image files to another image format! (eg. PNG to ICO)

1 Jan 16, 2022
This will help to read QR codes using Raspberry Pi and Pi Camera

Raspberry-Pi-Generate-and-Read-QR-code This will help to read QR codes using Raspberry Pi and Pi Camera Install the required libraries first in your T

Raspberry_Pi Pakistan 2 Nov 06, 2021
DP2 graph edit codes.

必要なソフト・パッケージ Python3 Numpy JSON Matplotlib 動作確認環境 MacBook Air M1 Python 3.8.2 (arm64) Numpy 1.22.0 Matplotlib 3.5.1 JSON 2.0.9 使い方 draw_time_histgram(

1 Feb 19, 2022
:rocket: A minimalist comic reader

Pynocchio A minimalist comic reader Features | Installation | Contributing | Credits This screenshots contains a page of the webcomic Pepper&Carrot by

Michell Stuttgart 73 Aug 02, 2022
Fast batch image resizer and rotator for JPEG and PNG images.

imgp is a command line image resizer and rotator for JPEG and PNG images.

Terminator X 921 Dec 25, 2022
SimpleITK is an image analysis toolkit with a large number of components supporting general filtering operations, image segmentation and registration

SimpleITK is an image analysis toolkit with a large number of components supporting general filtering operations, image segmentation and registration

672 Jan 05, 2023
This piece of code is a User Welcomer with Image Manipulation using Python and Pillow (PIL).

This piece of code is a User Welcomer with Image Manipulation using Python and Pillow (PIL).

Bero 4 Jan 11, 2022
Design custom QR codes with this web app!

My-QR.Art This web app lets users design their own QR codes to any domain. It can be acessed on my-qr.art. You can find some more background info abou

Marien Raat 406 Dec 20, 2022
Tweet2Image - Convert tweets to Instagram-friendly images.

Convert tweets to Instagram-friendly images. How to use If you want to use this repository as a submodule, don't forget to put the fonts d

Janu Lingeswaran 1 Mar 11, 2022
A tool for making simple-style text posters or wallpapers with high resolution.

PurePoster PurePoster is a fancy tool for making arbitrary-resolution, simple-style posters or wallpapers with text in center. Functionality PurePoste

Renyang Guan 4 Jul 09, 2022
Convert HDR photos taken by iPhone 12 (or later) to regular HDR images

heif-hdrgainmap-decode Convert HDR photos taken by iPhone 12 (or later) to regular HDR images. Installation First, make sure you have the following pa

Star Brilliant 5 Nov 13, 2022
Python framework for creating and scaling up production of vector graphics assets.

Board Game Factory Contributors are welcome here! See the end of readme. This is a vector-graphics framework intended for creating and scaling up prod

Adam Volný 5 Jul 13, 2022
SALaD (Semi-Automatic Landslide Detection) is a landslide mapping system

SALaD (Semi-Automatic Landslide Detection) is a landslide mapping system. SALaD utilizes Object-based Image Analysis and Random Forest to map landslides.

NASA 14 Jan 04, 2023