This is a significant re-work of the pythreejs extension that introduces an "autogen" script that generates the majority of the ipython-widget code to wrap each of three.js's types. It also takes a different view towards the pythreejs API. Whereas pythreejs adds custom functionality to the classes, sometimes renaming, etc., this approach attempts to mimic the low-level three.js API as closely as possible, opening up the possibility for others to build utility libraries on top of this.

This PR does not support all the functionality of the current pythreejs, but it is a significant step forward in terms of the potential to support the majority of three.js's features. Currently supported features:

• Basic building blocks of a scene:
  • Textures (both Image-based and Data-based)
  • Materials
  • Geometries
  • Meshs
  • Scene, Lights, Cameras, WebGLRenderer

The autogen script, generate-wrappers.js, takes advantage of a config file three-class-config.js to auto-generate both javascript and python files to define the ipywidget wrappers for each three.js class. The generated javascript files will have .autogen.js as the extension. The generated python files have _autogen.py as their extension. The script uses the handlebars template system to generate the various code files.

The autogen solution allows for overriding the default behavior of the generated classes. E.g., if Object3D.js is present, then it will be loaded into the namespace as opposed to loading Object3D.autogen.js. It is up to the author of the override classe to decide whether to inherit behavior from the autogen class or not. Same goes for the python modules. This allows for writing custom methods on both the python and javascript side when needed.

The autogen script relies on a json-like config file (three-class-config.js) to describe the classes. Reasonable defaults should take care of most, but it allows specifying the base class, constructor args, etc. for each of the wrappers. A base version of this file can be generated by generate-class-config.js, but beware, it overwrites any customization to the config file that has already been done.

List of relevant files:

js/package.json
js/webpack.config.js

# Files for auto-generation
js/scripts/clean-generated-files.js
js/scripts/generate-class-config.js
js/scripts/generate-wrappers.js
js/scripts/prop-types.js
js/scripts/templates/js_index.mustache
js/scripts/templates/js_param_obj.mustache
js/scripts/templates/js_wrapper.mustache
js/scripts/templates/py_top_level_init.mustache
js/scripts/templates/py_wrapper.mustache
js/scripts/three-class-config-defaults.js
js/scripts/three-class-config.js
js/scripts/three-class-config.js.backup

# New bases classes for supporting wrapper classes
js/src/_base/RendererPool.js
js/src/_base/Three.js
js/src/_base/enums.js
pythreejs/_base/Three.py
pythreejs/enums.py
pythreejs/traits.py

# Overridden classes
js/src/core/Object3D.js
js/src/textures/DataTexture.js
js/src/textures/ImageTexture.js # custom classed created to ease image texture loading
js/src/renderers/WebGLRenderer.js
pythreejs/core/Object3D.py
pythreejs/renderers/WebGLRenderer.py

# Updated examples notesbooks
examples/Examples.ipynb
examples/Geometries.ipynb
examples/Textures.ipynb
examples/renderer_limit.ipynb
examples/test.ipynb
examples/img/checkerboard.png
examples/img/earth.jpg

# Pulled in latest version of examples files from three.js source
js/src/examples/Detector.js
js/src/examples/controls/MomentumCameraControls.js
js/src/examples/controls/OrbitControls.js
js/src/examples/controls/TrackballControls.js
js/src/examples/js/controls/OrbitControls.js
js/src/examples/renderers/CanvasRenderer.js
js/src/examples/renderers/Projector.js

This PR will track the TODO's necessary to convert the autogen branch to master. Previous PR with discussion @jasongrout @SylvainCorlay @martinal @vidartf

• [x] Cleanup commented code
• [x] Get repo updated with latest ipywidgets changes
• [ ] Audit for critical unimplemented features
• [ ] (other todo's)

Here is the link:
https://colab.research.google.com/drive/1ZdOYamFGclGdPDbIkXExF8KRYQ6dAdGu

When running, cubeGeometry = BufferGeometry(...)

it results in "runtime disconnect."

Any idea why or how to fix it?

Since you mention it in the installation instructions, I thought I would let you know

conda install -c conda-forge pythreejs

installs version 0.4.1, not the current 1.0.0. Someone needs to update the condo-forge package.

I'm getting the following trace in the Chrome console under JLab 0.33.4 + ipywidgets 7.3.1 when trying to run pythreejs examples (e.g. Animation: sphere, cube, etc.). I guess widget declaration somehow has changed in this version of JLab.

default.js:928` Exception opening new comm
(anonymous) @ default.js:928
s @ default.js:7
Promise.then (async)
l @ default.js:8
(anonymous) @ default.js:9
YC29.n @ default.js:5
_handleCommOpen @ default.js:913
(anonymous) @ default.js:1020
(anonymous) @ default.js:9
YC29.n @ default.js:5
_handleMessage @ default.js:977
_msgChain._msgChain.then @ default.js:124
Promise.then (async)
_onWSMessage.e @ default.js:121
default.js:129 Error: Object 'jupyter.widget' not found in registry
    at M.Promise (default.js:1476)
    at new Promise (<anonymous>)
    at Object.M [as loadObject] (default.js:1455)
    at y.<anonymous> (default.js:921)
    at Generator.next (<anonymous>)
    at default.js:9
    at new Promise (<anonymous>)
    at YC29.n (default.js:5)
    at y._handleCommOpen (default.js:913)
    at y.<anonymous> (default.js:1020)
    at Generator.next (<anonymous>)
    at default.js:9
    at new Promise (<anonymous>)
    at YC29.n (default.js:5)
    at y._handleMessage (default.js:977)
    at _msgChain._msgChain.then (default.js:124)
_msgChain._msgChain.then.catch.e @ default.js:129
Promise.catch (async)
_onWSMessage.e @ default.js:126
default.js:928 Exception opening new comm
(anonymous) @ default.js:928
s @ default.js:7
Promise.then (async)
l @ default.js:8
(anonymous) @ default.js:9
YC29.n @ default.js:5
_handleCommOpen @ default.js:913
(anonymous) @ default.js:1020
(anonymous) @ default.js:9
YC29.n @ default.js:5
_handleMessage @ default.js:977
_msgChain._msgChain.then @ default.js:124
Promise.then (async)
_onWSMessage.e @ default.js:121
default.js:129 Error: Object 'jupyter.widget' not found in registry
    at M.Promise (default.js:1476)
    at new Promise (<anonymous>)
    at Object.M [as loadObject] (default.js:1455)
    at y.<anonymous> (default.js:921)
    at Generator.next (<anonymous>)
    at default.js:9
    at new Promise (<anonymous>)
    at YC29.n (default.js:5)
    at y._handleCommOpen (default.js:913)
    at y.<anonymous> (default.js:1020)
    at Generator.next (<anonymous>)
    at default.js:9
    at new Promise (<anonymous>)
    at YC29.n (default.js:5)
    at y._handleMessage (default.js:977)
    at _msgChain._msgChain.then (default.js:124)
_msgChain._msgChain.then.catch.e @ default.js:129
Promise.catch (async)
_onWSMessage.e @ default.js:126
default.js:928 Exception opening new comm
(anonymous) @ default.js:928
s @ default.js:7
Promise.then (async)
l @ default.js:8
(anonymous) @ default.js:9
YC29.n @ default.js:5
_handleCommOpen @ default.js:913
(anonymous) @ default.js:1020
(anonymous) @ default.js:9
YC29.n @ default.js:5
_handleMessage @ default.js:977
_msgChain._msgChain.then @ default.js:124
Promise.then (async)
_onWSMessage.e @ default.js:121
default.js:129 Error: Object 'jupyter.widget' not found in registry
    at M.Promise (default.js:1476)
    at new Promise (<anonymous>)
    at Object.M [as loadObject] (default.js:1455)
    at y.<anonymous> (default.js:921)
    at Generator.next (<anonymous>)
    at default.js:9
    at new Promise (<anonymous>)
    at YC29.n (default.js:5)
    at y._handleCommOpen (default.js:913)
    at y.<anonymous> (default.js:1020)
    at Generator.next (<anonymous>)
    at default.js:9
    at new Promise (<anonymous>)
    at YC29.n (default.js:5)
    at y._handleMessage (default.js:977)
    at _msgChain._msgChain.then (default.js:124)

Is there a way for producing a bitmap image of the rendnering ? Looking at some threejs examples I found that from threejs it is in principle possible to produce a png output, but if I am not mistaken pythreejs is not yet setup to do so.

I don't know if this is the right place to ask such questions, but I couldn't really find a Q/A space or forum for pythreejs.

Could anyone point me to how one can achieve geometry instancing with pythreejs?

I would basically want to reproduce the example described here using pythreejs.

Or to put it more simply, I would like to have 10,000 boxes, each with a different color and orientation, with maximum performance by minimizing the number of draw calls. I don't even need any lighting, boxes can just be made of BasicMaterial.

Any help would be greatly appreciated!

According to the docs the default background for a Scene should be null. However, when I explicitly set it to None, it translates to sth non-null on the frontend:

Using pipenv I roughly have installed jupyter and pythreejs using python 3.7. A snippet of the versioning:

pythreejs==2.2.0

• ipydatawidgets [required: >=1.1.1, installed: 4.0.1]
  • ipywidgets [required: >=7.0.0, installed: 7.5.1]
    • ipykernel [required: >=4.5.1, installed: 5.3.4]
      • ipython [required: >=5.0.0, installed: 7.18.1]

Widgets seem to work fine, but I have nothing rendered when I run the simple sphere example, just an empty output cell.

Checking the javascript console there is only a few warnings about threejs not being able to invert singular matrices. There is a healthy print that it rendered: THREE.WebGLRenderer 97

jupyter nbextension list shows everything enabled:

Known nbextensions: config dir: /home/bingjeff/.local/share/virtualenvs/primitive_hulls-T7x0kADv/etc/jupyter/nbconfig notebook section jupyter-datawidgets/extension enabled - Validating: OK jupyter-threejs/extension enabled - Validating: OK jupyter-js-widgets/extension enabled - Validating: OK

Should be reproducible with this repo: https://github.com/bingjeff/primitive_hulls

Anything else I should be doing?

I've been experimenting with the use of pythreejs to render 3D triangle-mesh models of brains in jupyter; everything works fine for low-resolution models (tested up to about 60k vertices), but for larger models, the display is empty when rendered. I've put the models I've been using for tests on filedropper in hdf5 format; I can reproduce the problem by doing the following:

1. Download the files to ~/Desktop/, then in bash: tar zxf brains.tar.gz (creates data in directory ~/Desktop/brains/)
2. In a running Jupyer notebook, import the brain mesh data:

   import h5py, numpy as np
   brains = {}
   for resolution in [32, 164]:
       data = {}
       with h5py.File('/Users/nben/Desktop/brains/brain%dk.hdf5' % resolution) as f:
           data['vertices'] = np.array(f['vertices'])
           data['faces'] = np.array(f['faces'])
           data['colors'] = np.array(f['colors'])
       brains[resolution] = data
   

3. Next, make a function to plot either mesh:

   def plot_brain_3D(data):
       from matplotlib.colors import to_hex
       import numpy as np
       import pythreejs as p3js
       vertices = data['vertices'].tolist()
       faces    = data['faces'].tolist()
       colors   = [to_hex(c) for c in data['colors']]
   
       faces = [f + [None, [colors[i] for i in f]] for f in faces]
   
       geo = p3js.Geometry(vertices=vertices, faces=faces, colors=colors)
       mesh = p3js.Mesh(
          geometry=geo,
          material=p3js.MeshLambertMaterial(vertexColors='VertexColors',
                                            side='DoubleSide'))
       cam = p3js.PerspectiveCamera(position=[1000,0,0], up=[0,0,1], fov=12)
       cam.lookAt(np.mean(vertices, axis=0).tolist())
       scene = p3js.Scene(
           children=([mesh, cam, p3js.AmbientLight(color='white', intensity=0.8)] +
                     [p3js.DirectionalLight(color='white', position=pos, intensity=0.6)
                      for pos in [(100,100, 100),(-100,100, 100),(100,-100, 100),(-100,-100, 100),
                                  (100,100,-100),(-100,100,-100),(100,-100,-100),(-100,-100,-100)]]),
           background='white')
       renderer = p3js.Renderer(camera=cam, scene=scene,
                                controls=[p3js.OrbitControls(controlling=cam)],
                                width=6*72, height=6*72)
       return renderer
   

4. Next, plot the 32k mesh (this works fine for me):

   from IPython.display import display
   display(plot_brain_3D(brains[32]))
   

   

5. Finally, plot the 164k mesh (this just displays as if the mesh didn't exist):

   from IPython.display import display
   display(plot_brain_3D(brains[164]))
   

   

I've also tested an intermediate-resolution mesh (~59k vertices) that displays fine.

I initially thought that this was a data-transfer issue, but I have set the following line in my ~/.jupyter/jupyter_notebook_config.py file: c.NotebookApp.iopub_data_rate_limit = 500000000.

The meshes should only substantively differ in that the latter mesh is a higher-resolution version of the former--Am I doing something wrong here, or are there fundamental limits to the complexity of what Jupyter can display using pythreejs?

(Apologies if this is user-error; I tried stackexchange first, but even with a bounty got no answers, and haven't been able to find any documentation about this.)

From the comments near the imports at the top of pythree.js, it seems that pythreejs should work under IPython 3.x. However, it tries to import widget_serialization, which appears to only exist in the new Jupyter notebook.

Is this bug fixable, or do you intend to have the development version of the Jupyter notebook as a minimum dependency?

Doing a right-click on a scene brings up the right-click Jupyter menu, while performing a translation operation on the camera.

As a comparison, doing a right-click on a Matplotlib figure using the ipympl backend does not bring up the menu. It would be nice if the right-click also did not show the Jupyter menu with pythreejs.

Thanks for any help!

Versions:

- pythreejs                 2.4.1              pyhc268e32_0    conda-forge
- python                    3.9.13          h2660328_0_cpython    conda-forge
- ipympl                    0.9.2              pyhd8ed1ab_0    conda-forge
- jupyter_client            7.4.5              pyhd8ed1ab_0    conda-forge
- jupyter_core              5.0.0            py39hf3d152e_0    conda-forge
- jupyter_server            1.23.2             pyhd8ed1ab_0    conda-forge
- jupyterlab                3.5.0              pyhd8ed1ab_0    conda-forge
- jupyterlab_pygments       0.2.2              pyhd8ed1ab_0    conda-forge
- jupyterlab_server         2.16.3             pyhd8ed1ab_0    conda-forge
- jupyterlab_widgets        3.0.3              pyhd8ed1ab_0    conda-forge

Hi,

I'm trying to add the inverse bind matrices to the skeleton but I am having the following problems (btw, I'm not an expect in JS. I'm trying to build some interactive 3D examples in Jupyter notebooks that use animations and this is the last step. Loading directly from GLTF sadly is not an option):

The Skeleton Model doesn't have the option to add it to it. I tried modifying the three-class-config to the following (adding boneInverses: new Types.ArrayBuffer('float32', [null, 16], {nullable: true}),):

Skeleton: {
        relativePath: './objects/Skeleton',
        properties: {
            bones: new Types.ThreeTypeArray('Bone'),
            boneInverses: new Types.ArrayBuffer('float32', [null, 16], {nullable: true}),
        },
        constructorArgs: ['bones', 'boneInverses'],
    },

1. When creating an instance, I get a flat array instead of a 2D array (i.e. if I send 3 matrices4x4 I would expect a dimension of [3,16] but I get [48] which causes problems in the init function of the Skeleton. a. Any way to force the dimensions to be respected as stated in the three-class-config? b. Would it be ok to create a Skeleton.js next to Skeleton_autogen.js and extend constructThreeObject ? So I make sure to send the correct 2D array? Would this cause any issues?

2. Even getting around problem 1 (which I tried and seemed to be working), I get an issue when I try to bind the Skeleton to the SkinnedMeshModel here:

    assignSkeleton(obj, key, value) {
        if (value) {
            obj.bind(value);
            obj.scale.multiplyScalar(1);
        }
    }

Calling the bind function for the SkinnedMesh expects the skeleton and bindMatrix but only the skeleton gets send:

bind( skeleton, bindMatrix ) {

		this.skeleton = skeleton;

		if ( bindMatrix === undefined ) {
			this.updateMatrixWorld( true );

			this.skeleton.calculateInverses();

			bindMatrix = this.matrixWorld;

		}

As it expects bindMatrix to be defined and it is not, it will override the inverse bind matrices by calling this.skeleton.calculateInverses() function

I haven't been able to get around this issue, as I don't want to break the current behavior. I've thought of maybe setting a boolean before assigning the skeleton using a different function but I don't really understand how the calls are made to JS, because a function I tried to set always got called after the assignment.

When trying to render anything in a new conda env with version 2.4.1, I get this Javascript error icon

Code to reproduce:

import pythreejs as p3
p3.BoxGeometry(
    width=5,
    height=10,
    depth=15,
    widthSegments=5,
    heightSegments=10,
    depthSegments=15)

The more detailed error after clicking:

[Open Browser Console for more detailed log - Double click to close this message]
Failed to load model class 'PreviewModel' from module 'jupyter-threejs'
Error: No version of module jupyter-threejs is registered
    at f.loadClass (http://localhost:8888/lab/extensions/@jupyter-widgets/jupyterlab-manager/static/134.083e6b37f2f7b2f04b5e.js?v=083e6b37f2f7b2f04b5e:1:74855)
    at f.loadModelClass (http://localhost:8888/lab/extensions/@jupyter-widgets/jupyterlab-manager/static/150.467514c324d2bcc23502.js?v=467514c324d2bcc23502:1:10721)
    at f._make_model (http://localhost:8888/lab/extensions/@jupyter-widgets/jupyterlab-manager/static/150.467514c324d2bcc23502.js?v=467514c324d2bcc23502:1:7517)
    at f.new_model (http://localhost:8888/lab/extensions/@jupyter-widgets/jupyterlab-manager/static/150.467514c324d2bcc23502.js?v=467514c324d2bcc23502:1:5137)
    at f.handle_comm_open (http://localhost:8888/lab/extensions/@jupyter-widgets/jupyterlab-manager/static/150.467514c324d2bcc23502.js?v=467514c324d2bcc23502:1:3894)
    at _handleCommOpen (http://localhost:8888/lab/extensions/@jupyter-widgets/jupyterlab-manager/static/134.083e6b37f2f7b2f04b5e.js?v=083e6b37f2f7b2f04b5e:1:73392)
    at b._handleCommOpen (http://localhost:8888/static/lab/jlab_core.9193dfb13484acaca919.js?v=9193dfb13484acaca919:2:994423)
    at async b._handleMessage (http://localhost:8888/static/lab/jlab_core.9193dfb13484acaca919.js?v=9193dfb13484acaca919:2:996413)

Versions:

ipydatawidgets            4.3.2              pyhc268e32_0    conda-forge
ipykernel                 6.15.1             pyh210e3f2_0    conda-forge
ipympl                    0.9.2              pyhd8ed1ab_0    conda-forge
ipython                   8.4.0              pyh41d4057_1    conda-forge
ipython_genutils          0.2.0                      py_1    conda-forge
ipywidgets                8.0.1              pyhd8ed1ab_0    conda-forge
jupyter_client            7.3.5              pyhd8ed1ab_0    conda-forge
jupyter_core              4.11.1           py39hf3d152e_0    conda-forge
jupyter_server            1.18.1             pyhd8ed1ab_0    conda-forge
jupyterlab                3.4.5              pyhd8ed1ab_0    conda-forge
jupyterlab_pygments       0.2.2              pyhd8ed1ab_0    conda-forge
jupyterlab_server         2.15.1             pyhd8ed1ab_0    conda-forge
jupyterlab_widgets        3.0.2              pyhd8ed1ab_0    conda-forge
pythreejs                 2.4.1              pyhc268e32_0    conda-forge

Hi, I am trying to use pythreejs on Google Colab, and I am seeing a failure related to DirectionalLight.

A minimal notebook to reproduce has the following content:

# Cell 1: installation
!pip install pythreejs

# Cell 2: allow custom widgets
from google.colab import output
output.enable_custom_widget_manager()

# Cell 3: import pythreejs
from pythreejs import *

# Cell 4: create a simple widget
BoxGeometry(
    width=5,
    height=10,
    depth=15,
    widthSegments=5,
    heightSegments=10,
    depthSegments=15)

# Cell 5: create a light object
DirectionalLight(color='white', position=[3, 5, 1], intensity=0.5)

Cell 4 runs correctly, the box shows up and the user can interact with it. Cell 5 runs, a black background gets rendered, but immediately below the black widget there are several instances of

---------------------------------------------------------------------------
TraitError                                Traceback (most recent call last)
[/usr/local/lib/python3.7/dist-packages/ipywidgets/widgets/widget.py](https://localhost:8080/#) in _handle_msg(self, msg)
    754                 if 'buffer_paths' in data:
    755                     _put_buffers(state, data['buffer_paths'], msg['buffers'])
--> 756                 self.set_state(state)
    757 
    758         # Handle a state request.

5 frames
[/usr/local/lib/python3.7/dist-packages/ipywidgets/widgets/widget.py](https://localhost:8080/#) in set_state(self, sync_data)
    623                     from_json = self.trait_metadata(name, 'from_json',
    624                                                     self._trait_from_json)
--> 625                     self.set_trait(name, from_json(sync_data[name], self))
    626 
    627     def send(self, content, buffers=None):

[/usr/local/lib/python3.7/dist-packages/traitlets/traitlets.py](https://localhost:8080/#) in set_trait(self, name, value)
   1436                                 (cls.__name__, name))
   1437         else:
-> 1438             getattr(cls, name).set(self, value)
   1439 
   1440     @classmethod

[/usr/local/lib/python3.7/dist-packages/traitlets/traitlets.py](https://localhost:8080/#) in set(self, obj, value)
    578 
    579     def set(self, obj, value):
--> 580         new_value = self._validate(obj, value)
    581         try:
    582             old_value = obj._trait_values[self.name]

[/usr/local/lib/python3.7/dist-packages/traitlets/traitlets.py](https://localhost:8080/#) in _validate(self, obj, value)
    610             return value
    611         if hasattr(self, 'validate'):
--> 612             value = self.validate(obj, value)
    613         if obj._cross_validation_lock is False:
    614             value = self._cross_validate(obj, value)

[/usr/local/lib/python3.7/dist-packages/traitlets/traitlets.py](https://localhost:8080/#) in validate(self, obj, value)
   1985                 except TraitError:
   1986                     continue
-> 1987         self.error(obj, value)
   1988 
   1989     def __or__(self, other):

[/usr/local/lib/python3.7/dist-packages/traitlets/traitlets.py](https://localhost:8080/#) in error(self, obj, value, error, info)
    690                     e = "The '%s' trait expected %s, not %s." % (
    691                         self.name, self.info(), describe("the", value))
--> 692                 raise TraitError(e)
    693 
    694     def get_metadata(self, key, default=None):

TraitError: The 'target' trait of a DirectionalLight instance expected an Uninitialized or an Object3D, not the str 'IPY_MODEL_[object Object]'.

I suspect that some of pythreejs dependencies already pre-installed on Colab are quite old, and by running pip install --upgrade on the right dependency the error would disappear. Unfortunately, I can't pinpoint which dependency I should upgrade. Can you help me with that?

The only dependency I am confident one must not upgrade is ipywidgets: it seems that doing so would make Cell 2 ineffective, and Cell 2 is fundamental for the widget in Cell 4 to correctly show up.

The same notebook, and much more complicated ones, run correctly on my local jupyter lab installation.

Does the memory of this part of BufferGeometry automatically reclaim? Or do I need a manual recycling method? I called this in a function, but left the function to show that memory kept growing. like that: