Potential Integration with Pyrender PBR

Hi @mikedh!

I’ve created a new repo for my changes to meshrender. For now, I called it pyrender – open to suggestions for a more catchy name 😃

Hoping to get a bit of feedback on what works and what you’d like to see change for it to integrate nicely with trimesh. Here are some points that I know will need work:

• For now, it seems to work well on Ubuntu! Performance can take a hit when shadowmapping for many lights – I can fix this with a deferred rendering pipeline, but that would have to come at a later date when I have some time to write one.
• The repo uses GLSL 3+ shaders, so you need to be able to get a core profile in Pyglet to use the viewer. This works without difficulty on Linux, but Pyglet’s code for MacOS is broken. I’ve submitted PRs to try to fix this, but they’re quite slow to respond and release new versions. I forked Pyglet and can point the setup.py to use my fork, but it’s not the prettiest solution.
• Shadowmapping still doesn’t work nicely on MacOS (something to do with retina displays, will need to debug).
• I still need to test things on Windows.

Best, Matt

[jackd]

Since 3.0 the trimesh viewer is visualizing objs very nicely. Unfortunately, I have two issues with it:

• lighting doesn’t seem to be customizable? Or maybe I’m misunderstanding… but when I add custom lights I get indistinguishable results whether the intensity is 10.0 or 0.0.
• offscreen rendering is broken (I keep getting saved screenshots of my desktop).

pyrender seems to resolve these, but it looks like the work that went into better obj loading/more generalized textures doesn’t carry across. I’m happy to try and hack something together with a bit of direction, but I’m really not a graphics guy…

Example below will download a shapenet category to tmp/shapenet_data/SYNSET_ID.zip - the hard-coded synset_id (for caps) is pretty small.

import os
import pyrender
import trimesh
import wget
import zipfile
import numpy as np


class ZipSubdirResolver(trimesh.visual.resolvers.Resolver):
    def __init__(self, archive, subdir):
        assert(hasattr(archive, 'read'))
        self.archive = archive
        self.subdir = subdir

    def get(self, name):
        name = name.lstrip()
        if name.startswith('./'):
            name = name[2:]
        with self.archive.open(os.path.join(self.subdir, name)) as fp:
            return fp.read()


def compare(zf, synset_id, model_id):
    resolution = (512, 512)
    camera_transform = np.array(
        [[0.58991882, -0.3372407,   0.73366511,  0.86073076],
         [0.04221561,  0.92024442,  0.38906047,  0.45644301],
         [-0.80635825, -0.19854197,  0.55710632,  0.65359322],
         [0.0,         0.0,         0.0,         1.0]]
    )
    fov = 45
    light_kwargs = dict(
        color=np.ones(3),
        intensity=10.0,
        innerConeAngle=np.pi/16.0)

    print('Visualizing %s / %s' % (synset_id, model_id))
    subdir = os.path.join(synset_id, model_id)
    obj_path = os.path.join(subdir, 'model.obj')
    print(subdir, obj_path)
    resolver = ZipSubdirResolver(zf, subdir)

    with zf.open(obj_path) as fp:
        mesh_or_scene = trimesh.load(fp, file_type='obj', resolver=resolver)
        if isinstance(mesh_or_scene, trimesh.Trimesh):
            meshes = [mesh_or_scene]
        else:
            assert(isinstance(mesh_or_scene, trimesh.Scene))
            meshes = mesh_or_scene.geometry.values()
    camera = trimesh.scene.cameras.Camera(resolution=resolution, fov=(fov,)*2)
    light = trimesh.scene.lighting.SpotLight(**light_kwargs)
    tm_scene = trimesh.scene.Scene(
        camera=camera,
        camera_transform=camera_transform,
        geometry=meshes,
        lights=[light])
    tm_scene.graph[light.name] = camera_transform

    pr_scene = pyrender.Scene.from_trimesh_scene(tm_scene)

    if isinstance(tm_scene, trimesh.Trimesh):
        pr_scene.add(pyrender.Mesh.from_trimesh(tm_scene))
        tm_scene = tm_scene.scene()
    else:
        for mesh in tm_scene.geometry.values():
            pr_scene.add(pyrender.Mesh.from_trimesh(mesh))

    dist = np.linalg.norm(camera_transform[:3, 3])

    camera = pyrender.PerspectiveCamera(
        yfov=np.pi / 180 * fov,
        znear=dist-0.5, zfar=dist+0.5
    )

    print('trimesh')
    tm_scene.show()

    camera_transform = np.array(tm_scene.camera_transform)
    pr_scene.add(camera, pose=camera_transform)
    light = pyrender.SpotLight(**light_kwargs)
    pr_scene.add(light, pose=camera_transform)
    print('pyrender')
    pyrender.Viewer(pr_scene, viewport_size=resolution)


DL_URL = (
    'http://shapenet.cs.stanford.edu/shapenet/obj-zip/ShapeNetCore.v1/'
    '{synset_id}.zip')

# synset_id = '02773838'  # suitcases
synset_id = '02954340'    # cap
dl_url = DL_URL.format(synset_id=synset_id)
dl_dir = os.path.join('/tmp', 'shapenet_data')
if not os.path.isdir(dl_dir):
    os.makedirs(dl_dir)
path = os.path.join(dl_dir, os.path.split(dl_url)[1])

if not os.path.isfile(path):
    wget.download(DL_URL.format(synset_id=synset_id), out=path)


with zipfile.ZipFile(path, 'r') as zf:
    model_ids = [
        n.split('/')[1] for n in zf.namelist() if n.endswith('model.obj')]
    for model_id in model_ids:
        compare(zf, synset_id, model_id)

[jjmontesl]

trimesh.Scene.show will add another viewer option, pyrender.

Was this finally implemented? Doesn’t seem to be available in my installation.