Rendering & Lighting

A Selection of Rendering Tasks

The following items on this page are a selection of rendering and lighting tasks I completed while working at Surgical Science / Mimic.

Click any thumbnail image to see the full-screen image or video.

Physically-based Rendering

I designed a physically-based lighting and shading model for our flagship simulator. The standard entity shader uses a metallic/roughness workflow. The diffuse shading model is based on Disney's diffuse model. The specular model is Trowbridge-Reitz (GGX) for the Normal Distribution Function (D), Smith’s height-correlated masking/shadowing function for the geometry term (G), and a modified version of Schlick's Fresnel (F). Indirect lighting uses pre-filtered mipmapped radiance environment maps (PMREM) and irradiance maps derived from HDR images. The following section of images shows some final results:

Video: simulated fatty tissue with a branching blood vessel and two robotic instruments
Simulated fatty tissue with a branching blood vessel, lit with wet specular highlights, and two robotic instruments
The same tissue scene with a robotic instrument grasping the blood vessel near yellow guide markers
Two robotic instruments holding the open ends of simulated bowel above coiled intestines
Curved suture needles with colored threads under a spotlight on a dark background
Pick-and-place exercise: two instruments above red, yellow and purple dishes and small jacks on a blue drape
Pick-and-place exercise in a simulated operating room, with instruments reaching into a spotlight on three colored dishes
Suturing exercise: robotic instruments driving a curved needle through a foam suture pad
Matching exercise: robotic instruments placing letter pieces on a wooden board

The next set of images demonstrate various components of the lighting model. All images were rendered in-engine. The Cerberus pistol asset is courtesy of Andrew Maximov:

Grid of green spheres varying in roughness and metallic in an outdoor environment
Row of metal spheres from smooth to rough on a desert road
Cerberus pistol lit by a desert environment
Cerberus pistol lit by a snowy outdoor environment
Cerberus pistol lit by an indoor environment
Blue tiled sphere on a pedestal with a glossy finish
Iron lion statue with a glossy finish
Wooden spheres on pedestals with a matte finish
Dark wooden spheres and lion statues with a matte finish
Spheres, shader balls and lion statues in worn yellow-and-black painted metal under a spotlight

Volume Rendering

The following video shows the integration of a volume renderer with a forward renderer in a prototyping environment. The second image is a screenshot of a volumetric texture synthesis tool I developed. The tool uses machine learning to procedurally generate a volume texture based on a given 2D example texture. Notice how the image pattern remains consistent no matter how you slice through the volume.

Video: volume rendering of the bones of a foot, standing on a blue drape in the simulator
Volume texture synthesis tool: a 2D example texture beside a cube cut from the synthesized volume

Rendering for Position Based Dynamics

I developed rendering support for a Position Based Dynamics (PBD) physics system. Three methods of visualizing the PBD objects included billboard rendering, screen-space particle surface rendering, and mesh generation. The following video demonstrates an early prototype of dissection using the billboard rendering method.

Video: two instruments dissecting a block of yellow fat drawn as billboard particles

Screen-Space Particle Fluid Rendering

The next video shows a physics-based particle system representing blood. A screen-space rendering technique is used to render the blood as a continuous surface. Notice how the surgical instrument can interact with and displace the blood, and the blood flows around the jaws of the instrument.

Video: particle blood rendered as a continuous surface, flowing along a vessel around an instrument

Segmentation-Mask Rendering for AI Image Recognition

I developed a segmentation-mask rendering mode in C++ and HLSL for synthetic training-data generation and image segmentation, producing the imagery used in the company's MICCAI conference presentation and opening a rendering path for its new medical-AI business. Individual frames were segmented, numbered, and streamed out with the standard rendered frame for correlation. The following images show two matched frames, one segmented and the other rendered as normal.

Segmentation mask: each instrument part and the suture needle drawn as a flat color on a dark background, with frame number 000000
The matching rendered frame: two needle drivers beside a suture block with a curved needle, with frame number 000000

Exponential Variance Shadow Mapping

For the shadows in the simulator, I implemented Exponential Variance Shadow Maps (EVSM). A few screenshots can be seen below. Also any video on this site showing the simulation can be used to view the EVSM shadows.

Hard-edged shadows of two instruments and a ring cast by a spotlight onto a pegboard
Soft, blurred shadows of two instruments cast onto a pegboard holding colored rings
Needle drivers passing a red ring over a pegboard, with soft EVSM shadows beneath them