shade.js:

Adaptive Material Descriptions

Kristian Sons, Felix Klein, Jan Sutter, Philipp Slusallek

```javascript function shade(env) { var smod = (env.texcoord.x() * env.frequency) % 1.0; var tmod = (env.texcoord.y() * env.frequency) % 1.0; var blackColor = env.blackColor || new Vec3(0); var whiteColor = env.whiteColor || new Vec3(1); var color = ((smod < 0.5 && tmod < 0.5) || (smod >= 0.5 && tmod >= 0.5)) ? whiteColor : blackColor; return Shade() .diffuse(color, env.normal); .phong(color, env.normal, env.shininess) } ```

Tutorial Agenda

  1. Welcome and introduction (5min)
  2. shade.js: motivation, idea, approach (20min)
  3. shade.js: Hands-on (60min)
  4. Future Work and Discussion (5min)

Tutorial Resources

Motivation

General Material Description

* Sharable and portable between different platforms * Assignable to different 3D assets

General & Portable Materials

  • Material: Visual concept
  • Conceptually equal appearances on all platforms
  • Works for multiple execution environments
  • Can be shared and reused across platforms

General & Portable Materials

  • Material: Visual concept
  • Conceptually equal appearances on all platforms
  • Works for multiple execution environments
  • Can be shared and reused across platforms

General & Portable Materials

  • Material: Visual concept
  • Conceptually equal appearances on all platforms
  • Works for multiple execution environments
  • Can be shared and reused across platforms

What do we need for such materials?

Self contained

and separated from other concerns

* All material related logic in a single place * Independent from the application * Material description only * Avoid mixing with other concerns (e.g. animation)

Polymorphism

* No explicit types and computation rates * Should work with multiple types * More like a template

Adaptation

```javascript kd = Color(0.8); if (exists(diffuseMap) && isTexture(diffuseMap) && exists(texcoords)) { kd = sample(diffuseMap, texcoords); } else if (exists(vertexColor)) { kd = vertexColor; } else if (exists(uniformColor)) { kd = uniformColor; } ```

* Need to adapt the control flow * Specialized shader languages (GLSL, HLSL, OSL): * Can not query the execution environment * Dynamic branching is not desired

Adaptation

* Need to specialize materials * Shader Permutations * Based on adaptation logic (Ubershader) * Must not be 'baked' into application!

Games

* Control over assets and platform * Game engine specific shader is part of the asset * Specialization within platform * Use Shader Compositing or Preprocessor * No general or portable materials!

Shade Trees [Cook 1984]

* Artist involved * Connects execution environment with shader tree manually * Full control over asset * Typically no way to adapt control flow

Our Approach

Main ideas

1. Make adaptation logic part of the material description 2. Do not declare types or computation frequencies

shade.js: JavaScript based DSL

```javascript function shade(env) { var color = new Vec3(0.8); return color; } ```

* Subset of JavaScript * No explicit type declarations and computation rates * Immutable objects: Vec3, Vec4 ... * Subset: Only homogeneous arrays with fixed size, no prototyping, ... * Integrates well into WebGL frameworks (xml3d.js)

Introspect Execution Environment

```javascript function shade(env) { var color = new Vec3(0.8); // Set color depending on present parameters if (env.texcoords && env.diffuseMap instanceof Texture) color = diffuseMap.sample2D(env.texcoords); else if (env.vertexColor) color = env.vertexColor; else if (env.uniformColor) color = env.uniformColor; ... } ```

Introspect Execution Environment

```javascript ... if (this.fwidth) { // The execution environment supports derivatives var fw = this.fwidth(env.texcoord); ... } ... ```

Interface to Lighting System

```javascript function shade(env) { var color = new Vec3(1, 0.5, 0.25); return Shade() .diffuse(color, env.normal) .phong(color, env.normal, env.shininess); } ```

* Based on radiance closures * Similar to OSL * Predefined closures: * Oren-Nayar, Phong, Cook Torrance, Ward ...

Compiler: Specialization

* Inject platform-specific light integrator * Specializes material based on specific execution environment * Compute types: Type Inference * Eliminate non-applicable branches * Constant Propagation * Dead Code Elimination

Compiler: Mapping to Stages

* Optimizations: * Extract uniform expressions * Extract (linear) coordinate space transformations

Overview

  • New Material Description Language: shade.js
    • Adaptive: Supports introspection of the execution environment
    • General: Non-explicit types
    • Portable: Interfaces to scene and lighting
  • Compiler Framework
    • Specialization: Static Code Analysis
    • Optimization: Maps computations to different stages
    • Code Generation: GLSL, OSL
    • Written in JavaScript as well

Results

Performance

* Specialization * Same instruction count compared to preprocessor specialization * Despite the much higher level of abstraction * Uniform Extraction * Instruction Count: Up to 71% of instructions mapped to CPU * Desktop / Nvidia GeForce GTX 660 Ti: 1.3 - 1.6x * Laptop / Nvidia NVS 5400M: 1.3 - 1.9x * Tablet / Qualcomm Adreno 330: 1.3 - 2.4x

XML3D - Forward Shading

XML3D - Deferred Shading

Cycles - Ray tracing

Cycles - Global Illumination

All in one

Conclusion

* Adaptive and more general material description language * All material-related logic in one place * Sharable and portable to different platforms * Compiler to specialize material and map computations * Higher abstraction level without sacrificing performance * System fully integrated into the web (OSS)

Conclusion

* Limitations * Full specialization: Combinatorial increase in number of shaders * Future Work * Extend to procedural displacement, lights, etc. * Multi-pass concepts (e.g. post-processing)

Thanks!

Questions?

* Supported by * [Intel Visual Computing Institute](http://www.intel-vci.uni-saarland.de/) * [FI-CONTENT2](http://www.fi-ppp.eu/projects/fi-content-2/) (EU-FP7) * [FI-CORE](http://www.fi-ppp.eu/projects/fi-ware/) (EU-FP7) * [DREAMSPACE](http://www.dreamspaceproject.eu/) (EU-FP7) * [CIMPLEX](https://www.cimplex-project.eu/) (EU-H2020)