Managing 3D Assets with Model I/O

NICK PORCINO: Hello, everybody. Welcome to managing 3D Assets with Model I/O. I'm Nick Porcino in the image, media, and graphics group.

Today, I'm really excited to introduce the new Model I/O framework to you. We are raising the bar on -- or actually we are enabling you to raise the bar on interactive and realistic graphics. Up until now, the graphics frameworks and hardware that you've been dealing with come from an age when hardware was a lot more limited than it is now. The thing you have in your pocket is absolutely amazing.

Now we got new frameworks that enable you to get the power out to your users, but you have to feed that framework with really good looking stuff. So in the new world of really high performance and low overhead graphics APIs, you need a way to get things that are realistic, realistic means physically derived and physically based things. And so there's a lot of research and knowledge that you need in order to pull that off. You need to understand the physics of cameras, the physics of materials, the physics of light and you can certainly go out and read all of that stuff and I encourage you to do so. And implementing each one of those things requires a fair amount of heavy work, a fair amount of reading and then a whole lot of integration.

So what we have done with Model I/O is we've gone out, we have done a whole bunch of research, we have done a whole bunch of integration, and we've prepared a unified set of data structures to describe these things in a consistent way and an easy to use API.

So as it says, Model I/O is a framework for handling 3D assets and data.

So at its most basic level, you can use Model I/O to bring in common file formats and export them.

You can describe in a physically realistic way lighting, materials, environments.

You can get assets and art work from your artists into Model I/O, do some interesting processes which we'll get into as the talk progresses. And there's a lot of tools now that you can get that are focused on physically-based rendering and materials. And Model I/O gives you access to those things in your own pipelines. Model I/O is integrated into Xcode 7, and the GameKit APIs, it's on iOS 9, and OS 10.11.

So in a nut shell, the big green box there is where Model I/O fits in your framework or in your application. We start in the contact creation tool. Your artist does some great work and you import it into Model I/O. Model I/O then does the annoying and tedious and error-prone step of creating buffers for various frameworks to render quickly. So those frameworks that we're supporting out of the box are SceneKit, Metal, and OpenGL. Now Model I/O doesn't just load files and save them.

It also lets you perform useful operations that are time consuming or whatever, that serve to improve the look of your assets, and you can take an asset. You can do one of these baking operations which we'll get into, bring it back into the unified representation and Model I/O and get it off to your hardware ready buffers. Also, you can just complete the loop here, export your asset, out of Model I/O after you did something exotic that Model I/O provides for you, send it all the way back to the content creation tool, let the artist noodle around on it and perhaps add some, you know, special sweetness or whatever, and then you can just keep iterating the cycle until you achieve the look and level of quality that you need for your app.

So what we're going to talk about today is what are the features? What are the data types? Where does the physical motivation for those data types come from? And we're going to talk about support for various geometrical features and voxels and some advanced lighting and our baking steps.

All right. So here we go, bread and butter, reading things in. So import formats, we start with some of the most commonly used formats.

Alembic is a very high performance format and it comes from the film industry and is heavily uses in games now as well. And it is the most modern of the formats listed up here. It includes information on animation and material bindings and all kinds of interesting things. Polygon is the standard polygon format that you are going to get out of a lot of academic research. Triangles are commonly originated and ingested by CAD and CAM applications and Wavefront .obj is universally available everywhere.

And for exporting at the moment, you can export out to your CAD/CAM stuff and . So importing is that easy. You get an MDLAsset, by initWithURL. And you can send it back out somewhere with exportAssetToURL.

So just a few words about physical realism without getting super detailed. Lights, historically have had like a position, and, you know, a cone angle and falloff and some other physically unrealistic parameters that just are mathematically united to the way hardware and graphics APIs used to be. What we are providing here is access to IES profiles. Now, if you go to the hardware store and find, like, a light fixture that you really like. Like, say I really like that light up there and I found it in the store. I can go to the manufacturers website after reading the label on the side and I can find an IES profile file which is a data file, where they have done measurements all the way around the light to get the irradiance from every different angle.

We read that in into a so-called light web, which is a set of data that's pretty easy to load up into a shader. So if you want, you can have physically motivated lights that match real world lights in your shaders. Now, when you buy a light at the hardware store, the light isn't specified as an RGB value or whatever. It's specified as a temperature like, you know 4,000K or whatever. So you can specify these heights in degrees Kelvin as well. We also provide image based lighting which is in play on the excavator there. The materials, historically, everything has been Lambertian, which means fall off according to angle, and with the Blinn-Phong specular which means a plastically shiny highlight. We also provide you with a baseline physical bidirectional reflection function, or a BRDF which is what you really need if you want real world materials. And once again, that excavator has got physical BDRF on it. Cameras, historically have been pinhole cameras. We are describing them all the way from the lens to the sensor and we are providing you some utilities for processing environments from photographs and procedural skies. Now, you are going to see tools in Xcode to do baking, and what I'm telling you here on this slide is that the tools that you have in Xcode to perform these operations are available to you in Model I/O through the framework. So if you want to make your own line -- your own offline pipeline to bake your own assets from beginning to end, in giant batches or on a farm, all of those things are available through the frameworks API.

We have introduced voxels. So you can take a big old mesh and turn it into giant cache of indexes that you can associate your own data with. More on that later.

Once again, it's very straightforward, create your voxels from an asset and find voxels in a particular region. You can do constructive solid geometry and you can turn the voxels back into a mesh using some type of smoothing algorithm. Some kind.

So system integration, Model I/O is integrated directly into SceneKit and it's utilized in MetalKit and GLKit.

Model I/O is used to do preview in the Finder and in Quick Look. So you can go find an Alembic .abc cache hit the space bar and it'll pop up in the Finder and you can tumble your asset and preview it without even opening up any type of auditioning environment. You can edit assets in the Xcode. The excavator here is loaded in Xcode in this picture. And Model I/O works in Playgrounds and with Swift. All right.

Down to the nitty-gritty of data types.

So MDLAsset is the thing that you get from a URL and it's the big overall container that you will often be working with. It's an index container, you know, for fast enumeration and stuff, it has transform hierarchies in it, meshes, cameras and lights and you can make them using the API or loading from a URL.

So a typical asset that you might pull out of like a big old Alembic file, is going to have a whole bunch of interesting things in it. This particular example here has got a couple of cameras and a light, a root transform, the blue box over on the right there. And underneath it, are the bits of the excavator and the orange boxes just indicate that, yes, you can indicate your material assignments and things like that as well. So that's what's in an asset. It's all the stuff that logically belongs together and you will get a bunch of assets and compose them into a scene. So an asset has allocators in case you need to manage memory yourself, you can add an allocator. It has descriptors and which teach about the things that are inside of the asset. There's the import and export facilities and a bunch of children. And the children are MDLObjects.

MDLObjects themselves can comprise a hierarchy. Now, a typical scene graph, of course, has transformation hierarchies. And so an MDLObject has got a transform component.

We are not implementing the transform in, you know, the standard way of putting a matrix everywhere. We actually have a transformation component and components are very interesting, because it allows us to make scene graphs that aren't limited just to the parent of or child of type relationships. Instead you can define your own components.

Now, I guess I should have mentioned that the nice thing is that the API is designed that you can write your own importers and exporters. You can write -- if you have your own custom file format, you can implement that and so back to this, I've got a custom component which is a TriggerComponent like a volume your character enters a region and some sort of action should occur. You can just make that yourself and define what the behavior is and what it connects to, the API lets you do that.

Now, a mesh contains one or more vertex buffers. That's positions and normals the thing that has to go off to the GPU for rasterization. And submeshers.

Submeshes, to get an idea of what exactly it is, you might have, like, a character who is going to drive the excavator and he might have some optional components like a hard hat, it shouldn't be optional, but it is. So in one index buffer, I might have the whole character without the hat, and in another index buffer, I might just have all indexes referring to the original mesh vertex buffers that have got his hat in it. So by rendering or not rendering that submesh, he does or does not have a hat.

Submeshes can share the data in the vertex buffers so this just allows you to have a single submission for the hardware. So the mesh, besides holding vertex and index buffers, also has utility functions, and there's generators to make all of your usual things, like boxes and spheres and what not. There's modifiers. So if a mesh that didn't have normals or tangent bases, or any of those things, you can generate those on demand, thinking back to that bake and export cycle that I showed earlier.

And there's, of course the bakers. Now, a mesh buffer is the thing that has to go to the hardware. It's got the data in it.

The actual data. How big the buffer is, how you allocated it. And you have to describe those vertex buffers. You have to say what the intention of the buffer is, like is this a position? How big is it? How many bytes does it take and so on and so forth.

So finally the stride from one vertex to the next, that's what the hardware needs to know. The same sort of thing for the submesh. You say what kind of indexes do you have. Are they 16s or 32s or whatever. The geometry type. Is it a triangle strip, or is it a point or a line, and finally a material.

Materials, as I said are going to be physically motivated if you use these APIs. And to tell you what that means, we have got bidirectional reflectance functions with ten simple parameters that are designed by artists to be intuitive. So one of the more important features is just to specify whether the material is dialectic like clay or if it's a metal. If you set this value all the way to one end, it's dialectic like clay all the way to the other end, it will behave like a metal. Here I have combined the two to put an acrylic clear coat on top of the metallic base and here I tweaked up one of the parameters to give a satin finish. And here's an actual artist-prepared piece of spaceship with all kinds of different materials on it, just to give you an idea that a small number of parameters can give you a wide variety of looks.

So, materials have a name, just like everything in Model I/O has properties. You specify the scattering function, whether you want it to be Lambert Blinn-Phong because you need compatibility with an older system, and, or physically plausible if you're going into this brave new world of physical motivation.

The materials are singly inherited and so what that means is you might have some kind of a uniform that only varies by a number or something on each character. So you can specify a base material and override properties and subclass materials. The material properties have names, and they have a semantic which means how are they to be used, a type, like is it a floating point value or a color and a value.

And lights. And lights have physical parameters and physical properties and they have geometry. They have, you know, an extent and a width. And the light itself emits light in a certain way. You can specify it with lumens and color temperature. One thing I'm really excited about overall in Model I/O is we've got support for color that can be specified using modern color pipelines. So if you want to use SRGB, we have a well-specified SRGB. If you want to use Rec.709 or the new Asus CG color profiles you can ensure that your color started in a content production app a certain way and it got all the way to the end, without going through odd transformations that might give you some, you know, color surprises.

So there we have our physically plausible light and various subclasses, procedure area light, I should say it procedural description of an area light. And the photometric light from an IES profile, and light probes that take reflective maps or irradiance maps and you can use them to compute spherical harmonics and stuff like that to compactly represent what's going on.

And our camera is no longer merely a pinhole camera with an infinite in focus projection. We describe a camera from one end to the other. My picture here is supposed to have a lens, shutter and sensor plain there. We describe what the lens can see, the properties of the lens, what kind of distortion, barrel distortion, or chromatic aberration that sort of thing, the geometry of the lens and how big is the glass, how long is the barrel. The exit aperture. How tightly have you closed the aperture, how big is the sensor and what is the exposure characteristics of the sensor. So the end result of all of that is, if you specify your cameras in this way, if you mount a 35 or say a 50-millimeter lens with an F1.8 aperture and go check Wikipedia, what are the characteristics of a lens like that, what is the field of view, and what's the out of focus light, highlight size and things like that, the utility functions on the MDLCamera will agree with what you find in a textbook. So that's a handy and fun thing. And I encourage you to incorporate those kinds of calculations into your shaders and pipelines.

Just a quick example here.

When I first load this thing up in my viewer, these are the default exposure settings. Some of the detail is lost, it's a bit washed out. We can't see a lot of detail on the dark areas of the excavator.

Sorry for the brightness here, but I underexposed this image and then flashed the sensor in order to bring out shadow detail, and raise the overall levels while reducing glare.

Just like a photographer might do, if you are playing around with settings on your own real camera. Now, skies.

We have two ways to create skies. The first way is to use a procedural sky generator. It uses physics. You give it time of day, essentially how high is the sun. What are the atmospheric conditions. Is there a lot of back scatter from the ground, is there a lot of junk in the air. We calculate exactly what the sky would look like through some fairly heavy math.

So the other way you can create -- it creates a cube that you can then just, you know, use. Now, the other way you can create sky for illumination in this physically based realm is through photography. You can take a spherical panorama with your phone or a DSLR. I made this picture with my iPhone. Then you can prepare it for rendering using the MDLTexture and initWithURL API, create a cube map for reflectance and irradiance. Irradiance is the incoming light that we can deduce from the image, so there it's been converted into a cube. And then, from that, you can compute the irradiance and what these three strips show is the original image, the middle line is a texture map showing the irradiance at a certain convolution or level of blur and the third one is really fun. The middle one is a texture and it uses a fair amount of memory. And the third one is actually spherical harmonic coefficients and so it's 27 floats that reproduce the look of the irradiant environment that previously took several dozen K to represent in the middle slide or the middle strip.

So putting that all together, this excavator is fairly well situated in its environment, in that I photographed and that's, I think, a pretty exciting result, and I hope you guys can think of cool things to do with that. Now, I just want to talk a little bit about how Model I/O integrates with SceneKit.

There's essentially a one-to-one correspondence between top level Model I/O elements and SceneKit elements. MDLAsset corresponds to SceneKit root node and MDLMesh coordinates with SCNNode with SCNGeometry. MDLLight with SCNLight camera camera, material material.

Where exact analogs don't exist between a SceneKit parameter and about a Model I/O parameter, we translate to get a close approximation.

Now, Model I/O doesn't actually do rendering for you. You probably gathered that since I mentioned all of these other APIs that do do rendering. So in MetalKit, you are much closer to the metal.

The correspondence that you get between Model I/O and MetalKit is an MDLMesh can be converted into a model kit array kit of meshes. Once you have that array of meshes, it's up to you to write shaders, traverse the mesh, find the materials, the lighting condition, and do the rendering. But getting those metal buffers prepared for you is absolutely transparent.

So with that, I would like to pass the mic to Claudia Roberts to talk about geometry and voxels.

CLAUDIA ROBERTS: Hello, everyone, my name is Claudia Roberts and as Nick mentioned, I will give you an overview of some of the different ways you can describe and characterize geometry in Model I/O.

The motivation being to help you create games and applications that have a more physically plausible look and feel. To give you all some context of where we are headed, first I will discuss how Model I/O supports normal smoothing then go into subdivision surfaces followed by a discussion on voxels and a quick demo. Okay. Let's get started. Normal smoothing.

Normal smoothing is a cool technique that tricks people into believing that your models have way more geometry than they actually do.

By default, the vertices of a polygon all share the same face normal. And thus all points on the face of the polygon have the same normal as well.

This creates a perceived crease between adjacent polygons which is the result of the abrupt change and vertex normals during the rasterization process.

This sharp contrast in colors can be mitigated by introducing a shared normal whose value is the average of the vertex normals that share the same coordinate position.

Now, during the GPU's lighting calculations the normal at each point on the face of the polygon will be the interpolation of vertex normals that are no longer all the same, giving a nice, smooth shading effect.

Using the MDLMesh API, you can add smoothed out normals to your object, by calling the addNormalsWithAttributeNamed method and you can control when normal smoothing is applied by setting the crease threshold value.

With our spaceship, we see our default mesh with the flat shading on the left and the smooth shading on the right. Next, subdivision surfaces. Surface subdivision is a common technique for using low detailed geometry to generate and render a smooth surface for rendering.

This technique allows you to use a simple polygon control mesh to create varying levels of detail as needed. For instance, it would make a lot of sense to render a character at a low polygon count when further away and increase the level of detail as the character gets closer and closer to the camera.

By varying the subdivision level of a model, you can generate these different meshes without needing to manually create them all.

In Model I/O, you can create subdivision surfaces by calling the newSubdividedMesh routine, also found in MDLMesh.

Here at the bottom, we achieve the smooth mesh on the right, by setting the subdivision level to two, significantly increasing the number of polygons.

Finally, voxels.

In addition to providing support for various advanced techniques for polygonal representations of 3D models, Model I/O also supports volumetric representations.

By representing a model as a close approximation of how it is actually found in the real world, that is a set of particles or discreet points in space with inherent properties such as volume, mass, velocity, color, the door becomes wide open to a range of physically realistic techniques, analysis, and manipulations.

Where as with polygon meshes it's difficult to model and represent surfaceless phenomena such as clouds, water, fire. It becomes much easier with the volume representation.

Now, instead of trying to mingle and twist a rigid shell of polygons, the model becomes a deformable mass that can change its properties at any time.

Along those same lines this representation allows for procedural generation and modeling, meaning it can make for exciting and novel opportunities and game play. Think modification and destruction of objects and terrain on the fly.

Because a voxel model is a more accurate representation of the real world, it lends itself to being analyzed, explored, and operated on in a more natural and real way like slicing and cutting.

This fact is proven to be particularly useful in the medical imaging field where, lucky for us, scientists have proven than our skulls are not really comprised of an empty shell of triangles.

And finally, given that you have a few solid voxelized models, you can perform Constructive Solid Geometry Boolean operations on them in order to create a slew of more interesting and complex models. In Model I/O, we expose the support of voxels via the MDLVoxelArray API.

Our implementation represents volume models as a sparse volume grid, where voxels can be accessed by using a simple spatial index.

This representation allows for quick neighbor finding and neighborhood traversal. In addition to the grid coordinates each voxel contains a shell level value which indicates how close or far a voxel is to the surface of the model, both in the positive exterior direction and the negative interior direction.

And Model I/O also supports the creation of closed meshes model cleanup and conversion back to a polygon mesh.

I will now show you the handful of API calls you will need in order to get started with voxel and Model I/Os. So given an initialized MDLVoxelArray you can generate its voxel data from an MDMMesh object by calling the setVoxelsForMesh method.

The divisions parameter is used to set the resolution of your voxel model by specifying the number of layers your model will be divided into on the vertical extent.

You can also specify how thick you want the interior and the exterior walls of your model to be with the last two parameters. Once you have your voxel array objects set up, you can perform various operations on them, such as intersect, union, and differenceWithVoxel which perform expected basic Boolean operations. To actually retrieve your voxel data for processing and inspection, simply call the getVoxelIndices method and once your done with your processing convert your voxel model back to a polygonal representation, using the meshUsingAllocator routine.

Simple as that.

And now, I will show you voxels in action with a quick demo.

CLAUDIA ROBERTS: So here we have this demo. It actually took about one hour to create from start to finish using SceneKit editor. We simply dragged and dropped in the ground and we did the same for our red panda which you actually saw yesterday at the state of the union. It's just a regular polygon mesh and you can use any mesh that you want for this.

And then using the MDLVoxelArray API, it only took two lines of code to turn this mesh into voxels.

And then for each voxel, we creates an SCN box, and this is what it looks like.

And now that we have our voxels in SceneKit, the exciting thing about that is we can take advantage of all the really cool things that SceneKit has to offer. For instance, with one line of code, we can turn all of these SCN boxes into SCN spheres. And just for fun, we will apply SceneKit body to all the nodes and explode him. Wee! I am hand it over to Remi now who will talk about advanced topics in lighting and Xcode support.

REMI PALANDRI: Hey, everyone, and thanks, Claudia. So hi. As Chris said today, I will be talking about advances in baking and how Model I/O does all of that for you.

So what exactly is is advanced lighting and baking. What's the goal here? The goal is to introduce to your frameworks and your rendering pipelines in your games is something called global illumination.

So what is that? It's not the way of saying, all right I have a point height and I have a triangle and let's light it up using dot products and be done with it. We are going to try to pretend that that scene is real. We're going to try to simulate how light would actually move in that scene how light will reflect off the walls and occlude because it can't go through triangles.

The issue is that it's very expensive. It's been used a long time in the movies because you can take a half hour to reframe if you want, but that doesn't work for us. If you look at the picture here, you can see, for example, that the wall here, the red wall on the left irradiates a bit of red light on that sphere. The wall isn't really the light per se, but lights as it does in real life reflection off the wall and irradiated the sphere. In the back of the sphere, it's a bit dark, because this sphere occludes some lighting from going there. It's not a real direct a shadow but there is still something going on.

The issue is that this is really hard to do in realtime. So we are going to show you ways to actually get some of that precomputed before you even launch your game during precompilation. So that you can get a really realistic rounded look without having any of the performance drawbacks and we will balance performance and quality so you can get the look that you want with very, very few performance overheads. So global illumination today will have two different heads. So we are going to first show you ambient occlusion and then light maps.

And to introduce ambient occlusion, I would like to show you an example. If you look at the spaceship it's the default SceneKit spaceship, it looks good. It's a great spaceship. I love to play the game, but it's a bit flat. If you look at the wing or the engine, it's not extremely clear where the wing ends or the engine starts. If you look at the two fins on the right image, it's weird because you have the same light as you have on the front of the ship but the fins should block out the lights.

If you were to add ambient occlusion it would look like this. Same shader, but the look is a bit more realistic, because there is now a shadow between an occlusion between the wing and the engine. If I were to add ambient occlusion to the second one, it would look like this, same thing, but you can see light that's occluded and it's a more compelling experience. It's a better looking picture.

What exactly is ambient occlusion? What ambient occlusion is is very simply a measure of geometry occlusion. What that means is, for my point or my mesh, how much of the light that arrives from my world can actually go to my point and how much of my light is actually blocked by my mesh and its surrounding meshes.

So it's basically a signal, one, for white, saying I have absolutely no blocking. The whole light arrives and/or commuting, most of my light can't go there. If we were to look at the signal, that's what it looks like.

Mostly white, because most light can go there. But you see some physical data there.

How do we compute that.

We compute that using offline raytracing. So your mesh gets into a and we send rays everywhere. And we calculate how much rays if I send rays all around my points. How much rays hit the mesh and how much rays go to the sky box. Difference between both is my ambient occlusion signal.

So what do we require from you guys? An input. It's a mesh, my spaceship, and a set of occlusion meshes. So here it's only a spaceship. If I were, for example, to represent that scene right there, and I wanted to make the ground. So it would compute ambient occlusion for the ground. I would also need all the chairs and the people and the floors and all of that to stop the rays so that I have a very nicely looking mesh. So that would be a nice set.

And what do I get as output, a set of occlusion values just for every point.

What is the occlusion of that point.

So how exactly do we store that? We have two ways, either vertices or textures. So if my mesh has lots of vertices for example a big spaceship with plenty of triangles we can just store that in vertices. It works very well because its very cheap it's nearly one float per vertex and using rasterization for your metal or pipeline, it's extremely cheap to render, but for example, that spaceship is actually fairly low triangles and so we need a texture.

So we actually built for you guys inside Model I/O, a UV mapper that creates 2D texture and wraps it around the 3D mesh, and so that corresponds. And then for every pixel of that texture we have an ambient occlusion. If we were to look at what that texture looks like for that spaceship, it looks like this. So you can see it's basically the texture wrapped around the spaceship. You can see the wings and the engine base and all of that. How do we compute that? It's very easy. It's literally a one liner. You can look at the top one for example, shipMesh generateAmbientOcclusion, here it's vertex. And so we have two parameters quality and attenuation factor.

If we increase quality, what it will do, it will send more rays to get a better looking signal but it's going to take a bit more time to compute.

Because it's baking, it's before you even launch the game, but still. And if we have a textural bake, then the texture will be bigger so it will increase a bit your memory cost. And then that attenuation will simply attenuate the signal so only the darker parts stay dark.

The really cool thing here is we actually partnered with the SceneKit team to integrate those features in both SceneKit but even in the SceneKit editor that you guys saw at the state of the union. And I would like to show you that right now.

So it works. So I just loaded here a little scene that literally has a spaceship and a sky box around. The ship is very nicely flying in space. I added no lights here, which is why is is very flat. The only thing that we are visualizing right now is ambient lighting. And as you can see the ambient lighting does not take what the ship looks like into account whatsoever. It doesn't give the user nice feedback in terms of where the geometry is, so its very flat. We're going to change that. So I'm going to click on my mesh and first, I will see that we actually have a fair amount of vertexes here so we will do a vertex bake. I will bring up the geometry tab. I'm going to go here and under occlusion bake and choose the vertex, those values are perfect, and press bake. So what's happening here? For every little vertex of that ship, we will send between 100 to 200 rays around and then it looks like this.

See? Way better! We had this and now we have got this. And it makes perfect sense. If you look at, for example, here, the top deck, created occlusion on the bottom one, because the light can't arrive there easily. If we look at the windows inside here, here the inner port of the windows have more occlusion than the outer parts.

If we look here. Let me zoom in.

If we look here at the cannons underneath, the top of the cannons are really dark because the whole ship stops the light from arriving there. If we were to look at the bottom parts, all white, makes sense, right? So by just adding one float per vertex, we were able to give our ambient light away to light up our scenes and give a better look.

And that's available in the of SceneKit.

So let's go back to the slides.

So that was ambient occlusion. I will finish today by introducing you to advanced lighting with light maps.

So what is light maps? What light maps are is a way to get your diffused lighting into the game and precomputing how the diffused lighting affects your textures to not have to do that in realtime. As you guys know, if you have done some game programming, lighting is extremely expensive. If you have ten lights then for every frame you need to compute that lights' aspect and how that interferes with your game. That's very expensive. So what we have here, I just dragged and dropped a plane and two boxes inside a simulator and put eight lights there and I computed, before I even launched the game, using light map baker how those light maps light up my scene and light up the texture.

And if we were to look at the scene, that's what it looks like. It's really realistic. We've got the shadows and the lights but this costs me literally one texture fetch.

That's it.

Usually rendering eight lights especially if you've got shadows and shadow maps they are very expensive. That's what the texture looks like. Very straightforward, you see my boxes in the middle and my shadows. The cool thing is this is just a texture fetch, right? So it supports lots of lights. I could have 100,000 lights if I wanted and it would have the exact same runtime costs. Even the shadows look really cool. When you do spotlights like this, that are really close to the ground then the shadow maps you reach kind of precision issues but with this, for every pixel on the thing, we send rays and see which ones are in the light and which ones are not. So your shadows look really realistic. Calculate it offline. And the cool thing is we support super complex slide that you couldn't even dream of doing at run time. For example our real lights have for a long time been really hard to do at runtime because they are really hard to do with normal point light to triangle illumination processes but here we are using ray tracing. So we just send the ray and see which ones arrive and which ones don't.

We also support the cool IS lights, that Nick talked to you before.

And that was light maps. So to close this talk, I would like to summarize a bit. Today we introduced a great new framework and we are super happy about it.

First it does the basics imports and exports 3D asset files. But is actually does so much more for you. It introduces concepts for physical basis for rendering, with models, lights, cameras, materials, and skies that aren't just defined with floats but are actually based on stuff in real life. We have integration of system in the frameworks and tools in Xcode that you guys can play with that and have fun.

For more information, we invite you to look at our documentation and videos and forums and our technical support and for any general inquiries, you can contact Allan Schaffer.

Related sessions are "Enhancements to SceneKit" tomorrow and "What's New In Metal" on Thursday. And we would like to see you at Model I/O Lab right now and tomorrow morning at 9AM. Thank you all for your time and have a great WWDC.

[ Applause ]