The illusion of space: how the new Spiderman renders spaces without geometry

In the recently released Marvel's Spider-Man game, many buildings outside the windows have interiors. They look great, but they seem to have been implemented using clever rendering - the geometry of the interiors actually does not exist and it is generated by the shader. I did not see any official statements by Insomniac about how they did it, but based on what the effect looks like, the interior mapping technique that I came up with in 2007 while working on my dissertation is likely to be implemented. Previously, I did not write about it in a blog, so now is the right time to explain the curious little shader that I came up with.

Let's start by watching Marvel's Spider-Man gameplay. The game looks just awesome. Site Kotaku recorded a separate video dedicated to the windows:


As you can see at about 40 seconds of video, in fact, the rooms are not part of the geometry: where there should obviously be a window, there is a door. In addition, looking in one room from different angles of the building, we see a different interior. In some cases, there is even a wall around the corner of the building. All this makes us realize that the rooms are simulated. However, in terms of perspective, they are displayed correctly and have real depth. I think the drawbacks of such rooms when playing are not very important, because players usually do not study rooms so closely: interiors are just the background, and not the subject of careful research. I believe that this way of creating rooms adds depth and life to a city without spending too much resources.
')

To save resources, buildings in games often do not have interiors, as can be seen in the screenshot of GTA V

First of all, I want to explain that my post is not a complaint: I am delighted with the fact that my technique was used in such a large-scale game and in no case accuse Insomniac of stealing. As I said in the first publication on interior mapping, it would be an honor for me if someone uses this technique. If Insomniac really took advantage of my idea in my technique, then I think that's great. If not used, then it seems she came up with something strangely similar. Then I would be interested in how this was done.

So how does interior mapping work? The idea is that the building itself does not contain any additional geometry. Interiors exist only in the shader. This shader performs raycasting with walls, ceilings and floors to calculate what the player should see in the interior.


From left to right: only windows with reflections, windows with Interior Mapping, frame model - Interior Mapping does not add any polygons.

A raycast used for raycast is simply a ray from a camera to a pixel. The pixel we are rendering is located on the outside of the building, so we only use part of the beam by pixel, because this is exactly the part that is actually inside the building.

Raycasting may seem like a complicated and costly operation, but in this particular case it is actually very simple and fast. The trick is that you can add a simple limitation: with interior mapping, the ceilings and floors are at a constant distance. Knowing this, we can easily figure out which room we are in and where the floor and ceiling are located in this room. Ceilings and floors themselves are infinite geometric planes. The calculation of the intersection between the infinite plane and the ray takes only a few steps and spends few resources.


The room has 6 planes: ceiling, floor and 4 walls. However, we need to take into account only three of them, because we know in which direction we are looking. For example, if we look up, we do not need to check the floor below, because we will see the ceiling above. Similarly, instead of 4 walls, we need to consider only two in the direction in which we are looking.

To determine what we see, we need to calculate the intersection of the beam with each of these three planes. The intersection with the beam nearest to the camera tells us which of the planes we see in this pixel. Then we use the intersection point as the coordinate of the texture to find the color of the pixel. For example, if the ray intersects at the (x, y, z) position with the ceiling, then we use the (x, y) textures as coordinates and ignore z.

Here I added a good optimization: a part of the intersection calculations for each of the three planes can be performed simultaneously. The shaders used worked with float4 at the same speed as with float, so thanks to the clever packaging of variables, all three rays could be intersected with the planes at the same time. This saved me some resources and helped me achieve a high frame rate with interior mapping even in 2007. I was told that modern video cards with float work faster than with float4; This means that this optimization does not work on the current hardware.


Interior Mapping without windows textures shows that rooms are rendered with the correct perspective and textures, although additional geometry is not required.

More details about the work of interior mapping can be found in my article . This article was published at the Computer Graphics International Conference in 2008. The presence of this peer-reviewed publication is my first (and only) application for the proud title of a scientist. This article also describes additional experiments on adding details, for example, changing the distance between walls for rooms of unequal size and randomly choosing textures from a texture atlas for greater variability of rooms. It also describes in detail the two variations shown in the images below.


The lighting in the rooms can be dynamically turned on and off to simulate the change of day and night. This is done using the noise texture from which we read, using the room index as the coordinates of the texture.

Since we just emit rays in the plane, all rooms are simple squares with textures. All the furniture in the room will be on the texture, and therefore remain flat. In Spiderman, this is noticeable in the case of approaching the camera: the tables in the rooms are in fact flat textures on the walls. As can be seen in the image below, you can supplement our technique with one or more additional layers of textures per room, but this is due to the additional cost of performance.


By performing raytracing on another plane parallel to the outer surface of the building, you can add furniture and people to the room. However, they will still remain flat.

After the publication of this post, one of the programmers at Simcity (2013) told me that the interior mapping technique was also used in this game. In her, she looks very cool, and the developers recorded a great video about it. They perfected my original idea, retaining all the textures in one texture and adding rooms of different depths. The interior mapping part starts at 1:00:


If you want to explore this technique more deeply, you can download my interior mapping demo with the source code. If you work in Unreal Engine 4, you can find interior mapping as a standard engine function as a function of InteriorCubeMap.

After so many years it is very cool to finally see that my interior mapping technique is used in the production of large-scale video games! If you are familiar with games that use something similar, then write about it to me.