Unreal Engine Tutorial: Cel Shading

Thanks to the physically accurate rendering in Unreal Engine 4, it is convenient to develop realistic games. The rendering model imitates the interaction of light with materials, which leads to the creation of a realistic image. However, if you want to develop a game with a stylized look, then you will have to explore other techniques.

One way to create styling is to use cel shading (also known as toon shading). This technique emulates the shading commonly used in cartoons and anime. Examples of its use can be seen in games such as Jet Set Radio , The Legend of Zelda: The Wind Waker and Gravity Rush .

In this tutorial you will learn the following:
')

• Create and use post processing material
• Create cel shader
• Isolate cel shader for individual meshes
• Manage color bars with lookup tables.

Note: This tutorial assumes that you are already familiar with the basics of the Unreal Engine. If you are new to the Unreal Engine, then you should study first the tutorial of ten parts of the Unreal Engine for beginners .

Getting Started

Download the project blank and unpack it. Navigate to the project folder and open CelShader.uproject . You will see the following scene:


This is a character to which we will apply cel shading. Before you begin, you need to figure out what is cel shading.

What is Cel Shading?

Cel shading is a rendering process using multiple bands of color, rather than a continuous gradient.


Below is an example of using cel shading in The Legend of Zelda: Breath of the Wild . Note that cel shading is implemented only for the character, and the background remains normal.


There are three bands in this image: one for the shadows, one for the mid-tones, and one for the light areas.

It is often mistakenly thought that cel shading is applied if objects have contours . An example of this is the Borderlands . Although this game has a stylized look, in fact there is no cel shading. This can be seen in the image below. Note that the color of the character does not use color bars.


Contours are not cel shading, but they are often shared. Thanks to this, the picture becomes similar to painted with ink or ink. This technique is often used in anime styling games such as Guilty Gear Xrd and Dragon Ball FighterZ .


In the next section, we will learn how to implement cel shading.

Method of implementing Cel Shading

The most common method of implementation is to compare the direction of the surface (known as “normal”) and the direction of light. By calculating the scalar product between the normal and the direction of light, we obtain a value from -1 to 1.

A value of -1 means that the surface and light have opposite directions. 0 means they are perpendicular. 1 means they are directed in the same way.


By setting threshold values ​​for scalar products, you can create multiple bands. For example, if the dot product is greater than -0.8, then the surface can be assigned a dark color, and if the dot product is less than -0.8, then the light color. So we will create a two-way cel-shader.


The limitation of this method is that objects with cel-shading cannot be affected by other sources of illumination. In addition, objects cannot cast shadows on objects with cel-shading.


To solve this problem, we must use another method. Instead of calculating the dot product, we will calculate the surface illumination. Then, when setting the threshold values, you can use this value instead of the dot product.


Now that you know what a cel shader is and how it works, it's time to create it.

Create Cel Shader

In this tutorial, cel shading will be a post-processing effect. Post processing allows you to change the image after the engine has finished rendering it. Post-processing is commonly used for effects such as depth of field, motion blur and bloom.

To create our own post-processing effect, we need to use the post process material . Go to the Materials folder and create a new Material . Rename it to PP_CelShader and open it.

To convert a material into a post-processing material, you need to change its domain . Go to the Details panel and change the Material Domain to Post Process .


The first step in creating a cel shader is to calculate the illumination of each pixel. We will call this the lighting buffer .

Light Buffer Calculation

When Unreal renders an image to the screen, it saves passes to the buffers. To calculate the lighting buffer, we will need to access two such buffers:

1. Post Process Input: After Unreal has done the lighting and post-processing, it saves the image to this buffer. It is his Unreal will be displayed to the player, if you do not perform further post-processing.
2. Diffuse Color: This is a scene without any lighting and post-processing. It will contain the diffuse color of everything on the screen.

To access these buffers you need to use the SceneTexture node. Create it, select it and go to the Details panel. To access the Post Process Input buffer, change the Scene Texture Id to PostProcessInput0 .


To access Diffuse Color, create another SceneTexture node. Change its Scene Texture Id to DiffuseColor .


The illumination buffer should contain only values ​​in grayscale (describing the degree of illumination of pixels). This means that we do not need color information from both buffers. To discard the colors, connect the Color output of both SceneTexture nodes to the Desaturation . This will completely discolor both buffers.


To calculate the light buffer, simply divide SceneTexture: PostProcessInput0 into SceneTexture: DiffuseColor . Order is important here!


Then use Clamp to keep the values ​​in the range from 0 to 1. This will simplify the creation of thresholds, because we will know the possible values.


Here is the visualization of the lighting buffer:


As you can see, the illuminated areas are closer to white, and the unlit areas are closer to black.

Then we use the light buffer to create the threshold.

Create threshold

In our cel shader, any pixel with a value greater than 0.5 will use a normal diffuse color. Pixels with values ​​less than 0.5 will use a half-brightness diffuse color.

First, create an If node. He will allow us to compare two values. Depending on the results of the comparison, we will be able to specify different outputs.


Next, connect the Clamp to the input A. Then create a Constant with a value of 0.5 and connect it to the input B.

Note: to change the threshold, you can change the value of input B.

To get the colors, create a SceneTexture and set its Scene Texture Id to Diffuse Color . Then multiply the Color by 0.5 to get a diffuse color of half brightness.


And finally, let's connect everything as follows:


Summarize:

1. Desaturation converts Post Process Input and Diffuse Color to grayscale images.
2. Divide divides Post Process Input to Diffuse Color . So we create a lighting buffer.
3. Clamp limits the values ​​from 0 to 1
4. If displays the normal diffuse color if the light value is greater than 0.5 . If it is less than 0.5 , then it displays a diffuse color of half brightness.

Now that we have a cel shader, we need to apply it to the scene.

Use of post processing materials

To use post-processing materials, we need to create a Post Process Volume . It is commonly used to control post-processing effects such as white balance, saturation, and contrast.

Click on Apply and return to the main editor. To create a Post Process Volume, go to the Modes panel and select the category Volumes . Then drag Post Process Volume to Viewport to create it.


Now we need to tell Post Process Volume to use the cel shader. After selecting Post Process Volume, go to the Details panel. Then find the Rendering Features \ Post Process Materials and click on the + icon. This will add a new element to the array.


Then click on the Choose drop-down list and select Asset Reference .


This will allow you to choose a material. Click on the None drop-down list and select PP_CelShader .


By default, the Post Process Volume only affects when we are inside. However, in our case, it is necessary that it affect the whole world. To do this, scroll to Post Process Volume Settings and turn on Infinite Extent (Unbound) .


Now that the cel shader is applied to the whole game, we will see the following:


"Wait a minute, it doesn't look like the cel shader that you showed before!"

The main reason for this difference is that the engine uses a cel-shader after tone mapping. To fix this, we need to ask the engine to use a cel-shader before tone mapping.

Cel Shading before tone mapping

Before showing the image to the player, Unreal performs a process known as “tonemapping”. One of the goals of tonal compression is to make the image more natural. It takes the input color and then uses the curve to shift it to a new value.

Here are two images, before and after tone mapping:


As you can see, the light areas before tone mapping are too bright. However, after tonal compression, bright areas become softer.

Although tone mapping is useful for images that need to be displayed, we do not have to perform tone mapping for images that we want to use in calculations. Because of the offset values, we will not use the values ​​that we expect.

Open PP_CelShader and make sure nothing is selected. Then go to the panel and find the Post Process Material section. Set the Blendable Location to Before Tonemapping .


Click on Apply , and then return to the main editor. The colors now look much better!


In the next section, we will learn how to apply cel shading only to individual objects.

Isolating the Cel Shader

To isolate the effects of post-processing, we need to use a function called Custom Depth . Like Post Process Input with Diffuse Color, this is also a buffer that can be used in post-processing materials.

Before you understand what Custom Depth is, you need to deal with the Scene Depth buffer. Scene Depth keeps the distance of each pixel from the camera. Here's what the Scene Depth visualization looks like:


Custom Depth stores the same information, but only for the meshes you select. Here is his visualization with a Viking rendered to Custom Depth:


By comparing Scene Depth with Custom Depth, we can isolate objects. If Scene Depth is smaller than Custom Depth, then we use a regular image. If Scene Depth is larger than Custom Depth, then an image with cel shading is used.

The first step is to render the viking to Custom Depth.

Using Custom Depth

Go to World Outliner and select SK_Viking . Then go to the Details panel and find the Rendering section. Then turn on the Render CustomDepth Pass .


Next we need to perform a depth comparison. Open PP_CelShader and create the following schema:

Note: Mask (R) nodes are Component Masks. They allow you to convert multi-channel data to scalar values. We need to put a mask on Scene Depth and Custom Depth because the If node for inputs A and B takes only scalar values.

Then connect the output of the cel shading network to A> B. Finally, connect the output of the newly created If to the Emissive Color .


Now cel shading will only be applied to meshes rendered in Custom Depth.

Click on Apply , and then return to the main editor. You will see that cel shading is now performed only for the viking.


The Cel Shader works great, but it’s pretty simple. What if we need more lanes? What if we want to create smoother transitions between lanes? All this can be done using lookup tables (LUT).

What is a “lookup table”?

In childhood we learned what multiplication is. However, the young brain could not always perform such calculations. Therefore, instead of calculations, you could use the multiplication table to “search” for answers.


In essence, this is LUT. This is an array of values ​​(usually precomputed) that can be accessed using input data. In the case of a multiplication table, the input data was a multiplier and a multiplicand.

In the context of our cel-shader, LUT is a texture with some kind of gradient. Here are four examples of what LUT might look like:


For now, we calculate the color of the shadow by multiplying the diffuse color by 0.5. Instead of multiplying by the constant 0.5, we will use the value from the LUT. Due to this we can manage the number of lanes and their transitions. To understand how shading will look, you can by the appearance of the LUT.

Before using LUT, you need to change some of its texture parameters.

Change LUT Parameters

Go to the Textures folder and open T_Lut_01 . This is what LUT looks like:


The first parameter we need to change is sRGB . When rendering, Unreal converts all textures with sRGB on to a linear color. This makes it easier for the engine to perform rendering calculations.

The sRGB parameter is useful for textures that describe appearance. However, textures like normal maps and LUTs contain data for mathematical calculations. Therefore, Unreal should consider their values ​​already true. If you disable sRGB, Unreal will not convert to a linear color.

To do this, uncheck sRGB . This parameter is located in the Texture section.


The next parameter we need to change is the texture tiling method. Since we will not display this texture, she does not need tiling. Moreover, if you leave the tiling enabled, it will add problems when sampling at the edges of the texture. For example, if we sample a pixel from the left edge, then because of the tiling, it will try to blend with the right edge.

To disable tiling, change the value of the X-axis Tiling Method to Clamp . Do the same for the Y-axis Tiling Method .


And on this we are done with the parameters. Now we need to use LUT in the postprocessing material.

Using LUT

Close T_Lut_01 and open PP_CelShader . First, delete the selected nodes:


Then create a Texture Sample and change its Texture to T_Lut_01 . This LUT table will create three bands with a smooth transition.


As we remember, LUTs use input data to determine the values ​​to be output. In our case, the light buffer will be used as input.

To do this, connect the Clamp to the UVs in the Texture Sample .


This works because the lighting buffer values ​​and texture coordinates are between 0 and 1. For example, if a pixel from the lighting buffer is 0.5, then the LUT will output the pixel value from the middle of the texture.

Next we need to multiply the diffuse color by LUT. To do this, recreate the following scheme:


We use Append to convert Texture Sample’s output to a four-channel vector. We need this because we cannot multiply a three-channel vector by a four-channel ( SceneTexture ).

Finally, connect everything as follows:


Now, instead of multiplying the diffuse color by a constant, we multiply it by the value from the LUT. So we control the number of color bars and their transitions (depending on LUT). The output value of the LUT is determined by the light buffer.

Click Apply , and then close PP_CelShader . Now shading will have three lanes with smoother transitions between lanes.


Below is a comparison of how different LUTs may look. These LUTs are also added to the project.

Where to go next?

The finished project can be downloaded from here .

As you can see, post-processing materials are a very powerful tool. They allow you to create many realistic and stylized effects. If you want to learn more about the post-processing, then study the documentation for the post-processing UE .