Frame buffer

To date, we have already managed to use several types of screen buffers: a color buffer, in which the color values ​​of the fragments are stored; depth buffer storing information about the depth of the fragments; stencil buffer, allowing you to drop some fragments according to a specific condition. The combination of these three buffers is called the frame buffer (framebuffer) and is stored in a specific area of ​​memory. OpenGL is flexible enough to allow us to create our own frame buffers ourselves, by defining our own color buffers and, optionally, depth and stencil buffers.





All the rendering operations that we have performed so far have been executed within the framework of buffers attached to the basic frame buffer. The base frame buffer is created and configured when the application window is created (GLFW does the hard work for us). By creating our own framebuffer, we get additional space where we can direct the render.



At first glance, it may not seem obvious how to use your own frame buffers, but displaying an image in an additional buffer allows at least creating mirror effects or post-processing. But first, we will understand how the frame buffer is arranged, and then we will consider the implementation of some interesting post-processing effects.

Create frame buffer

Like any other object in OpenGL, the framebuffer object (abbreviated FBO from FrameBuffer Object ) using the following call:

unsigned int fbo; glGenFramebuffers(1, &fbo); 

We already have a familiar and dozens of times applied approach to the creation and use of objects of the OpenGL library: create a frame buffer object, bind it as the current active frame buffer, perform the necessary operations, and untie the frame buffer. Binding is as follows:

 glBindFramebuffer(GL_FRAMEBUFFER, fbo); 

After binding our frame buffer to the GL_FRAMEBUFFER bind point, all subsequent read and write operations for the frame buffer will use it. It is also possible to bind a frame buffer for read-only or write-only by binding to special anchor points GL_READ_FRAMEBUFFER or GL_DRAW_FRAMEBUFFER, respectively. The buffer bound to GL_READ_FRAMEBUFFER will be used as the source for all read operations of type glReadPixels . And the buffer associated with GL_DRAW_FRAMEBUFFER will be the receiver of all render operations, buffer cleaning and other write operations. However, for the most part you do not have to use these anchor points by applying the GL_FRAMEBUFFER anchor point .



Unfortunately, we are not yet ready to use the frame buffer for us, since it is not complete. To become complete, the frame buffer must meet the following requirements:

• At least one buffer must be connected (color, depth, or stencil).
• There must be at least one color attachment.
• All connections must also be completed (provided with dedicated memory).
• Each buffer must have the same number of samples.

Do not worry so far about what samples are - this will be discussed in the next lesson.

So, from the list of requirements it is clear that we have to create some “attachments” and connect them to the frame buffer. If we have fulfilled all the requirements, we can check the completion status of the frame buffer by calling glCheckFramebufferStatus with the parameter GL_FRAMEBUFFER . The procedure checks the current associated frame buffer for completeness and returns one of the values ​​specified in the specification . If GL_FRAMEBUFFER_COMPLETE is returned, then work can continue:

 if (glCheckFramebufferStatus(GL_FRAMEBUFFER) == GL_FRAMEBUFFER_COMPLETE) { //  ,   ! } 

All subsequent rendering operations will output to the frame buffer connections currently attached. Since our frame buffer is not basic, the output to it will not have any effect on what is displayed in the window of your application. That is why the render in its own personnel buffer is called an off-screen renderer. In order for output commands to take effect again on the output window of the application, we need to screw in the basic frame buffer to the active place:

 glBindFramebuffer(GL_FRAMEBUFFER, 0); 

It is transmission 0 as the frame buffer identifier that indicates to bind the base buffer as active. After completing all the necessary actions with the created frame buffer, remember to delete its object:

 glDeleteFramebuffers(1, &fbo); 

So, let's go back a step to check the buffer completeness: you need to create and connect at least one attachment to our personnel buffer. An attachment is an area in memory that can act as a receiver buffer for a frame buffer, making it easier to imagine it as an image. When creating an attachment, we have a choice: use textures or a render buffer.

Texture Attachments

After connecting the texture to the frame buffer, the result of all subsequent commands will be written into this texture as if it is a normal color, depth or stencil buffer.



The advantage of using a texture object is that the results of the render operations will be saved in a texture format, making them easily accessible for processing in shaders.



The process of creating a texture for use in the frame buffer roughly coincides with that for a regular texture object:

 unsigned int texture; glGenTextures(1, &texture); glBindTexture(GL_TEXTURE_2D, texture); glTexImage2D(GL_TEXTURE_2D, 0, GL_RGB, 800, 600, 0, GL_RGB, GL_UNSIGNED_BYTE, NULL); glTexParameteri(GL_TEXTURE_2D, GL_TEXTURE_MIN_FILTER, GL_LINEAR); glTexParameteri(GL_TEXTURE_2D, GL_TEXTURE_MAG_FILTER, GL_LINEAR); 

The main difference is that texture sizes are set equal to the screen size (although this is not necessary), and instead of a pointer to an array of texture values, NULL is passed. Here we only allocate memory for the texture, but do not fill it with something, since the filling will occur by itself when the render is directly called into this frame buffer. Also note the lack of texture repeat mode settings and mipmapping settings, since in most cases the use of offscreen buffers is not required.

If you want to render the entire screen into a smaller or larger texture, you must additionally call glViewport before directly rendering it , and transfer the dimensions of the used texture to it. Otherwise, only a fragment of the screen image gets into the frame buffer, or the frame buffer texture will be filled with the screen image only partially.

Having created a texture object, you need to attach it to the frame buffer:

 glFramebufferTexture2D(GL_FRAMEBUFFER, GL_COLOR_ATTACHMENT0, GL_TEXTURE_2D, texture, 0); 

The function takes the following parameters:

• target - the type of the frame object to which the texture is connected (read only, write only, read / write).
• attachment - the type of attachment that we plan to connect. In this case, we connect the color attachment. Note the 0 at the end of the attachment identifier - its presence implies the ability to connect more than one attachment to the buffer. More this moment is considered later.
• textarget is the type of texture you plan to mount.
• texture is the texture object itself.
• level - used for outputting the MIP-level.

In addition to the color attachments, we can also connect the depth and stencil textures to the frame buffer object. To attach a depth, we set the attachment type GL_DEPTH_ATTACHMENT . Do not forget that the format and internalformat parameters of the texture object must take the value GL_DEPTH_COMPONENT to be able to store the depth values ​​in the appropriate format. To attach a stencil, the type is set to GL_STENCIL_ATTACHMENT , and the texture format parameters are set to GL_STENCIL_INDEX .



It is also possible to connect both the depth buffer and the stencil simultaneously using only one texture. For this configuration, each 32-bit texture value consists of a 24-bit depth value and 8 bits of stencil information. To connect the depth buffer and the stencil as a single texture, the attachment type GL_DEPTH_STENCIL_ATTACHMENT is used , and the texture format is configured to store the combined depth and stencil values. An example of connecting the depth buffer and stencil in the form of a single texture is shown below:

 glTexImage2D( GL_TEXTURE_2D, 0, GL_DEPTH24_STENCIL8, 800, 600, 0, GL_DEPTH_STENCIL, GL_UNSIGNED_INT_24_8, NULL ); glFramebufferTexture2D(GL_FRAMEBUFFER, GL_DEPTH_STENCIL_ATTACHMENT, GL_TEXTURE_2D, texture, 0); 

Rendered Buffer Attachments

Chronologically, the rendered buffer objects as another type of frame buffer attachments were added to the library later than textural ones, which were the only option for working with offscreen buffers in ancient days. Like the texture, the renderbuffer object is a real buffer in memory, i.e. an array of bytes, integers, pixels, or something else.



However, it has an additional advantage - the data in the render buffer is stored in a special, comprehensible library format, which makes it optimized for an off-screen render.



Render objects save the render data directly, without additional transformations into specific texture data formats, which ultimately gives a significant advantage in speed during writing to the buffer. Unfortunately, in a general sense, the render buffer is write-only. You can read something from it only indirectly, through a call to glReadPixels , and then this will return the pixel data of the currently used frame buffer, and not the renderbuffer itself.



Since data is stored in an internal library format, renderbuffers are very fast when writing to them or when copying their data to other buffers. Buffer switching operations are also quite fast when using renderboover objects. So, the function glfwSwapBuffers , which we used at the end of each render cycle, can also be implemented using renderboover objects: write to one buffer, then switch to another after rendering is completed. In such tasks, the renderbuffer is clearly on horseback.



Creating a renderbuffer object is quite similar to creating a frame buffer object:

 unsigned int rbo; glGenRenderbuffers(1, &rbo); 

Expectedly, we need to bind the renderboover object so that subsequent rendering operations direct the results to it:

 glBindRenderbuffer(GL_RENDERBUFFER, rbo); 

Since the render buffer objects are generally not available for reading, they are often used to store depth and stencil data - for the most part we don’t often need specific values ​​of these buffers, but in general we need their functions. More precisely, we need a depth buffer and a stencil for the corresponding tests, but we do not plan to make a sample of them. In cases where the selection of buffers is not planned, the render buffer is an excellent choice, because the bonus is also a great performance.



Creating a Render Buffer for Depth and Stencil:

 glRenderbufferStorage(GL_RENDERBUFFER, GL_DEPTH24_STENCIL8, 800, 600); 

Creating a renderboover object is similar to texture objects. The only difference is that the renderbuffer was designed for direct storage of the image, unlike the general purpose buffer, which is a texture object. Here we specify the internal format of the buffer GL_DEPTH24_STENCIL8 , which corresponds to 24 bits per depth value and 8 bits per stencil.



Do not forget that the object must be connected to the frame buffer:

 glFramebufferRenderbuffer(GL_FRAMEBUFFER, GL_DEPTH_STENCIL_ATTACHMENT, GL_RENDERBUFFER, rbo); 

Using a render buffer can give some performance benefit to processes using offscreen buffers, but it is important to understand when to use them and when to use textures. The general principle is this: if you never plan to make selections from the buffer, then use the renderboover object for it. If at least sometimes you need to make a selection from the buffer, such as, for example, the color or depth of a fragment, then you should refer to the texture attachments. In the end, the performance gain will not be huge.

Render texture

So, armed with the knowledge of how (in general) the personnel buffers work, we begin to use them directly. Let's try to render the scene in the texture attachment of the frame buffer, and then draw one full-screen quad using this texture. Yes, we will not see any differences - the result will be the same as without the use of a frame buffer. What is the profit of such a venture? Wait for the next section and find out.



To begin, create a frame buffer object and tie it right there:

 unsigned int framebuffer; glGenFramebuffers(1, &framebuffer); glBindFramebuffer(GL_FRAMEBUFFER, framebuffer); 

Next, we will create a texture object, which we will attach to attaching the frame buffer color. Again, we set texture sizes equal to the size of the application window, and leave a pointer to the data empty:

 //    unsigned int texColorBuffer; glGenTextures(1, &texColorBuffer); glBindTexture(GL_TEXTURE_2D, texColorBuffer); glTexImage2D(GL_TEXTURE_2D, 0, GL_RGB, 800, 600, 0, GL_RGB, GL_UNSIGNED_BYTE, NULL); glTexParameteri(GL_TEXTURE_2D, GL_TEXTURE_MIN_FILTER, GL_LINEAR ); glTexParameteri(GL_TEXTURE_2D, GL_TEXTURE_MAG_FILTER, GL_LINEAR); glBindTexture(GL_TEXTURE_2D, 0); //        glFramebufferTexture2D(GL_FRAMEBUFFER, GL_COLOR_ATTACHMENT0, GL_TEXTURE_2D, texColorBuffer, 0); 

We would also like to be able to conduct a depth test (and a stencil test, if you need it), so let's not forget about the task of attaching a depth (and stencil) to our personnel buffer. Since we plan to make only samples from the color buffer, we can use the render buffer as a data carrier for depth and stencil.



Creating a renderbuffer object is trivial. It is worth remembering only that we are going to create a combined depth and stencil buffer. Therefore, we expose the internal format of the renderboover object in GL_DEPTH24_STENCIL8 . For our tasks, 24-bit depth accuracy is enough.

 unsigned int rbo; glGenRenderbuffers(1, &rbo); glBindRenderbuffer(GL_RENDERBUFFER, rbo); glRenderbufferStorage(GL_RENDERBUFFER, GL_DEPTH24_STENCIL8, 800, 600); glBindRenderbuffer(GL_RENDERBUFFER, 0); 

As soon as we have requested a memory for an object, we can untie it.

Then we attach the renderbuffer object to the combined point of attachment of the depth and stencil of the frame buffer:

 glFramebufferRenderbuffer(GL_FRAMEBUFFER, GL_DEPTH_STENCIL_ATTACHMENT, GL_RENDERBUFFER, rbo); 

The final chord is to check the frame buffer for completeness with a debug message, if it is not:

 if(glCheckFramebufferStatus(GL_FRAMEBUFFER) != GL_FRAMEBUFFER_COMPLETE) std::cout << "ERROR::FRAMEBUFFER:: Framebuffer is not complete!" << std::endl; glBindFramebuffer(GL_FRAMEBUFFER, 0); 

Do not forget to unbind the frame buffer object at the end in order not to accidentally start the render in the wrong direction.



Well, we have a frame buffer object, fully prepared for rendering into it, instead of the default frame buffer. All that remains to be done is to bind our buffer and all subsequent render commands will affect the bound frame buffer. All operations with the depth and stencil buffers will also use the appropriate attachments of the currently attached frame buffer (if you have, of course, created such). If, for example, you forgot to add a depth buffer to the frame buffer, then the depth test will no longer work, since there will simply be no source data for it in the frame buffer.



So, we list the steps required to output a scene to the texture:



1. Bind our frame buffer object as current and output the scene in the usual way.

2. Bind the frame buffer to the default.

3. Display a full-screen quad with texture overlay from the color buffer of our frame buffer object.



We will draw the scene taken from the lesson about the depth test, but this time using the familiar texture of the container.



To display full-screen quad, we will create a new set of trivial shaders. There will not be any intricate matrix transformations, since we will immediately transfer the coordinates of the vertices to them in the form of normalized device coordinates ( NDC ). Let me remind you that in this form they can be immediately transferred to the output of the fragment shader:

 #version 330 core layout (location = 0) in vec2 aPos; layout (location = 1) in vec2 aTexCoords; out vec2 TexCoords; void main() { gl_Position = vec4(aPos.x, aPos.y, 0.0, 1.0); TexCoords = aTexCoords; } 

Nothing fancy, is it? The fragment shader will be even simpler, since all that he does is fetching from the texture:

 #version 330 core out vec4 FragColor; in vec2 TexCoords; uniform sampler2D screenTexture; void main() { FragColor = texture(screenTexture, TexCoords); } 

The code responsible for creating and configuring VAO for the quad itself remains on your conscience. The iteration of the render has the following structure:

 //   glBindFramebuffer(GL_FRAMEBUFFER, framebuffer); glClearColor(0.1f, 0.1f, 0.1f, 1.0f); glClear(GL_COLOR_BUFFER_BIT | GL_DEPTH_BUFFER_BIT); //     glEnable(GL_DEPTH_TEST); DrawScene(); //   glBindFramebuffer(GL_FRAMEBUFFER, 0); //      glClearColor(1.0f, 1.0f, 1.0f, 1.0f); glClear(GL_COLOR_BUFFER_BIT); screenShader.use(); glBindVertexArray(quadVAO); glDisable(GL_DEPTH_TEST); glBindTexture(GL_TEXTURE_2D, textureColorbuffer); glDrawArrays(GL_TRIANGLES, 0, 6); 

A few comments. First, since the created frame buffer object has its own set of buffers, it is necessary to clear each of them by setting the appropriate flags for the glClear function. Secondly, in the derivation of quad, we disable the depth test, since it is redundant when rendering a simple pair of triangles. However, the test should be enabled when directly rendering the scene itself.



Phew! Decent stages of work in which it is easy to make a mistake. If your program does not display anything - try to debug where it is possible, and also re-read the relevant parts of this lesson. If everything worked successfully, the output will be similar to this result:





On the left, the result is identical to the image from the lesson on the depth test, but this image is displayed on a full-screen quad. If you switch the render mode to skeleton (glPolygonMode (GL_FRONT_AND_BACK, GL_LINE) - enter the mode, glPolygonMode (GL_FRONT_AND_BACK, GL_FILL)), you can see that a pair of triangles is drawn into the frame buffer by default .

The source code of the example is here .



Well, what is the use of all this? Since we now have a texture with the contents of the finished frame, we can easily access the value of each pixel and implement many intricate effects in the fragment shader! Collectively, this approach is called post-processing or post -processing .

Post processing

Having a texture that contains an image of the entire frame, we can implement a variety of effects using simple operations with texture data. In this section, some common postprocessing techniques will be demonstrated, as well as ideas on how to make your effect with a little imagination.



Let's start with the simplest effect.

Color inversion

Since we have full access to the color data of the final frame, in a fragmentary shader, it is easy to get the color value opposite to the original one. To do this, take a sample of color from the texture and subtract from the unit:

 void main() { FragColor = vec4(vec3(1.0 - texture(screenTexture, TexCoords)), 1.0); } 

Inversions of color, despite the simplicity of the effect, can bring quite entertaining results:





All the colors in the scene were inverted with just a single line of code in the shader, not bad, huh?

Grayscale translation

Another interesting effect is the removal of all color information with the image being converted to grayscale. A naive solution is obvious; it is enough to sum up the brightness values ​​of each color channel and average it, replacing the original values ​​with the average:

 void main() { FragColor = texture(screenTexture, TexCoords); float average = (FragColor.r + FragColor.g + FragColor.b) / 3.0; FragColor = vec4(average, average, average, 1.0); } 

The results of this approach are quite acceptable, but the nature of the human eye implies greater sensitivity to the green part of the spectrum and less to the blue. So a more physically correct reduction to grayscale uses color averaging with weighting factors for individual channels:

 void main() { FragColor = texture(screenTexture, TexCoords); float average = 0.2126 * FragColor.r + 0.7152 * FragColor.g + 0.0722 * FragColor.b; FragColor = vec4(average, average, average, 1.0); } 

At first glance, the difference is not obvious, but in more saturated scenes weighted reduction to grayscale gives a better result.

Application of convolutional core

Another advantage of post-processing using a texture map is the fact that we can access any part of the texture. For example, take a small area around the current texture coordinate and sample values ​​around the current texel. And combining the values ​​of these samples is easy to create certain special effects.



A convolutional core (convolution matrix) is a small array of magnitudes like a matrix, the central element of which corresponds to the current pixel being processed, and its surrounding elements to adjacent texture texels. During processing, the core values ​​surrounding the central one are multiplied by the values ​​of the samples of adjacent texels, and then everything is added together and written into the current (central) texel. By and large, we simply add a small offset of the texture coordinates in all directions from the current texel and calculate the final result using values ​​from the core. Take, for example, the following convolution kernel:

$$

$$\begin{bmatrix}2 & 2 & 2 \\ 2 & -15 & 2 \\ 2 & 2 & 2 \end{bmatrix}$$

This core multiplies the values ​​of neighboring texels by 2, and the current texel by -15. In other words, the kernel multiplies all neighboring values ​​by the weighting factor stored in the kernel, and “equalizes” this operation by multiplying the value of the current texel by the large negative weighting factor.

Most convolutional matrices that you find in the network will have the sum of all coefficients equal to 1. If this is not the case, then the image after processing will either become brighter or darker than the original.

Convolutional kernels are an incredibly useful tool for creating post-processing effects, as they are fairly simple to implement, easy to experiment with, and many ready-made examples are already available on the net.



To support the convolutional kernel, we will have to modify the fragment shader code a little. We assume that only 3x3 kernels will be used (most of the known kernels do have this dimension):

 const float offset = 1.0 / 300.0; void main() { vec2 offsets[9] = vec2[]( vec2(-offset, offset), // top-left vec2( 0.0f, offset), // top-center vec2( offset, offset), // top-right vec2(-offset, 0.0f), // center-left vec2( 0.0f, 0.0f), // center-center vec2( offset, 0.0f), // center-right vec2(-offset, -offset), // bottom-left vec2( 0.0f, -offset), // bottom-center vec2( offset, -offset) // bottom-right ); float kernel[9] = float[]( -1, -1, -1, -1, 9, -1, -1, -1, -1 ); vec3 sampleTex[9]; for(int i = 0; i < 9; i++) { sampleTex[i] = vec3(texture(screenTexture, TexCoords.st + offsets[i])); } vec3 col = vec3(0.0); for(int i = 0; i < 9; i++) col += sampleTex[i] * kernel[i]; FragColor = vec4(col, 1.0); } 

9 vec2 , . , . , . , . , , , .

:





, .

,

$$

$$\begin{bmatrix} 1 & 2 & 1 \\ 2 & 4 & 2 \\ 1 & 2 & 1 \end{bmatrix} / 16$$

Since the total amount of elements is equal to 16, it is necessary to divide the result by 16 to avoid an extreme increase in brightness. Kernel Definition:

 float kernel[9] = float[]( 1.0 / 16, 2.0 / 16, 1.0 / 16, 2.0 / 16, 4.0 / 16, 2.0 / 16, 1.0 / 16, 2.0 / 16, 1.0 / 16 ); 

Changing the elements of the array of numbers representing the core itself led to a complete transformation of the image:





The blur effect has ample opportunities to apply. For example, you can change the amount of blur over time to simulate the intoxication of the character, or lift up the amount of blur in scenes where the hero forgot to wear glasses. Also, blurring allows you to make color transitions smooth, which will be useful in subsequent lessons.

I think it is already clear that by preparing the code for using the convolution kernel, you can easily and quickly create new post-processing effects. In conclusion, we will deal with the last of the most popular convolutional effects.

Definition of boundaries

Below is the core to identify the boundaries:

$$

$$\begin{bmatrix} 1 & 1 & 1 \\ 1 & -8 & 1 \\ 1 & 1 & 1 \end{bmatrix}$$

It resembles the core for sharpening, but in this case highlights all the borders in the image, while shading the rest of the parts. It is very useful if you are only interested in borders in the image:





I think you will not be surprised by the fact that convolutional kernels are used in image processing programs and filters, such as Adobe Photoshop. Pixel-based modification of images in real time becomes quite accessible due to the outstanding speed of parallel processing of fragments. That is why lately graphic packages are increasingly using the capabilities of video cards in the field of image processing.



PS From comments to the original: an excellent interactive demonstration of various bundles.

PPS : We have a telegram-konf to coordinate transfers. If there is a desire to fit into the cycle, then you are welcome!