UI components on pixel shaders: write your first shader
Who can be called "pixel shaders chief and pixel commander"? Denis Radin , working in Evolution Gaming on photo-realistic web games using React and WebGL: he is known to many just by the name Pixels Commander.
In December, at our HolyJS conference, he gave a talk on how using GLSL can improve work with UI components as compared to “regular javascript”. And now for Habr we have prepared a text version of this report - welcome under the cut! At the same time we attach a video of the performance:
First, a question to the audience: how many languages ​​are well supported on the web? (Voice from the audience: “Not one!”) Well, languages ​​in the browser, let's say. Three? Let's assume that there are four of them: HTML, CSS, JS and SVG. SVG can also be considered a declarative language, another type, it is still not HTML. ')
But in fact they are even more. There is VRML, he died, it can not be considered. And there is GLSL (“OpenGL Shading Language”). And GLSL is a very special language for the web.
Because the rest (JS, CSS, HTML) originated on the web, and from web pages they started a victorious march on other platforms (for example, mobile). And GLSL originated in the world of computer graphics, in C ++, and came to the web from there. And what's great about it: it works wherever OpenGL works, so if you learn it, you can use it anywhere (in Unity, Swift, Java, and so on).
GLSL is behind crazy special effects in computer games. And I like that it can be used to develop interesting and unusual UI components, later we will talk about them. It is also a technology for parallel computing, which means that you can mine cryptocurrency using GLSL. What are interested?
Story
Let's start with the history of GLSL. When and why did he appear? This chart displays the OpenGL rendering pipeline:
Initially, in the first version of OpenGL, the rendering pipeline looked like this: vertices are fed to the input, primitives are gathered from the vertices, the primitives are rasterized, clipping occurs and then the framebuffer is output.
There is a problem here: it is not customized. Since we have a well-defined pipeline, textures can be loaded there, but you can’t do anything special with any exact request.
Let's look at the simplest example: draw a fog. There is a scene. It all consists of vertices, texture is applied to them. In the first version of OpenGL, it looked like this:
How can I make a fog? In the formula, fogvalue is the distance to the camera multiplied by the fog density, and the pixel color is equal to the current pixel color multiplied by the fog color and the amount of fog. If we perform this operation for each pixel on the screen, we will get the following result:
GLSL shaders appeared in 2004 in OpenGL v2, and this was the biggest breakthrough in the history of OpenGL. It appeared in 1991, and now, after 13 years, the next version was released.
From now on, the rendering pipeline began to look like this:
Vertices are fed to the input, here the first vertex shader is executed, which allows you to change the geometry of the object, then primitives are built, they are rasterized, then a fragmentary shader is executed (“fragmentary” means “pixel”, in English terminology the “fragment shader” is often used), then clipped and displayed on screen.
Okay, let's talk about some of the features of GLSL, because it has so many, many things that are unusual and sound strange, for JS developers so accurately.
Well, first, what is important for us, for web developers: GLSL is part of the WebGL specification. The gateway to GLSL will be <canvas />.
GLSL is compiled using a GPU driver. Because of this, it is cross-platform, since it is compiled for each specific platform, and it is amazingly fast. It is very fast, thousands of times faster than JavaScript, because it compiles specifically for the platform and runs on a special piece of hardware for which it was intended.
At the same time, it is launched into many processes, for example, a card; if you follow the news of iron, a GTX 970 card, it simultaneously runs 1664 shader processes. Imagine how much you can mine?
In general, this way the mining is performed, and everything else, all parallel computations are CUDA platforms, they work through shaders. They come in many forms, not always GLSL, but on the web we have GLSL shaders, part of the OpenGL specification.
There are certain features associated with the fact that data is sent only once. Since the execution is parallel, for the entire passage, for the entire screen, the data is loaded into the shader once, and this should be taken into account.
GLSL is a strongly typed language. There are types float, integer, boolean, vectors 2-3-4 component (which, in fact, are 2-3-4-element arrays), 2-3-4-dimensional matrices also exist.
It is a language sharpened by mathematics, and it has all the wonderful trigonometric and mathematical functions that you can imagine: radians to degrees, degrees to radians, sine, arc cosine, tangent, matrix multiplication, multiplication of vectors, various derivatives, etc. .
Practice
Good. From theory let's move on to practice. Consider the simplest pixel shader.
First we set the radius of the circle, it is a variable of type float. Then the center is a two-component vector. Notice that the origin in GLSL is not in the upper left, but in the lower left. You can play around a bit with the coordinates, move this circle somewhere.
Then comes the main function, which is the entry point to any shader. It first calculates the distance to the center using the built-in function distance, the coordinates of the current pixel and the coordinates of the center.
Further, the inCircle float variable is calculated: if our pixel is inside the circle, it is equal to one, and if outside it is zero.
And the last operation is the outgoing parameter gl_FragColor, which determines the color that will be on the screen. We assign him the above inCircle. That is, if we are inside a circle, there will be one here.
This is a very interesting shorthand: a four-component vector is created from one float variable, the value of this variable is assigned to all the components of the vector at once. It uses RGBA notation, that is, four components are RGB and alpha channel.
And you can change it like this:
What's going on here? We assign the resulting value not to all channels at once, but only to green.
Okay, let's move on from the simplest example, which is practically useless, to solving a practical problem. One day, on a sad Amsterdam autumn morning, I received a task at JIRA, which made my life a little more fun.
The task was about spinner. We wrote an operating system on JS, and we had such a cool spinner in the OS. It worked, but there was one small problem: when there was some kind of background-process, the spinner sometimes twitched. I was asked to figure it out.
I began to dig and saw that the spinner was implemented using a sprite sheet: the element had a background-position change, and all these frames were scrolling.
In principle, it worked, but you probably know that if the background-position changes, what happens? Repaint. There is a constant ripaint, and it loaded the processor, it did not work very fast.
How can I fix this? It is possible through CSS. Naturally, I did not immediately go into the wilds of GLSL, at first we did it all under the simplest possible way, through CSS, through hardware-accelerated properties. Many of you know that there are hardware-accelerated properties that allow you to perform some kind of animation without repainting. Here it can all be changed to opacity, that is, with the background-position we move to opacity.
How can this be done using opacity? To decompose all frames into layers and using opacity to gradually hide them and show, in general, the same effect is obtained, but without any repaints. Hurray, QA-department confirmed the increase in speed, everyone is happy.
The next day I received another task at JIRA. Denis, we know that you are already an expert in spinners, we have the exact same spinner, only blue and a bit different width.
I knew that there were many spinners, and I realized that there was a small problem there. Firstly, this 150-frame spinner in video memory unfolds at more than 8 megabytes, I specifically counted on the resolution and bit depth of these textures (because for each frame, a texture is created as a result. And it takes 10 megabytes in RAM. download 100 kilobytes. In general, each spinner costs about 20-30 megabytes, considering that it needs to be released. For a spinner, 30 megabytes is, frankly, a lot. If there are 3-4 of them, this is 100 MB of RAM per spinner.
We had a limit of 256 megabytes in the browser: as soon as they were reached, the entire system collapsed. I think on mobile phones even 100 megabytes per spinner is also an unaffordable luxury.
Okay, I get it, we have a problem. It can be solved with GLSL. As far as pragmatic, we will analyze later.
Write a shader
And now we can write a pixel shader together. Spinner can be represented as an animated arch, which collapses and splits open: it changes the angle of beginning and end, and this arch rotates with a certain periodicity, attenuation, acceleration. Therefore, we need to learn how to draw an arch in GLSL with the help of mathematics.
First, install the Chrome Refined GitHub extension, it needs to copy and paste diff from commits. If you do not put it, you will have to copy line numbers when you try to copy the text of the diff, and you will have to delete them manually. Therefore, Refined Github helps a lot: it lists line numbers in a separate list, and it's cool.
What we start with is a copy-paste in the GLSL editor, the first step, thanks to which we will have a circle:
What's going on here? At the top of the new unit, it was not in the past example, here comes a uniform variable. "Uniform" means a variable sent from javascript. We see u_time, u_resolution, and u_mouse from JavaScript here. The most interesting of them is u_resolution. What she says is the dimension of the canvas. JavaScript took the dimension of the canvas and sent us a two-component vector in GLSL, now we know the size of the canvas in GLSL.
In PI, we determined the number pi, so as not to write it constantly with our hands. Then u_resolution was multiplied by 0.5: this is a two-component vector (where width and height are there), and when multiplying a vector by 0.5, all its components are multiplied by 0.5 at once. So we found half of our dimension. After that, we took the radius as the minimum of the width and height.
Now we have the Circle function: earlier we simply defined in main, whether we were lying inside the circle or not, but now we took it into a separate function, where we start the current pixel coordinate, center and radius.
And in main we get isFilled as a result of the execution of the Circle function, and subtract from the isFilled unit, because we want it not to be a white circle, but a background. That is, inverted all this wealth.
Now the second step : we will cut the sector on the circle.
We add a function that draws a sector, and a function that says whether the angle lies between two given angles. And besides, isFilled, we are now doing the product of the results circle and sector. If in both cases the unit, then we are within our figure. If circle were not taken into account, then the sector would be infinite, not bounded by a circle. The result is as follows:
This is where a new arch feature is added. Now we need to know its thickness, for this we will calculate the internal radius and what do we see?
Now we have isFilled - this is the result of the execution of the arc function, with which we transfer the starting angle, the ending angle, the inner and outer radii. Here it is all built inside on the sector that we already have, and on the two functions of the circle, which invert each other. That is, two circles are cut off, one hides the other.
Everything is great, everything is good, we have an arch, we are almost ready, but if you take a closer look, and I will try to help you, then you will notice that the arch is pixelated, there are "cloves" without smoothing:
This is because, there is no anti-aliasing, this is because when we draw a circle, we use the step function here, when we determine whether a point lies in a circle or not, and the step function hard cuts off discretely, 0 or 1 if the value is lower given, then this is 1, if higher is 0. Accordingly, our pixel can be either black or white.
Let's get rid of it, this will be step 4. Add anti-aliasing.
We replace the step function with smoothstep. A smoothstep does not just say “either 0 or 1”, but interpolates between two values. Here we have “distanceToCenter minus two pixels” and there is just distanceToCenter, that is, we have anti-aliasing by smearing 2 pixels. Here you can argue about the terms, but really we just added anti-aliasing to our shader.
And the arch became smooth and silky.
We now turn to the most difficult - to the animation. To draw an arch is, in general, class 5 trigonometry, and there is nothing complicated. With animation, everything is a bit more complicated, because you first need to recognize and decompose it.
Decomposing the animation of the spinner, we find that there are actually two animations. One is the flapping-rapping animation, and the second is the rotation animation. In addition, at the beginning of the cycle, the animation accelerates, and at the end it slows down. This is very similar to the behavior of the sine function: in the interval from - pi / 2 to pi / 2, acceleration first takes place, sharply soars up, and then slows down.
Step five. We will apply this function to our corners of the beginning and end of the arch. We get a collapse-splash animation, albeit a little blunt for now (let's fix this). What's going on here? Time closes in the period from - pi / 2 to pi / 2, then the sine function is applied to it, and all the time we get a value from zero to one - as far as we collapse-slammed. That is, in fact, the easing function is used here, this is what in tweens, in CSS is used everywhere, here it is implemented on GLSL. Then we multiply 360 by the result of the execution of this easing-function and get the angle of the beginning, the angle of the end, which we transfer to the function of the arch, which we wrote earlier.
The next step is to rotate the whole spinner.
With rotation, everything is simple, we already have a theoretical base prepared, we know that the sine rules and we add to startAngle and endAngle the value, which is obtained, again, from the sine, but with a twice as long period, because we have two slamming-flapping turns out just one turn.
Thus, we received a spinner, which is almost in line with our technical task. It remains to add a little parameterization:
To do this, you need the RGB function. It is not necessary to use it, but it’s good, because we usually take colors from Photoshop, and they have byteable channel values ​​from 0 to 255, you saw that in GLSL from 0 to 1, and this function allows you to send to it we are familiar with 255/255/255 and get 1/1/1 at the output.
This function is used in main, and there is also added customization of the background, just in case of fire.
The result was a wonderful animated vector spinner, in which you can change the width and color. The component is ready, it works, it is rendered on the GPU, and all this stuff takes 70 lines of code. If you lean on, you can probably shrink up to 5 lines, which, of course, cannot be compared with the amount of information we transmitted in the sprite sheet - just heaven and earth. If we have 30 megabytes there were just pictures, plus we need to initialize the same contexts for textures and so on, then there is an obvious progress.
What can we do about it
How to use the GLSL component in your web application? As already mentioned, this is done through the WebGL context.
There is a simple way. There is a web component called the GLSL component, and you put it in the right place on your page, this tag, inside put this GLSL code that we got in the editor. And you get in the amount that you have this block, you get your GLSL component that works online.
Previously, we implemented what could be done with a stretch on CSS through a sprite sheet or other tricks, though not always quickly. But in fact, shaders are much cooler: they give control over each pixel.
Here is the gif that shows the spinner reacting to the cursor:
And on the video you can see an even more impressive example of how GLSL gives disproportionately more features and allows you to manage each pixel. There the spinner has already become something else.
That is, by applying a fairly simple math, you can get some component, and after working a little more, we can add new unusual properties to it. The possibilities of pixel shaders, in fact, are endless and limited only by your knowledge of mathematics and your skills in writing shaders.
And what else is good for GLSL: in addition to these endless possibilities, it gives JavaScript developers, front-end developers, a breath of fresh air. You have been writing JavaScript for some years, you understand that you are good in it, and you want something new, but you don’t want to be in the backend. In this case, GLSL is a good way to change and diversify your life.
Thank you very much!
If you liked the report, please note: next week will be held HolyJS 2018 Piter , and there Denis will also speak , now with the topic “Mining crypto in browser: GPU, WebAssembly, JavaScript and all the good things to try”. And in the discussion area after the report, it will be possible to question him properly both on the topic of the new report and on shaders. In addition to Denis, there will be dozens of other speakers there - see all the details on the HolyJS website .