Network Render in a Blender

picture for those in the tank

A couple of years ago I was asked to write an educational program on the network renderer in a blender.
I got a little carried away and got what I bring to your attention.
The review was not published until the end on the website blender3d.org.ua , although the pdf can be found there. Please save, maybe someone will be interested.

Network Render in a Blender

This guide was written to demonstrate the capabilities of the blender in this.
relatively new role for him.
For many years now, such programs as 3Dstudio have been used over the network for collaboration or rendering, but in Blender this opportunity appeared relatively recently. With the release of the short film Sintel, the authors demonstrated the render studio at which they worked when making the film, and also complained that for the network render they had to use third-party development instead of the built-in 2.5 Blender network render, since by that time it was unable to provide a studio with a stable and fast result.
Let's see how things are now, and at the same time compare the internal network render of Blender
with LuxBlend plug-in, which recently received OpenCL support.

Internal netrendering
')
With the advent of Blender 2.5, the need to have at least some network render within the program caused the netrender code to be instantly included in it, written, like most plugins, on Python by Martin Poirier (Martin Poirier).
The idea turned out to be quite good: to use the internal render of Blender, which was to guarantee the identity of the image, processed locally or remotely; centralization of control on the master server; fairly simple server management and quick, not complicated installation.
However, the first tests showed that much remains to be done, and at the moment the plug-in, though supplied inside the distribution, needs to be activated manually.

Installation

Netrender in a blender is divided into three components:



Client - Sending a job to the master server
Master - Getting work from customers
Slave - A master - dependent server that processes render frames.

Customer

This can be any machine with Blender installed; when the scene is ready, the ip and port of the master that receives the requests are configured, the number of frames required for rendering is set, then the task is sent to the master, which distributes it among the other servers.

The following options are present in the client interface:

Animation on network - the entire work is sent as soon as the first frames are ready, they are transferred to the client in .exr format for display in the editor.

Send job - sends all selected frames to the master server, after clicking this button, “Job send” should appear in the information window, it may take some time if your .blend and textures take up enough space. It will take even longer if the master is outside the local network and the channel to it is limited.
It may even seem that Blender is frozen, check the network traffic and make sure that the client has started transferring data over the network. This option immediately releases Blender for the next task, since it does not wait for the result of the wizard.

Send current frame job - Sends only the selected frame, but keep in mind that, as in other cases, you need to send all the information (.blend and textures), so it will take as much time as it does when sending all the work. . The result is not waiting, instantly freeing the client.

After the wizard accepts the work, the transfer of the entire file and textures to the free slave will begin, it follows that it is better to locate them in one local network, otherwise we may get a large expenditure of traffic and another point of potential timeout.
What we are trying to achieve is, by pressing a button, to continue working without thinking about what kind of server will be engaged in rendering, as well as problems that may arise.

From time to time, you can glance at the web-interface built into the wizard (you can access it by clicking on “Open Master Monitor” or typing in the browser (http: // host: ip) and retrieving the completed frames. There is no slavery in the work of the slave, this is the Blender distribution, which is passed through the file parameters and frame numbers to render, and as soon as the task is ready, it will transfer the result in .exr format to the master, .log file and accept the next batch of frames.
The master, on the other hand, is a much more complex code whose task, in addition to interacting with the client, is to distribute the work and its frames among the available slaves. The balance of the load has not yet been polished, and it is difficult to predict who exactly will get the next frame, nevertheless, in general, he copes with his task, at least I did not have to see idle slaves in accessible work.

Pick up the result

When several frames are ready, we can take them from the master, here at the moment there are at least two details, which, in my opinion, are worth telling.
Firstly, the blender works in the .exr format, in fact, this means that each finished frame will be saved in an open HDR photo format, which gives good post-processing capabilities, but is far from ideal in terms of size. Thus, our photo taken on a single remote server can very easily take more time to load, process and transfer than if we rendered it locally.
Say, our .blend file takes 1mb + 512kb for textures, it is 1.5mb only for “lifting” the work for the master, the master gives these 1.5mb for the slave (it’s still not possible to use a larger amount for one frame), the slave renders the render for 10 minutes and gives the .exr file to the master, if FullHD-like resolution was set, then except for the case of the machine of the 90s or a very good client-master connection, we will get a lot more later than if we used a local render.

On this note, let me make a small digression: the fact is that, first, the network render is most often used by studios on the local network, with a channel of at least 1 gb / s, which greatly changes the situation. Secondly, in many well-established programs for editing three-dimensional images, it is possible to use several machines when processing a single frame, there is a banal division of a picture into parts equal to the number of servers, or other methods are used that reduce the time spent on rendering in proportion to the number of servers involved .
Since this function is not yet available in the blender, obviously, advantageous results can be achieved only when animating scenes with a sufficient number of frames to compensate for delays in loading and receiving files. In addition, you should not forget about the possibility to completely unload the client after transferring the material to the render, which can be very useful if it is necessary to continue working on another project.

Instruments

For minimal installation, one client computer and one master / slave are required, since both functions can be implemented by one computer with the master and slave servers running at the same time. Render speed will directly depend on the speed of the slave machine.

pros

• Easy installation - you only need to activate the plug-in and settings, as it is done through the blender interface
• Full unloading of the client immediately after returning work
• Scalability up to the number of servers equal to the number of frames
• Identical to local render - there is no need to customize something with an eye to work on the network
• Theoretical compatibility between all systems. - You can easily change from windows to linux and send the work, while one of the Linux slaves has turned into a mac (by the way, this is the merit of the python on which blender plugins are written)
• Promising collective opportunities - theoretically, an unlimited number of clients can theoretically send work to one master, the master himself will distribute them among the available slaves
• Priority tasks - when receiving a small work, the master will give it to one of the slaves as soon as he finishes a chunk-size portion of frames, even if his turn is not empty
• We take only the shots we need as they are ready from any browser or blender.
• The ability to pick up only .jpg frames without loading the channel, leaving heavy .exr files on the server

Minuses

• Some difficulties when working on the Internet - if the connection with the master is interrupted, you have to start the transfer again, removing the unsuccessful attempts.
• Arbitrary timeouts - (in recent versions there are noticeably fewer)
• There is no possibility of multiple slave machines working on one frame.
• Nuances with external files - (for example, textures) when working together on different operating systems. There have been annoying errors when connecting Linux to Win and Mac.
• Some functions do not work yet - (from examples you can mark video instead of texture)
• Before the rendering, the slave will be idle for some time, doing preprocessing (identical to the usual renderer, this is a minus only when compared with deluxeder)
• When the master stops, all the slave servers instantly fall off and have to be lifted by hands, this is all the more strange when you consider that when lifting the slave it will humbly wait for the connection regardless of whether it finds the master right away or not
• Any change in the settings of the master or server slave entails the pre-setting of these in the blender and then copying the finished .blend files to each server
• If the master server process aborts, we will lose the last job; it is necessary to complete the wizard correctly so that the render can continue from the same place or pick up the result without direct intervention in the file structure (all finished frames in the .exr format will most likely be in place)

Lux-Render

Luxrender is of interest in this review only to compare its capabilities with the capabilities of the internal network render in a blender, and there are many advantages to its advantages and disadvantages.
Starting with version 0.8, the suite has the opportunity to use the power of modern video cards to calculate the intersections of light through OpenCL.
The network render in the suite is promoted under the slogan “CPU vs. GPU? CPU + GPU + network render is better ”, in the future, the developers promise to expand the use of video cards in the renderer, for now you shouldn’t hope for a miracle, the video card is unlikely to load more than 30%, although you can already observe a certain performance increase now.
Unlike the internal network render, the Suite can combine several servers to work on one frame, the master server is not provided, the client partially performs its functions. The whole system is reduced to one client and several slave servers, as soon as after using the suites of the plug-in work gets on the slave, it calculates several samples. The result will be transferred back to the client after a certain period of time, where it will intersect with the result of the local render, and at the output we will get a picture with more samples than if the slave had not participated in the process.
Tools: One client and one server, while they work in pairs and theoretically provide linear performance gains.



pros

• Combining any number of slave machines to work on one frame
• A visual representation of the degree of completion of the overall work on the frame
• The ability to alternate the rendering of several frames for animation
• The ability to use the power of not only the CPU, but also the video card of each slave
• If necessary, you can save the result and continue rendering later.

Minuses

• The client cannot be unloaded, as soon as it is stopped, the deluxe server will clear the queue and go on waiting for a new client.
• The selfish system of collective work "who first got up - that and sneakers", two clients for one slave can not connect
• With a large number of servers, adding them to the list can be compared with the work of the hosts file, which is at least inconvenient
• If, without waiting for the work to finish, after pressing the quit button to close the client, a particularly loaded slave may fall off, it will be necessary to restart manually, otherwise it will continue to wait for the nonexistent client

Installation

Taking into account the fact that the network render of the suite will be used in some cases, when it is necessary to get a picture with ray tracing, and in others the internal network render, we can supply both.

Installation of Luxrader

First we go to the site and download the tip.zip file, this is a version under development, if you have a blender version newer than 2.58, it’s better to use it rather than a stable build. Next, unpack the archive into the plugins folder and launch the blender, activate the suite in the plugins menu. We will need the suite itself, which can be taken here , it seems to me that Archiver is not better suited for our purposes, but this is to the taste.
Those who have already dealt with this renderer simply customize everything, not forgetting that OpenCL support is available only with the choice: “Render: Hybrid (CPU + GPU)”, “Surface integrator: Path” and in “Advanced > Light strategy: one ”, otherwise select options at your discretion and leave“ Sampler: Traditional CPU ”. Next, set the path to our render in “Path to LuXrender installation”, and we can assume that half the work has been done.
The server installation is described in detail on the wiki pages , the main thing is to open the corresponding port on the server for access from the outside, in principle this command is enough: luxconsole -s -p 10001 The suite will listen on TCP port 10001 (-p) in server mode ( -s), and we do not need more.
Returning to the client, we set up LuxRender networking by adding our server to the list, for example: hostname: 10001 or IP: port.
When you press F12, the luxury window should appear and in the Network tab we will see our server. There should be no connection errors in the Log tab.
We check the server load, if we see the processors under 100%, then as they say “water went to the hut” and the master will begin to give samples after a period of time specified in the “Upload interval” blender options. If the connection has not been established, it will be possible to find out about it from the logs, most likely the port on the wizard has not opened, we can check by running telnet on it and write something in the console, you should see the typed text on the server.

In short, the log contains entries similar to:

[192.168.2.3:10001] 00:00:36 - 6T: 3.67 S / p 78.79 kS / s 99% Eff 77.86 kC / s - Using CPU
Server: port, time (during which the server does the work), number of cores / tricks, number of samples per pixel, number of samples per second (in thousands), percentage of frame efficiency (average value between the number of samples per second and efficiency), the latter means Is it used on the OpenCL server for rendering?

Here is another example:

[192.168.2.4:10001] 00:06:38 - 4T: 25.92 S / p 51.20 kS / s 99% Eff 50.51 kC / s - Using CPU
Translate:
After 6:38 minutes of rendering, the server 192.168.2.4 on port 10001 managed to get 22.92 samples per pixel at a speed of 51.20 thousand samples per second and 99% of its efficiency is estimated at 50.51 thousand contributions per second using only the central processor.

Installing an internal network render

Everything you need to work is part of the blender, activate the netrender plugin in the settings, in the appropriate section, select the netrender and proceed to setup.
First, it is easier to configure the wizard, select the port, leave the servername empty so that you can also receive requests on new interfaces without having to configure everything again.
The temporary folder in Linux is better to give a separate partition on the hard disk, the main thing is not to forget to create it, otherwise the server will not start. We leave Broadcast turned on, if the master is on the local network, then the client will be able to find it himself, but to do this you will need to open another UDP port (8000). I do not recommend Clear on exit, it is better to be able to resume work, and saving on free space is not our task.
Further, users of graphics can safely click "Start service" and continue to configure the slave server.
The rest save the .blend file with the name master.blend, this is the configuration for our environment, it will be useful to us if the server does not have a graphical interface and there is no possibility to configure it. We give the finished file to the server, we also need the startup.blend file, the user profile is saved in it, without it, the blender simply does not understand what it needs to activate the network render plugin, copy it to the user’s directory: (.blender / blender version / config / startup.blend), then we can start the server itself from the blender folder: (./ blender -b master.blend -a)
It just means to start the blender (./blender), read the file (-b) and render all frames (-a). The last parameter is answered by the netrender himself, (-a) we write only for the start, and this does not mean that all frames will always be rendered.
After this, a legend should appear that the server is waiting for work, check by going to our ip: our port is the web interface of the wizard, through it we will be able to manage the work and monitor their progress, also it will allow us to quickly pick up the finished .jpg files but more on that later.
So, we start the blender, configure it to work as a Client in the network render settings, register the IP and port, or press the button with two arrows, if we have specified Broadcast on the wizard. As soon as we connect, we can send the work, in the browser window we will see that it was assigned a number and that it is in the Queued state - the waiting mode, in which the render does not occur, since the master is not suitable for this purpose, and we’ve not configured. Well, let's correct this omission.

Slave tuning

In the graphical interface, everything is simple, choose Slave in the options of the non-tender and find the master. The number of threads should be equal to the number of processor cores (owners of intel can experiment). We leave Auto-detect except for the cases when it is necessary to do something other than the render behind the slave, since in this case we will get a full load of all cores. Thumbnails are a separate topic, the fact is that if we have several slave machines, in the event of a crash it is more convenient to collect information from one place, and not to go to each server, it is better to leave this option off, Clear on exit in this case we can activate Since filling up the hard disk at every slave with duplicates is a dubious task.
If you need to start a slave on a server without a graphical interface, simply save the .blend file and upload it to the server, the startup method is identical to the master, it's easier not to get confused to use the name, say slave.blend.

Trial run

After connecting to the master, our slave should take his job and start rendering, after the number of frames specified by the “chunk-size” parameter in the client, he will give the finished part of the work to the master and you can look at the result via the web, or load it into the blender with the “get frame ”. There are many programs that allow you to download a list of files from the server, flashget under Windows or wget under linux, for the latter the command loads the entire list:
wget :/thumb_ _{ .. }.jpg
eg:
wget 192.168.2.3:10001/thumb_1_{1..20}.jpg
Via flashget: The right button on the icon> Add new> Add batch download> http://192.168.2.3:10001/thumb_1_ (*) ​​.jpg, then below we set the condition for frame loading.

Additional settings

In case of insufficient memory on the slave server, by increasing the number of “chunks” of the frame in “Render> Performance> Tiles”, you can achieve the release of part of the RAM by sacrificing increased idle time when loading the processor thread.
If the geometry on the scene or the speed of the server does not allow to render in less than five minutes, the wizard and the slave servers will most likely be disconnected.
To solve this problem, inside the file "blender-version-linux-glibc27-x86_64 / version / scripts / addons / netrender / master.py" we find the variable "self.slave_timeout" and set the value equal to the number of minutes that we expect to take the render " chink-size “frames of each slave (we naturally focus on the weak link).
If you need to see the general statistics of the work, you can use this script (only under * NIX):
blenderartists.org/forum/showthread.php?233628-Netrender-AWK-statistics-script

That's all, beautiful to all renders.

useful links

Official documentation of the network render
Note on Cycles network render support