Record video from Mac OS X screen with open-source tools

Sometimes you need to record a demo of the program, but there are no suitable tools at hand. Moreover, free utilities for these purposes are not found at all, and the paid ones are not the fact that they will cope as they should. As they corrected me in comments, the regular version of QuickTime Player writes great videos. Further it is recommended to read only to fans of open-source and unusual solutions.
In my case, it became necessary to record the work of the game on the iPhone and Android simulator. The programmer inside me immediately offered to write a bunch of code, both for iOS / Android, and for Mac himself, dump the frames via OpenGL, etc. Having stopped these urges, I decided to find ready-made solutions and then issue an article here as a reminder.

Prelude

Searching the forums has shown that people with similar tasks fall into two categories: those who buy specialized programs with the ability to capture (QuickTime Pro, etc.) and those who use hacked / trial / beta versions of the same programs. Having dug a little deeper, I learned that in the Linux / Unix world there is no such problem at all since the addition of the x11grab module to ffmpeg. This module intercepts the buffer of X applications and then through ffmpeg the video is compressed into an intermediate format (qtrle, raw, x264 lossless, etc.). The Mac also has an X server, but only programs that are ported from Linux, but not adapted for Cocoa, are displayed through it. This was the starting point of my experiments.

Tools: ffmpeg + x11grab + Xvfb

In MacPorts, in ffmpeg, the x11grab module is basically missing. It didn't work out from the first time to collect it entirely from the source, so I decided to patch the port file for ffmpeg-devel:
/opt/local/var/macports/sources/rsync.macports.org/release/tarballs/ports/multimedia/ffmpeg-devel/Portfile
I added the --enable-x11grab and --enable-shared lines to the nonfree configuration to activate x11grab.
Looking ahead to say that there was a flaw in this method, because I do not post the corrected Portfile, but describe it here for reference only.
')
When the X program is displayed in the session (display): 0.0 by default, video recording gives a small frame rate, mainly due to the screen size (2560x1440 from the iMac), and most likely also because of the graphics rendering, so I decided to redirect programs in a virtual display with a low resolution. This is done through the Xvfb project, which was installed from the ports without any problems.
It runs quite simply through the terminal (size with a margin):
Xvfb :1 -screen 0 1024x800x15 -shmem

It also just connects to this virtual ffmpeg display:
ffmpeg -r 30 -s 1024x768 -f x11grab -i :1.0+0,20 -vcodec qtrle target.mov

At this stage, I was waiting for an error with the Shared Memory overflow, which for some reason in OS X is set to indecent value in 4mb. A temporary increase in its size is described in Apple’s recommendations for servers and other sources :
sudo sysctl -w kern.sysv.shmmax=67108864
sudo sysctl -w kern.sysv.shmall=67108864

Record VirtualBox Session

The next step I decided to display something useful through the X server. The first thought was to collect the X version of VirtualBox, and already there in the virtual machine to show literally anything, but VirtualBox for Macs had long ago migrated to Cocoa, because it was a dead end. The second sound idea was to connect via RDP to a virtual machine and record the rdesktop session, since it works under X. Activating RDP under VirtualBox is quite simple, but it requires installing the Extension Pack from the official site .
Connection rdesktop with output through the screen: 0.1
DISPLAY=:1.0 rdesktop -xl localhost

After these actions, ffmpeg starts writing a video to the .mov file quite successfully. In my case, this was a handy Android x86:


Unfortunately, the video is quite jerky, it affects the compression rdp, because the animation is not so good to shoot.

The next step I decided to move from RDP to VNC. VirtualBox has a VNC server built in, but not in public builds, but in collected from ports or sources. I didn’t have to do any port manipulations, after assembling the virtualbox port, I got version 4.1.14, which is quite possible to work with.

The unpleasant moment was only that VNC is not accessible through the interface, but only when launched in headless mode:
VBoxHeadless -startvm 'Android x86' -v on --vnc
This mode has to be managed either through the second VNC session or again via RDP, which is not very convenient, but generally tolerable. To capture the VNC stream, vncviewer was used, redirected to the virtual X display:
vncviewer localhost -ViewOnly -display :1.0 -PreferredEncoding raw -FullColor

The result of all these delights was a 5-minute video with an honest 30 fps at a resolution of 1024x768:

(I apologize in advance for the quality of the content, I'm still not a professional player)

If you look closely, there are sometimes pauses for a few seconds. Unfortunately, this problem did not manage to be won, and the approach itself came out rather cumbersome. For the simplest demonstration of the game in Android, this is generally sufficient, so I switched to the next task - shooting a video of the iPhone simulator.

Capture VNC for the entire screen

OS X has built-in remote access that works simultaneously under two protocols, ARD and VNC. Before the release of Lion 10.7, it was possible to turn on Screen Sharing in the system settings and connect to the current session with any VNC client. Starting from 10.7, serious changes began: all types of compression were thrown out, except for ZRLE, not all clients can connect, and after connecting we see a gray login screen, and only after entering the user's password we connect to the session. This is great for administrators, but for my task, on the contrary, it created only obstacles. The vncviewer program (also known as RealVNC) in the latest versions already knows how to connect to OS X, but does not know how to enter a user password, because this way also turned out to be a dead end.
As an alternative, I took the free VNC server from TestPlant (from the same osxvnc and also Vine). Version 3.0 with sourceforge is outdated, so you need to build a new one from source or take it from the TestPlant website

An unimportant bug of this server is that occasionally the client disappears with the error "unknown message type 131". It is treated by restarting the server.

With the already well-established bundle of ffmpeg + x11grab + Xvfb and vncviewer, it turned out to shoot the full-screen video of the current OS X session, where the iPhone simulator was launched:


The size of the virtual buffer, I obviously chose a smaller screen resolution to remove the upper left corner. The result came out quite decent, but unfortunately, with low FPS - the animation in the game frankly braked. Moreover, the pauses, which were quite rare with the schema from VirtualBox, became much more pronounced.
At this stage, I conducted several experiments, reassembled the VNC server and client from source, put the minimum screen resolution, but did not achieve a good result. After a few hours, it became clear that the server itself gives frames with some delay. After digging into the code, it turned out that the server intentionally pauses between screen updates:

  /* OK, now, to save bandwidth, wait a little while for more updates to come along. */ /* REDSTONE - Lets send it right away if no rfbDeferUpdateTime */ if (rfbDeferUpdateTime > 0 && !cl->immediateUpdate && !cl->needNewScreenSize) { pthread_mutex_unlock(&cl->updateMutex); usleep(rfbDeferUpdateTime * 1000); pthread_mutex_lock(&cl->updateMutex); } 

It turned out that the rfbDeferUpdateTime variable has an initial value of 40 ms, but it is completely controllable and is set via the command line. In the Vine Server itself there is a separate field for this:

I set the value with a margin of 15, which gives a maximum frame rate of 66 fps. After that, the lags stopped, but there were still noticeable pauses in the video. Theoretically, from such a video sequence they can be cut and collected something acceptable, but I wanted a more universal solution.

Ace up your sleeve: vnc2flv

Now I had a video stream in vnc format of excellent quality, it only remained to write it to a file. I again suppressed the primordially programmer desire to write my own dumper and found the project vnc2swf , and then its successor vnc2flv, in the depths of the Internet. Skeptical grabber on Python passed immediately after the first results - the program records video stream in lossless quality and WQHD resolution with 15+ fps! I run it without perversions with Xvfb, directly connecting to the VNC server:
flvrec.py -r 30 127.0.0.1

To increase fps, it is enough to reduce the resolution to 1280x720. What's interesting is that you can restart Vine VNC, it will pick up the current screen resolution, and then you can safely switch to the native resolution and start recording.
The installation of vnc2flv is very simple and is described on the website, there are no special pitfalls here.
The finished video can be processed in your favorite video editor, cut off the excess and convert it to the desired format. I use VirtualDub running under wine, but this is a matter of habit.


Here is the result of this whole epic:

The video is quite clear, without jerks and lags. Animation is recorded normally. As for me, it is quite possible to use to write literally anything from the Mac OS screen. There is literally just a cursor missing, but this can be experienced as it is.

Afterword: dead-end branches

I will list here the dead ends and rakes that I met, in case someone will look for their own method.

VLC Player - since OS X 10.7 can not record video from the screen
CaptureMe - crashes, writes blank video
SIMBL plugin for iPhone Simulator - requires modifying the version of the simulator in the code and rebuilding, does not write iPad video, in general did not work for me at all
osxvnc 3.0 c SourceForge - crash on 10.7
RealVNC Server - requires a license key
VirtualBox, video recording mode - died 3 years ago, not supported and not working
ffmpeg + x11grab is perspective and flexible, but it gives incurable pauses in the video. I cite a link to the blog, from which I underlined this idea.
Apple HTTP Live Streaming - does not contain streaming tools