Raspberry Pi: Encode H.264 live video
In one of the projects of the Itseez company connected with computer vision, we use the Raspberry Pi to process the video stream from a webcam, and recently we encountered the problem of recording video on a flash card. The difficulty was that the CPU resources were eaten by other more important tasks, but it was still necessary to save the video. Moreover, there were no preferences as to which codec to compress and which format to use, as long as it doesn’t affect fps (frames per second). Having tried a large number of software codecs from RAW to H.264 ( OpenCV wrapper was used over FFmpeg), we came to the conclusion that nothing will come of this, since with a high load, fps sagged from 20 to 5 frames per second, while the picture is black and white with a resolution of 320x240. A little googling, we found out that the Raspberry Pi processor has a hardware encoder with support for the H.264 standard (as far as I know, the license was purchased only for it). Plus to all was that the interaction with the coder is implemented according to the OpenMAX standard, so it was decided to start writing code using OpenMAX , and see what happens. It turned out, by the way, very even not bad!
Below is an example of a video before applying hardware acceleration:
.
OpenMAX (Open Media Acceleration) is a cross-platform API that provides a set of tools for hardware accelerated video and audio processing and work with various multimedia systems, designed to be used independently of the OS or hardware platform. At once I will make a reservation that not the “pure” OpenMAX IL (Integration Layer) API, but some adapted version for the Broadcom chip is implemented on the Raspberry Pi. Therefore, an attempt to reuse the code on another board may fail. In addition, it was decided to use a wrapper over OpenMAX, provided by the developers of Raspberry Pi - ilcient. In the Raspbian wheezy distribution kit , by default there are ready-made libraries and examples of using OpenMAX, which are located in the / opt / vc / directory. The subdirectory / opt / vc / src / hello_pi / libs / ilclient / contains the source of the wrappers over OpenMAX. These are the ilclient.c files ilclient.h and ilcore.c.
Let's return to the task. There is an image from the camera, single-channel (that is, black and white), with a resolution of 320x240, in our case it is the IplImage structure from OpenCV, and you need to save it to the AVI container, previously running through the H.264 codec. This implies the following subtasks and the ways in which they were solved:
')
So the points:
Everything is simple here: create a conversion context and two AVPicture structures. The first is for the single-channel image, the second is for the YUV420p:
Conversion accordingly looks like this:
Where buf is the input buffer for the codec, and frame is the IplImage * from the camera.
Here it is more complicated, it is especially important to initialize the encoder correctly and in the right order:
Here, the client is created and the input buffer parameters are set: the height and width of the image, fps and color scheme. Port 200 is the developer-defined input port for the driver component video_encode, 201 is the output port of this component. For other operations (video decoding, audio encoding-decoding, etc.), other ports are used accordingly.
Above, the output buffer and bitrate parameters are set. Parameter format.eCompressionFormat = OMX_VIDEO_CodingAVC, just determines that the image will be encoded in H.264. The optimal bitrate was calculated manually as described here: www.ezs3.com/public/What_bitrate_should_I_use_when_encoding_my_video_How_do_I_optimize_my_video_for_the_web.cfm .
Next, turn on the buffers and put the driver in the execution state.
Actually, the coding itself:
Here, too, nothing complicated for those who used FFmpeg. Initialization of the output format context:
Next, open the file for recording and record the header and information about the content format:
The process of saving the encoded image:
The av_rescale_q function converts the codec timestamp to the corresponding frame timestamp in the container.
For the assembly you will need to include the following header files:
Accordingly, you will also have to build or install FFmpeg and OpenCV, although nothing prevents you from using other libraries to save the video to a file. The files “bcm_host.h” and “ilclient.h” can be found in the subdirectories of the path / opt / vc /. ilclient.c and ilcore.c, in which the OpenMAX client code is located, get together with the project.
For linking you will definitely need the following libraries:
Well, plus you will need to specify the FFmpeg and OpenCV libraries, for example, as shown below:
That's all. I will add only that with the use of the built-in encoder, the fps of our system with the video saving function turned on and without it practically do not differ, while earlier when using the soft codecs, fps dropped by 40-60%. See for yourself:
Below is an example of a video before applying hardware acceleration:
.
OpenMAX (Open Media Acceleration) is a cross-platform API that provides a set of tools for hardware accelerated video and audio processing and work with various multimedia systems, designed to be used independently of the OS or hardware platform. At once I will make a reservation that not the “pure” OpenMAX IL (Integration Layer) API, but some adapted version for the Broadcom chip is implemented on the Raspberry Pi. Therefore, an attempt to reuse the code on another board may fail. In addition, it was decided to use a wrapper over OpenMAX, provided by the developers of Raspberry Pi - ilcient. In the Raspbian wheezy distribution kit , by default there are ready-made libraries and examples of using OpenMAX, which are located in the / opt / vc / directory. The subdirectory / opt / vc / src / hello_pi / libs / ilclient / contains the source of the wrappers over OpenMAX. These are the ilclient.c files ilclient.h and ilcore.c.
Let's return to the task. There is an image from the camera, single-channel (that is, black and white), with a resolution of 320x240, in our case it is the IplImage structure from OpenCV, and you need to save it to the AVI container, previously running through the H.264 codec. This implies the following subtasks and the ways in which they were solved:
')
- Before encoding, you need to bring the image to any color model, for example YUV420p, we will do this using the swscale module from the FFmpeg version 0.7.1 library set.
- We encode the received buffer using OpenMAX, having previously configured it so that the input will be the buffer containing the image in YUV420p, and the output buffer with the image, after being processed by the H.264 codec.
- We save the compressed image in an AVI container using the same FFmpeg.
So the points:
Conversion
Everything is simple here: create a conversion context and two AVPicture structures. The first is for the single-channel image, the second is for the YUV420p:
#define WIDTH 320 #define HEIGHT 240 AVFrame *input_frame = avcodec_alloc_frame(); r = avpicture_alloc((AVPicture *) input_frame, PIX_FMT_GRAY8, WIDTH, HEIGHT); AVFrame *omx_input_frame = avcodec_alloc_frame(); r = avpicture_alloc((AVPicture *) omx_input_frame, PIX_FMT_YUV420P, WIDTH, HEIGHT); SwsContext *img_convert_ctx = sws_getContext(WIDTH, HEIGHT, PIX_FMT_GRAY8, WIDTH, HEIGHT, PIX_FMT_YUV420P, SWS_BICUBIC, NULL, NULL, NULL); Conversion accordingly looks like this:
avpicture_fill ((AVPicture *) input_frame, (uint8_t *) frame->imageData, PIX_FMT_GRAY8, WIDTH, HEIGHT); buf->nFilledLen = avpicture_fill ((AVPicture *) omx_input_frame, buf->pBuffer, PIX_FMT_YUV420P, WIDTH, HEIGHT); sws_scale(img_convert_ctx, (const uint8_t* const*)input_frame->data, input_frame->linesize, 0, HEIGHT, omx_input_frame->data, omx_input_frame->linesize); Where buf is the input buffer for the codec, and frame is the IplImage * from the camera.
Coding
Here it is more complicated, it is especially important to initialize the encoder correctly and in the right order:
OMX_VIDEO_PARAM_PORTFORMATTYPE format; OMX_PARAM_PORTDEFINITIONTYPE def; COMPONENT_T *video_encode; ILCLIENT_T *client; OMX_BUFFERHEADERTYPE *buf; // OMX_BUFFERHEADERTYPE *out; // int r = 0; #define VIDEO_ENCODE_PORT_IN 200 #define VIDEO_ENCODE_PORT_OUT 201 #define BITRATE 400000 #define FPS 25 bcm_host_init(); client = ilclient_init(); OMX_Init(); ilclient_create_component(client, &video_encode, "video_encode", (ILCLIENT_CREATE_FLAGS_T)(ILCLIENT_DISABLE_ALL_PORTS | ILCLIENT_ENABLE_INPUT_BUFFERS | ILCLIENT_ENABLE_OUTPUT_BUFFERS)); memset(&def, 0, sizeof(OMX_PARAM_PORTDEFINITIONTYPE)); def.nSize = sizeof(OMX_PARAM_PORTDEFINITIONTYPE); def.nVersion.nVersion = OMX_VERSION; def.nPortIndex = VIDEO_ENCODE_PORT_IN; OMX_GetParameter(ILC_GET_HANDLE(video_encode), OMX_IndexParamPortDefinition, &def); def.format.video.nFrameWidth = WIDTH; def.format.video.nFrameHeight = HEIGHT; def.format.video.xFramerate = FPS << 16; def.format.video.nSliceHeight = def.format.video.nFrameHeight; def.format.video.nStride = def.format.video.nFrameWidth; def.format.video.eColorFormat = OMX_COLOR_FormatYUV420PackedPlanar; r = OMX_SetParameter(ILC_GET_HANDLE(video_encode), OMX_IndexParamPortDefinition, &def); Here, the client is created and the input buffer parameters are set: the height and width of the image, fps and color scheme. Port 200 is the developer-defined input port for the driver component video_encode, 201 is the output port of this component. For other operations (video decoding, audio encoding-decoding, etc.), other ports are used accordingly.
memset(&format, 0, sizeof(OMX_VIDEO_PARAM_PORTFORMATTYPE)); format.nSize = sizeof(OMX_VIDEO_PARAM_PORTFORMATTYPE); format.nVersion.nVersion = OMX_VERSION; format.nPortIndex = VIDEO_ENCODE_PORT_OUT; format.eCompressionFormat = OMX_VIDEO_CodingAVC; r = OMX_SetParameter(ILC_GET_HANDLE(video_encode), OMX_IndexParamVideoPortFormat, &format); OMX_VIDEO_PARAM_BITRATETYPE bitrateType; memset(&bitrateType, 0, sizeof(OMX_VIDEO_PARAM_BITRATETYPE)); bitrateType.nSize = sizeof(OMX_VIDEO_PARAM_PORTFORMATTYPE); bitrateType.nVersion.nVersion = OMX_VERSION; bitrateType.eControlRate = OMX_Video_ControlRateVariable; bitrateType.nTargetBitrate = BITRATE; bitrateType.nPortIndex = VIDEO_ENCODE_PORT_OUT; r = OMX_SetParameter(ILC_GET_HANDLE(video_encode), OMX_IndexParamVideoBitrate, &bitrateType); ilclient_change_component_state(video_encode, OMX_StateIdle); Above, the output buffer and bitrate parameters are set. Parameter format.eCompressionFormat = OMX_VIDEO_CodingAVC, just determines that the image will be encoded in H.264. The optimal bitrate was calculated manually as described here: www.ezs3.com/public/What_bitrate_should_I_use_when_encoding_my_video_How_do_I_optimize_my_video_for_the_web.cfm .
ilclient_enable_port_buffers(video_encode, VIDEO_ENCODE_PORT_IN, NULL, NULL, NULL); ilclient_enable_port_buffers(video_encode, VIDEO_ENCODE_PORT_OUT, NULL, NULL, NULL); ilclient_change_component_state(video_encode, OMX_StateExecuting); Next, turn on the buffers and put the driver in the execution state.
Actually, the coding itself:
buf = ilclient_get_input_buffer(video_encode, VIDEO_ENCODE_PORT_IN, 1); OMX_EmptyThisBuffer(ILC_GET_HANDLE(video_encode), buf); out = ilclient_get_output_buffer(video_encode, VIDEO_ENCODE_PORT_OUT, 1); OMX_FillThisBuffer(ILC_GET_HANDLE(video_encode), out); Saving video
Here, too, nothing complicated for those who used FFmpeg. Initialization of the output format context:
AVCodecContext *cc; char *out_file_name; // .avi AVOutputFormat *fmt; AVFormatContext *oc; AVStream *video_st; av_register_all(); fmt = av_guess_format(NULL, out_file_name, NULL); oc = avformat_alloc_context(); oc->debug = 1; oc->start_time_realtime = AV_NOPTS_VALUE; oc->start_time = AV_NOPTS_VALUE; oc->duration = 0; oc->bit_rate = 0; oc->oformat = fmt; snprintf(oc->filename, sizeof(out_file_name), "%s", out_file_name); video_st = avformat_new_stream(oc, NULL); cc = video_st->codec; cc->width = WIDTH; cc->height = HEIGHT; cc->codec_id = CODEC_ID_H264; cc->codec_type = AVMEDIA_TYPE_VIDEO; cc->bit_rate = BITRATE; cc->profile = FF_PROFILE_H264_HIGH; cc->level = 41; cc->time_base.den = FPS; cc->time_base.num = 1; video_st->time_base.den = FPS; video_st->time_base.num = 1; video_st->r_frame_rate.num = FPS; video_st->r_frame_rate.den = 1; video_st->start_time = AV_NOPTS_VALUE; cc->sample_aspect_ratio.num = video_st->sample_aspect_ratio.num; cc->sample_aspect_ratio.den = video_st->sample_aspect_ratio.den; Next, open the file for recording and record the header and information about the content format:
avio_open(&oc->pb, out_file_name, URL_WRONLY); avformat_write_header(oc, NULL); if (oc->oformat->flags & AVFMT_GLOBALHEADER) oc->streams[0]->codec->flags |= CODEC_FLAG_GLOBAL_HEADER; av_dump_format(oc, 0, out_file_name, 1); The process of saving the encoded image:
AVPacket pkt; AVRational omxtimebase = { 1, FPS}; OMX_TICKS tick = out->nTimeStamp; av_init_packet(&pkt); pkt.stream_index = video_st->index; pkt.data= out->pBuffer; pkt.size= out->nFilledLen; if (out->nFlags & OMX_BUFFERFLAG_SYNCFRAME) pkt.flags |= AV_PKT_FLAG_KEY; pkt.pts = av_rescale_q(((((uint64_t)tick.nHighPart)<<32) | tick.nLowPart), omxtimebase, oc->streams[video_st->index]->time_base); pkt.dts = AV_NOPTS_VALUE; av_write_frame(oc, &pkt); out->nFilledLen = 0; The av_rescale_q function converts the codec timestamp to the corresponding frame timestamp in the container.
For the assembly you will need to include the following header files:
#include "opencv2/core/core_c.h" #include "opencv2/imgproc/imgproc_c.h" #include "libavcodec/avcodec.h" #include "libavformat/avformat.h" #include "libswscale/swscale.h" #include "libavutil/opt.h" #include "libavutil/avutil.h" #include "libavutil/mathematics.h" #include "libavformat/avio.h" #include "bcm_host.h" #include "ilclient.h" Accordingly, you will also have to build or install FFmpeg and OpenCV, although nothing prevents you from using other libraries to save the video to a file. The files “bcm_host.h” and “ilclient.h” can be found in the subdirectories of the path / opt / vc /. ilclient.c and ilcore.c, in which the OpenMAX client code is located, get together with the project.
For linking you will definitely need the following libraries:
-L/opt/vc/lib -lbcm_host -lopenmaxil -lbcm_host -lvcos -lvchiq_arm –lpthread Well, plus you will need to specify the FFmpeg and OpenCV libraries, for example, as shown below:
-L/usr/local/lib -lavcodec -lavformat -lavutil -lswscale \ -L/usr/local/lib -lopencv_imgproc -lopencv_core That's all. I will add only that with the use of the built-in encoder, the fps of our system with the video saving function turned on and without it practically do not differ, while earlier when using the soft codecs, fps dropped by 40-60%. See for yourself:
Source: https://habr.com/ru/post/207314/
All Articles