Scaling operations in the Intel Media SDK

This article covers all scaling operations in the Intel Media SDK. Scaling is one of the most common operations in video processing. The application can set the desired area for each video using the video processing pipeline (VPP). Using the Intel Media SDK VPP, you can perform various scaling operations. Here we describe the two most frequently used operations and their results.

A simple flowchart showing the pipeline of the functions involved in scaling is shown on the KDPV.

Detection of the free surface of a frame — surfaces from a pool of surfaces are used, blocked surfaces cannot be used, therefore, an unused surface is searched for.
')

int GetFreeSurfaceIndex(mfxFrameSurface1** pSurfacesPool, mfxU16 nPoolSize) { if (pSurfacesPool) for (mfxU16 i = 0; i < nPoolSize; i++) if (0 == pSurfacesPool[i]->Data.Locked) return i; return MFX_ERR_NOT_FOUND; }

Load the raw frame to the surface - for the raw frame, the light and chromaticity planes are read, then loaded to the surface.

 mfxStatus LoadRawFrame(mfxFrameSurface1* pSurface, FILE* fSource) { w = pInfo->Width; h = pInfo->Height; pitch = pData->Pitch; ptr = pData->Y; //read luminance plane for (i = 0; i < h; i++) { nBytesRead = (mfxU32) fread(ptr + i * pitch, 1, w, fSource); if (w != nBytesRead) return MFX_ERR_MORE_DATA; } mfxU8 buf[2048]; // maximum supported chroma width for nv12 w /= 2; h /= 2; ptr = pData->UV; // load U sts = ReadPlaneData(w, h, buf, ptr, pitch, 0, fSource); if (MFX_ERR_NONE != sts) return sts; // load V ReadPlaneData(w, h, buf, ptr, pitch, 1, fSource); if (MFX_ERR_NONE != sts) return sts; }

The write operation on the output surface is after the RunFrameVPPSync operation, which asynchronously processes the frame. A synchronization operation is called to get all the output data to write back to the raw frame.

 mfxStatus WriteRawFrame(mfxFrameSurface1* pSurface, FILE* fSink) { mfxFrameInfo* pInfo = &pSurface->Info; mfxFrameData* pData = &pSurface->Data; mfxU32 i, j, h, w; mfxStatus sts = MFX_ERR_NONE; for (i = 0; i < pInfo->Height; i++) sts = WriteSection(pData->Y, 1, pInfo->Width, pInfo, pData, i, 0, fSink); h = pInfo->Height / 2; w = pInfo->Width; for (i = 0; i < h; i++) for (j = 0; j < w; j += 2) sts = WriteSection(pData->UV, 2, 1, pInfo, pData, i, j, fSink); for (i = 0; i < h; i++) for (j = 1; j < w; j += 2) sts = WriteSection(pData->UV, 2, 1, pInfo, pData, i, j, fSink); return sts; }

We will use sample_vpp from the tutorial on the Media Solution Portal page. The input file foreman.yuv can be obtained here. The downloaded file will be in Y4M format, it will need to be converted to YV12 format using ffmpeg.

 ffmpeg -i input.y4m output.yuv

For scaling operations, six VPP parameters are used.

CropX , CropY , CropW , CropH determine the location of the input and output frames, which must be explicitly set to obtain the result.
The parameters Width and Height should be set explicitly for both input and output frames. The values of height and width must be a multiple of 16 for frame images and a multiple of 32 for fields.

Trimming is one of the most common video processing operations, and is often used to determine the working area (ROI). With it, you can also change the aspect ratio of the video. The most common changes are 16: 9-> 4: 3 and 4: 3-> 16: 9. When changing the aspect ratio, black bars are added to the image, either on the top and bottom, or on the right and left. Below is a table of input parameters for trimming, changing the aspect ratio with the addition of black bars above and below, to the right and left in sample_vpp.

	CropX	CropY	Cropw	CropH	Width	Height
Input	128	128	1024	464	1280	720
Output_Crop	0	0	1024	464	1024	464
Output_PillarBoxing	128	0	1024	720	1280	720
Output_LetterBoxing	0	128	1280	464	1280	720

Here are the results you get when using these parameters.

Resizing is another video processing operation used to obtain a video image of the desired size. But in this case no stripes are added to the image, its output resolution simply changes. Resizing is possible in two dimensions, and only in one (you can only change the width or only the height of the video, which corresponds to horizontal and vertical stretching). Below is a table of input parameters for resizing and stretching in sample_vpp.

	Cropw	CropH	Width	Height
Input	640	480	640	480
Output_re-size	1280	720	1280	720
Output_VerticalStretch	640	608	640	608
Output_HorizontalStretch	720	480	720	480

Here are the results you get when using these parameters.

More information about the parameters and functions can be found in the user manual, which is located in the documents folder of the installed copy of MSDK or is available here .

For more information on compiler optimization, see our optimization notice .

Additional links

Source: https://habr.com/ru/post/255551/

All Articles

Scaling operations in the Intel Media SDK

Additional links

More articles: