Compress and stream TCP video over OpenCV

At work, I develop image processing algorithms and, in particular, algorithms for automatically tracking objects in video for special applications. Recently, it was necessary to make a model of the algorithm, controlled from a remote computer, to debug the logic of work in a complex system. Previously, such a task was not, because All algorithms were implemented as a result on FPGA. I have been working with OpenCV for a long time and, rubbing my hands, approached the writing of the program. But the enthusiasm quickly went out when he encountered direct video transmission over the network.

The task was as follows:

1. Write a program server that downloads video from a file, compresses it into JPEG and sends it to the client via TCP protocol.
2. Write a client program that receives video over TCP, decodes and displays.

The tasks described above are elementary and serve to “work out” the technology. It seems that this topic has already been described a long time ago, but after some time in searching for a ready answer (a piece of code), I realized that not everything is obvious. Therefore, here I present my experience in this matter. Perhaps someone my experience will be useful.
')
The program was developed in Visual Studio 2015 using the OpenCV version 3.1 library. I will omit the steps of creating a project, connecting libraries and writing code that is responsible for networking. At the end of the article I will give a link to the source code of the project with full comments in the code for quick understanding. Let's focus on the main problem: how to get a video from a file, compress it in JPEG with the necessary degree of compression and transmit over the network with further decoding and display on the receiving side. Below is a small piece of code showing how to compress a video frame and transmit it over the network (taking into account that the connection with the receiving side is established).

//   Mat srcMat; //    vector<uchar> imgBuf; //     vector<int> quality_params = vector<int>(2); //     quality_params[0] = CV_IMWRITE_JPEG_QUALITY; //  JPEG quality_params[1] = 20; //     (95) 0-100 //    cd.frame = cvQueryFrame(cd.videocap); //     srcMat = cv::cvarrToMat(cd.frame); //    JPEG imencode(".jpg", srcMat, imgBuf, quality_params); //   send(clientSocket, (const char*)&imgBuf[0], static_cast<int>(imgBuf.size()), 0); 

Now we will describe how to unpack the image on the receiving side. Below is a fragment of client program code.

 //   int iResult; //     const int MAX_BUF_SIZE = 2073600; //      unsigned char *buf = new unsigned char[MAX_BUF_SIZE]; //     vector<uchar> videoBuffer; //    Mat jpegimage; //    IplImage img; //    //    iResult = recv(connectSocket, (char *)&buf[0], MAX_BUF_SIZE, 0); if (iResult > 0) { //    ,   videoBuffer.resize(iResult); memcpy((char*)(&videoBuffer[0]), buf, iResult); //   jpegimage = imdecode(Mat(videoBuffer), CV_LOAD_IMAGE_COLOR); img = jpegimage; //   cvShowImage("Recieved Video", &img); //    ( 5 ms) cvWaitKey(5); }//if (iResult > 0)... 

The described operations are performed for each video frame. I hope this small example will help solve the difficulties of compressing and streaming video over TCP using the OpenCV library. Below is a link screenshot of the client and server.
www.dropbox.com/s/ikisl8rjxxd5d0f/%D0%91%D0%B5%D0%B7%D1%8B%D0%BC%D1%8F%D0%BD%D0%BD% .png
Due to the fact that the compression ratio is 20 (high compression) on the receiving side, you can notice significant distortion of the picture.

The source codes of the server and client programs with detailed comments you can see by the links:

www.dropbox.com/s/3ucjsdes7khcr24/Server.cpp?dl=0
www.dropbox.com/s/14mat8bhlonz392/Client.cpp?dl=0

Good luck in all your endeavors!