Qt and mobile camera. Part 1, Symbian

Good day, Habr!

During my work in the development of mobile applications, in particular for Symbian, several solutions have been created for working with the phone's camera. Over time, these decisions evolved, which I want to talk about in the next two articles.
The first one deals with non-trivial, but flexible image acquisition from a Symbian phone using QtMobility 1.1.3, the second deals with problems and their solutions when porting code to the Meego 1.2 Harmattan platform under which the Nokia N900, N950, N9 are currently working .
This material may be useful for novice Qt-developers of mobile applications.

So, Qt for working with the Symbian-camera.

The task I was working on remained the same: it was necessary to get an image from the camera, display it in my own viewfinder, draw some widget over the output image and transfer several video frames to a remote server. An important aspect of the task is that the execution of all actions should occur without user intervention - it only requires the launch of the application and camera guidance.
')
For the first time, the solution was born in wild agony, due to some features of QtMobility. If we use the simplest direct method of displaying images on the viewfinder (QVideoWidget or its successor QCameraViewfinder), we encounter the following problems:

• QVideoWidget does not allow rendering semi-transparent widgets on top of itself (only black rectangles are obtained for transparency)
• You can get an image as an object only as a photo with its mandatory saving in the file system (using QCameraImageCapture)
• It follows from the above that it is impossible to control the image “inside” the viewfinder, that is, on the screen we will see only what the camera itself sees

In addition, if you try to get an image after it is output to the viewfinder via QWidget :: render (), then again we will see only a black rectangle, since QVideoWidget itself does not redraw on each frame. For the same reason, it is useless to override the QWidget :: paintEvent () method for the viewfinder.

When the problem was solved for the first time, we finally came to the use of native Symbian code. The result was a QObject, sending a singal with a QPixmap on each new frame. This pixmap was processed in the application - it was drawn, sent, etc.
And although this option was fully operational, with it it was possible to easily and quickly port to other platforms, more precisely to another - Meego.

Next, I propose a solution to the problem using only Qt and QtMobility, which, by the way, is simpler (especially for those who are not familiar with Symbian.C ++).
The code below is partly based on an official example , in which for us there is a lot of superfluous, irrelevant to the case.

So, we get video frames from the camera on the Symbian ^ 3 or Symbian 5th edition phone using Qt as separate objects. To do this, we write our own viewfinder, rendered anew on each new frame.
The scheme of operations is as follows:

1. We create a descendant of the abstract class QAbstractVideoSurface and define two pure virtual methods QAbstractVideoSurface :: supportedPixelFormats () and QAbstractVideoSurface :: present () in it. This will be a frame handler in which data from the camera will be transmitted.
2. In the present () method, we convert the video frame into a QImage object and pass it into a callback.
3. We connect the camera to our application and install a new “surface” (frame handler) for it - an object of our class from the first item
4. When we receive a QImage object in the callback method, we process it at our discretion, in this case we will draw it in paintEvent () of our viewfinder

To begin with, we will create a callback, to the method of which the image (frame) will be transmitted. In principle, it can be replaced by a signal-slot pair, but I will use this method. This class must be inherited by our final viewfinder.
PS I will not include the connection of libraries - all the necessary Qt-classes can be connected as usual - by name, for example

#include <QAbstractVideoSurface> 

> videosurfaceimageobserver.h

 class VideoSurfaceImageObserver { public: virtual void newImage(QImage) = 0; }; 

Next - create a class inherited from QAbstractVideoSurface:
> myvideosurface.h

 class myVideoSurface : public QAbstractVideoSurface { Q_OBJECT public: myVideoSurface(VideoSurfaceImageObserver *mObserver, QObject *parent = 0); bool present(const QVideoFrame &frame); QList<QVideoFrame::PixelFormat> supportedPixelFormats(QAbstractVideoBuffer::HandleType type=QAbstractVideoBuffer::NoHandle ) const; private: QVideoFrame m_frame; QImage::Format m_imageFormat; QVideoSurfaceFormat m_videoFormat; VideoSurfaceImageObserver *observer; }; 

> myvideosurface.cpp

 myVideoSurface::myVideoSurface(VideoSurfaceImageObserver *mObserver, QObject *parent) : QAbstractVideoSurface(parent) { //   ,   newImage()  //    observer = mObserver; m_imageFormat = QImage::Format_Invalid; } /** *  ,     *  video surface' * Symbian      */ QList<QVideoFrame::PixelFormat> myVideoSurface::supportedPixelFormats( QAbstractVideoBuffer::HandleType handleType) const { if (handleType == QAbstractVideoBuffer::NoHandle) { return QList<QVideoFrame::PixelFormat>() << QVideoFrame::Format_RGB32 << QVideoFrame::Format_ARGB32 << QVideoFrame::Format_ARGB32_Premultiplied << QVideoFrame::Format_RGB565 << QVideoFrame::Format_RGB555; } else { return QList<QVideoFrame::PixelFormat>(); } } /** *   . *,       QVideoFrame, *     QImage     */ bool myVideoSurface::present(const QVideoFrame &frame){ m_frame = frame; // " " -       //,    surface'. //       , //       if(surfaceFormat().pixelFormat() != m_frame.pixelFormat() || surfaceFormat().frameSize() != m_frame.size()) { stop(); return false; } else { //   QImage... if (m_frame.map(QAbstractVideoBuffer::ReadOnly)) { QImage image( m_frame.bits(), m_frame.width(), m_frame.height(), m_frame.bytesPerLine(), m_imageFormat); //...      observer->newImage(image); } return true; } } 

Finally, create our viewfinder class.
> myviewfinder.h

 class myViewFinder: public QWidget, public VideoSurfaceImageObserver { Q_OBJECT public: explicit myViewFinder(QWidget *parent = 0); virtual ~myViewFinder(); //     void newImage(QImage); private: void paintEvent(QPaintEvent *); private: QCamera* camera_; myVideoSurface *surface; QImage *pix; }; 

> myviewfinder.cpp

 myViewFinder::myViewFinder(QWidget *parent) : QWidget(parent), camera_(0), viewfinder_(0), pix(0) { //,   -  camera_ = new QCamera; //  video surface     //   surface = new myVideoSurface(this, this); //     pix = new QImage(); //---      //---  video surface QVideoRendererControl *control = qobject_cast<QVideoRendererControl *>( camera_->service()->requestControl("com.nokia.Qt.QVideoRendererControl/1.0")); if(control){ control->setSurface(surface); } //--- // ,     // myVideoSurface    camera_->start(); } /** *      , *  QImage */ void myViewFinder::newImage(QImage img){ // ... &pix = img; //...   . //  update() //   paintEvent() //  update(); } void myViewFinder::paintEvent(QPaintEvent *event){ QPainter painter(this); //  . // -  . painter.fillRect(this->geometry(), Qt::black); if(!pix) return; //     //  .    //    video surface //    if(!pix->isNull() && this->geometry().width() != 0 && this->geometry().height() != 0 && pix->width() != 0 && pix->height() != 0) { //   ... QRect pixR(pix->rect()); pixR.moveCenter(this->geometry().center()); //... ! painter.save(); painter.drawImage(pixR, *pix); painter.restore(); } } myViewFinder::~myViewFinder() { camera_->unload(); delete viewfinder_; delete camera_; delete pix; } 

Now, why fill the viewfinder with black before rendering the image. The screen size on Symbian ^ 3 and Symbian 5th edition 640x360 phones, and the camera resolution is at least 640x480. So if you have, for example, in a full-screen application there is nothing but the viewfinder, the aspect ratio will not allow you to fill the entire screen with the camera image. To do this, first fill the canvas with black (or any other color or picture) so that the desktop or other widgets do not shine around the viewfinder.

To fill the entire widget with the viewfinder without black bars along the edges, but with cropping the edges of the image, painEvent () can be modified as follows:

 void myViewFinder::paintEvent(QPaintEvent *event){ QPainter painter(this); if(!pix) return; if(!pix->isNull() && this->geometry().width() != 0 && this->geometry().height() != 0 && pix->width() != 0 && pix->height() != 0) { //  ,      QPixmap newPix = pix->scaled(this->size(), Qt::KeepAspectRatioByExpanding); QRect pixR(newPix.rect()); pixR.moveCenter(this->geometry().center()); //... ! painter.save(); painter.drawImage(pixR, newPix ); painter.restore(); } } 

Thus, we got a full viewfinder, which can be used as a regular widget in the future. At the same time, after the frame is received in newImage (), you can do anything with it, which gives an advantage over QVideoWidget.

In the next article I will talk about porting this code to Meego 1.2 Harmattan and its modifications related to the hardware features of the camera in devices running the latter.

Used sources:

1. Same example
2. Remaining official Qt and QtMobility docks

Thanks for attention.