Qt and Direct3D 12 – First Encounter

The landscape of graphics APIs is changing. Qt Quick 2, released with Qt 5.0 in 2012, was betting heavily on OpenGL and OpenGL ES 2.0. There have been changes and improvements since then – the Qt Quick 2D Renderer was introduced, experiments with software rasterizers were made, and enablers for newer OpenGL versions got added all over the Qt graphics stack. However, as Lars mentioned in his Qt World Summit 2015 keynote, the situation is changing: new, low-level, more efficient APIs like Vulkan, Metal and Direct3D 12 are about to become widely available. Some of these are tied to certain platforms, making them the best choice when targeting the platform in question, while others are expected to be present on a wider variety of systems. At the same time betting solely on a complex, accelerated API is not always the best choice: traditional, desktop user interfaces running on older hardware are sometimes better served with plain old CPU-based rendering.

Therefore, unsurprisingly enough, one of the research areas for upcoming Qt versions is making the graphics stack, and in particular, Qt Quick, more flexible, with support for different graphics APIs as well as software rendering.

Such research work often leads to useful side effects: this post is about one of these, a simple Qt module enabling easy integration of Direct3D 12 rendering into a standalone Qt window – or alternatively QWidget-based applications – and making it easier to get started and experiment with modern D3D techniques while continuing to enjoy the familiar APIs, tools, and development environment Qt and its ecosystem offer.

What it is

The QtD3D12Window module, living in a qt-labs repository, is a Qt 5.6 module providing a QD3D12Window class similar to QOpenGLWindow, a handy qmake rule for offline HLSL shader compilation via fxc, and a comprehensive set of examples for a number of basic use cases (inspired by Microsoft’s own Hello World samples).

What it is not

Before going further, let’s reiterate what this module is not: it is not a way to run existing Qt applications on Direct3D (that is exactly what ANGLE provides when it comes to D3D9 and 11), nor does it support Qt Quick in any way. It is also not something that will get added to Qt in its current form, and it is not a complete engine or framework of any kind (the interface QD3D12Window offers is likely insufficient for more complex, multi-threaded approaches).

Window

The first step for integrating custom rendering code using a new graphics API is to make the rendering target a QWindow’s underlying native window. Currently OpenGL is the only choice and it is deeply integrated into Qt both internally (QPA interfaces, platform plugins) and externally (public APIs, see the QOpenGL classes in QtGui for instance). This is expected to change in the future, but for now, for our standalone D3D experiment, we will just do everything on our own. Qt’s windows platform plugin makes it easy to obtain the native window handle (HWND) as that is exactly what QWindow::winId() returns in Qt applications running on Windows. We can then use DXGI to enumerate the available adapters, pick either a hardware one or WARP, and create a swap chain. QD3D12Window handles all this, with defaults that are good enough for the typical basic applications.

Speaking of DXGI, this marks our first foray into graphics adapter and device management: while with OpenGL and its usual windowing system interfaces such capabilities are limited, other APIs (especially the ones tied to platforms with a suitable driver model) may offer a lot more when it comes to discovering graphics adapters and managing device changes, removals and resets during the lifetime of the application. Such functionality, on platforms and APIs where available, is expected to get more focus in Qt in the future.

While implementing QD3D12Window, it turned out there was no need to do much generic plumbing – thanks to the way QOpenGLWindow and QRasterWindow had been introduced back in Qt 5.4: the common base class QPaintDeviceWindow (somewhat misleading in the D3D case as our window here is not suitable for QPainter-based painting) provides all the necessary infrastructure so subclasses can focus on the graphics API specifics while getting the basic, consistent Qt functionality – like update() – out of the box. This is good news since it allows easy experimenting with other APIs as well in the future (QMetalWindow, QVulkanWindow, you name it).

Class

QD3D12Window mirrors QOpenGLWindow which in turn is based on QOpenGLWidget/QGLWidget’s well-known initializeGL – resizeGL – paintGL interface, with GL becoming D3D, naturally. There are two new functions subclasses may reimplement: releaseD3D and afterPresent. The latter is invoked every time after issuing a Present call: most simple applications will wait for the GPU via a fence here. The former is used to make applications able to survive device removals: when the graphics device becomes unavailable, this function is invoked and is expected to release all resources it has created during initialize/resize/paint. QD3D12Window will then then take care of starting over and invoking initializeD3D again. This way the application can remain functional even when a driver update or a timeout in shader execution occurs.

The best is probably to dive straight into the examples, looking at hellotriangle for instance shows that if you haved used QOpenGLWindow or QOpenGLWidget before, QD3D12Window will present no unexpected surprises either.

Widgets

So now we can have a top-level window with nothing but our awesome D3D12-rendered content in it. This is excellent but what about having some traditional user interface controls in there? For now, the approach that is usable with QD3D12Window is to make it a native child window via QWidget::createWindowContainer(). This comes with the usual limitations so be sure to read the documentation first. Nonetheless it will work fine for most simple purposes.

hellooffscreen, one of the QD3D12Window examples

Shaders

The handling of shader code and compilation is another very interesting topic. With OpenGL, Qt and Qt Quick bet on runtime compilation due to that being the only universally available solution not involving vendor and platform specifics. With other graphics APIs in the future, it is expected that Qt will focus more on offline compilation, integrating it into the build system as much as possible. Besides the obvious performance benefits, this drastically improves the development workflow as well: getting a proper compiler error right when hitting Ctrl+B in Qt Creator can easily feel infinitely superior to the “old” way of browsing the debug output while running the application.

The QtD3D12Window module comes with a simple qmake rule that allows invoking fxc.exe during build time. All the examples use this to generate header files where the compiled bytecode is provided as a plain char array. Take a look at the the .pro file for one of them and the shader setup for the pipeline state. Simple and easy, isn’t it?

All this is not completely new to Qt: the bundled ANGLE in qtbase performs shader compilation in the same manner. However, that is hidden to and not usable by application-level D3D code, while the hlsl.prf file here can be copied over to the Qt SDK’s mkspecs/features folder, making it available to any Qt application.

Graphics Debugging

QD3D12Window automatically enables the D3D12 debug layer. This is extremely useful in practice as many of the common mistakes made while getting started with D3D12 result in human readable, verbose and actually helpful debug messages. However, you may want to disable this when doing performance testing or overhead comparison.

Another helpful tool is the graphics debugger included in Visual Studio. One way to launch this for Qt apps is doing devenv /debugexe qtapplication.exe from a developer command prompt and hitting Alt+F5. (alternatively, generating Visual Studio project files with qmake -tp vc may work too) This proved to be quite useful while developing even our simple examples – for instance the ability to inspect graphics resources and see if we managed to correctly generate all mipmap levels is immensely helpful.

Examples

As mentioned before, the module comes with a set of examples that cover the basics and may be useful to anyone getting started with D3D12 development. See the readme for an overview.

That’s it for now, hope you find our little labs module useful. Happy hacking!

The post Qt and Direct3D 12 – First Encounter appeared first on Qt Blog.

Source :- Qt Blog http://ift.tt/1SkPVcs

Embedded Linux news in Qt 5.6

With Qt 5.6 approaching, it is time to look at some of the new exciting features for systems running an Embedded Linux variant, just like we did for Qt 5.5 a while ago.

Support for NVIDIA Jetson Pro boards

Boards like the Jetson TK1 Pro running a Yocto-generated Vibrante Linux image have support for X11, Wayland, and even running with plain EGL without a full windowing system. The latter is accomplished by doing modesetting via the well-known DRM/KMS path and combining it with the EGL device extensions. This is different than what the eglfs platform plugin’s existing KMS backend, relying on GBM, offers. Therefore Qt 5.5 is not functional in this environment. For more information on the details, check out this presentation.

Wayland presents a similar challenge: while Weston works fine due to having been patched by NVIDIA, Qt-based compositors built with the Qt Compositor framework cannot function out of the box since the compositor has to use the EGLStream APIs instead of Mesa’s traditional EGLImage-based path.

With Qt 5.6 this is all going to change. With the introduction of a new eglfs backed based on EGLDevice + EGLOutput + EGLStream, Qt applications will just work, similarly to other embedded boards:

The well-known Qt Cinematic Experience demo running with Qt 5.6 and eglfs on a Jetson K1 Pro

Cross-compilation is facilitated by the new device makespec linux-jetson-tk1-pro-g++.

Wayland is going to be fully functional too, thanks to the patches that add support for EGL_KHR_stream, EGL_KHR_stream_cross_process_fd, and EGL_KHR_stream_consumer_gltexture in the existing wayland-egl backend of Qt Compositor.

The qwindow-compositor example running on the Jetson with some Qt clients

All this is not the end of the story. There is room for future improvements, for example when it comes to supporting multiple outputs and direct rendering (i.e. skipping GL-based compositing and connecting the stream directly to an output layer à la eglfs to improve performance). These will be covered in future Qt releases.

Note that Wayland support on the Jetson should be treated as a technical preview for the time being. Compositors using the unofficial C++ APIs, like the qwindow-compositor example shown above, will work fine. However, QML and Qt Quick support is still work in progress at the time of writing.

Support for Intel NUC

Some of the Intel NUC devices make an excellent embedded platform too, thanks to the meta-intel and the included meta-nuc layers for Yocto. While these are ordinary x86-64 targets, they can be treated and used like ARM-based boards. When configuring Qt for cross-compilation, use the new linux-nuc-g++ device spec. Graphics-wise everything is expected to work like on an Intel GPU-based desktop system running Mesa. This includes both eglfs (using the DRM/KMS/GBM backend introduced in Qt 5.5) and Wayland.

Wayland on boards based on the i.MX6

Systems based on Freescale’s i.MX6 processors include a Vivante GC2000 GPU and driver support for Wayland. Qt applications have traditionally been working fine on the Weston reference compositor, see for example this previous post for Qt 5.4, but getting Qt-based compositors up and running is somewhat tricky due to some driver specifics that do not play well with QPA and eglfs. With Qt 5.6 this issue is eliminated as well: in addition to the regular Vivante-specific backend (eglfs_viv), eglfs now has an additional backend (eglfs_viv_wl) which transparently ensures proper functionality when running compositor applications built with the Qt Compositor framework. This backend will need to be requested explicitly, so for example to run the qwindow-compositor example, do QT_QPA_EGLFS_INTEGRATION=eglfs_viv_wl ./qwindow-compositor -platform eglfs (the -platform can likely be omitted since eglfs is typically the default).

OpenGL ES 3.0 and 3.1

As presented earlier, OpenGL ES 3 support is greatly enhanced in Qt 5.6. Using the new QOpenGLExtraFunctions class applications targeting embedded devices with GLES 3 capable drivers can now take the full API into use in a cross-platform manner.

libinput

Qt 5.5 introduced support for libinput when it comes to getting input events from keyboards, mice, touchpads, and touchscreens. Qt 5.6 takes this one step further: when libinput is available at build time, it will be set as the default choice in eglfs and linuxfb, replacing Qt’s own evdevkeyboard, mouse, and touch backends.

In some rare cases this will not be desirable (for example when using evdevkeyboard-specific keyboard layouts from the Qt 4 QWS times), and therefore the QT_QPA_EGLFS_NO_LIBINPUT environment variable is provided as a means to disable this and force the pre-5.6 behavior.

That’s it for now. Hope you will find the new Embedded Linux features useful. Happy hacking!

P.S. the Qt World Summit 2015 had a number of exciting talks regarding embedded development, for example Qt for Device Creation, Choosing the right Embedded Linux platform for your next project and many more. Browse the full session list here.

The post Embedded Linux news in Qt 5.6 appeared first on Qt Blog.

Source :- Qt Blog http://ift.tt/1LjjXFO

Using modern OpenGL ES features with QOpenGLFramebufferObject in Qt 5.6

QOpenGLFramebufferObject is a handy OpenGL helper class in Qt 5. It conveniently hides the differences between OpenGL and OpenGL ES, enables easy usage of packed depth-stencil attachments, multisample renderbuffers, and some more exotic formats like RGB10. As a follow up to our previous post about OpenGL ES 3 enhancements in Qt 5.6, we are now going to take a look at an old and a new feature of this class.

Qt 5.6 introduces support for multiple color attachments, to enable techniques requiring multiple render targets. Even though support for this is available since OpenGL 2.0, OpenGL ES was missing it up until version 3.0. Now that mobile and embedded devices with GLES 3.0 support are becoming widely available, it is time for QOpenGLFramebufferObject to follow suit and add some basic support for MRT.

Our other topic is multisample rendering to framebuffers. This has been available for some time in Qt 5, but was usually limited to desktop platforms as OpenGL ES does not have this as a core feature before version 3.0. We will now go through the basic usage and look at some common pitfalls.

Multisample framebuffers

By default our framebuffer will have a texture as its sole color attachment. QOpenGLFramebufferObject takes care of creating this texture, applications can then query it via textures() or can even take ownership of and detaching it from the FBO by calling takeTexture().

What if multisampling is desired? Requesting it is simple by using the appropriate constructor, typically like this:

    QOpenGLFramebufferObjectFormat format;
    format.setAttachment(QOpenGLFramebufferObject::CombinedDepthStencil);
    format.setSamples(4);
    fbo = new QOpenGLFramebufferObject(width, height, format);

Here 4 samples per pixel are requested, together with depth and stencil attachments, as the default is to have color only. The rest of the defaults in QOpenGLFramebufferObjectFormat matches the needs of most applications so there is rarely a need to change them.

Now, the obvious question:

What if support is not available?

When support for multisample framebuffers is not available, either because the application is running on an OpenGL ES 2.0 system or the necessary extensions are missing, QOpenGLFramebufferObject falls silently back to the non-multisample path. To check the actual number of samples, query the QOpenGLFramebufferObjectFormat via format() and check samples(). Once the framebuffer is created, the returned format contains the actual, not the requested, number of samples. This will typically be 0, 4, 8, etc. and may not match the requested value even when multisampling is supported since the requests are often rounded up to the next supported value.

    ...
    fbo = new QOpenGLFramebufferObject(width, height, format);
    if (fbo->isValid()) {
        if (fbo->format().samples() > 0) {
            // we got a framebuffer backed by a multisample renderbuffer
        } else {
            // we got a non-multisample framebuffer, backed by a texture
        }
    }

What happens under the hood?

QOpenGLFramebufferObject’s multisample support is based on GL_EXT_framebuffer_multisample and GL_EXT_framebuffer_blit. The good thing here is that glRenderbufferStorageMultisample and glBlitFramebuffer are available in both OpenGL 3.0 and OpenGL ES 3.0, meaning that relying on multisample framebuffers is now feasible in cross-platform applications and code bases targeting desktop, mobile, or even embedded systems.

As usual, had Qt not cared for older OpenGL and OpenGL ES versions, there would have been some other options available, like using multisample textures via GL_ARB_texture_multisample or GL_ARB_texture_storage_multisample instead of multisample renderbuffers. This would then require OpenGL 3.2 (OpenGL 4.3 or OpenGL ES 3.1 in case of the latter). It has some benefits, like avoiding the explicit blit call for resolving the samples. It may be added as an option to QOpenGLFramebufferObject in some future Qt release, let’s see.

A note for iOS: older iOS devices, supporting OpenGL ES 2.0 only, have multisample framebuffer support via GL_APPLE_framebuffer_multisample. However, API-wise this extension is not fully compatible with the combination of EXT_framebuffer_multisample+blit, so adding support for it is less appealing. Now that newer devices come with OpenGL ES 3.0 support, it is less relevant since the standard approach will work just fine.

So what can I do with that renderbuffer?

In most cases applications will either want a texture for the FBO’s color attachment or do a blit to some other framebuffer (or the default framebuffer associated with the current surface). In the multisample case there is no texture, so calling texture() or similar is futile. To get a texture we will need a non-multisample framebuffer to which the multisample one’s contents is blitted via glBlitFramebuffer.

Fortunately QOpenGLFramebufferObject provides some helpers for this too:

    static bool hasOpenGLFramebufferBlit();
    static void blitFramebuffer(QOpenGLFramebufferObject *target, const QRect &targetRect,
                                QOpenGLFramebufferObject *source, const QRect &sourceRect,
                                GLbitfield buffers,
                                GLenum filter,
                                int readColorAttachmentIndex,
                                int drawColorAttachmentIndex);

Combined with some helpful overloads, this makes resolving the samples pretty easy:

    ...
    GLuint texture;
    QScopedPointer tmp;
    if (fbo->format().samples() > 0) {
        tmp.reset(new QOpenGLFramebufferObject(fbo->size()));
        QOpenGLFramebufferObject::blitFramebuffer(tmp.data(), fbo);
        texture = tmp->texture();
    } else {
        texture = fbo->texture();
    }
    ...

Checking for the availability of glBlitFramebuffer via hasOpenGLFramebufferBlit() is not really necessary because the multisample extension requires the blit one, and, on top of that, QOpenGLFramebufferObject checks for the presence of both and will never take the multisample path without either of them. Therefore checking format().samples(), as shown above, is sufficient. It is worth noting that creating a new FBO (like tmp above) over and over again should be avoided in production code. Instead, create it once, together with the multisample one, and reuse it. Note also that having depth or stencil attachments for tmp is not necessary.

And now on to the shiny new stuff.

Multiple render targets

In Qt 5.6 QOpenGLFramebufferObject is no longer limited to a single texture or renderbuffer attached to the GL_COLOR_ATTACHMENT0 attachment point. The default behavior does not change, constructing an instance will behave like in earlier versions, with a single texture or renderbuffer. However, we now have a new function called addColorAttachment():

void addColorAttachment(const QSize &size, GLenum internalFormat = 0);

Its usage is pretty intuitive: as the name suggests, it will create a new texture or renderbuffer and attach it to the next color attachment point. For example, to create a framebuffer object with 3 color attachments in addition to depth and stencil:

fbo = new QOpenGLFramebufferObject(size, QOpenGLFramebufferObject::CombinedDepthStencil);
fbo->addColorAttachment(size);
fbo->addColorAttachment(size);

The sizes and internal formats can differ. The default value of 0 for internalFormat leads to choosing a commonly suitable default (like GL_RGBA8). As for the sizes, the main rule is to keep in mind that rendering will get limited to the area that fits all attachments.

After this we will have a color attachment at GL_COLOR_ATTACHMENT0, 1 and 2. To specify which ones are written to from the fragment shader, use glDrawBuffers. As introduced in the previous post, QOpenGLExtraFunctions is of great help for invoking OpenGL ES 3.x functions:

QOpenGLExtraFunctions *f = QOpenGLContext::currentContext()->extraFunctions();
GLenum bufs[] = { GL_COLOR_ATTACHMENT0, GL_COLOR_ATTACHMENT1, GL_COLOR_ATTACHMENT2 };
f->glDrawBuffers(3, bufs);

We are all set. All we need now is a fragment shader writing to the different attachments:

...
layout(location = 0) out vec4 diffuse;
layout(location = 1) out vec4 position;
layout(location = 2) out vec4 normals;
...

The rest is up to the applications. QOpenGLFramebufferObject no longer prevents them from doing deferred shading and lighting, or other MRT-based techniques.

To check at runtime if multiple render targets are supported, call hasOpenGLFeature() with MultipleRenderTargets. If the return value is false, calling addColorAttachment() is futile.

Finally, now that there can be more than one texture, some of the existing QOpenGLFramebufferObject functions get either a new overload or a slightly differently named alternative. They need no further explanation, I believe:

QVector textures() const;
GLuint takeTexture(int colorAttachmentIndex);
QImage toImage(bool flipped, int colorAttachmentIndex) const;

That’s all for now, happy hacking!

The post Using modern OpenGL ES features with QOpenGLFramebufferObject in Qt 5.6 appeared first on Qt Blog.

Source :- Qt Blog http://ift.tt/1FXgDzd

Cross-platform OpenGL ES 3 apps with Qt 5.6

Now that the alpha release for Qt 5.6 is here, it is time to take a look at some of the new features. With the increasing availability of GPUs and drivers capable of OpenGL ES 3.0 and 3.1 in the mobile and embedded world, targeting the new features these APIs provide has become more appealing to applications. While nothing stops the developer from using these APIs directly (that is, #include <GLES3/gl3.h>, call the functions directly, and be done with it), the cross-platform development story used to be less than ideal, leading to compile and run-time checks scattered across the application. Come Qt 5.6, this will finally change.

Support for creating versioned OpenGL (ES) contexts has been available in Qt 5 for some time. Code snippets like the following should therefore present no surprises:

int main(int argc, char **argv) {
    QSurfaceFormat fmt;
    fmt.setVersion(3, 1);
    fmt.setDepthBufferSize(24);
    QSurfaceFormat::setDefaultFormat(fmt);

    QGuiApplication app(argc, argv);
    ...
}

It is worth pointing out that due to the backwards compatible nature of OpenGL ES 3 this may seem unnecessary with many drivers because requesting the default 2.0 will anyway result in a context for the highest supported OpenGL ES version. However, this behavior is not guaranteed by any specification (see for example EGL_KHR_create_context) and therefore it is best to set the desired version explicitly.

The problem

So far so good. Assuming we are running on an OpenGL ES system, we now have a context and everything ready to utilize all the goodness the API offers. Except that we have no way to easily invoke any of the 3.0 or 3.1 specific functions, unless the corresponding gl3.h or gl31.h header is included and Qt is either a -opengl es2 build or the application explicitly pulled in -lGLESv2 in its .pro file. In which case we can wave goodbye to our sources’ cross-platform, cross-OpenGL-OpenGL ES nature.

For OpenGL ES 2.0 the problem has been solved for a long time now by QOpenGLFunctions. It exposes the entire OpenGL ES 2.0 API and guarantees that the functions are resolved correctly everywhere where an OpenGL context compatible with either OpenGL ES 2.0 or OpenGL 2.0 plus the FBO extension is available.

Before moving on to introducing the counterpart for OpenGL ES 3.0 and 3.1, it is important to understand why the versioned OpenGL function wrappers (for example, QOpenGLFunctions_3_2_Core) are not a solution to our problem here. The versioned wrappers are great when targeting a given version and profile of the OpenGL API. However, they lock in the application to systems that support that exact OpenGL version and profile, or a version and profile compatible with it. Building and running the same source code on an OpenGL ES system is out of question, even when the code only uses calls that are available in OpenGL ES as well. So it turns out that strict enforcement of the compatibility rules for the entire OpenGL API is not always practical.

Say hello to QOpenGLExtraFunctions

To overcome all this, Qt 5.6 introduces QOpenGLExtraFunctions. Why “extra” functions? Because adding all this to QOpenGLFunctions would be wrong in the sense that everything in QOpenGLFunctions is guaranteed to be available (assuming the system meets Qt’s minimum OpenGL requirements), while these additional functions (the ES 3.0/3.1 API) may be dysfunctional in some cases, for example when running with a real OpenGL (ES) 2.0 context.

The usage is identical to QOpenGLFunctions: either query a context-specific instance from QOpenGLContext via QOpenGLContext::extraFunctions() or subclass and use protected inheritance. How the functions get resolved internally (direct call, dynamic resolving via dlsym/GetProcAddress, or resolving via the extension mechanism, i.e. eglGetProcAddress and friends) is completely transparent to the applications. As long as the context is OpenGL ES 3 or a version of OpenGL with the function in question available as an extension, it will all just work.

As an example let’s try to write an application that uses instanced drawing via glDrawArraysInstanced. We want it to run on mobile devices with OpenGL ES 3.0 and any desktop system where OpenGL 3.x (compatibility profile) is available. Needless to say, we want as little branching and variation in the code as possible.

For the impatient, the example is part of Qt and can be browsed online here.

Now let’s take a look at the most important pieces of code that allow the cross-OpenGL-OpenGL ES behavior.

Context versioning

int main(int argc, char *argv[])
{
    QSurfaceFormat fmt;
    fmt.setDepthBufferSize(24);
    if (QOpenGLContext::openGLModuleType() == QOpenGLContext::LibGL) {
        fmt.setVersion(3, 3);
        fmt.setProfile(QSurfaceFormat::CompatibilityProfile);
    } else {
        fmt.setVersion(3, 0);
    }
    QSurfaceFormat::setDefaultFormat(fmt);
    QGuiApplication app(argc, argv);
    ...
}

This looks familiar. Except that now, in addition to opting for version 3.0 with OpenGL ES, we request 3.3 compatibility when running with OpenGL. This is important because we know that instanced drawing is available there too. Therefore glDrawArraysInstanced will be available no matter what.

The fact that we are doing runtime checks instead of ifdefs is due to the dynamic OpenGL implementation loading on some platforms, for example Windows, introduced in Qt 5.4. There it is not necessarily known until runtime if the implementation provides OpenGL (opengl32.dll) or OpenGL ES (ANGLE).

Note that it may still be a good idea to check the actual version after the QOpenGLContext (or QOpenGLWidget, QQuickView, etc.) is initialized, just to be safe. QOpenGLContext::format(), once the context is sucessfully create()’ed, always contains the actual, not the requested, version and other information.

Astute readers may now point out that it should also be possible to request an OpenGL ES context unconditionally in case GLX_EXT_create_context_es2_profile or similar is supported. This would mean that instead of branching based on openGLModuleType(), one could simply set the format’s renderableType to QSurfaceFormat::OpenGLES. The disadvantage is obvious: that approach just won’t work on many systems. Hence we stick to compatibility profile contexts when running with OpenGL.

Shader version directive

    if (QOpenGLContext::currentContext()->isOpenGLES())
        versionedSrc.append(QByteArrayLiteral("#version 300 es\n"));
    else
        versionedSrc.append(QByteArrayLiteral("#version 330\n"));

What’s this? Our shader code is written in the modern GLSL syntax and is simple and compatible enough between GLSL and GLSL ES. However, there is a one line difference we have to take care of: the version directive. This is easy enough to fix up.

Note that with OpenGL implementations supporting GL_ARB_ES3_compatbility this is not needed as these should be able to handle 300 es shaders too. However, in order to target the widest possible range of systems, we avoid relying on that extension for now.

Calling ES 3 functions

    QOpenGLExtraFunctions *f = QOpenGLContext::currentContext()->extraFunctions();
    ...
    f->glDrawArraysInstanced(GL_TRIANGLES, 0, m_logo.vertexCount(), 32 * 36);

And here’s why this is great: the actual API call is the same between OpenGL and OpenGL ES, desktop and mobile or embedded platforms. No checks, conditions, or different wrapper objects are needed.

Below a screenshot of the application running on a Linux PC and on an Android tablet with no changes to the source code. (What’s better than a single Qt logo? A Qt logo composed of 1152 Qt logos of course!)

The hellogles3 example running on Linux. Make sure to check it out live in all its animated glory as the photos do no justice to it.

The exact same app running on Android

Summary

To summarize which API wrapper to use and when, let’s go through the possible options:

• QOpenGLFunctions – The number one choice, unless OpenGL 3/4.x features are desired and the world outside the traditional desktop platforms is not interesting to the application. Cross-platform applications intending to run on the widest possible range of systems are encouraged to to stick to this one, unless they are prepared to guard the usage of OpenGL features not in this class with appropriate runtime checks. QOpenGLFunctions is also what Qt Quick and various other parts of Qt use internally.
• QOpenGLExtraFunctions – Use it in addition to QOpenGLFunctions whenever OpenGL ES 3.0 and 3.1 features are needed.
• Versioned wrappers (QOpenGLFunctions_N_M_profile) – When an OpenGL 3/4.x core or compatibility profile is needed and targeting OpenGL ES based systems is not desired at all.

Bonus problem: the headers

Before going back to coding, an additional issue needs explaining: what about the GL_* constants and typedefs? Where do the OpenGL ES 3.x specific ones come from if the application does not explicitly include GLES3/gl3.h or gl31.h?

As a a general rule Qt applications do not include OpenGL headers themselves. qopengl.h, which is included by the QOpenGL class headers, takes care of this.

• In -opengl es2 builds of Qt, which is typical on mobile and embedded systems, qopengl.h includes the header for the highest possible ES version that was found when running configure for Qt. So if the SDK (sysroot) came with GLES3/gl31.h, then applications will automatically have everything from gl31.h available.
• In -opengl desktop and -opengl dynamic builds of Qt the ES 3.0 and 3.1 constants are available because they are either part of the system’s gl.h or come from Qt’s own internal copy of glext.h, where the latter conveniently gives us all constants up to OpenGL 4.5 and is included as well from qopengl.h.

So in many cases it will all just magically work. There is a problematic scenario, in particular on mobile, though: if only gl2.h was available when building Qt, then applications will not get gl3.h or gl31.h included automatically even if the SDK against which applications are built has those. This can be a problem on Android for example, when Qt is built against an older NDK that does not come with ES 3 headers. For the time being this can be worked around in the applications by explicitly including gl3.h or gl31.h guarded by an ifdef for Q_OS_ANDROID.

That is all for now. For those wishing to hear more about the exciting news in Qt’s graphics world, Qt World Summit is the place to be. See you there in October!

The post Cross-platform OpenGL ES 3 apps with Qt 5.6 appeared first on Qt Blog.

Source :- Qt Blog http://ift.tt/1Fxz8dj

Qt 5.5, computer vision, and the Nitrogen6X

In a previous post we have introduced one of the many new features of the upcoming Qt 5.5: the ability to easily integrate image processing and vision algorithms with camera and video streams via Qt Multimedia. It is now time to see the briefly mentioned OpenCV-based example in more detail.

Qt Quick, Multimedia, and OpenCV on the Nitrogen6X board

Nitrogen6X is a single board computer from Boundary Devices based on the well-known Freescale i.MX6 platform. Combined with the 5 megapixel MIPI camera and the 7″ multi-touch display, all of which Qt supports out of the box, it provides an excellent platform for modern user interfaces and multimedia applications.

To see it all in its full glory, check out the following video:

As always, the application is cross-platform. Below it is seen running on an ordinary Linux PC, with QT_QUICK_CONTROLS_STYLE set to Flat. This allows having the exact same style for the controls as on the actual devices. This previously commercial-only style is now available under LGPLv3 for anyone, on any platform, starting from Qt 5.5. This is excellent news when targeting embedded devices, as those do not have a native look and feel to begin with, and for anyone who is after a consistent, unified experience across desktop, mobile, and embedded platforms.

Qt OpenCV demo on the desktop

The post Qt 5.5, computer vision, and the Nitrogen6X appeared first on Qt Blog.

Source :- Qt Blog http://ift.tt/1eQM8Uy

Integrating custom OpenGL rendering with Qt Quick via QQuickFramebufferObject

Integrating custom OpenGL rendering code, for example to show 3D models, with Qt Quick is a popular topic with Qt users. With the release of Qt 5.0 the standard approach was to use the beforeRendering and afterRendering signals to issue the custom OpenGL commands before or after the rendering of the rest of the scene, thus providing an over- or underlay type of setup.

With Qt 5.2 a new, additional approach has been introduced: QQuickFramebufferObject. This allows placing and transforming the custom OpenGL rendering like any other Quick item, providing the most flexible solution at the expense of rendering via OpenGL framebuffer objects. While it was little known at the time of its original release, it is becoming more and more popular in applications built with Qt 5.4 and 5.5. It is also one of the building blocks for Qt 3D, where the Scene3D element is in fact nothing but a QQuickFramebufferObject under the hood. We will now take a look at this powerful class, look at some examples and discuss the issues that tend to come up quite often when getting started.

First things first: why is this great?

textureinsgnode is the standard example of QQuickFramebufferObject. The Qt logo is rendered directly via OpenGL while having animated transformations applied to the QQuickFramebufferObject-derived item.

This QQuickWidget example also features QQuickFramebufferObject: the Qt logo is a real 3D mesh with a basic phong material rendered via a framebuffer object. The textured quad, rendered using the framebuffer’s associated texture, becomes a genuine item in the scene with other standard Qt Quick elements above and below it. It also demonstrates the usage of custom properties to control the camera.

The API

QQuickFramebufferObject is a QQuickItem-derived class that is meant to be subclassed further. That subclass is then registered to QML and used like any other standard Quick element.

However, having the OpenGL rendering moved to a separate thread, as Qt Quick does on many platforms, introduces certain restrictions on OpenGL resource management. To make this as painless as possible, the design of QQuickFramebufferObject follows the well-known split: subclasses are expected to reimplement a createRenderer() virtual function. This factory function is always called on the scenegraph’s render thread (which is either the main (GUI) thread, or a separate, dedicated one). The returned QQuickFramebufferObject::Renderer subclass belongs to this render thread. All its functions, including the constructor and destructor, are guaranteed to be invoked on the render thread with the OpenGL context bound and therefore it is always safe to create, destroy, or access OpenGL resources.

In practice this is no different than the QQuickItem – QSGNode or the QVideoFilter – QVideoFilterRunnable pattern.

The basic outline of the C++ code usually becomes something like the following:

class FbItemRenderer : public QQuickFramebufferObject::Renderer
{
    QOpenGLFramebufferObject *createFramebufferObject(const QSize &size)
    {
        QOpenGLFramebufferObjectFormat format;
        format.setAttachment(QOpenGLFramebufferObject::CombinedDepthStencil);
        // optionally enable multisampling by doing format.setSamples(4);
        return new QOpenGLFramebufferObject(size, format);
    }

    void render()
    {
        // Called with the FBO bound and the viewport set.
        ... // Issue OpenGL commands.
    }
    ...
}

class FbItem : public QQuickFramebufferObject
{
    QQuickFramebufferObject::Renderer *createRenderer() const
    {
        return new FbItemRenderer;
    }
    ...
}

int main(int argc, char **argv)
{
    ...
    qmlRegisterType("fbitem", 1, 0, "FbItem");
    ...
}

In the QML code we can import and take the custom item into use:

import fbitem 1.0

FbItem {
    anchors.fill: parent
    ... // transform, animate like any other Item
}

This is in fact all you need to get started and have your custom OpenGL rendering shown in a Qt Quick item.

Pitfalls: context and state

When getting started with QQuickFramebufferObject, the common issue is not getting anything rendered, or getting incomplete rendering. This can almost always be attributed to the fact that QQuickFramebufferObject::Renderer subclasses use the same OpenGL context as the Qt Quick scenegraph’s renderer. This is very efficient due to avoiding context switches, and prevents unexpected issues due to sharing and using resources from multiple contexts, but it comes at the cost of having to take care of resetting the OpenGL state the rendering code relies on.

In earlier Qt versions the documentation was not quite clear on this point. Fortunately in Qt 5.5 this is now corrected, and the documentation of render() includes the following notes:

Do not assume that the OpenGL state is all set to the defaults when this function is invoked, or that it is maintained between calls. Both the Qt Quick renderer and the custom rendering code uses the same OpenGL context. This means that the state might have been modified by Quick before invoking this function.

It is recommended to call QQuickWindow::resetOpenGLState() before returning. This resets OpenGL state used by the Qt Quick renderer and thus avoids interference from the state changes made by the rendering code in this function.

What does this mean in practice?

As an example, say you have some old OpenGL 1.x fixed pipeline code rendering a not-very-3D triangle. Even in this simplest possible case you may need to ensure that certain state is disabled: we must not have a shader program active and depth testing must be disabled. glDisable(GL_DEPTH_TEST); glUseProgram(0); does the job. For rendering that is more 3D oriented, a good example of a very common issue is forgetting to re-enable depth writes via glDepthMask(GL_TRUE).

Although not necessary in many cases, it is good practice to reset all the state the Qt Quick renderer potentially relies on when leaving render(). Since Qt 5.2 a helper function is available to do this: resetOpenGLState(). To access this member function, store a pointer to the corresponding item in the Renderer and do the following just before returning from render():

    m_item->window()->resetOpenGLState();

Note that this must be the last thing we do in render() because it also alters the framebuffer binding. Issuing further rendering commands afterwards will lead to unexpected results.

Custom properties and data synchronization

Accessing the Renderer object’s associated item looks easy. All we need is to store a pointer, right? Be aware however that this can be very dangerous: accessing data at arbitrary times from the Renderer subclass is not safe because the functions there execute on the scenegraph’s render thread while ordinary QML data (like the associated QQuickFramebufferObject subclass) live on the main (GUI) thread of the application.

As an example, say that we want to expose the rotation of some model in our 3D scene to QML. This is extremely handy since we can apply bindings and animations to it, utilizing the full power of QML.

class FbItem : public QQuickFramebufferObject
{
    Q_OBJECT
    Q_PROPERTY(QVector3D rotation READ rotation WRITE setRotation NOTIFY rotationChanged)

public:
    QVector3D rotation() const { return m_rotation; }
    void setRotation(const QVector3D &v);
    ...
private:
    QVector3D m_rotation;
};

So far so good. Now, how do we access this vector in the FbItemRenderer’s render() implementation? At first, simply using m_item->rotation() may look like the way to go. Unfortunately it is unsafe and therefore wrong.

Instead, we must maintain a copy of the relevant data in the renderer object. The synchronization must happen at a well-defined point, where we know that the main (GUI) thread will not access the data. The render() function is not suitable for this. Fortunately, QQuickFramebufferObject makes all this pretty easy with the Renderer’s synchronize() function:

class FbItemRenderer : public QQuickFramebufferObject::Renderer                                                         
{                                                                                                                       
public:                                                                                                                 
    void synchronize(QQuickFramebufferObject *item) {
        FbItem *fbitem = static_cast<FbItem *>(item);
        m_rotation = fbitem->rotation();
    }
    ...
private:
    QVector3D m_rotation; // render thread's copy of the data, this is what we will use in render()
}

When synchronize() is called, the main (GUI) thread is guaranteed to be blocked and therefore it cannot read or write the FbItem’s data. Accessing those on the render thread is thus safe.

Triggering updates

To schedule a new call to render(), applications have two options. When on the main (GUI) thread, or inside JavaScript code, use QQuickItem::update(). In addition, QQuickFramebufferObject::Renderer adds its own update() function, which is only usable from the render thread, for example inside render() to schedule a re-rendering of the FBO content on the next frame.

In practice the most common case is to schedule a refresh for the FBO contents when some property changes. For example, the setRotation() function from the above example could look like this:

void FbItem::setRotation(const QVector3D &v)
{
    if (m_rotation != v) {
        m_rotation = v;
        emit rotationChanged();
        update();
    }
}

Internals

Finally, it is worth noting that QQuickFramebufferObject is also prepared to handle multisampling and screen changes.

Whenever the QOpenGLFramebufferObject returned from createFramebufferObject() has multisampling enabled (the QOpenGLFramebufferObjectFormat has a sample count greater than 0 and multisampled framebuffers are supported by the OpenGL implementation in use), the samples will be resolved with a glBlitFramebuffer call into a non-multisampled FBO after each invocation of render(). This is done transparently to the applications so they do not have to worry about it.

Resizing the window with the Qt Quick content will lead to recreating the FBO because its size should change too. The application will see this as a repeated call to createFramebufferObject() with the new size. It is possible to opt out of this by setting the textureFollowsItemSize property to false.

However, there is a special case when the FBO has to get recreated regardless: when moving the window to a screen with a different device pixel ratio. For example moving a window between a retina and non-retina screen on OS X systems will inherently need a new, double or half sized, framebuffer, even when the window dimensions are the same in device independent units. Just like with ordinary resizes, Qt is prepared to handle this by requesting a new framebuffer object with a different size when necessary. One possible pitfall here is the applications’ caching of the results of the factory functions: avoid this. createFramebufferObject() and createRenderer() must never cache their return value. Just create a new instance and return it. Keep it simple. Qt takes care of managing the returned instance and destroying it when the time comes. Happy hacking!

The post Integrating custom OpenGL rendering with Qt Quick via QQuickFramebufferObject appeared first on Qt Blog.

Source :- Qt Blog http://ift.tt/1zS7ASf

Qt Quick with the power of OpenCL on Embedded Linux devices

Many Qt users encounter a need to integrate GPU compute solutions into their Qt-based applications. What is more, with the advent of compute API implementations and powerful GPUs on embedded devices, using OpenCL or CUDA on an Embedded Linux device is a reality now. In a previous post we looked at NVIDIA’s Jetson TK1 board and discovered how easy it is to get started with CUDA development in Qt applications using OpenGL. When it comes to OpenCL, developers are not left out in the cold either, thanks to Hardkernel’s ODROID-XU3, where the ARM Mali-T628 graphics processor provides full OpenCL 1.1 support with CL-GL interop in addition to OpenGL ES 3.0.

In this post we will take a look at a simple and powerful approach to integrating Qt Quick applications and OpenCL. We will focus on use cases that involve sharing OpenGL resources like textures or buffers between OpenGL and OpenCL. The examples demonstrate three standard compute use cases and we will see them running on an actual ODROID board.

ODROID-XU3

Why OpenCL and Qt?

The ability to perform complex, highly parallel computations on embedded devices while keeping as much data on the GPU as possible and to visualize the results with Qt Quick and touch-friendly Qt Quick Controls open the door for easily creating embedded systems performing advanced tasks in the domain of computer vision, robotics, image and signal processing, bioinformatics, and all sorts of heavyweight data crunching. As an example, think of gesture recognition: with high resolution webcams, Qt Multimedia, Qt Quick, Qt Quick Controls, and the little framework presented below, applications can focus on the things that matter: the algorithms (OpenCL kernels) performing the core of the work and the C++ counterpart that enqueues these kernels. The rest is taken care of by Qt.

Looking back: Qt OpenCL

OpenCL is not unknown to Qt – once upon a time, back in the Qt 4 days, there used to be a Qt OpenCL module, a research project developed in Brisbane. It used to contain a full 1:1 API wrapper for OpenCL 1.0 and 1.1, and some very helpful classes to get started with CL-GL interop.

Today, with the rapid evolution of the OpenCL API, the availability of an official C++ wrapper, and the upcoming tighter C++ integration approaches like SYCL, we belive there is little need for straightforward Qt-ish wrappers. Applications are encouraged to use the OpenCL C or C++ APIs as they see fit. However, when it comes to the helpers that simplify common tasks like choosing an OpenCL platform and device so that we get interoperability with OpenGL, they turn out to be really handy. Especially when writing cross-platform applications. Case in point: Qt Multimedia 5.5 ships with an OpenCL-based example as presented in the video filters introduction post. The OpenCL initialization boilerplate code in that example is unexpectedly huge. This shows that the need for modern, Qt 5 based equivalents of the old Qt OpenCL classes like QCLContextGL has not gone away. In fact, with the ubiquity of OpenCL and OpenGL on all kinds of devices and platforms, they are more desirable than ever.

Qt 5.5 on the ODROID-XU3

Qt 5.5 introduces support for the board in the device makespec linux-odroid-xu3-g++. Just pass -device odroid-xu3 to configure.

For example, to build release mode binaries with a toolchain borrowed from the Raspberry Pi, assuming a sysroot at ~/odroid/sysroot:

./configure -release -prefix /usr/local -extprefix ~/odroid/sysroot/usr/local -hostprefix ~/odroid/qt5-build -device odroid-xu3 -device-option CROSS_COMPILE=~/odroid/toolchain/arm-bcm2708/gcc-linaro-arm-linux-gnueabihf-raspbian-x64/bin/arm-linux-gnueabihf- -sysroot ~/odroid/sysroot -nomake examples -nomake tests -opengl es2

This will configure the Qt libraries and target tools like qmlscene to be deployed under /usr/local in the sysroot, while the host tools – like the x86 build of qmake that is to be used when building applications afterwards – get installed into ~/odroid/qt5-build.

When it comes to the platform plugins, both xcb and eglfs are usable, but only one at a time: the Mali graphics driver binary is different for X11 and fbdev, and has to be switched accordingly. The Ubuntu image from Hardkernel comes with X11 in place. While OpenGL is usable under X too, the usage of eglfs and the fbdev drivers is recommended, as usual.

For more information on the intricacies and a step by step guide to deploying Qt on top of the Hardkernel image, see this wiki page.

It is worth noting that thanks to Qt’s Android port, running a full Android system with Qt apps on top is also feasible on this board.

Time for some action

Now to the fun part. Below are three examples running on the framebuffer in full HD resolution with the fbdev Mali driver variant, Qt 5.5 and the eglfs platform plugin. All of them utilize OpenCL 1.1, CL-GL interop, and are regular Qt Quick 2 applications. They all utilize the little example framework which we call Qt Quick CL for now.

OpenGL texture to OpenGL texture via OpenCL

An OpenCL-based alternative for ShaderEffect

First, let’s take a look at a standard image processing use case: we will execute one or more OpenCL kernels on our input, which can be a Qt Quick Image element, a (potentially invisible) sub-tree of the scene, or any texture provider, and generate a new texture. With CL-GL interop the data never leaves the GPU: no pixel data is copied between the CPU and the GPU. Those familiar with Qt Quick have likely realized already that this is in fact an OpenCL-based alternative to the built-in, GLSL-based ShaderEffect items.

By using the easy-to-use base classes to automatically and transparently manage OpenCL and CL-GL initialization, and to hide the struggles and gotchas of Qt Quick’s dedicated render thread and OpenGL contexts, the meat of the above application gets reduced to something like the following:

class CLRunnable : public QQuickCLImageRunnable
{
public:
    CLRunnable(QQuickCLItem *item)
        : QQuickCLImageRunnable(item)
    {
        m_clProgram = item->buildProgramFromFile(":/kernels.cl");
        m_clKernel = clCreateKernel(m_clProgram, "Emboss", 0);
    }
    ~CLRunnable() {
        clReleaseKernel(m_clKernel);
        clReleaseProgram(m_clProgram);
    }
    void runKernel(cl_mem inImage, cl_mem outImage, const QSize &size) Q_DECL_OVERRIDE {
        clSetKernelArg(m_clKernel, 0, sizeof(cl_mem), &inImage);
        clSetKernelArg(m_clKernel, 1, sizeof(cl_mem), &outImage);
        const size_t workSize[] = { size_t(size.width()), size_t(size.height()) };
        clEnqueueNDRangeKernel(commandQueue(), m_clKernel, 2, 0, workSize, 0, 0, 0, 0);
    }
private:
    cl_program m_clProgram;
    cl_kernel m_clKernel;
};

class CLItem : public QQuickCLItem
{
    Q_OBJECT
    Q_PROPERTY(QQuickItem *source READ source WRITE setSource)
public:
    CLItem() : m_source(0) { }
    QQuickCLRunnable *createCL() Q_DECL_OVERRIDE { return new CLRunnable(this); }
    QQuickItem *source() const { return m_source; }
    void setSource(QQuickItem *source) { m_source = source; update(); }
private:
    QQuickItem *m_source;
};

...
qmlRegisterType("quickcl.qt.io", 1, 0, "CLItem")
...

import quickcl.qt.io 1.0

Item {
    Item {
        id: src
        layer.enabled: true
        ...
    }
    CLItem {
        id: clItem
        source: src
        ...
    }
}

Needless to say, the application works on a wide variety of platforms. Windows, OS X, Android, and Linux are all good as long as OpenGL (ES) 2.0, OpenCL 1.1 and CL-GL interop are available. Getting started with OpenCL in Qt Quick applications won’t get simpler than this.

OpenGL texture to arbitrary data via OpenCL

Histogram in Qt Quick directly on the GPU

And now something more complex: an image histogram. Histograms are popular with Qt, and the recent improvements in Qt Multimedia introduce the possibility of efficiently calculating live video frame histograms on the GPU.

In this example we take it to the next level: the input is an arbitrary live sub-tree of the Qt Quick scene, while the results of the calculation are visualized with a little Javascript and regular OpenGL-based Qt Quick elements. Those 256 bars on the right are nothing else but standard Rectangle elements. The input image never leaves the GPU, naturally. All this with a few lines of C++ and QML code.

OpenGL vertex buffer generation with OpenCL

VBO generation from OpenCL

Last but not least, something other than GL textures and CL image objects: buffers! The position of the vertices, that get visualized with GL by drawing points, are written to the vertex buffer using OpenCL. The data is then used from GL as-is, no readbacks and copies are necessary, unlike with Qt Quick’s own GL-based particle systems.

To make it all more exciting, the drawing happens inside a custom QQuickItem that functions similarly to QQuickFramebufferObject. This allows us to mix our CL-generated drawing with the rest of the scene, including Qt Quick Controls when necessary.

Looking forward: Qt Quick CL

QtQuickCL is a small research and demo framework for Qt 5 that enables easily creating Qt Quick items that execute OpenCL kernels and use OpenGL resources as their input or output. The functionality is intentionally minimal but powerful. All the CL-GL interop, including the selection of the correct CL platform and device, is taken care of by the module. The QQuickCLItem – QQuickCLRunnable split in the API ensures easy and safe CL and GL resource management even when Qt Quick’s threaded render loop is in use. Additional convenience is provided for the cases when the input, output or both are OpenGL textures, like for instance the first two of the three examples shown above.

The code, including the three examples shown above, is all available on Gerrit and code.qt.io as a qt-labs repository. The goal is not to provide a full-blown OpenCL framework or wrapper, but rather to serve as a useful example and reference for integrating Qt Quick and OpenCL, and to help getting started with OpenCL development. Happy hacking!

The post Qt Quick with the power of OpenCL on Embedded Linux devices appeared first on Qt Blog.

Source :- Qt Blog http://ift.tt/1Jm5oCd

Introducing video filters in Qt Multimedia

Qt Multimedia makes it very easy to get a video stream from the camera or a video file rendered as part of your application’s Qt Quick scene. What is more, its modularized backend and video node plugin system allows to provide hardware accelerated, zero copy solutions on platforms where such an option is available. All this is hidden from the applications when using Qt Quick elements like Camera, MediaPlayer and VideoOutput, which is great. But what if you want to do some additional filtering or computations on the video frames before they are presented? For example because you want to transform the frame, or compute something from it, on the GPU using OpenCL or CUDA. Or because you want to run some algorithms provided by OpenCV. Before Qt 5.5 there was no easy way to do this in combination with the familiar Qt Quick elements. With Qt 5.5 this is going to change: say hello to QAbstractVideoFilter.

QAbstractVideoFilter serves as a base class for classes that are exposed to the QML world and are instantiated from there. They are then associated with a VideoOutput element. From that point on, every video frame the VideoOutput receives is run through the filter first. The filter can provide a new video frame, which is used in place of the original, calculate some results or both. The results of the computation are exposed to QML as arbitrary data structures and can be utilized from Javascript. For example, an OpenCV-based object detection algorithm can generate a list of rectangles that is exposed to QML. The corresponding Javascript code can then position some Rectangle elements at the indicated locations.

Let’s see some code

    import QtQuick 2.3
    import QtMultimedia 5.5
    import my.cool.stuff 1.0

    Item {
        Camera {
            id: camera
        }
        VideoOutput {
            source: camera
            anchors.fill: parent
            filters: [ faceRecognitionFilter ]
        }
        FaceRecognizer {
            id: faceRecognitionFilter
            property real scaleFactor: 1.1 // Define properties either in QML or in C++. Can be animated too.
            onFinished: {
                console.log("Found " + result.rects.length + " faces");
                ... // do something with the rectangle list
            }
        }
    }

The new filters property of VideoOutput allows to associate one or more QAbstractVideoFilter instances with it. These are then invoked in order for every incoming video frame.

The outline of the C++ implementation is like this:

QVideoFilterRunnable *FaceRecogFilter::createFilterRunnable()
{
    return new FaceRecogFilterRunnable(this);
}
...
QVideoFrame FaceRecogFilterRunnable::run(QVideoFrame *input, const QVideoSurfaceFormat &surfaceFormat, RunFlags flags)
{
    // Convert the input into a suitable OpenCV image format, then run e.g. cv::CascadeClassifier,
    // and finally store the list of rectangles into a QObject exposing a 'rects' property.
    ...
    emit m_filter->finished(result);
    return *input;
}   
...
int main(..)
{
    ...
    qmlRegisterType<FaceRecogFilter>("my.cool.stuff", 1, 0, "FaceRecognizer");
    ...
}

Here our filter implementation simply passes the input video frame through, while generating a list of rectangles. This can then be examined from QML, in the finished signal handler. Simple and flexible.

While the registration of our custom filter happens from the main() function in the example, the filter can also be provided from QML extension plugins, independently from the application.

The QAbstractVideoFilter – QVideoFilterRunnable split mirrors the approach with QQuickItem – QSGNode. This is essential in order to support threaded rendering: when the Qt Quick scenegraph is using its threaded render loop, all rendering (the OpenGL operations) happen on a dedicated thread. This includes the filtering operations too. Therefore we have to ensure that the graphics and compute resources live and are only accessed on the render thread. A QVideoFilterRunnable always lives on the render thread and all its functions are guaranteed to be invoked on that thread, with the Qt Quick scenegraph’s OpenGL context bound. This makes creating filters relying on GPU compute APIs easy and painless, even when OpenGL interop is involved.

GPU compute and OpenGL interop

All this is very powerful when it comes to avoiding copies of the pixel data and utilizing the GPU as much as possible. The output video frame can be in any supported format and can differ from the input frame, for instance a GPU-accelerated filter can upload the image data received from the camera into an OpenGL texture, perform operations on that (using OpenCL – OpenGL interop for example) and provide the resulting OpenGL texture as its output. This means that after the initial texture upload, which is naturally in place even when not using any QAbstractVideoFilter at all, everything happens on the GPU. When doing video playback, the situation is even better on some platforms: in case the input is already an OpenGL texture, D3D texture, EGLImage or similar, we can potentially perform everything on the GPU without any readbacks or copies.

The OpenCL-based example that comes with Qt Multimedia demonstrates this well. Shown below running on OS X, the input frames from the video already contain OpenGL textures. All we need to do is to use OpenCL’s GL interop to get a CL image object. The output image object is also based on a GL texture, allowing us to pass it to VideoOutput and the Qt Quick scenegraph as-is.

Real-time image transformation on the GPU with OpenCL, running on OS X

While the emboss effect is not really interesting and can also be done with OpenGL shaders using ShaderEffect items, the example proves that integrating OpenCL and similar APIs with Qt Multimedia does not have to be hard – in fact the code is surprisingly simple and yet so powerful.

It is worth pointing out that filters that do not result in a modified image and are not interested in staying in sync with the displayed frames do not have to block until the computation is finished: the implementation of run() can queue the necessary operations without waiting for them to finish. A signal indicating the availability of the computation results is then emitted later, for example from the associated event callback in case of OpenCL. All this is made possible by the thread-awareness of Qt’s signals: the signal emission will work equally well regardless of which thread the callback is invoked on.

CPU-based filtering

Not all uses of video filters will rely on the GPU. Today’s PCs and even many embedded devices are powerful enough to perform many algorithms on the CPU. Below is a screenshot from the finished version of the code snippet above. Instead of faces, we recognize something more exciting, namely Qt logos:

Qt logo recognition with OpenCV and a webcam in a Qt Quick application using Qt Multimedia and Qt Quick Controls.

The application’s user interface is fully QML-based, even the rectangles are actual Rectangle elements. It is shown here running on a desktop Linux system, where the video frames from the camera are provided as YUV image data in system memory. However, it functions identically well on Embedded Linux devices supported by Qt Multimedia, for example the i.MX6-based Sabre SD board. Here comes the proof:

The same application, using the on-board MIPI camera

And it just works, with the added bonus of the touch-friendly controls from the Flat style.

The post Introducing video filters in Qt Multimedia appeared first on Qt Blog.

Source :- Qt Blog http://ift.tt/1C3NEL8

Qt Weekly #28: Qt and CUDA on the Jetson TK1

NVIDIA’s Jetson TK1 is a powerful development board based on the Tegra K1 chip. It comes with a GPU capable of OpenGL 4.4, OpenGL ES 3.1 and CUDA 6.5. From Qt’s perspective this is a somewhat unorthodox embedded device because its customized Linux system is based on Ubuntu 14.04 and runs the regular X11 environment. Therefore the approach that is typical for low and medium-end embedded hardware, running OpenGL-accelerated Qt apps directly on the framebuffer using the eglfs platform plugin, will not be suitable.

In addition, the ability to do hardware-accelerated computing using CUDA is very interesting, especially when it comes to interoperating with OpenGL. Let’s take a look at how CUDA code can be integrated with a Qt-based application.

The board

Building Qt

This board is powerful enough to build everything on its own without any cross-compilation. Configuring and building Qt is no different than in any desktop Linux environment. One option that needs special consideration however is -opengl es2 because Qt can be built either in a GLX + OpenGL or EGL + OpenGL ES configuration.

For example, the following configures Qt to use GLX and OpenGL:

configure -release -nomake examples -nomake tests

while adding -opengl es2 requests the usage of EGL and OpenGL ES:

configure -release -opengl es2 -nomake examples -nomake tests

If you are planning to run applications relying on modern, non-ES OpenGL features, or use CUDA, then go for the first. If you however have some existing code from the mobile or embedded world relying on EGL or OpenGL ES then it may be useful to go for #2.

The default platform plugin will be xcb, so running Qt apps without specifying the platform plugin will work just fine. This is the exact same plugin that is used on any ordinary X11-based Linux desktop system.

Vsync gotchas

Once the build is done, you will most likely run some OpenGL-based Qt apps. And then comes the first surprise: applications are not synchronized to the vertical refresh rate of the screen.

When running for instance the example from qtbase/examples/opengl/qopenglwindow, we expect a nice and smooth 60 FPS animation with the rendering thread throttled appropriately. This unfortunately isn’t the case. Unless the application is fullscreen. Therefore many apps will want to replace calls like show() or showMaximized() with showFullScreen(). This way the thread is throttled as expected.

A further surprise may come in QWidget-based applications when opening a popup or a dialog. Unfortunately this also disables synchronization, even though the main window still covers the entire screen. In general we can conclude that the standard embedded recommendation of sticking to a single fullscreen window is very valid for this board too, even when using xcb, although for completely different reasons.

CUDA

After installing CUDA, the first and in fact the only challenge is to tackle the integration of nvcc with out Qt projects.

Unsurprisingly, this has been tackled by others before. Building on this excellent article, the most basic integration in our .pro file could look like this:

... # QT, SOURCES, HEADERS, the usual stuff 

CUDA_SOURCES = cuda_stuff.cu

CUDA_DIR = /usr/local/cuda
CUDA_ARCH = sm_32 # as supported by the Tegra K1

INCLUDEPATH += $$CUDA_DIR/include
LIBS += -L $$CUDA_DIR/lib -lcudart -lcuda
osx: LIBS += -F/Library/Frameworks -framework CUDA

cuda.commands = $$CUDA_DIR/bin/nvcc -c -arch=$$CUDA_ARCH -o ${QMAKE_FILE_OUT} ${QMAKE_FILE_NAME}
cuda.dependency_type = TYPE_C
cuda.depend_command = $$CUDA_DIR/bin/nvcc -M ${QMAKE_FILE_NAME}
cuda.input = CUDA_SOURCES
cuda.output = ${QMAKE_FILE_BASE}_cuda.o
QMAKE_EXTRA_COMPILERS += cuda

In addition to Linux this will also work out of the box on OS X. Adapting it to Windows should be easy. For advanced features like reformatting nvcc’s error messages to be more of Creator’s liking, see the article mentioned above.

A QOpenGLWindow-based application that updates an image via CUDA on every frame could now look something like the following. The approach is the same regardless of the OpenGL enabler in use: QOpenGLWidget or a custom Qt Quick item would operate along the same principles: call cudaGLSetGLDevice when the OpenGL context is available, register the OpenGL resources to CUDA, and then do map – invoke CUDA kernel – unmap – draw on every frame.

Note that in this example we are using a single pixel buffer object. There are other ways to do interop, for example we could have registered the GL texture, got a CUDA array out of it and bound that either to a CUDA texture or surface.

...
// functions from cuda_stuff.cu
extern void CUDA_init();
extern void *CUDA_registerBuffer(GLuint buf);
extern void CUDA_unregisterBuffer(void *res);
extern void *CUDA_map(void *res);
extern void CUDA_unmap(void *res);
extern void CUDA_do_something(void *devPtr, int w, int h);

class Window : public QOpenGLWindow, protected QOpenGLFunctions
{
public:
    ...
    void initializeGL();
    void paintGL();

private:
    QSize m_imgSize;
    GLuint m_buf;
    GLuint m_texture;
    void *m_cudaBufHandle;
};

...

void Window::initializeGL()
{
    initializeOpenGLFunctions();
    
    CUDA_init();

    QImage img("some_image.png");
    m_imgSize = img.size();
    img = img.scaled(m_imgSize).convertToFormat(QImage::Format_RGB32); // BGRA on little endian
    
    glGenBuffers(1, &m_buf);
    glBindBuffer(GL_PIXEL_UNPACK_BUFFER, m_buf);
    glBufferData(GL_PIXEL_UNPACK_BUFFER, m_imgSize.width() * m_imgSize.height() * 4, img.constBits(), GL_DYNAMIC_COPY);
    glBindBuffer(GL_PIXEL_UNPACK_BUFFER, 0);

    m_cudaBufHandle = CUDA_registerBuffer(m_buf);

    glGenTextures(1, &m_texture);
    glBindTexture(GL_TEXTURE_2D, m_texture);

    glTexImage2D(GL_TEXTURE_2D, 0, GL_RGBA8, m_imgSize.width(), m_imgSize.height(), 0, GL_BGRA, GL_UNSIGNED_BYTE, 0);

    glTexParameteri(GL_TEXTURE_2D, GL_TEXTURE_MIN_FILTER, GL_LINEAR);
    glTexParameteri(GL_TEXTURE_2D, GL_TEXTURE_MAG_FILTER, GL_LINEAR);
}

void Window::paintGL()
{
    glClear(GL_COLOR_BUFFER_BIT);

    void *devPtr = CUDA_map(m_cudaBufHandle);
    CUDA_do_something(devPtr, m_imgSize.width(), m_imgSize.height());
    CUDA_unmap(m_cudaBufHandle);

    glBindBuffer(GL_PIXEL_UNPACK_BUFFER, m_buf);
    glBindTexture(GL_TEXTURE_2D, m_texture);
    // Fast path due to BGRA
    glTexSubImage2D(GL_TEXTURE_2D, 0, 0, 0, m_imgSize.width(), m_imgSize.height(), GL_BGRA, GL_UNSIGNED_BYTE, 0);
    glBindBuffer(GL_PIXEL_UNPACK_BUFFER, 0);

    ... // do something with the texture

    update(); // request the next frame
}
...

The corresponding cuda_stuff.cu:

#include <stdio.h>
#ifdef Q_OS_MAC
#include <OpenGL/gl.h>
#else
#include <GL/gl.h>
#endif
#include <cuda.h>
#include <cuda_gl_interop.h>

void CUDA_init()
{
    cudaDeviceProp prop;
    int dev;
    memset(&prop, 0, sizeof(cudaDeviceProp));
    prop.major = 3;
    prop.minor = 2;
    if (cudaChooseDevice(&dev, &prop) != cudaSuccess)
        puts("failed to choose device");
    if (cudaGLSetGLDevice(dev) != cudaSuccess)
        puts("failed to set gl device");
}

void *CUDA_registerBuffer(GLuint buf)
{
    cudaGraphicsResource *res = 0;
    if (cudaGraphicsGLRegisterBuffer(&res, buf, cudaGraphicsRegisterFlagsNone) != cudaSuccess)
        printf("Failed to register buffer %u\n", buf);
    return res;
}

void CUDA_unregisterBuffer(void *res)
{
    if (cudaGraphicsUnregisterResource((cudaGraphicsResource *) res) != cudaSuccess)
        puts("Failed to unregister resource for buffer");
}

void *CUDA_map(void *res)
{
    if (cudaGraphicsMapResources(1, (cudaGraphicsResource **) &res) != cudaSuccess) {
        puts("Failed to map resource");
        return 0;
    }
    void *devPtr = 0;
    size_t size;
    if (cudaGraphicsResourceGetMappedPointer(&devPtr, &size, (cudaGraphicsResource *) res) != cudaSuccess) {
        puts("Failed to get device pointer");
        return 0;
    }
    return devPtr;
}

void CUDA_unmap(void *res)
{
    if (cudaGraphicsUnmapResources(1,(cudaGraphicsResource **) &res) != cudaSuccess)
        puts("Failed to unmap resource");
}

__global__ void run(uchar4 *ptr)
{
    int x = threadIdx.x + blockIdx.x * blockDim.x;
    int y = threadIdx.y + blockIdx.y * blockDim.y;
    int offset = x + y * blockDim.x * gridDim.x;

    ...
}

void CUDA_do_something(void *devPtr, int w, int h)
{
    const int blockSize = 16; // 256 threads per block
    run<<<dim3(w / blockSize, h / blockSize), dim3(blockSize, blockSize)>>>((uchar4 *) devPtr);
}

This is all that’s needed to integrate the power of Qt, OpenGL and CUDA. Happy hacking!

Source :- Qt Blog http://ift.tt/1DO8VYE