Qt Quick on Vulkan, Metal, and Direct3D

Now that the first beta of Qt 5.14 is getting closer, it is time to start talking about one of the big new features. We cannot possibly cover all the details around the graphics stack improvements and the road to Qt 6 in one post, so in part 1 (this blog) and part 2 we will describe the background and take a closer look at what 5.14 will ship with, and then dive into the technical details and future directions in another set of posts later on.

The 5.14 new features page mentions: Added the first preview of the graphics API independent scenegraph renderer as an opt-in feature. This allows running qualifying Qt Quick applications on top of Vulkan, Metal, or Direct3D 11 instead of OpenGL.

What does this mean in practice?

As outlined in the Qt 6 Technical Vision post, one of the main goals in Qt 6 is to move away from direct OpenGL usage in the most places in Qt, and, with the appropriate abstractions in place, allow operating on a wider variety of graphics APIs, such as, Vulkan, Metal, and Direct3D. OpenGL (and OpenGL ES) remain an option, naturally. The primary motivation behind this is not gaining performance, but ensuring Qt Everywhere stays true in the future too, even on platforms and devices where OpenGL is either not available or not desired anymore. At the same time, being able to build on the modern, lower-level, explicit APIs can also open up possibilities when it comes to improving performance (e.g., lower CPU usage due to less API overhead) and new ways of doing things in the rendering engines behind Qt Quick and other modules, like the recently announced Qt Quick 3D.

In addition, being able to render user interfaces with the platform's primary, best supported graphics API is great news for applications that do their own native 2D or 3D graphics rendering while using Qt to render the UI. In such a case Qt is often not in the driving seat when it comes to deciding what graphics API to use: for example, if a desktop application on macOS wishes to use Metal for its own 3D content while relying on Qt Quick to render the 2D UI elements, it is then highly beneficial if Qt Quick also renders via Metal. This will sound familar to those who have been following the evolution of graphics in Qt 5.x: conceptually this is no different from when support for operating in OpenGL core profile contexts got introduced in the Qt Quick renderer - Qt Quick itself has no need for or interest in that, but in order to allow integrating external rendering code that is tied to core profile features, Qt Quick had to be made aware and capable to deal with it. So in this sense the story is a natural continuation of what we had in Qt 5, now being expanded to cover graphics APIs other than OpenGL.

All this can now raise two obvious questions:

• How is this relevant to Qt 5.x? Isn't this all Qt 6 material?
• Why don't you just use <name of some graphics API translation solution> and standardize on <API name>?

So what's there in Qt 5.14?

Rolling out the complete overhaul of the graphics bits in all (or at least most) places in Qt is indeed something targeted for Qt 6. However, stopping work on 5.x and attempting to invent, develop and refactor everything in one big go, hoping everything will turn out to be fine, is not very appealing in practice. As Qt developers used to (and still do) say, the first iteration of any API is likely to be suboptimal. So instead we take the develop side-by-side approach, focusing on one certain UI technology in Qt: Qt Quick.

Qt 5.14 is expected to ship with a preview of the new Qt Quick rendering path. By default this is inactive and so there are no changes visible whatsoever to applications - internally they then go through the same direct OpenGL based code path as they did in earlier versions. Those who wish to try out the new approach can however opt-in either by setting an environment variable or requesting it via a C++ API in main(). The hope here is to get feedback early on, allowing us to iterate and evolve without having to wait for the release of Qt 6.0.

Taking a look at the snapshot of the Qt 5.14 documentation reveals the following:

Not all application will work out of the box when running with QSG_RHI set. Custom QQuickItem implementations with scenegraph nodes performing direct OpenGL calls or containing GLSL shader code in custom materials will not be functional when enabling RHI-based rendering. The same applies to ShaderEffect items with GLSL source code in them. The solutions for doing custom materials and effects the modern way are already there, mostly, but these need migrating applications accordingly. Early adopters may experiment with this already in the 5.14 and 5.15 time frame, but widespread adoption and migration is naturally not expected before Qt 6.0. On the other hand, many existing QML applications will likely just work, even with the underlying rendering engine going through a completely different API, like Vulkan or Metal.

Why not translation layer XYZ?

First of all, it is important to state that the possibility of using API and shader translation layers like MoltenVK, MoltenGL, ANGLE, Zink, and others is still there, even if not always available out of the box. MoltenVK, for example, allows rendering Qt Quick UIs via Vulkan on macOS as well. If a Qt Quick application wishes to only ever use Vulkan, and still wants to operate on macOS, MoltenVK is an option. (as long as an appropriately configured Qt build is used an deployed, MoltenVK is available on the users' systems, etc.)

Making such a translation layer a mandatory dependency, and therefore including and deploying it with Qt, is a very different story.

• Needless to say, changing Qt Everywhere to Qt Only Where External Dependencies Allow is not ideal.
  
  Qt targets a lot more platforms and environments than one typically would think. It can only really rely on mandatory 3rd party dependencies that compile and work in "exotic" environments too, and are easily adaptable should special needs arise. (think INTEGRITY, QNX, customized embedded Linux environments, systems with broken or in-development graphics stacks, the occasional need to adapt to proprietary bits and pieces, or sometimes having to interoperate - perhaps in a vendor-specific, non-standard way - with various graphics or composition APIs you thought were long dead, etc. - all this needs flexibility and customizability on every level in Qt's rendering stack)
  
  
• Doing translation between shading languages (or intermediate formats) at run time is not quite ideal. The shader pipeline in Qt 6 is expected to focus more on doing things off line, or at application build time at latest. With a translation layer in the middle, hiding reality (what API, what shading languaged is really used), that effort quickly becomes futile in practice as we cannot prepare for, or cannot feed in, the shaders or bytecode for the real underlying API.
  
  
• Some of the options mentioned are out of question due to the current state of reality: OpenGL (ES) is and will still be the primary workhorse in the foreseeable future on many devices. Therefore keeping on directly using a single API could only mean that API being OpenGL, or a translation layer that can target OpenGL (and is efficient enough for low performing devices as well).
  
  
• The cases where Qt's rendering engines  are supplemented by the applications' custom, native rendering code tend to work best when both parties use the same APIs directly. Using a translation layer is not always a blocker in this respect, if they allow accessing the underlying native objects (think for example how Direct3D - Qt Quick interop is made possible by EGL extensions when running with Qt 5 on top of ANGLE on Windows), but this is not always in place, and when it is, it can present another set of hassles to deal with.
  
  
• There are licensing implications and problems as well. Think Apache 2.0 being incompatible with GPLv2. Relying on commercial-only solutions would be out of question in any case.
  
  
• From experience (some with ANGLE and a little with MoltenVK), using such solutions is never as simple as one initially hopes. At some point the effort needed to keep all the options up and running may become too big - that effort is then better invested in doing things "right" directly with the native API instead. The intrinsically platform dependent nature of some of these translation solutions is not ideal either - if we need to pull in another one for every platform Qt targets, the situation quickly becomes untenable.

So instead of going the route of relying on low-level API translators, Qt defines its own high level abstraction for 3D graphics (for internal use, not exposed to applications for the time being). This is then backed by API-specific backend implementations, a pattern familiar from many components in Qt. In some cases the backend is platform specific by nature (Metal, D3D), while in some others one backend targets one API but multiple platforms (Vulkan, OpenGL). This is complemented by a new shader management pipeline, building on a few 3rd party projects like glslang and SPIRV-Cross. More details about all this is going to come in later posts. For now let's look a bit higher in the stack, and see what this enables on the level of Qt Quick in Qt 5.14.

Does it really work?

Let's look at an example, namely the well-known Qt5 Cinematic Experience demo application from QUIt Coding. We are using the slightly modified version where the few ShaderEffect items are updated to be functional with both rendering paths of the Qt Quick scenegraph. This version can be found here.

Launching the application normally, with QSG_INFO=1 set, we get:

Like the logs printed on the debug output suggest, this is running on OpenGL on a Linux desktop:

qt.scenegraph.general: threaded render loop
qt.scenegraph.general: Using sg animation driver
qt.scenegraph.general: Animation Driver: using vsync: 16.95 ms
qt.scenegraph.general: opengl texture atlas dimensions: 2048x1024
qt.scenegraph.general: GL_VENDOR: X.Org
qt.scenegraph.general: GL_RENDERER: AMD Radeon (TM) R9 M360 (VERDE, DRM 3.23.0, 4.15.0-62-generic, LLVM 8.0.1)
qt.scenegraph.general: GL_VERSION: 4.5 (Compatibility Profile) Mesa 19.2.0-devel (git-08f1cef 2019-07-25 bionic-oibaf-ppa)
qt.scenegraph.general: GL_EXTENSIONS: ...
qt.scenegraph.general: Max Texture Size: 16384
qt.scenegraph.general: Debug context: false

How does this change if we set QSG_RHI=1?

qt.scenegraph.general: Using QRhi with backend OpenGL
  graphics API debug/validation layers: 0
  QRhi profiling and debug markers: 0
qt.scenegraph.general: threaded render loop
qt.scenegraph.general: Using sg animation driver
qt.scenegraph.general: Animation Driver: using vsync: 16.95 ms
qt.rhi.general: Created OpenGL context QSurfaceFormat(version 4.5, options QFlags<QSurfaceFormat::FormatOption>(DeprecatedFunctions), depthBufferSize 24, redBufferSize 8, greenBufferSize 8, blueBufferSize 8, alphaBufferSize 0, stencilBufferSize 8, samples -1, swapBehavior QSurfaceFormat::DoubleBuffer, swapInterval 1, colorSpace QSurfaceFormat::DefaultColorSpace, profile QSurfaceFormat::CompatibilityProfile)
qt.rhi.general: OpenGL VENDOR: X.Org RENDERER: AMD Radeon (TM) R9 M360 (VERDE, DRM 3.23.0, 4.15.0-62-generic, LLVM 8.0.1) VERSION: 4.5 (Compatibility Profile) Mesa 19.2.0-devel (git-08f1cef 2019-07-25 bionic-oibaf-ppa)
qt.scenegraph.general: MSAA sample count for the swapchain is 1. Alpha channel requested = no.
qt.scenegraph.general: rhi texture atlas dimensions: 2048x1024

Not much different, at first glance. Still seems it is going through OpenGL. However, internally there is no direct OpenGL usage and no GLSL shader sources flying around in the Qt Quick scenegraph anymore. Instead, rendering goes thorough QRhi, the Qt Rendering Hardware Interface (a private API in the QtGui module for the time being).

Let's make it real interesting now. Let's set QSG_RHI_BACKEND=vulkan:

qt.scenegraph.general: Using QRhi with backend Vulkan
  graphics API debug/validation layers: 0
  QRhi profiling and debug markers: 0
qt.scenegraph.general: threaded render loop
qt.scenegraph.general: Using sg animation driver
qt.scenegraph.general: Animation Driver: using vsync: 16.95 ms
WARNING: radv is not a conformant vulkan implementation, testing use only.
qt.rhi.general: Physical device 0: 'AMD RADV CAPE VERDE (LLVM 8.0.1)' 19.1.99
qt.rhi.general:   using this physical device
qt.rhi.general: queue family 0: flags=0xf count=1
qt.rhi.general: queue family 1: flags=0xe count=2
qt.rhi.general: 55 device extensions available
qt.scenegraph.general: MSAA sample count for the swapchain is 1. Alpha channel requested = no.
qt.scenegraph.general: rhi texture atlas dimensions: 2048x1024
qt.rhi.general: Creating new swapchain of 3 buffers, size 1280x720, presentation mode 2

Nice. Apparently it is now rendering through Vulkan. Yet even the more exotic Qt Quick features, like distance field text rendering, shader effects, and particles are all there as expected.

Running the application in RenderDoc and capturing a frame gives us something like the following. Qt Quick is indeed building Vulkan pipeline state objects and command buffers, with the shader code being provided as SPIR-V bytecode.

That's it for now. In the second part of this series we will be looking at what Qt 5.14 has to offer for macOS and Windows. After that we will be moving on to looking into how all this works under the hood and what the consequences are for applications that require custom materials and effects.

Exciting times ahead!