Back to Blog home

Pimp my video: shader effects and multimedia

Published on Wednesday February 29, 2012 by Gareth Stockwell in Qt Quick OpenGL Multimedia | Comments

Introduction

A topic which has been receiving quite a bit of attention recently is the use of shader programs to apply effects within Qt Quick applications. Once you have grasped the basics of the shader programming language, embedding the shaders in your application is made really easy by Qt Quick, with the result that stunning visual effects can be achieved from a surprisingly small amount of code.

What hasn't been mentioned previously in this blog is that applying shader effects to multimedia content (video playback, or a camera viewfinder) is just as easy as transforming any other QML element. This post shows some of the results you can achieve by combining shader programming with QtMultimedia.

The best way to demonstrate some of the effects you can achieve is ... with a demo, so that's where we'll start.

http://www.youtube.com/watch?v=SMRln8FJvKc&w=1280&h=720&hd=1
qmlvideofx demo running on Qt 5, on desktop Linux

The videos above show a Qt Quick 2 demo app, running on a desktop Linux machine. Aside from some ancillary C++ code (listening to scene graph signals in order to calculate the frame rate, and reading shader programs from the file system), the app is entirely written in QML.

Implementation details

As previously described, Qt Quick 2 comes with built-in support for shader effects via the ShaderEffect QML element. Using this to apply an effect to video or viewfinder content really isn't any more difficult than applying the same effect to any other QML element. To illustrate this, the code snippet below shows a time-varying wobble effect being applied to a video clip which is played from a file.

import QtQuick 2.0
import QtMultimedia 5.0

Rectangle {
width: 600
height: 400

MediaPlayer {
id: mediaPlayer
autoplay: true
source: "test.ogg"
}

VideoOutput {
id: videoOutput
anchors.fill: parent
source: mediaPlayer
}

ShaderEffect {
anchors.fill: parent

// Properties which will be passed into the shader as uniforms
property real amplitude: 0.02
property real frequency: 20
property real time: 0

NumberAnimation on time {
loops: Animation.Infinite
from: 0
to: Math.PI * 2
duration: 600
}

property variant source: ShaderEffectSource {
sourceItem: videoOutput
hideSource: true
}

fragmentShader: "
uniform highp float amplitude;
uniform highp float frequency;
uniform highp float time;
uniform sampler2D source;
uniform lowp float qt_Opacity;
varying highp vec2 qt_TexCoord0;
void main() {
highp vec2 p = sin(time + frequency * qt_TexCoord0);
highp vec2 tc = qt_TexCoord0 + amplitude * vec2(p.y, -p.x);
gl_FragColor = qt_Opacity * texture2D(source, tc);
}
"
}
}

Applying the effect to a viewfinder stream rather than to video playback is simply a matter of replacing the Video element with a Camera element.

In the qmlvideofx demo, each effect is implemented as a QML element which inherits from ShaderEffect; these elements are dynamically loaded and applied to the video content when the user selects the corresponding effect from the menu. Similarly, the different inputs (image, video and camera) are represented by QML elements which are dynamically loaded when required.

The set of parameters supported by each effect (e.g. 'granularity' for Pixelate; 'radius and diffraction' for Magnify) is exposed as a ListModel which is used to construct sliders via which the parameter values can be adjusted.

The code (Qt 5.x)

The source for the qmlvideofx demo can be found in the qtmultimedia repository here:

https://qt.gitorious.org/qt/qtmultimedia/trees/master/examples/video/qmlvideofx

Until Qt 5.0 is released, you'll have to build it from source in order to build and run the demo. Instructions for how to do that can be found on the wiki here. The required subset of Qt modules can be cloned as follows:

$QTDIR/init-repository --module-subset=qtbase,qtdeclarative,qtjsbackend,qtmultimedia,qtxmlpatterns

The code (Qt 4.x)

While Qt Quick 1 does not have built-in support for shader effects, it is provided by the Qt.labs.shaders plugin, which is shipped with Qt from version 4.7.4 onwards. A Qt 4 version of the demo is available here:

https://qt.gitorious.org/qt-mobility/qt-mobility/trees/master/demos/video/qmlvideofx

In addition to running on desktop platforms (tested on Linux and Windows), the Qt 4 version of the demo will also run on mobile devices - the video below shows it running on a Symbian device (Nokia C-701).

http://www.youtube.com/watch?v=aBuBlpUSajM&w=1280&h=720&hd=1
qmlvideofx demo running on Qt 4, on Symbian (Nokia C-701)

Keen-eyed viewers will notice that the app has a slightly different UI to the desktop version - thanks to the flexibility of Qt Quick, that is achieved by substituting a single QML file which describes the layout.

It should be noted that the app will not run on currently available versions of the Symbian platform. The reason is that it requires the output of the video decoder to be available to the OpenGLES engine as a texture. An EGL extension which allows the video and graphics stacks to cooperate in this way (EGL_NOK_image_endpoint2) will be added in a future Symbian release. Once this is available, QtMultimediaKit will automatically make use of it (see QTMOBILITY-1818), and the demo will start working.

You can, however, run the app today on the Nokia N9 - as shown by the video below.

http://www.youtube.com/watch?v=Uy88gM-ItKA&w=1280&h=720&hd=1
qmlvideofx demo running on Qt 4, on MeeGo Harmattan (Nokia N9)

Further information

If you weren't already excited about the potential of shader effects, hopefully you are now. Below are some links to other examples of their use in conjunction with Qt/QML.

Subscribe to Our Blog

Stay up to date with the latest marketing, sales and service tips and news.

The blog comment system has been migrated to a new platform. If you face any issues, please let us know via feedback@qt.io.