Qt 5.8 benefited from significant performance efforts in Qt QML. Some of us spent our entire time figuring out why things are slow and what could be done. This is a continued effort that pretty much started with the first Qt QML release and I don't think will ever stop. After a bit of deliberation we decided to make it a bit more obvious what kind of effort is ongoing. Due to this long standing effort we have accumulated a lot of performance tools and benchmarks from internal and external contributors. This enables us to put some hard figures to the general claim.
Startup Times
We compared general startup performance and ran graphical and non-graphical benchmarks for Qt 5.6.2 and Qt 5.8.0. As platforms we used Windows, Ubuntu, macOS and an iMX6 board (the only 32 bit architecture in the mix). The startup figures were taken for the following Qt configurations:
- Qt 5.6.2
- Qt 5.6.2 with commercial Qt Quick Compiler (qmlcompiler)
- Qt 5.8.0 without QML caching
- Qt 5.8.0 with QML caching (first startup)
- Qt 5.8.0 with QML caching (subsequent starts too)
The last two configurations take into account that the cache is generated the first time an application is started and reused on subsequent runs. This assumes that the application does not change in between runs. The application under scrutiny was the Qt Quick Controls Gallery example. There is no particular reason for this selection other than that it is fairly large and has a lot of JavaScript in it. Here are the main take aways:
- On macOS and Ubuntu, using caches gives the same results as using the qmlcompiler.
- On iMX6, the caching is an actual improvement of 60% over the qmlcompiler.
- On Windows, something other than the caching has regressed. Further checks resulted in QTBUG-59360.
- The figures show the negative impact of initial cache generation (between 10% to 180%).
QML Benchmarks
While the startup figures above give a very detailed overview for a chosen application, it is likely to not match the next application. This is where more detailed benchmarks come in. The QML benchmarks can be categorized into two areas. The first category is related to QML binding performance (create, read & write different types of properties) and the second category measures graphical drawing performance. What can we see?
- QML property reads and writes are on average 25% faster on desktop and 35% faster on iMX6
- especially low end hardware benefits dis-proportionally from recent changes
- some iMX6 tests have regressions. Looking at the code, in particular the use of C++11 atomics seems to have a bad influence (Qt 5.7 and later require a C++11 toolchain). A toolchain update might help.
Summary
We started these regular measurements in October and each round of improvements actually sparked new rounds as new issues appeared. It taught us that we have to run those tests more regularly or even automate them. For transparency purposes I attached a more detailed summary of the benchmarks results.
It is fair to say that there are still open issues which need further resolution (e.g. the startup performance on Windows or some of the discovered regressions in individual performance tests). Besides toolchain/compiler updates it clearly shows that more areas besides Qt QML need to be looked at.
To summarize we have a very broad improvement across the board in Qt 5.8.0 and the figures even validate some of our already ongoing plans such as shifting of the cache generation to compile time or improvements to the garbage collector. Stay tuned ;)
