In the last post, we discussed a new approach to design time and build time integration of external tools in Visual Studio using MSBuild rules and targets. This will be included in the upcoming release of version 2.2 of the Qt VS Tools. In this post, we will discuss the performance improvements that are also included in this new version.
We'll focus on two situations where users of the current version (2.1) of the Qt VS Tools have reported sub-standard performance:
- Adding new header files, especially to projects with many configurations
- Converting Qt projects (.pro files) with many files
We'll look at the scale of these problems and why they occur, then discuss the steps that have been taken to fix the problems and the improvements that have been achieved.
Adding Header Files to a Project
To understand the scale of this problem and the effectiveness of our solution, we ran the same set of tests using both the current and the new version. These tests consisted of adding several header files containing Qt macros to projects with varying numbers of configurations. For more information on the test setup and a detailed analysis of the results, please refer to the section "Measuring Performance", at the end of this post.
Looking into the results of testing version 2.1, we found that the time to add files to a project got progressively worse, up to several minutes, the more configurations were defined in the projects. The worst case scenario was just under 45 minutes for adding 10 files to a project with 20 configurations. It was possible to determine that the performance degradation is closely related to the approach previously followed, where files generated by moc were explicitly added to the project in order to be included in the C++ compilation. The time spent modifying properties of generated files accounted for 98% of total time.
As we discussed in the previous post, the new approach based on MSBuild rules and targets allows source files to be added to the build process while it is ongoing. This way, instead of adding the files generated by moc as static project items, they are added dynamically to the C++ compilation during the processing of the moc target. This should represent a significant improvement to the cost of adding new header files to a project. The test results from version 2.2 show that this is indeed the case: the time spent on each test case was, on average, 95% less than that of version 2.1, and what was previously the worst case scenario (10 files added to a project with 20 configurations) now took only 3,2 seconds to complete.
This improvement in performance is not limited to the action of adding header files to projects. It will have an impact in all operations that handle generated files as project items. The following is a list of other use cases that should also benefit from the increase in performance (especially in projects with several configurations):
- Importing Qt projects (see next section);
- Creating new Qt classes;
- Manually adding/removing the Q_OBJECT macro in a file;
- Changing the Qt version;
- Changing the path to the moc, rcc and uic generated files.
Importing Qt Projects
The problem described above also applies when importing a .pro file, since properties of generated files are also accessed and modified during import; in fact, the import procedure modifies properties of all project files. To avoid this problem, in the new version of the Qt VS Tools, the Visual Studio project that is generated by qmake is not immediately loaded into Visual Studio; all modifications are carried out as an XML transformation on the project file. Only then is it loaded into Visual Studio. This way we avoid any project-wide processing while the project is loaded.
To test this optimization, we imported the Qt example project demobrowser. Below is a recording of this test using version 2.1 (left) and version 2.2 (right) of the Qt VS Tools (the recording also includes building and running the program). We found that the time to import the project decreased from over 30 seconds in the current version to less than 6 seconds in the new version.
Measuring Performance
To measure the scale of the performance problem when adding new header files, we ran a series of tests with an instrumented build of the Qt VS Tools that recorded the number and duration of calls to every function. As the performance seemed to be influenced by the number of project configurations, we tested with several projects containing respectively 2, 5, 10, 20 and 40 configurations. As the number of files added also seemed to be a factor, we tested each project with 5, 10, 20 and 40 files added, for a total of 20 test cases. We will use the variable C to represent the number of configurations, and the variable F to represent the number of files added.
The results of testing with version 2.1 of the Qt VS Tools show that the performance degrades significantly as we increase the number of configurations and the number of files added. The graph below summarizes the test results. Some tests were interrupted after 45 minutes; this is noted in the graph by grayed-out bars. The blue bar on the left serves as a 1 minute scale.
Looking further into the test data, we find that most of the elapsed time was spent calling functions of the Visual Studio SDK. For example, in the test case C=20,F=10, which took almost 45 minutes to complete, 98% of that time was spent in the following calls:
These functions were called while adding generated files to the project – files generated by moc must be added to the project in order to be included in the C++ compilation. For every header file added, one generated file per configuration will also be added to the project, i.e. F×C generated files. In the test case of C=20,F=10, this accounts for the 200 calls, e.g. to the AddFile function.
Each one of the F×C generated files can only be compiled if the corresponding project configuration is active. For all other configurations, the file must be set to "excluded from build". This is the reason for the larger number of calls to the set_ExcludedFromBuild function, i.e. F×C×(C-1) calls to that function. In the example of the C=20,F=10 test case, this accounts for the 3800 calls to set_ExcludedFromBuild.
We've also found that the functions of the Visual Studio SDK tend to become slower as the number of calls increases. In the case of set_ExcludedFromBuild, we see that the average time per call went from 11 milliseconds, when calling the function 10 times, to 535 milliseconds per call for 3800 calls:
With the approach followed in version 2.1, which requires that generated files be added to the project, there doesn't seem to be much room for improvement in performance: almost all time is spent inside external functions and these functions are called exactly the number of times needed. The problem of the performance degradation seems then directly linked to the approach itself.
To test the performance of the new approach, we ran the same 20 test cases using the new version 2.2 of the Qt VS Tools, which uses the Qt/MSBuild targets; the results are shown in the graph below. As before, the blue bar on the left represents the same 1 minute scale.
Comparing the results of both tests, in version 2.2 the time spent was, on average, 95% less than that of version 2.1. The average cost of adding header files to a project, per file and configuration (i.e. total time divided by F×C), decreased from 300 milliseconds to 40 milliseconds.
