Size matters

No, this is not your email client and this is not one of the regular spams you get.

Recently we have been investigating on file size for Qt/WinCE. As some of you might know, Qt is quite big compared to other solutions, thus we investigate on strategies to reduce the dll sizes.
One of the documented ways is to use qfeatures to strip out classes you do not need for your application. But unfortunately there is a chance that this will lead to unexpected side-effects, as well as we cannot give runtime support for any of these configurations. Qt heavily uses itself (which is a good thing), and stripping parts out ends in a lot of dependencies not being included either.

So one (beside other ways) idea was to play around with exports of the dll, and specifically ordinals. Ordinals are basically numbers which represent the decorated name of a function (like ??4ThisIsTheExportedClass@@QAEAAV0@ABV0@@Z). The advantage is, that obviously an integer takes less size than the whole string and optionally you can remove the information about the original name. So, when linking against the library, it does not search for the operator=() function of ThisIsTheExportedClass, it only looks for the number it represents. Luckily this mapping is stored inside the .lib file of the library, so you as the developer have to do nothing, and the linker does all the magic for you. Taking into account all the exports and classes we have inside Qt, this sounds like a very useful thing.

Following you will see what we got so far, and when and where we got stucked.

To specify the ordinals, you need to create a .def file, which contains all the symbols you want to export. Problem is, that we cannot use the __declspec(dllexport) anymore, as you should not export twice. A small change in qglobal.h and the exporting is basically disabled. But first, we need to get a list with all the exports we have in Qt. The command
link.exe /DUMP /EXPORTS
returns exactly this. Writing a small application which converts that output to a valid .def file is written within minutes and shouldn't be too hard to do for yourself. So the file starts like this:

EXPORTS
??0?$QList@VQItemSelectionRange@@@@QAE@ABV0@@Z @1 NONAME
??0?$QList@VQItemSelectionRange@@@@QAE@XZ @2 NONAME
??0AttributeSelector@QCss@@QAE@ABU01@@Z @3 NONAME
??0AttributeSelector@QCss@@QAE@XZ @4 NONAME
??0BasicSelector@QCss@@QAE@ABU01@@Z @5 NONAME
??0BasicSelector@QCss@@QAE@XZ @6 NONAME
??0Declaration@QCss@@QAE@ABU01@@Z @7 NONAME
...

Now that we have the .def file, we need to tell the linker to use it. Either we hack on the Makefile itself (remember this is plain research), or we use the DEF_FILE option for .pro files.

You will need to completely rebuild the project. Unfortunately this will end up in lots of undefined references. Basically lib.exe will complain about stuff like
??4QAbstractButton@@QAEAAV0@ABV0@@Z
is missing. So where do all these functions come from? Firstly we should know, what this function actually is. Using undname.exe (a tool to undecorate function names) gives us this:
Undecoration of :- "??4QAbstractButton@@QAEAAV0@ABV0@@Z"
is :- "public: class QAbstractButton & __thiscall QAbstractButton::operator=(class QAbstractButton const &)"

Having a look at qabstractbutton.h you will see that there is no QAbstractButton::operator= declared. Ah right, remembering some basics, this function must have been auto-generated during the build process and as inside Qt we export the whole class

class Q_GUI_EXPORT QAbstractButton : public QWidget
{
Q_OBJECT
...

also generated functions will be exported. But why does it fail now? The point is, that during the point in time, where the .def is being used to create the exports, the actual code for operator= does not exist,yet. So it seems like there is some magic in the background, which does a second iteration and then adds these functions to the EXPORTS list.
Unfortunately there has been no way to identify an export to be generated or a function been declared manually. The initial attempt was to remove all ??4 functions (??4 is the operator= identifier for classes, ??0 constructor, ??1 destructor, etc.), but that obviously removed the operators we really do use inside of Qt and which we want to export.

To shorten things up, this principle also caused a lot of problems with inline functions. Initially I thought, that there is nothing inside the library unless it is not getting inlined (the Microsoft compiler is very free to decide, whether it should inline or not). But that was wrong, and here is the reason why. As soon as you export an inline function, it will really do that and put stuff into the library. Never the less, when you are building your application, MSVC might still decide to inline that code-chunk into your code. But still you have a valid export, and it is getting exported as we do that for the whole class, remember?

To summarize, we cannot leave these parts in the .def file as they are unknown during the lib step. And also we cannot leave these out as we will miss exports being needed on application side.

Just to have some results, I played around with another hack.
- First create a .def file with all exports available in a regular build.
- Try to use this .def file and store the error output.
- Filter the output of all symbols and remove those from the definition file
- Now, reenable the __declspec(dllexport) in qglobal.h again and still use the new definition file.

As the LIB step comes before LINK, the redefinition warnings created will complain about the declspec export trying to redefine the definition file export and ignoring this one. This ends in using the stripped ordinal version for the exports we can use, but still generate the exports for the whole class (meaning the functions missing). If you take a look with depends on the created library, it looks like this:

Ordinal        Hint        Function
13(0x000C)    N/A        N/A
14(0x000C)    N/A        N/A
15(0x000C)    N/A        N/A
16(0x000C)    N/A        N/A
17(0x000C)    0 (0x0000)    ??0Parser@QCss@@QAE@ABV01@@Z
18(0x000C)    N/A        N/A
19(0x000C)    N/A        N/A
...
1783 (0x06F7)    N/A        N/A
1784 (0x06F8)    362 (0x016A)    ??4QAbstractButton@@QAEAAV0@ABV0@@Z
...

This resulted in QtGui4.dll being reduced from a size of 7,766,016 to 7,114,752 bytes, which is ~10% of the size. These numbers are taken from the desktop libraries, but I guess that gain would be approximately the same on Windows CE. And this is a far more important topic on embedded.

Unfortunately this is not a solution, it is just a intermediate step. Reasons for this being problematic are:
- For QtGui we generated 17000+ warnings
- We rely on the behavior of prefering the definition file over code export, which is not guaranteed anywhere
- It only provides a half baked solution (still some functions are not ordinals)
- The hinting, where to find functions gets a big offset (see 1784 compared to 362). We have not checked on potential side effects on this.

So here we are, waiting for your input. Anything I have missed, anything else to consider? Feel free to comment.