The last week I looked at some issues regarding Qt's font rendering on X11, triggered by some bug reports comparing Qt 4's font rendering to Qt 3's and GTK's font rendering, specifically when subpixel antialiasing is enabled. When rendering fonts with subpixel antialiasing Qt asks Freetype to render into an alpha mask that is three times wider than the desired glyph size (or taller in the case of a vertical LCD display). We then convert this alpha mask into an ARGB mask based on subpixel ordering and a smoothing filter. The smoothing filter is necessary to avoid color fringes; without it the glyphs would look like this:
With the default Qt 4 smoothing filter the fonts instead look like this (slight improvement, don't you think?):
After doing some digging I found out that we changed the smoothing filter in the transition from Qt 3 to Qt 4 to use an interpixel as opposed to intrapixel filter. Interpixel means we do the smoothing on the subpixels directly, ignoring pixel boundaries, which is arguably more correct from a technical perspective. However, since Qt 4 was released, Freetype has added support for doing subpixel rendering and filtering itself, along with several settings for which filter to use. In Qt 4.5 we will use Freetype's filtering if available, and thus honor the lcdfilter setting in font config. Note that the filtering we use in Qt 4.4 and earlier is pretty much the same as Freetype's default filtering, but unlike before we now also support the other filtering settings. For Qt 4.5 we've also fixed an issue where we'd ignore the "slight" and "medium" hinting options in font config, leaving only "full" or "none" as the real options.
One of our goals was to make Qt 4 applications with QGtkStyle have the same font rendering as native GNOME applications, but apparently there's a known issue in GNOME/Cairo/GTK that makes Cairo use the legacy filter as soon as medium or full hinting is enabled in the GNOME font settings dialog: https://bugs.launchpad.net/ubuntu/+source/fontconfig/+bug/200707
As the legacy smoothing filter looks really ugly with certain fonts and/or slight hinting there is no way we want to use it by default in Qt, but if you really want Qt 4's fonts to be rendered as in Qt 3, you can enable the legacy subpixel filtering as described on http://johan.kiviniemi.name/blag/2008/01/12/ubuntu-hardy-fonts/ (note it seems you might have to use the value "legacy" or "lcdlegacy" instead of "lcdfilterlegacy" depending on your system). Note that this can cause severe color fringes with certain fonts, while making other fonts crisper. This is how the font above looks with the legacy filter and full hinting:
With the "slight" hinting option however it turns quite ugly (note the color fringes):
To sum up, in Qt 4.5 we should honor more of the font config settings, including lcd filter and hinting style. However, unless you explicitly change the LCD filter or hinting style things should look basically the same as in Qt 4.4. By tweaking the LCD filter and hinting settings it should be possible to get Qt applications with QGtkStyle to look even closer to native on GNOME, though hopefully the configuration issues on GNOME will be fixed eventually and you won't have to change anything to achieve this. We could of course try to mimic GNOME's behavior in Qt, but as long as there are open issues regarding this in Gtk/Cairo it doesn't make sense to change Qt's behavior.
Note: all the screenshots in this article were taken on an LCD monitor with horizontal RGB subpixel ordering, and will probably look quite ugly on other display types.