Kitware Inc. is a company that uses an open source development model to produce high quality leading edge software in the fields of computer vision, medical imaging, visualization, 3D data publishing, and software development. They are probably best known in the graphics programming world for the Visualization Toolkit (VTK), and in the programming world more generally for their build tool, CMake.
We were very pleased to speak with Bill Hoffman, Kitware’s Vice President and CTO, who gave us some insight into why his company uses Squish and the benefits they get from doing so. Bill has a lot of experience to draw on—he’s been doing software testing for almost twenty years.
Kitware is based in the U.S., and has around seventy employees. The company develops and maintains open source collaboration platforms, including the development of software for sale to other businesses, web applications, and software used in-house. Kitware uses in-house testing tools written in Tcl and in Python, and for their commercial customer projects they have started to use Squish, with two Qt 4-based projects using Squish so far. (Kitware cannot reveal any more details due to customer confidentiality requirements.)
Kitware use Squish to test their commercial Qt 4 projects on all the platforms that they deploy on: Windows, Mac OS X, Linux, and some Unix variants (Sun, HP, Irix).
Why Squish?
Kitware have been testing their software for over a decade, and as Bill told us:
Things that are not tested do not work!
They chose Squish for their Qt testing because they consider Squish to be the only viable tool for Qt GUI testing. Kitware haven’t yet taken advantage of the availability of Squish’s source code, but they are glad that it is available. And regarding licensing, they have found Squish’s flexible licensing to be particularly attractive.
Before choosing Squish, Kitware did consider some other possibilities, including an in-house developed record/play event system that they had developed for another project, and several screen click/capture programs. None of these provided the convenience and power that is available from Squish. Bill told us that key persuaders were how easy Squish makes it to add a new test, and the versatility Squish provides in that tests are recorded using a standard language—Kitware use Python—and can be edited after they’ve been recorded.
Kitware make great use of the Squish Spy: this is a tool that can access a running application. The Spy can be used to navigate through all the application’s objects (menus, widgets, items in lists, tables, trees, and so on), and to list any particular object’s properties—such as whether it is enabled, its text, font, color, and so on. This tool can also be used to set up verification points so that at particular points in a test run the value of specified object properties can be evaluated to see if they are what they ought to be.
Squish in Practice
One feature that Kitware have found especially beneficial is Squish’s robustness in the face of GUI changes. And the ability to write new tests from scratch or to edit recorded tests all using an off the shelf language they are familiar with has proved to be a particular boon.
Bill was straight to the point about Squish’s technical support, when he told us that it is simply “very good”.
The main problem that Kitware have encountered is test robustness in the face of changes to the application’s GUI. Squish’s Object Map is designed specifically to help in such situations since it allows test engineers to modify how application objects are identified—by adding/changing property values that have changed as a result of GUI changes—without needing to change the test code that refers to these objects. Of course, when an application changes so much that application objects are added or removed, some amount of test editing may be necessary to account for the changes, but this is usually both easy and painless.
Many customers prefer property-based verifications (for example, comparing numeric or string values), and Kitware is no different in using them. However, Kitware also make considerable use of Squish’s screenshot verification facility where essentially “photographs” of the application are taken in the course of recording a test and these are then compared when the test is run again at a later date.
In addition to Squish and various in-house tools, Kitware use a variety of other tools as part of their testing effort. These include CDash, Kitware’s own web-based software testing server. In fact, Kitware developed a Squish integration for CDash, something that they found quite easy to do.
Conclusion
Kitware have found their use of Squish provides much better test coverage than they had before. They have also found that Squish testing has had a direct effect on improving the quality of the applications they develop and deliver. And they have also found Squish's technical support to be a genuine help whenever they’ve needed it.
The Qt Quality Assurance team would like to thank Bill for taking the time to share his and his team and company’s experience with Squish, and we look forward to a continued successful relationship.
