Shortened Testing Cycles and Increased Testing Coverage using Squish in the Marine Electronics Industry
Introduction
The marine electronics industry manufactures a comprehensive range of tools for recreational boating and commercial marine markets.
Each designed and manufactured marine tool is driven by software exclusively developed for the specific device. The success of each manufacturer relies on the proven accuracy and reliability of the marine electronics and its software.
The Application Under Test
One marine electronics company developed a Qt-based product embedding four individual applications. Each application produces engine data, visualizes radar data, provides chart plotting, and helps find the fish! The applications are designed for mariners: each built using Qt 4 and deployed on custom hardware.
Why Squish?
This particular marine electronics company wanted to test the accuracy of their user interfaces and the data presented within the applications. Lacking time to develop their own testing software, as well as wishing to avoid adding intrusive instrumentation into their applications, they began searching for an alternate solution.
After searching for a solution online, they quickly found Squish. Initially it appeared Squish would provide everything they needed: the ability to test Qt-based GUI applications without changing the applications they wanted to test.
Following further examination of Squish’s capabilities, three features stood out as decisive factors in their decision to evaluate it with their applications. First, Squish’s support for writing test scripts in a non-proprietary language—they use Python for their test scripts. (Squish also supports JavaScript, Perl, and Tcl and Ruby.) Second, Squish’s ability to record and replay test scripts, providing immediate automated testing out of the box. And third, Squish provided a comprehensive framework making it very quick and easy to create tests.
Squish Implemented
The marine applications are tested on Linux and Windows systems; despite the deployed hardware devices being solely Linux-based.
One of the top features upon implementing was the Spy (a built-in tool for examining the properties of application objects in live running applications), and the support for remote testing.
While one of the most valuable, the initial remote testing configuration also proved to be the biggest challenge. Once the initial hurdle was overcome, the manufacturer make extensive use of the feature in their testing process.
When they hit problems starting out with Squish, they choose a pragmatic approach. First experimenting, and if that didn’t reveal a solution, referencing the documentation, and if the solution still evaded them, an email to support resulted in a quick and excellent response.
With several hundred test cases implemented for their Qt-based applications, with extensive use of data driven testing, they’ve discovered another advantage: Easily adding or modifying test data to cover new cases without requiring a change to their test scripts (Squish has excellent support for both data driven and sophisticated keyword driven testing).
Combined with an automated configuration management system (CM) to build their applications, Squish has been fully integrated into their build process. Changes in the software that introduce regressions are detected immediately and the relevant developer is notified. Using Squish has dramatically reduced the turnaround time for detecting and fixing regressions—formerly days or weeks is now hours.
For new application features, tests are created in advance but labeled as “not testable”, then automatically skipped when the Squish regression test suite runs. Once the application feature is implemented, the tests are activated and kept to guard against regressions. The goal: keeping the Results column and Test column entries all green, meaning all the tests passed.
Going forward, the ideal approach is for the user interface team to create a specification as data, which is read by Squish for data-driven tests, enabling the developers doing the implementation to immediately test their work (i.e., doing test driven development). Plans are also underway to enhance the process by automatically entering the test cases into a test management tool for regression testing.
Conclusion
Made possible with their reliable Squish test framework and great froglogic support, they’re experiencing extended testing coverage, reduced testing cycles and reduced the person-hours devoted to testing, not to mention a measurable improvement in application quality.
