Raymarine is a world leader in marine electronics. Raymarine develops and manufactures a comprehensive range of electronic equipment for the recreational boating and light commercial marine markets. Raymarine employs around 400 people around the world.
Raymarine creates software exclusively for the marine instruments it designs and manufactures.
We were delighted to have the opportunity to discuss GUI software testing with Gary Welch, a Software Test Development Engineer at Raymarine. Gary is based in the UK, although Raymarine also have development and testing teams in China, India, Poland, Sweden, and the USA.
The Application Under Test
Raymarine have developed one Qt-based product which embeds four separate applications. These applications present engine data, visualize radar data, provide chart plotting, and help to find fish! The applications are designed for use by mariners: they are all built using Qt 4 and deployed on Raymarine’s custom hardware.
Why Squish?
Raymarine wanted to test the correctness of their user interfaces and of the data their applications presented. But they did not have the time to develop their own testing software—and in any case, they wanted to avoid adding intrusive instrumentation into their applications.
They searched the Internet for a solution, and quickly found Squish. On paper Squish looked like it would provide everything they needed: the ability to test Qt-based GUI applications without needing to change the applications they wanted to test.
When Raymarine looked more carefully into Squish’s capabilities, three features stood out as decisive factors in their decision to try it. First, Squish’s support for writing test scripts in a non-proprietary language—Gary told us that at Raymarine they use Python for their test scripts. (Squish also supports JavaScript, Perl, and Tcl.) Second, Squish’s ability to record and replay test scripts to provide automated testing straight out of the box. And third, that Squish provided a comprehensive framework within which it is very quick and easy to create tests.
Squish at Raymarine
At Raymarine, the applications are tested on Linux and Windows sytems; although the hardware devices they are deployed on are all Linux-based.
Gary told us that for Raymarine, Squish’s best features are the Spy (a built-in tool for examining the properties of application objects in live running applications), and the support for remote testing.
One of Raymarine’s biggest challenges in making the most of Squish was the setting up of remote testing. Now that the initial hurdle has been overcome, Raymarine make extensive use of this feature in their testing process.
Gary told us that when he hit problems when starting out with Squish he took a pragmatic approach. First he’d experiment, and if that didn’t reveal a solution he read the documentation, and if he still couldn’t find an answer he would email for support. On the subject of froglogic’s technical support, Gary said it was:
“Mostly excellent, always quick and responsive.”
Raymarine now have several hundred test cases for their Qt-based applications. And they make extensive use of data driven testing. This has the advantage that they can easily add or modify the test data they use to cover new cases without having to change their test scripts. (Squish has excellent support for both data driven and sophisticated keyword driven testing.)
At Raymarine they use an automated configuration management (CM) to build their applications and have fully integrated Squish into their build process. (See the screenshot below. The Squish integration was achieved using Perl scripts written by Gary’s colleague, Kevin Conroy.) This means that changes to software that introduce regressions are detected immediately and the relevant developer notified. Using Squish has dramatically reduced the turnaround time for detecting and fixing regressions—from days or weeks previously to hours now.
When it comes to new application features, Gary creates the tests in advance but marks them “not testable” so that Squish won’t run them. Then, once the application feature is implemented the tests are activated and kept to guard against regressions. Gary’s aim is to keep the Results column’s Test column’s entries all green, meaning that all the tests have passed.
For the future, Gary told us that his ideal approach would be to have the user interface team create a specification as data that is read by Squish for data-driven tests so that the developers doing the implementation could immediately test their work (i.e., doing test driven development). And going beyond that, he would like to see the test cases automatically entered into a test management tool for regression testing.
Conclusion
Gary told us:
“I have had great support from froglogic.”
He also told us that Raymarine’s use of Squish has made it possible for them to extend their test coverage and to reduce their testing cycles and the person-hours devoted to testing. They have also found Squish tests to be reliable. In addition, their Squish-based test process has resulted in improved application quality.
froglogic’s team would like to thank Raymarine’s Gary Welch for taking the time to share Raymarine’s experience with Squish, and we are looking forward to a continued successful relationship.
