Ten years ago CERN developed a toolkit called the JCOP (Joint Controls Projects) Framework for the development of industrial control systems. This framework was designed to support the creation of systems for controlling experiments, particle accelerators, and CERN’s technical infrastructure.
The JCOP Framework is a software layer designed to simplify and unify access to the underlying SCADA (Supervisory Control and Data Acquisition) technology built with WinCC-OA, itself built on Qt. The JCOP Framework consists of both code libraries and also a substantial amount of GUI software. Since the Framework is so widely used throughout CERN, it was necessary to create a formalized automated quality assurance process to provide confidence and ensure stability. This demanded a tool that would reliably support daily regression testing on both Windows and Linux.
After evaluating various GUI test tools, CERN test engineers, Paul Burkimsher, Manuel Gonzalez-Berges, and Stefan Klikovits chose to use Squish, and CERN acquired its first Squish license.
The CERN test engineers’ initial approach was to make extensive use of Squish’s test recording facility (in their case they chose to have the tests recorded in Python). Unfortunately, this wasn’t always as robust as they had hoped because changes to their complex software stack (Qt, WinCC-OA, JCOP itself), or improvements in newer versions of Squish, could sometimes affect Squish’s reliability of playback.
To meet this challenge the CERN test engineers analysed their recorded Squish tests and used this as the basis for creating their own high level object-oriented test library. Their library contains a class for every GUI widget they test. Having the library has allowed them to drastically cut down the duplicate code produced by simply recording tests, and to abstract GUI interactions into calls to their library. This has resulted in them being able to reduce their tests to library calls and verifications. And this in turn has made test development much faster and easier, since test engineers can build on all the predefined functionality. Of course, this work has also made all of their test cases far more robust, since any change to the underlying software stack that affects the tests can now be addressed by updating a single library, rather than having to check and change a large body of test cases.
Another important matter for the CERN test engineers is the ability to test on both Windows and Linux. They tried a number of different solutions and in the end adopted a CI (Continuous Integration) approach using the Hudson tool (and for which there is a Squish integration). Initially they ran all their tests on different physical machines, but they have now switched to using Virtual Machines hosted in the CERN Computer Centre—a change which made no difference to their testing or to Squish.
CERN test engineers published a brief technical paper describing their use of Squish, the challenges that arose, and the solutions they found. This paper was presented at the 13th International Conference on Accelerator and Large Experimental Physics Control Systems:
Technical Paper
Multi-platform SCADA GUI Regression Testing at CERN
Slides
Multi-platform SCADA GUI Regression Testing at CERN (Slides)
