It's very important to make automated GUI tests a part of a Continuous Integration (CI) process. Squish offers add-ons for many popular CI tools like Jenkins, Bamboo and TeamCity. The full list is here. If your tool is not on the list, it can be still integrated as Squish tests because we support a command-line interface for all of our tools. Moreover, Squish can generate JUnit reports, which can be parsed easily by many CI or Test Management tools.
This article demonstrates how to run Squish tests on merge requests in GitLab. Squish will generate a JUnit report and we will configure a job in GitLab to use this report. Later GitLab will display a test summary on the merge request page. It contains information about newly failed tests, existing failures and resolved failures. Based on the above data, a decision about merge is made.
The first step is to configure GitLab Runner. We will configure the following environment variables:
- SQUISH_DIR - Squish Installation directory
- SQUISH_LICENSEKEY_DIR - Squish license location
- SQUISH_SERVER_PORT - port at which squishserver process is started
- DISPLAY - display used to display an application GUI. We use VNC Server to provide a headless display
Next step is to define a pipeline in
.gitlab-ci.yml file. Pipelines are defined by specifying jobs that run in stages. In our example, we define job named "squish-tests1" which is run in a test stage.
- echo DISPLAY=$DISPLAY
- echo $SQUISH_DIR
- echo "Starting squishserver..."
- $SQUISH_DIR/bin/squishserver --port $SQUISH_SERVER_PORT 1>server.log 2>&1 &
- sleep 5
- echo "Register AUT..."
- $SQUISH_DIR/bin/squishserver --port $SQUISH_SERVER_PORT --config addAUT addressbook /home/name/Squish/squish-for-qt-6.4.0/examples/qt/addressbook
- $SQUISH_DIR/bin/squishrunner --port $SQUISH_SERVER_PORT --testsuite tests/suite_addressbook --reportgen junit,report.xml --reportgen stdout
- echo "Stopping squishserver..."
- $SQUISH_DIR/bin/squishserver --stop --port $SQUISH_SERVER_PORT &
The job performs the following actions:
squishserverand redirect its stdout and stderr output to a
- Register the Application Under Test (AUT)
squishrunnerto run the Test Suite and generate JUnit report and stdout report (so results are also printed to a screen)
- Collect artifacts
While working on a dev branch, we introduced a defect that is causing a failure in a test case. When committing a change, Squish Tests are executed.
Later in GitLab, we create a merge request to the master branch. On the Merge Request Page, a Test Summary shows a difference between test results on the master versus the merged branch.
In the picture above, a Test Summary shows that there is one new failed test. Based on this information, a decision about the merge can be made.