One App, Endless System Combinations
So you've developed an app. One important step that shouldn't be forgotten is making sure it works on whatever device, browser and OS combination your potential customers might be using. Sound easy? Well, for a truly cross-platform, -browser, -device app, the number of possible system combinations hits easily in the several thousands. Provisioning and managing a device lab massive enough to test all possibilities (and let's not forget about maintaining and updating these devices) is prohibitively expensive and time-consuming.
Cloud-based Device Farm
Enter the device farm. More aptly, the cloud-based device farm. Essentially a remote testing environment, device farms let engineers test their apps on any number of real devices, all in the cloud.
Advantages abound. First, you can parallelize your testing, running tests concurrently on multiple real devices. Second, you can test your app's behavior under real-device operating conditions. And third, you gain the confidence of knowing your app works seamlessly on all potential system combinations your customers are using.
BitBar is one such cloud-based device farm. The platform enables engineers to perform mobile app testing on an array of mobile devices, has support for several frameworks, including custom frameworks for those not natively supported, and integrates in the DevOps toolchain for Continuous Integration, issue tracking and build automation.
Here, we demonstrate the steps to run automated GUI tests created with Squish, for an Android app on the BitBar device farm. Let's get started.
- Step 1: Select a target OS type
Choose the Android Operating System.
- Step 2: Select a framework
Choose Appium Android Server Side, as this type will allow a custom test framework, namely Squish, not natively supported by BitBar.
- Step 3: Choose files
This step requires two uploads: the .apk of the Application Under Test (AUT) and the test package (.zip).
Our AUT is the Android version of the AddressBook application packaged with every Squish download.
The test package must contain the following minimum components:
- Squish for Android bundle
- Test suites
- A run script (see below)
The Squish for Android bundle must be built for a Linux OS, and, for faster uploads to BitBar, you can eliminate the IDE and the documentation. A license file is also required, which we've included within this folder.
The test suites contain your functional GUI tests, which we've authored in Squish. Here, we're using example test scripts made available with every Squish download.
The run script, run-tests.sh, is called automatically and contains the commands needed to prepare the test execution environment and start the Squish test runs. More specifically, it sets the required environment variables, starts the squishrunner process and generates a JUnit report to display execution results in BitBar. The shell script is listed here for completeness:
#!/bin/bash
echo "=> SET ENVIRONMENT VARIABLES"
export SQUISH_PREFIX=$PWD/squish-android-6.5
export SQUISH_LICENSEKEY_DIR=$SQUISH_PREFIX/license
echo "=> RUN SQUISH TEST"
$SQUISH_PREFIX/bin/squishrunner --testsuite $SQUISH_PREFIX/examples/android/AddressBook/suite_py --reportgen junit,$PWD/TEST-all.xml --exitCodeOnFail=99 --local EXITCODEF=$?
exit $EXITCODEF
Note: This shell script must be located within the package's root directory.
Once you've uploaded the above, select the following actions:
- For the .apk file, set Install on the device
- For the test package, set Use to run the test
- Step 4: Choose devices
Here, select the mobile device on which to run the test. Selecting "Use chosen devices" brings you to a large selection of available Android mobile devices. Here, we've selected only one device for speed, namely the Motorola Google Nexus 6.
Then, click "Create and run automated test".
Reviewing the Results
When the results finish, click "Go to results." The web interface brings you to an overview of your test runs:
From there, you can explore further: review the logs, visualize the JUnit result output, and analyze run times.
Next Steps
BitBar has lots of great features not shown in this article. One such feature being that you can stream a screen feed of your mobile devices as the tests are executed. And, as we touched on above, integrations for build automation and CI systems are available with BitBar, too. In a future blog, we'll take a look at setting up Jenkins CI with BitBar + Squish for start-to-finish test automation. Be on the lookout for that.
Wrap-up
Using a device farm, like BitBar, is a fast, affordable and easy-to-configure approach to parallelizing your mobile and web app testing. The above example used a single test on a single Android device, but the steps shown can be easily modified to automate multiple Squish GUI tests on an array of device types and operating systems.
