Enabling Large Scale Execution and Testing of Android Applications - JSS Online Appendix

Purpose

The main purpose of this site is to describe solutions to common issues that research teams and professional developers will encounter when trying to enable the large-scale execution of Android applications for testing purposes. Specifically, this site offers detailed step-by-step methodologies, downloads, and external links to enable the solutions discussed in the experience report. All of the solutions outlined below were developed in the course of our research group enabling an infrastructure for parallel testing and execution of Android apps in collaboration with Huawei. To navigate the guide, browse the Table of Contents below and simply click on the section you would like to view to be taken to that section.

V) Useful ADB and aapt commands

I) Concurrent Execution of AVDs

As stated in the Experience Report, because mobile developers are typically required to test their applications against several combinations of platform and hardware configurations, automated solutions are required. The most economically feasible and practical method for accomplishing this for both development teams and academic researchers is using Android Virtual Devices (AVDs), which we define as any instantiation of a physical android device in a virtualized manner. There are several options for creating and running AVDs, such as the official Google emulator, Genymotion emulators, Android Virtual machines based on the android-x86 platform, and niche solutions such as Bluestacks. For reasons covered in the Experience Report, the best choice when attempting to enable large scale parallel execution is Android Virtual Machines based on the android-x86 project using VirtualBox as the managing hypervisor.

Installation of Android-x86 via VirtualBox

(See complete Installation Instructions with screenshots see mobilefun, and android-x86)

To install a particular version of android-x86 in a VirtualBox VM, perform the following steps:

Download the .iso image of the desired version of android you wish to run in a Virtual Machine from the android-x86 site.
If you don't have it already, download the latest version of VirtualBox and create a new VM with your desired name and the following settings:
- Type: Linux
- Version: Linux 2.6 / 3.x /4.x (32-bit)
- Memory: >512MB
- Virtual Hard Disk: Fixed Size, >= 4GB, VDI type
Next, you need to mount the .iso file you downloaded earlier to the VM. To do this go to right-click on the VM you just created in the sidebar, and click on the storage tab at the top. Then click on the "Controller" Disk Drive, and select the .iso file.
After performing these steps, you are ready to start the VM, boot from the live android-x86 .iso, and install the image to your VM permanently. The steps for this are very straightforward:
- Start the VM
- Select the "Installation – Install Android-x86 to harddisk" option from the boot menu.
- Select the "Create/Modify partitions" option from the installation menu, this will bring you to the disk partitioning screen.
- Select the "New" option, and press enter.
- Follow the instructions to make the partition you are creating the primary boot partition.
- After you arrive back at the disk partitioning screen, make sure the "bootable" option is enabled and select the "write" option to write the changes to the VM, and select the "Quit" option to return to the installer.
- Choose the disk partition you created in the installer window, and hit enter.
- Follow the on-screen installation instructions to install the GRUB boot-loader and make the /system directory read/write.
- Wait for the installation to finish and then reboot.
After the image has been installed to the VM, shut it down and eject the .iso from the "Controller" optical drive in the settings menu.
Start the VM and enjoy your new Android Virtual Machine!

Cloning Additional AVDs

Once you have configured a single AVD, it is easy to clone additional VMs quickly without following the entire setup procedure again. This can be done either through the command line or through the GUI. This is especially helpful as it allows all hardware and software settings for an AVD to be configured once, and then copied multiple times with relative ease.

Cloning via the GUI

Right Click on the VM you wish to clone.
Select the "Clone..." option from the context menu.
Enter a name for new VM and ensure that the "Reinitialize the MAC address of all network cards: option is enabled to allow for VMs with different Network Addresses.
Select "Continue", and ensure that the "Full Clone" option is checked to give the cloned VM a separate Virtual Hard Disk.
Click the "Clone" button, and wait for the files to be copied.

Cloning via the Command Line

Use the following command:

$ VBoxManage clonevm <legacy-vm-name> --name "<desired-name-of-clone>" --register

Configuring a Custom Screen Size

One of the key components for enabling testing a large variety of hardware configurations for GUI-based testing is custom screen resolutions, sizes, and pixel densities. To enable custom screen settings for an android-x86 based VM, follow the instructions below modified from this Stack Overflow post:

Add a desired custom screen resolution to VirtualBox using the following command:
- ```
VBoxManage setextradata "VM\_NAME\_HERE" "CustomVideoMode1" "320x480x16"
```
etermine the hex-value for the added video mode by stating the VM, entering "a" into the GRUB menu, typing "vga=ask" and pressing enter, and writing down/copying the hex value for the "Mode column of your custom resolution.
Translate the hex value to a decimal value
Enter debug mode from the GRUB menu upon startup of the VM, modify menu.lst using the following commands:
- ```
# mount -o remount,rw /mnt;
```
- ```
# cd /mnt/grub;
```
- ```
# vi menu.lst;
```
Modify the menu.lst file to include "vga=" followed by the decimal value of your custom video mode obtained above (e.g. vga=864 for a hex value of 360). The Resolution should match your custom video mode (e.g. UVESA_MODE=320x480) and the DPI can be adjusted to match the device screen you are attempting to mirror (e.g. DPI=160 UVESA_MODE=320x480). Example menu.lst file contents:
- ```
kernel /android-2.3-RC1/kernel quiet root=/dev/ram0 androidboot_hardware=eeepc 
 acpi_sleep=s3_bios,s3_mode DPI=160 UVESA_MODE=320x480 SRC=/android-2.3-RC1 
 SDCARD=/data/sdcard.img vga=864
```
unmount and reboot the device using:
- ```
# cd /;
```
- ```
# umount /mnt;
```
- ```
# reboot -f;
```
The AVD should now boot into your custom resolution. *Note that it may take some experimentation to ensure that the resolution, resolution and pixel density match a target device perfectly.

Configuring the Network to Allow for adb and Internet Access

In order to utilize an android-x86 AVD for testing purposes, it is important to enable a simultaneous connection to the adb and Internet using the port-forwarding capabilities built into VirtualBox. To accomplish this, please follow the steps below modified from this Stack Overflow post:

To enable via the VirtualBox GUI:

Right-Click on the Virtual-Machine for which you wish to enable adb access and select the "Settings" option from the context menu.
Click the "Network" tab at the top of the new window.
Click the "Advanced" button to enable more actions.
Click the "Port-Forwarding" button to launch a new window.
Add two rules to forward ports 5555 (adb) and 5554 (console) from the guest to your host machine. To make an easy rule for potential multiple AVDs, you can simply add a digit to the end – e.g. 5555→55551 and 5554→55541 for this machine (add a "2" for your second, etc).
Launch the AVD and after it has finished booting, connect to it with the following command:
- ```
# adb connect localhost:55551
```

To enable via the command-line:

Use the following commands:

# VBoxManage modifyvm "<AVD-Name>" --natpf1 "adb1,tcp,,55551,,5555"

# VBoxManage modifyvm "<AVD-Name>" --natpf1 "adb2,tcp,,55541,,5554"

To execute an adb command on a specific AVD, use the following command:
- ```
# adb -s localhost:<adb-port-assigned> shell <command>
```
```
 
```

Running Multiple adb Servers

Due to the default port limitations imposed by Google's adb interface, only 15 devices (virtual or physical) can be connected to any single adb server. However, there are two potential solutions to overcome this limitation: 1) Use custom ports when invoking Google's default emulators, or 2) Instantiate multiple adb servers, with each server hosting up to 15 devices on the default ports. In our infrastructure for parallel execution and testing of android apps, we use the second option, creating and destroying adb servers as need for the load on the execution engine.

Option 1: Using Custom adb Ports (works only with Google emulators)

Use the following command when invoking an emulator (assuming the Android SDK is in your $PATH):
- ```
# emulator -avd <your-avd-name> -ports <port-for-adb><port-for-console>
```
It should be noted that in our experience, using this solution is not stable, because, once around 30 devices are operating on a single adb port, the server becomes unstable, devices can disconnect without notice, and running the # adb devices command can crash the server.

Option 2: Instantiate Multiple adb Servers (works with any Android device)

The default adb server port is 5037. To create a sever on a new port use the following command:
- ```
# adb -P <desired-server-port> start-server
```
To kill a server and disconnect all currently connected devices use the following command:
- ```
# adb -P <desired-server-port> kill-server
```

To execute shell commands to a particular device use the following command:

# adb -P shell <desired-server-port> -s localhost:<adb-port-assigned> shell <command>

II) Modifying UiAutomator To Enable Accurate GUI-Testing

In GUI-based testing of mobile apps, the extraction of the currently displayed GUI-hierarchy on a device is a fundamental operation for data collection and future event generation, depending on the automated technique being used. In the development of our large-scale automated execution and testing framework, we have relied on uiautomator to extract GUI-related information from devices under test. Uiautomator is a powerful android framework tool that is capable of extracting a GUI hierarchy from a physical or virtual android device, and dumping this information to an xml file which can then be parsed. However, as described in the experience report, one major problem with the tool's current implementation is that information for dynamically changing GUIs (e.g. a countdown timer is running on the screen), cannot be reliably extracted. The bug that causes this problem persists across all recent OS versions. To fix this problem, we crafted a solution that involves downloading and modifying the the DumpCommand.java file in the AOSP v4.2.2 (although the fix should apply to any recent OS version). The custom uiautomator files can then be pushed to the device without re-flashing the OS. To download our recompiled, modified files for OS v4.4.2, please use the button below and skip to step to the "Installation of the Modified uiautomator files to Devices" subsection. Otherwise, simply follow the steps in the tutorial below to recreate the fix.

Click here to download the modified uiautomator files

Preparing the AOSP Build Environment and Downloading the Code

To apply our fix for the uiautomator bug, you must set up an environment for downloading and compiling the AOSP code base for your target platform. For the sake of simplicity, we cater this tutorial for users on linux machines, since this OS has resulted in the most consistent and fastest compilation results. Many of the steps for this and the other subsections of this section of the guide are taken from Google's AOSP site, which should be referred to for more details, this is simply just a repost of the steps to make this tutorial easier to follow (Credit source.android.com).

First you must choose which version and branch of the project you would like to check out, as this determines some of the setup required. For the sake of this guide will assume the user is attempting to compile a version of Android since Gingerbread (2.3.x) or newer on the Ubuntu Linux distribution version 14.04.
Next you must obtain the correct version of Java for the version of Android that you wish to compile:
- Java 7: for Lollipop through Marshmallow
- Java 6: for Gingerbread through KitKat
- Java 5: for Cupcake through Froyo

Change to the correct version of Java by using the following commands:

$ sudo update-alternatives --config java

$ sudo update-alternatives --config javac

Install the following compilation dependencies:

$ sudo apt-get install git-core gnupg flex bison gperf build-essential \
  zip curl zlib1g-dev gcc-multilib g++-multilib libc6-dev-i386 \
  lib32ncurses5-dev x11proto-core-dev libx11-dev lib32z-dev ccache \
  libgl1-mesa-dev libxml2-utils xsltproc unzip

By default, the AOSP compilation process will write the output of the compilation to the /out/ directory of the established build directory. To change this default setting to a different desired location, use the following command:
- ```
$ export OUT_DIR_COMMON_BASE=<path-to-your-out-directory>
```
To speed up the rebuild process, we highly recommend enabling the ccache option, which acts as a compiler cache that dramatically speeds up rebuilds, especially if you use the make clean command frequently. To enabled this option put the following command in your .bashrc or equivalent:
- ```
$ export USE_CCACHE=1
```
- Then you will need to specify a directory where the cache will be stored, with enough space for the recommended cache size of 50-100GB. To do this use the following command:
- ```
$ export CCACHE_DIR=<path-to-your-cache-directory>
```
- Then after the source has been downloaded, you need to run following command before compilation:
- ```
$ prebuilts/misc/linux-x86/ccache/ccache -M 50G
```
Now you are ready to download the source. To accomplish this you first must install the repo tool. This is simply a tool that makes it easier to work with the Git version control system in the context of the Android source code. To install the repo tool:
- First make sure you have a bin/ directory in your home directory and that it is included in your $PATH:
  - ```
  $ mkdir ~/bin
```
- ```
$ PATH=~/bin:$PATH
```
- Then download the repo tool and ensure that it is executable:
  - ```
  $ curl https://storage.googleapis.com/git-repo-downloads/repo > ~/bin/repo
```
- ```
$ chmod a+x ~/bin/repo
```
After installing repo, you need to set up your machine to access the Android source repository. To do this:
- First create your working directory for the AOSP source code, this should done on a case sensitive filesystem (most Linux distros default to this) and should have ample space ~50-100GB. Example commands:
  - ```
  $ mkdir WORKING_DIRECTORY
```
- ```
$ cd WORKING_DIRECTORY 
```
- Next configure git with your name and email address:
  - ```
  $ git config --global user.name "Your Name"
```
- ```
$ git config --global user.email "you@example.com"
```
- Next run the repo init command to download the latest version of repo and specify a url for the manifest, which specifies where the various repositories included with the Android source will be placed in your working directory.
  - ```
  $ repo init -u https://android.googlesource.com/platform/manifest
```
- To checkout a branch other than the master branch, specify it with -b as follows:
  - ```
  $ repo init -u https://android.googlesource.com/platform/manifest -b android-4.0.1_r1
```
- To download the Android source tree into your working directory run:
  - ```
  $ repo sync
```
- This will download the version of the source specified in the manifest file set up in the previous sub-step. This can take quite some time, depending on the connection speed. To avoid connection interruptions on remote machines, it is best practice to utilize a utility like screen to connect and disconnect to the remote session.

Modifying the uiautomator Framework Files

After the AOSP source code download for your target version has completed, the next step is to modify the uiautomator files to correct the bug explained at the beginning of this section. For ease of use we include a download of our modified DumpCommand.java file below, which will be used in the steps outlined below. Simply replace the DumpCommand.java files specified in the downloaded Android source with the modified version. Note that the file we included is specifically for version 4.4.2, thus if you are trying to update a different version, you may need to adopt our changes to fit updated code of more recent versions of Android.

Click here to download the Modified DumpCommand.java File

After downloading the above file, use the following commands to make the file changes:

cp PATH_TO_DOWNLOADED_FILE/DumpCommand.java \
PATH_TO_AOSP_WORKING_DIR/frameworks/testing \
/uiautomator/cmds/uiautomator/src/com/android/commands/uiautomator/

cp PATH_TO_DOWNLOADED_FILE/DumpCommand.java \
PATH_TO_AOSP_WORKING_DIR/frameworks/uiautomator \
/cmds/uiautomator/src/com/android/commands/uiautomator/

Alternatively, you can modify the files yourself, adopting our changes manually:

vim AOSP_WORKING_DIR/frameworks/testing/uiautomator \
/cmds/uiautomator/src/com/android/commands/uiautomator/

vim AOSP_WORKING_DIR/frameworks/testing/uiautomator/cmds \
/uiautomator/src/com/android/commands/uiautomator/

Compiling AOSP

After the changes to the DumpCommand.java file have been made, the next step is to compile the source code to obtain the modified executable, .jar and .odex files that an Android device needs to use the uiautomator framework. To compile the source follow these steps, as outlined at Google's AOSP site, which should be referred to for more details, this is simply just a repost of the steps to make this tutorial easier to follow (Credit source.android.com).

Initialize the build environment using the envsetup.sh script in the build directory of your AOSP working directory. (Note that this script must be called from the root of the working in order to function properly):
- ```
$ . build/envsetup.sh
```
Next choose a target device to build for using the lunch command. For example, if you would like to build for the Intel-x86 based Android Virtual Devices discussed in the first section of this guide, you would use the following command:
- ```
$ lunch aosp_x86-eng
```
Build everything with make. GNU make can handle parallel tasks with a -jN argument, and it's common to use a number of tasks N that's between 1 and 2 times the number of hardware threads on the computer being used for the build. For example, on a dual-E5520 machine (2 CPUs, 4 cores per CPU, 2 threads per core), the fastest builds are made with commands between make -j16 and make -j32. It should be noted that android can only be built by version 3.81 or 3.82 of GNU make.
- ```
$ make -j4
```

Installation of the Modified uiautomator to Devices

After you have successfully built the source code yourself, or downloaded our pre-modified and compiled uiautomator files above, the next step is to install the files onto the target device. To accomplish this follow the steps below:

First you must make the system directory writeable by performing the following command:
- ```
$ adb shell
```
- ```
$ mount -o remount,rw /system
```

Next, remove the old uiautmoator files from the device:

$ adb shell rm -rf /system/framework/uiautomator.jar

$ adb shell rm -rf /system/framework/uiautomator.odex

$ adb shell rm -rf /system/bin/uiautomator

Finally, add the modified files to the device:

$ adb push PATH_TO_MODIFIED_FILES/uiautomator.jar /system/framework/uiautomator.jar

$ adb push PATH_TO_MODIFIED_FILES/uiautomator.odex /system/framework/uiautomator.odex

$ adb push PATH_TO_MODIFIED_FILES/uiautomator /system/bin/uiautomator

After the modified files have been pushed to the device the permissions for these files may need to be changed:

```
$ adb shell
```

$ chmod 776 /system/framework/uiautomator.jar

$ chmod 776 /system/framework/uiautomator.odex

```
$ chmod 776 /system/bin/uiautomator
```

After these steps have completed you should be able to successfully execute a ui-dump on the device using the following command:
- ```
$ adb shell /system/bin/uiautomator dump /sdcard/ui_dump.xml
```

III) Enabling Dynamic Inferral Of GUI Idle States

When automatically executing GUI or system events in Android apps, a critical property must be met: after sending an event to the device, the next event to be executed should not be sent until the previous event has been received and processed. This is usually achieved using either a manually hardcoded or automatically inferred wait-time between GUI events. In the course of enabling a large-scale execution engine for the testing of Android applications we have developed our own solution. It relies on the

```
adb shell dumpsys
```

command to query the device about the current state of the animation transitions on the screen. As we show below this will enable us to detect wether or not the GUI state of a device is idle, helping to solve the problem of inferring wait times between executing actions, or gathering information from a device.

Getting Familiar with the Dumpsys Command

Specifically, we use the following command to query the state of the screen:

```
adb shell dumpsys window -a
```

This command gives a lengthy output that contains detailed information from the Window Manager framework running on an Android device. Since we are interested in the foreground application’s transition state at the time of the running the command, we need to read the value of the mAppTransitionState property listed towards the end of the output from this command.

This property has one of the four following states [This information was taken from this blogpost, and inferring information from the Android Source]:

APP_STATE_READY (1): indicates that an application transition is imminent, and thus it would not be safe to execute a new GUI input command as the app under test’s GUI will likely change shortly.
APP_STATE_IDLE (2): indicates that any and all application transition animations have completed and the current state of the GUI as seen by the Window Manager Service is idle, thus signifying a safe state for executing a new GUI input command.
APP_SATE_TIMEOUT (3): indicates that the Activity Manager prepares for an application transition but it fails and a 5 second timeout is hit.
APP_STATE_RUNNING (4): indicates that the Window Manager has begun executing an app transition. During our testing, this state was never observable from the dumpsys command presumably because the Window Manager Service is in a “locked” state while the transition is occur- ring and the dumpsys window command does not return until after the lock is released.

Enabling Idle State Detection

To enable device Idle state detection, The mAppTransitionState property can be queried by using the following command directly after a GUI-input event has been sent to the device:

adb shell dumpsys window -a | grep ‘mAppTransitionState’

This will typically result in either State 1 (APP_STATE_READY) or 2 (APP_STATE_IDLE) being returned. If State 1 is returned, it means the app is not ready yet to process another event, therefore, the command should be executed continuously until State 2 is observed. Once State 2 has been observed, the Window Manager has indicated that the currently displayed GUI is idle and that it is safe to continue with the next GUI input command.

IV) Effectively Cleaning Applicaiton Data During Testing

One often overlooked problem regarding the testing of Android applications is ensuring the same execution environment at the beginning of each test run. As described in Section 2.2, this can be a more difficult practice than researchers and testers expect. “Stale” application data can cause problems with automated testing approaches; when performing automated GUI-testing of Android applications, a desired feature of an automated execution/testing framework is to ensure that the initial application state is the same (e.g., clean) before each test-case is executed. This guarantees that sequences of GUI commands sent to the device will have the same outcome during each run, which is of crucial importance when evaluating or verifying test-cases through techniques such as mutation analysis. There are two types of application data that need to be cleaned: 1) Internal Data (e.g. User settings and sqLite DBs), and 2) External Data (e.g. data stored on an sdcard). The following subsections explain how to clear such data.

Cleaning Internal Data

Cleaning internal application data is as easy as uninstalling and installing the application from the target Device under test. This can be done programmatically using the following adb commands:

To install an app:
- ```
$ adb shell install PATH_TO_APK
```

To uninstall an app:

$ adb shell pm uninstall APP_PACKAGE_NAME

However, completely uninstalling the application may not always be desired by developers or testers, therefore, an alternative way to clear this data is the use the following command:

```
adb shell pm clear APP_PACKAGE_NAME
```

Cleaning External Data

Files that are saved externally are slightly more difficult to handle properly. First, a tester must determine where such files are saved on a particular device and whether these files affect app execution in a derogatory manner with respect to the testing goals to be achieved. If it is determined by the tester that the persistence of an external file is causing unexpected or unwanted behavior between executions of subsequent test cases, then this data must be cleared by a command such as:

```
adb shell rm -rf PATH_TO_UNWANTED_FILES
```

V) Useful ADB and aapt Commands

In this section we offer some useful adb and aapt commands for controlling and interfacing with android devices. The adb tool can be found in platform-tools directory of the Standalone Android SDK, whereas the aapt tool can be found in the current version of the build-tools tools folder (e.g. /build-tools/21.1.2/aapt).

Clear app data

# adb shell pm clear <package-name>

Get screen orientation

# adb shell dumpsys input | grep 'SurfaceOrientation' | awk '{ print $2 }'

Get screen dimensions

# adb shell dumpsys window | grep "mUnrestrictedScreen"

Get current Activity

# adb shell dumpsys window windows | grep -E 'mCurrentFocus|mFocusedApp'

# adb shell dumpsys activity | grep -E “mFocusedActivity” | awk '{ print $4 }'

Capture Screen

# adb shell /system/bin/screencap -p /sdcard/test.png

Execute Tap

# adb shell input tap <x> <y>

Execute Long tap

# adb shell input touchscreen swipe <x> <y> <x> <y> 2000

Execute Swipe

# adb shell input touchscreen swipe <x1> <y1> <x2> <y2>

Enter Text

# adb shell input text <string>

Clean logcat

# adb shell logcat -c

Get logcat with Activity lifecyle (short including time)

# adb shell logcat -v time ActivityManager:V *:S

Get logcat with Activity lifecyle (detailed log including time)

# adb shell logcat -v time -b events *:V

Get logcat with Activity lifecyle( detailed log including time) for specific app

# adb logcat -v time -b events *:V | grep ‘adb shell ps | grep <app_package_name_or_pid> | cut -c10-15’

Rotate Screen

Disable Rotation via accelerometer:

adb shell content insert --uri content://settings/system --bind name:s:accelerometer_rotation --bind value:i:0

Rotate to portrait mode:

adb shell content insert --uri content://settings/system --bind name:s:user_rotation --bind value:i:0

Rotate to landscape mode:

adb shell content insert --uri content://settings/system --bind name:s:user_rotation --bind value:i:1

Get package name and main activity from APK

./aapt dump badging <apk> | grep -E 'package: name='
./aapt dump badging <apk> | grep -E 'package: name=' | awk -F " " '/1/ {print $2}' 
./aapt dump badging <apk> | grep -E 'launchable-activity: name=' | awk -F " " '{print $2}'

VI) Authors

Mario Linares-Vásquez - Universidad de los Andes, Bogotá, Colombia
E-mail: m.linaresv at uniandes dot edu dot co
Kevin Moran - The College of William and Mary, VA, USA.
E-mail: kpmoran at cs dot wm

dot

Carlos Bernal-Cárdenas - The College of William and Mary, VA, USA.
E-mail: cebernal at cs dot wm

dot

Denys Poshyvanyk - The College of William and Mary.
E-mail: denys at cs dot wm dot edu