edribeirojunior/android-sdk-mvn repository overview

⁠AndroidSDK

Android SDK development environment Docker image

⁠Android-SDK Installed Packages

⁠Goals

• It contains the complete Android SDK enviroment, is able to perform all regular Android jobs.

• Solves the problem of "It works on my machine, but not on XXX machine".

• Some tool (e.g. Infer⁠), which has complex dependencies might be in conflict with your local environment. Installing the tool within a Docker container is the easiest and perfect solution.

• Directly being used as Android CI build enviroment.

⁠Philosophy

Provide only the barebone SDK (the latest official minimal package) gives you the most flexibility in tailoring your own SDK tools for your project. You can maintain an external persistent SDK directory, and mount it to any container. In this way, you don't have to waste time on downloading over and over again, meanwhile, without having any unnecessary package.

⁠Caveat

Run Android SDK update directly within the Dockerfile or inside the container would fail if the storage driver is AUFS (by default), it is due to some file operations (during updating) are not supported by this storage driver, but changing it to Btrfs would work.

What happens if the update fails?

ls $ANDROID_HOME/tools/
#=> empty, nothing is there
# tools such as: android, sdkmanager, emulator, lint and etc. are gone

android
#=> bash: android: command not found

sdkmanager
#=> bash: /opt/android-sdk/tools/bin/sdkmanager: No such file or directory

To know more about the storage driver:

• Check Docker's current storage driver option

  docker info | grep 'Storage Driver'
  

  Copy

• Check which filesystems are supported by the running host kernel

  cat /proc/filesystems
  

  Copy

To prevent this problem from happening, and you don't wanna bother dealing with storage driver. The only solution is to mount an external SDK volume from host to container. Then you are free to try any of below approaches.

• Update SDK in the usual way but directly inside container.

• Update SDK from host directory (Remember: the host machine must be the same target architecture as the container - x86_64 Linux).

If you by accident update SDK on a host machine which has a mismatch target architecture than the container, some binaries won't be executable in container any longer.

gradle <some_task>
#=> Error: java.util.concurrent.ExecutionException: java.lang.RuntimeException: AAPT process not ready to receive commands

$ANDROID_HOME/build-tools/x.x.x/aapt
#=> aapt: cannot execute binary file: Exec format error

adb
#=> adb: cannot execute binary file: Exec format error

⁠Getting Started

# build the image
# set the working directory to the project's root directory first
docker build -t android-sdk android-sdk
# if you need to send the image to a registry
docker build -t android-sdk REGISTRY-NAME/android-sdk:TAG
# or you can also pass specific tool version as you wish (optional, while there is default version)
docker build --build-arg ANDROID_SDK_VERSION=<android_sdk_version> -t android-sdk android-sdk
# or pull the image instead of building on your own
docker pull edribeirojunior/android-sdk-mvn

# below commands assume that you've pulled the image

# copy the pre-downloaded SDK to the mounted 'sdk' directory
docker run -it --rm -v $(pwd)/sdk:/sdk edribeirojunior/android-sdk-mvn bash -c 'cp -a $ANDROID_HOME/. /sdk'

# go to the 'sdk' directory on the host, update the SDK
# ONLY IF the host machine is the same target architecture as the container
# JDK required on the host
sdk/tools/bin/sdkmanager --update
# or install specific packages
sdk/tools/bin/sdkmanager "build-tools;x.y.z" "platforms;android-x" ...

# mount the updated SDK to container again
# if the host SDK directory is mounted to more than one container
# to avoid multiple containers writing to the SDK directory at the same time
# you should mount the SDK volume in read-only mode
docker run -it -v $(pwd)/sdk:/opt/android-sdk:ro edribeirojunior/android-sdk-mvn /bin/bash

# you can mount without read-only option, only if you need to update SDK inside container
docker run -it -v $(pwd)/sdk:/opt/android-sdk edribeirojunior/android-sdk-mvn /bin/bash

# to keep and reuse Gradle cache
docker run -it -v $(pwd)/sdk:/opt/android-sdk -v $(pwd)/gradle_caches:/root/.gradle/caches edribeirojunior/android-sdk-mvn /bin/bash

# to stop and remove container
# when the image was pulled from a registry
docker stop $(docker ps -aqf "ancestor=edribeirojunior/android-sdk-mvn") &> /dev/null && docker rm $(docker ps -aqf "ancestor=edribeirojunior/android-sdk-mvn") &> /dev/null
# when the image was built locally
docker stop $(docker ps -aqf "ancestor=android-sdk") &> /dev/null && docker rm $(docker ps -aqf "ancestor=android-sdk") &> /dev/null
# more flexible way - doesn't matter where the image comes from
docker stop $(docker ps -a | grep 'android-sdk' | awk '{ print $1 }') &> /dev/null && docker rm $(docker ps -a | grep 'android-sdk' | awk '{ print $1 }') &> /dev/null

⁠SSH

It is also possible if you wanna connect to container via SSH. There are three different approaches.

• Build an image on your own, with a built-in authorized_keys

  # Put your `id_rsa.pub` under `android-sdk/authorized_keys` (as many as you want)
  
  # Build an image
  docker build -t android-sdk android-sdk
  
  # Run a container
  docker run -d -p 2222:22 -v $(pwd)/sdk:/opt/android-sdk:ro android-sdk
  

  Copy

• Mount authorized_keys file from the host to a container

  docker run -d -p 2222:22 -v $(pwd)/authorized_keys:/root/.ssh/authorized_keys edribeirojunior/android-sdk-mvn
  

  Copy

• Copy a local authorized_keys file to a container

  # Create a local `authorized_keys` file, which contains the content from your `id_rsa.pub`
  
  # Run a container
  docker run -d -p 2222:22 -v $(pwd)/sdk:/opt/android-sdk:ro edribeirojunior/android-sdk-mvn
  
  # Copy the just created local authorized_keys file to the running container
  docker cp $(pwd)/authorized_keys `docker ps -aqf "ancestor=edribeirojunior/android-sdk-mvn"`:/root/.ssh/authorized_keys
  
  # Set the proper owner and group for authorized_keys file
  docker exec -it `docker ps -aqf "ancestor=edribeirojunior/android-sdk-mvn"` bash -c 'chown root:root /root/.ssh/authorized_keys'
  

  Copy

That's it! Now it's up and running, you can ssh to it

ssh root@<container_ip_address> -p 2222

And, in case you need, you can still attach to the running container (not via ssh) by

docker exec -it <container_id> /bin/bash

⁠VNC

Remote access to the container's desktop might be helpful if you plan to run emulator inside the container.

# pull the image with VNC support
docker pull thyrlian/android-sdk-vnc

# spin up a container with SSH
# won't work when spin up with interactive session, since the vncserver won't get launched
docker run -d -p 5901:5901 -p 2222:22 -v $(pwd)/sdk:/opt/android-sdk thyrlian/android-sdk-vnc

When the container is up and running, use your favorite VNC client to connect to it:

• <container_ip_address>:5901

• Password (with control): android

• Password (view only): docker

# setup and launch emulator inside the container
# create a new Android Virtual Device
echo "no" | avdmanager create avd -n test -k "system-images;android-25;google_apis;armeabi-v7a"
# launch emulator
emulator64-arm -avd test -noaudio -no-boot-anim -gpu offscreen

For more details, please refer to Emulator section⁠.

⁠VNC client recommendation

• macOS: Screen Sharing⁠

⁠NFS

You can host the Android SDK in one host-independent place, and share it across different containers. One solution is using NFS (Network File System).

To make the container consume the NFS, you can try either way below:

• Mount the NFS onto your host machine, then run container with volume option (-v).

• Use a Docker volume plugin, for instance Convoy plugin⁠.

And here are instructions for configuring a NFS server (on Ubuntu):

sudo apt-get update
sudo apt-get install -y nfs-kernel-server
sudo mkdir -p /var/nfs/android-sdk

# put the Android SDK under /var/nfs/android-sdk
# if you haven't got any, run below commands
sudo apt-get install -y wget zip
cd /var/nfs/android-sdk
sudo wget -q $(wget -q -O- 'https://developer.android.com/sdk' | grep -o "\"https://.*android.*tools.*linux.*\"" | sed "s/\"//g")
sudo unzip *tools*linux*.zip
sudo rm *tools*linux*.zip
sudo mkdir licenses
echo 8933bad161af4178b1185d1a37fbf41ea5269c55 | sudo tee licenses/android-sdk-license > /dev/null
echo 84831b9409646a918e30573bab4c9c91346d8abd | sudo tee licenses/android-sdk-preview-license > /dev/null
echo d975f751698a77b662f1254ddbeed3901e976f5a | sudo tee licenses/intel-android-extra-license > /dev/null

# configure and launch NFS service
sudo chown nobody:nogroup /var/nfs
echo "/var/nfs         *(rw,sync,no_subtree_check,no_root_squash)" | sudo tee --append /etc/exports > /dev/null
sudo exportfs -a
sudo service nfs-kernel-server start

⁠Emulator

ARM emulator is host machine independent, can run anywhere - Linux, macOS, VM and etc. While the performance is a bit poor. On the contrary, x86 emulator requires KVM, which means only runnable on Linux.

According to Google's documentation⁠:

⁠Preconditions on the host machine (for x86 emulator)

Read How to Start Intel Hardware-assisted Virtualization (hypervisor) on Linux⁠ for more details.

Read KVM Installation⁠ if you haven't got KVM installed on the host yet.

• Check the capability of running KVM

  egrep -c '(vmx|svm)' /proc/cpuinfo
  # or
  egrep -c ' lm ' /proc/cpuinfo
  # when output > 0 it means the host CPU supports hardware virtualization.
  
  sudo kvm-ok
  # seeing below info means you can run your virtual machine faster with the KVM extensions
  INFO: /dev/kvm exists
  KVM acceleration can be used
  

  Copy

• Load KVM module on the host

  modprobe kvm_intel
  

  Copy

• Check if KVM module is successfully loaded

  lsmod | grep kvm
  

  Copy

⁠How to run emulator

• Check available emulator system images from remote SDK repository

  sdkmanager --list --verbose
  

  Copy

• Make sure that the required SDK packages are installed, you can find out by above command. To install, use the command below. Whenever you see error complains about ANDROID_SDK_ROOT, such as PANIC: Cannot find AVD system path. Please define ANDROID_SDK_ROOT or PANIC: Broken AVD system path. Check your ANDROID_SDK_ROOT value, it means that you need to install following packages.

  sdkmanager "platform-tools" "platforms;android-<api_level>" "emulator"
  

  Copy

• Download emulator system image(s) (on the host machine)

  sdkmanager "system_image_1" "system_image_2"
  # e.g.:
  # system-images;android-24;android-tv;x86
  # system-images;android-24;default;arm64-v8a
  # system-images;android-24;default;armeabi-v7a
  # system-images;android-24;default;x86
  # system-images;android-24;default;x86_64
  # system-images;android-24;google_apis;arm64-v8a
  # system-images;android-24;google_apis;armeabi-v7a
  # system-images;android-24;google_apis;x86
  # system-images;android-24;google_apis;x86_64
  # system-images;android-24;google_apis_playstore;x86
  

  Copy

• Run Docker container in privileged mode (not necessary for ARM emulator)

  # required by KVM
  docker run -it --privileged -v $(pwd)/sdk:/opt/android-sdk:ro edribeirojunior/android-sdk-mvn /bin/bash
  

  Copy

• Check acceleration ability (not necessary for ARM emulator)

  emulator -accel-check
  
  # when succeeds
  accel:
  0
  KVM (version 12) is installed and usable.
  accel
  
  # when fails (probably due to unprivileged mode)
  accel:
  8
  /dev/kvm is not found: VT disabled in BIOS or KVM kernel module not loaded
  accel
  

  Copy

• Create a new Android Virtual Device

  echo "no" | avdmanager create avd -n <name> -k <sdk_id>
  # e.g.:
  echo "no" | avdmanager create avd -n test -k "system-images;android-24;default;armeabi-v7a"
  

  Copy

• List existing Android Virtual Devices

  avdmanager list avd
  # ==================================================
  Available Android Virtual Devices:
      Name: test
      Path: /root/.android/avd/test.avd
    Target: Default Android System Image
            Based on: Android 7.0 (Nougat) Tag/ABI: default/armeabi-v7a
  # ==================================================
  
  # or
  
  emulator -list-avds
  # 32-bit Linux Android emulator binaries are DEPRECATED
  # ==================================================
  test
  # ==================================================
  

  Copy

• Launch emulator in background

  # use different command per architecture
  emulator64-arm -avd <virtual_device_name> -noaudio -no-boot-anim -no-window -accel on &
  
  # or
  emulator64-mips -avd <virtual_device_name> -noaudio -no-boot-anim -no-window -accel on &
  
  # or
  emulator64-x86 -avd <virtual_device_name> -noaudio -no-boot-anim -no-window -accel on &
  
  # if the container is not running in privileged mode, you should see below errors:
  #=> emulator: ERROR: x86_64 emulation currently requires hardware acceleration!
  #=> Please ensure KVM is properly installed and usable.
  #=> CPU acceleration status: /dev/kvm is not found: VT disabled in BIOS or KVM kernel module not loaded
  # or it's running on top of a VM
  #=> CPU acceleration status: KVM requires a CPU that supports vmx or svm
  

  Copy

• Check the virtual device status

  adb devices
  # ==================================================
  List of devices attached
  emulator-5554	offline
  # "offline" means it's still booting up
  # ==================================================
  
  # ==================================================
  List of devices attached
  emulator-5554	device
  # "device" means it's ready to be used
  # ==================================================
  

  Copy

Now you can for instance run UI tests on the emulator (just remember, the performance is POOR):

<your_android_project>/gradlew connectedAndroidTest

⁠Access the emulator from outside

Default adb server port: 5037

# spin up a container
# with SSH
docker run -d -p 5037:5037 -p 2222:22 -v $(pwd)/sdk:/opt/android-sdk thyrlian/android-sdk-vnc
# or with interactive session
docker run -it -p 5037:5037 -v $(pwd)/sdk:/opt/android-sdk thyrlian/android-sdk-vnc /bin/bash

# launch emulator inside the container...

Outside the container:

adb connect <container_ip_address>:5037
adb devices
#=> List of devices attached
#=> emulator-5554	device

Make sure that your adb client talks to the adb server inside the container, instead of the local one on the host machine. This can be achieved by running adb kill-server (to kill the local server if it's already up) before firing adb connect command above.

⁠Android Device

You can give a container access to host's USB Android devices.

# on Linux
docker run -it --privileged -v /dev/bus/usb:/dev/bus/usb -v $(pwd)/sdk:/opt/android-sdk edribeirojunior/android-sdk-mvn /bin/bash

# or
# try to avoid privileged flag, just add necessary capabilities when possible
# --device option allows you to run devices inside the container without the --privileged flag
docker run -it --device=/dev/ttyUSB0 -v $(pwd)/sdk:/opt/android-sdk edribeirojunior/android-sdk-mvn /bin/bash

Note:

• Connect Android device via USB on host first;

• Launch container;

• Disconnect and connect Android device on USB;

• Select OK for "Allow USB debugging" on Android device;

• Now the Android device will show up inside the container (adb devices).

Don't worry about adbkey or adbkey.pub under /.android, not required.

Docker for Mac FAQ⁠ says:

⁠Android Commands Reference

• Check installed Android SDK tools version

  cat $ANDROID_HOME/tools/source.properties | grep Pkg.Revision
  cat $ANDROID_HOME/platform-tools/source.properties | grep Pkg.Revision
  

  Copy

  The "android" command is deprecated. For command-line tools, use tools/bin/sdkmanager and tools/bin/avdmanager.

• List installed and available packages

  sdkmanager --list
  # print full details instead of truncated path
  sdkmanager --list --verbose
  

  Copy

• Update all installed packages to the latest version

  sdkmanager --update
  

  Copy

• Install packages

  The packages argument is an SDK-style path as shown with the --list command, wrapped in quotes (for example, "extras;android;m2repository"). You can pass multiple package paths, separated with a space, but they must each be wrapped in their own set of quotes.

  sdkmanager "extras;android;m2repository" "extras;google;m2repository" "extras;google;google_play_services" "extras;google;instantapps" "extras;m2repository;com;android;support;constraint;constraint-layout;1.0.2" "build-tools;26.0.0" "platforms;android-26"
  

  Copy

• Stop emulator

  adb -s <device_sn> emu kill
  

  Copy

⁠Demythologizing Memory

⁠OOM behaviour

Sometimes you may encounter OOM (Out of Memory) issue. The issues vary in logs, while you could find the essence by checking the exit code (echo $?).

For demonstration, below examples try to execute MemoryFiller⁠ which can fill memory up quickly.

• Exit Code 137 (= 128 + 9 = SIGKILL = Killed)

  Example code:

  # spin up a container with memory limit (128MB)
  docker run -it -m 128m -v $(pwd)/misc/MemoryFiller:/root/MemoryFiller edribeirojunior/android-sdk-mvn /bin/bash
  # fill memory up
  cd /root/MemoryFiller && javac MemoryFiller.java
  java MemoryFiller
  

  Copy

  Logs:

  Killed
  

  Copy

  Commentary: The process was in extreme resource starvation, thus was killed by the kernel OOM killer. This happens when JVM max heap size > actual container memory. Similarly, the logs could look like this when running a gradle task in an Android project: Process 'Gradle Test Executor 1' finished with non-zero exit value 137.

• Exit Code 1 (= SIGHUP = Hangup)

  Example code:

  # spin up a container with memory limit (or without - both lead to the same result)
  docker run -it -m 128m -v $(pwd)/misc/MemoryFiller:/root/MemoryFiller edribeirojunior/android-sdk-mvn /bin/bash
  # fill memory up
  # enable Docker memory limits transparency for JVM
  cd /root/MemoryFiller && javac MemoryFiller.java
  java -XX:+UnlockExperimentalVMOptions -XX:+UseCGroupMemoryLimitForHeap MemoryFiller
  

  Copy

  Logs:

  Exception in thread "main" java.lang.OutOfMemoryError: Java heap space at MemoryFiller.main(MemoryFiller.java:13)
  

  Copy

  Commentary: With enabling Docker memory limits transparency for JVM⁠, JVM is able to correctly estimate the max heap size, and it won't be killed by the kernel OOM killer any more. Similarly, the logs could look like this when running a gradle task in an Android project: Process 'Gradle Test Executor 1' finished with non-zero exit value 1. In this case, you should either check your code or tweak your memory limit for container (or JVM heap parameters, or even the host memory size).

• Exit Code 3 (= SIGQUIT = Quit)

  Example code:

  # spin up a container without memory limit
  docker run -it -v $(pwd)/misc/MemoryFiller:/root/MemoryFiller edribeirojunior/android-sdk-mvn /bin/bash
  # fill memory up
  cd /root/MemoryFiller && javac MemoryFiller.java
  # make sure that Docker memory resource is big enough > JVM max heap size
  # otherwise it's better to run with UnlockExperimentalVMOptions & UseCGroupMemoryLimitForHeap enabled
  java -XX:+ExitOnOutOfMemoryError MemoryFiller
  

  Copy

  Logs:

  Terminating due to java.lang.OutOfMemoryError: Java heap space
  

  Copy

  Commentary: JRockit JVM exits on the first occurrence of an OOM error. It can be used if you prefer restarting an instance of JRockit JVM rather than handling OOM errors.

• Exit Code 134 (= 128 + 6 = SIGABRT = Abort)

  Example code:

  # spin up a container without memory limit
  docker run -it -v $(pwd)/misc/MemoryFiller:/root/MemoryFiller edribeirojunior/android-sdk-mvn /bin/bash
  # fill memory up
  cd /root/MemoryFiller && javac MemoryFiller.java
  # make sure that Docker memory resource is big enough > JVM max heap size
  # otherwise it's better to run with UnlockExperimentalVMOptions & UseCGroupMemoryLimitForHeap enabled
  java -XX:+CrashOnOutOfMemoryError MemoryFiller
  

  Copy

  Logs:

  Aborting due to java.lang.OutOfMemoryError: Java heap space
  #
  # A fatal error has been detected by the Java Runtime Environment:
  #
  #  Internal Error (debug.cpp:308), pid=63, tid=0x00007f208708d700
  #  fatal error: OutOfMemory encountered: Java heap space
  #
  # JRE version: OpenJDK Runtime Environment (8.0_131-b11) (build 1.8.0_131-8u131-b11-2ubuntu1.16.04.3-b11)
  # Java VM: OpenJDK 64-Bit Server VM (25.131-b11 mixed mode linux-amd64 compressed oops)
  # Failed to write core dump. Core dumps have been disabled. To enable core dumping, try "ulimit -c unlimited" before starting Java again
  #
  # An error report file with more information is saved as:
  # /root/MemoryFiller/hs_err_pid63.log
  #
  # If you would like to submit a bug report, please visit:
  #   http://bugreport.java.com/bugreport/crash.jsp
  #
  Aborted
  

  Copy

  Commentary: JRockit JVM crashes and produces text and binary crash files when an OOM error occurs. When JVM crashes with a fatal error, an error report file hs_err_pid***.log will be generated in the same working directory.

⁠Facts

• JVM is not container aware, and always guesses about the memory resource.

• Many tools (such as free, vmstat, top) were invented before the existence of cgroups⁠, thus they have no clue about the resources limits.

• MaxRAMFraction: maximum fraction (1/n) of real memory used for maximum heap size, the default value is 4.

• MaxMetaspaceSize: where class metadata reside. MaxPermSize is deprecated in JDK 8. It used to be Permanent Generation space before JDK 8, which could cause java.lang.OutOfMemoryError: PermGen problem.

• -XshowSettings:category: s