Public | Automated Build

Last pushed: 10 months ago
Short Description
ProjectOdd OpenWhisk Controller image
Full Description

OpenWhisk Deployment for Kubernetes

This repo can be used to deploy OpenWhisk to a Kubernetes cluster.
To accomplish this, we have created a Kubernetes job responsible for
deploying OpenWhisk from inside of Kubernetes. This job runs through
the OpenWhisk Ansible playbooks with some modifications to "Kube-ify"
specific actions. The reason for this approach is to try and streamline
a one size fits all way of deploying OpenWhisk.

Currently, the OpenWhisk deployment is going to be a static set of
Kube yaml files. It should be easy to use the tools from this
repo to build your own OpenWhisk deployment job, allowing you to
set up your own configurations if need be.

The scripts and Docker images should be able to:

  1. Build the Docker image used for deploying OpenWhisk.
  2. Uses a Kubernetes job to deploy OpenWhisk.

Table of Contents


A number of requirements must be met for OpenWhisk to deploy on Kubernetes.


  • Kubernetes version 1.5.6 and 1.6.2
  • Kubernetes has KubeDNS deployed
  • (Optional) Kubernetes Pods can receive public addresses.
    This will be required if you wish to reach Nginx from outside
    of the Kubernetes cluster network.


  • Docker version 1.12+

NOTE: If you do not have an environment that meets these requirements then you can
set one up following the Setting up Kubernetes section.

Setting up Kubernetes

If you are deploying Kubernetes from scratch to try out OpenWhisk, there are a number of
ways to setup a Dev/Test environmnet depending on your host OS. To mitigate that, we are using
VirtualBox with a Ubuntu VM. For the full instructions on setting an environment up,
take a look at these instructions.

Configure OpenWhisk

To configure OpenWhisk on Kubernetes, you will need to target a Kubernetes
environment. If you do not have one up and running, then you can look
at the Setting up Kubernetes section. Once you
are successfully up, running, and targetd, you will then need to create a
namespace called openwhisk. To do this, you can just run the following command.

kubectl apply -f configure/openwhisk_kube_namespace.yml

From here, you should just need to run the Kubernetes job to
setup the OpenWhisk environment. The only caveat is that
the default image is used to deploy to kube v1.5.6.
Take a look
if you wish to change to kube v1.6.2 by replacing v1.5.2 to v1.6.2.

NOTE Unfortunately Kube does not have backward compatibility
requirements between the cli and Kube api server. However,
the v1.5.6 image will probably work with any Kube v1.5+
and the v1.6.2 image will probably work with any Kube v1.6+.
If the configuration image does return compatibility
issues then try building a custom image.

kubectl apply -f configure/configure_whisk.yml

To see what is happening during the deployment process, you can view
the logs from the configuration Pod creted by the previous command.

kubectl -n openwhisk logs configure-openwhisk-XXXXX

Once the configuration job successfully finishes, you will need the
default user auth tokens provided by OpenWhisk. As part of the deployment
process, we store these tokens in Kubernetes
To get these tokens, you can run the following command:

kubectl -n openwhisk get secret openwhisk-auth-tokens -o yaml

To use the secrets, you will need to base64 decode them. E.g:

export AUTH_SECRET=$(kubectl -n openwhisk get secret openwhisk-auth-tokens -o yaml | grep 'auth_whisk_system:' | awk '{print $2}' | base64 --decode)

From here, you will now need to get the publicly available address
of Nginx.

  1. Obtain the IP address of the Kubernetes nodes.

    kubectl get nodes
  2. Obtain the public port for the Kubernetes Nginx Service

    kubectl -n openwhisk describe service nginx

    From here you should note the port used for the api endpoint. E.g:

    export WSK_PORT=$(kubectl -n openwhisk describe service nginx | grep https-api | grep NodePort| awk '{print $3}' | cut -d'/' -f1)

Now you should be able to setup the wsk cli like normal and interact with

wsk property set --auth $AUTH_SECRET --apihost https://[nginx_ip]:$WSK_PORT


At some point there might be a need to cleanup the Kubernetes
environment. For this, we want to delete all the OpenWhisk deployments,
services, jobs and whatever else might be there. For this, you can run the following script:




When inspecting kafka logs of various components and they are not able to
send/receive message then Kafka is the usual problem. If everything is deployed
on a single machine, then you might need to allow Kube Pods to communicate with
themselves over a Kube Service. Setting a network to promiscous mode can be the
solution will enable network traffic to route in a loop back to itself. E.g:

ip link set docker0 promisc on


When deploying the configuration pod, if it fails with a
error validating data: the server does not allow access to the requested resource;
error then you probably do not have permissions to create Pods from a Pod running
in the Kube cluster. You will need to create a ClusterRoleBinding with proper
security settings. For information about the role bindings,
take a look at the info here.

Manually Building Custom Docker Files

There are two images that are required when deploying OpenWhisk on Kube,
Nginx and the OpenWhisk configuration image. Right now the the configuration
images built will work with a Kube version 1.5.6 and 1.6.2. To build the
configuration image with a custom Kube version you can edit the build script

To build these images, there is a helper script that installs all
required dependencies and the Docker images themselves. For example,
one the required dependencies is the wsk cli and to build it you will need
to download the OpenWhisk repo
and setup your invironment to build the docker images via gradle. That
setup can be found here.

To use the script, it takes in 2 arguments:

  1. (Required) The first argument is the Docker account to push the built images
    to. For Nginx, it will tag the image as account_name/whisk_nginx:latest
    and the OpenWhisk configuration image will be tagged
    account_name/whisk_config:dev-v1.5.6 and account_name/whisk_config:dev-v1.6.2.

    NOTE: log into Docker before running the script or it will
    fail to properly upload the docker images.

  2. (Optional) The second argument is the location of where the
    repo is installed locally. By default it assumes that this repo exists at

If you plan on building your own images and would like to change from danlavine's,
then make sure to update the
configure_whisk.yml and
nginx with your images.

To run the script, use the command:

docker/build <Docker username> <(optional) openwhisk dir>

Editing the Openwhisk Kube Deployment

Kubernetes Deployments and Services

The current Kube Deployment and Services files that define the OpenWhisk
cluster can be found here. Only one
instance of each OpenWhisk process is created, but if you would like
to increase that number, then this would be the place to do it. Simply edit
the appropriate file and
Manually Build Custom Docker Files


Debugging OpenWhisk on Kubernetes Configuration Pod

When in the process of creating a new deployment, it is nice to
run things by hand to see what is going on inside the container and
not have it be removed as soon as it finishes or fails. For this,
you can change the command of configure_whisk.yml
to command: [ "tail", "-f", "/dev/null" ]. Then just run the
original command from inside the Pod's container.

Limitations and Enhancements


During the deployment process, OpenWhisk needs to generate a CA-cert
for Nginx and currently it has a static dns entry. Because of this, you
will need to connect to OpenWhisk using the insecure mode (e.g. wsk -i).
There is future work to make this CA-cert configurable.

For now, OpenWhisk relies on part of the underlying infrastructure that Kube
is running on. When deploying the Invoker for OpenWhisk, it mounts the hosts
Docker socket. This way OpenWhisk can quickly provision actions and does not
have to run Docker inside of Docker.

A couple of components for OpenWhisk on Kube deployment strategy requires custom
built Docker images. One such component is Nginx and currently resides at
danlavine/whisk_nginx. There
is currently and open issue
to make a public image and once it is resolved, then we can switch to the public image.

The second Docker image this deployment strategy relies on is the OpenWhisk
configuration image. For now, it is hosted at
but ideally an official images can be built an maintained at some point.
If you would like to build your own deployment image, see
Manually Build Custom Docker Files

Lastly, since OpenWhisk is configured/deployed via a Kubernetes Pod it requires
the correct kubectl version to be built into danlavine/whisk_config. For now,
there is only a version for Kube 1.5, and one can be built for 1.6, but there
is no CI to test it against at the moment.

Minikube is not supported at this time because it uses an old version
of docker (1.11.x). See the the Requirements
section for more info.


  • Allow users to provide custom certs for Nginx
  • Enable the configuration job to run any number of times. This way it updates an already running
    OpenWhisk deployment on all subsequent runs
  • Use a public Edge Docker image once this issue
    is resolved


Report bugs, ask questions and request features here on GitHub.

You can also join our slack channel and chat with developers. To get access to our slack channel, request an invite here.

Docker Pull Command