jetski/hawkular-openshift-agent repository overview

= Hawkular OpenShift Agent image:https://travis-ci.org/hawkular/hawkular-openshift-agent.svg["Build⁠ Status", link="https://travis-ci.org/hawkular/hawkular-openshift-agent⁠"] :toc: macro :toc-title:

toc::[]

== Introduction

Hawkular OpenShift Agent is a Hawkular feed implemented in the Go Programming Language. Its main purpose is to monitor a node within an OpenShift environment, collecting metrics from Prometheus and/or Jolokia endpoints deployed in one or more pods within the node. It can also be used to collect metrics from endpoints outside of OpenShift. The agent can be deployed inside the OpenShift node it is monitoring, or outside of OpenShift completely.

Watch link:https://www.youtube.com/watch?v=jvOPlz7lzyM[this⁠ quick 10-minute demo] to see the agent in action.

Note that the agent does not collect or store inventory at this time - this is strictly a metric collection and storage agent that integrates with Hawkular Metrics.

=== Docker Image

Hawkular OpenShift Agent is published as a docker image on https://hub.docker.com/r/hawkular/hawkular-openshift-agent[Docker⁠ hub at hawkular/hawkular-openshift-agent]

=== License and Copyright

.... Copyright 2016 Red Hat, Inc. and/or its affiliates and other contributors.

Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at

   http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. ....

== Building

[NOTE] These build instructions assume you have the following installed on your system: (1) link:http://golang.org/doc/install[Go⁠ Programming Language], (2) link:http://git-scm.com/book/en/v2/Getting-Started-Installing-Git[git⁠], (3) link:https://docs.docker.com/installation/[Docker⁠], and (4) make. To run Hawkular OpenShift Agent inside OpenShift after you build it, it is assumed you have a running OpenShift environment available to you. If you do not, you can find a set of link:#setting-up-openshift[instructions on how to set up OpenShift below].

To build Hawkular OpenShift Agent:

• Clone this repository inside a GOPATH. These instructions will use the example GOPATH of "/source/go/hawkular-openshift-agent" but you can use whatever you want. Just change the first line of the below instructions to use your GOPATH.

⁠[source,shell]

⁠export GOPATH=/source/go/hawkular-openshift-agent mkdir -p $GOPATH cd $GOPATH mkdir -p src/github.com/hawkular cd src/github.com/hawkular git clone [email protected]⁠:hawkular/hawkular-openshift-agent.git export PATH=${PATH}:${GOPATH}/bin

• Install Glide - a Go dependency management tool that Hawkular OpenShift Agent uses to build itself

⁠[source,shell]

⁠cd ${GOPATH}/src/github.com/hawkular/hawkular-openshift-agent make install_glide

• Tell Glide to install the Hawkular OpenShift Agent dependencies

⁠[source,shell]

⁠cd ${GOPATH}/src/github.com/hawkular/hawkular-openshift-agent make install_deps

• Build Hawkular OpenShift Agent

⁠[source,shell]

⁠cd ${GOPATH}/src/github.com/hawkular/hawkular-openshift-agent make build

• At this point you can run the Hawkular OpenShift Agent tests

⁠[source,shell]

⁠cd ${GOPATH}/src/github.com/hawkular/hawkular-openshift-agent make test

== Running

=== Running Inside OpenShift

==== Setting up OpenShift The following section assumes that the user has link:https://github.com/openshift/origin[OpenShift⁠ Origin] installed with link:https://github.com/openshift/origin-metrics[metrics⁠] enabled.

The link:https://docs.openshift.org/latest/welcome/index.html[OpenShift⁠ Origin Documentation] will outline all the steps required. Please make sure to follow the link:https://docs.openshift.org/latest/install_config/cluster_metrics.html[steps⁠ involved to deploying metrics].

If you wish to forgo installing and configuring OpenShift with metrics yourself, the oc cluster up command can be used to get an instance of OpenShift Origin with metrics enabled:

⁠[source,shell]

⁠oc cluster up --metrics

[NOTE] In order to install the agent into OpenShift you will need to have admin priviledges for the OpenShift cluster.

==== Building the Docker Image

Create the Hawkular OpenShift Agent docker image through the "docker" make target:

⁠[source,shell]

⁠cd ${GOPATH}/src/github.com/hawkular/hawkular-openshift-agent make docker

==== Deploying the Agent Inside OpenShift

[NOTE] The following sections assume that the oc command is available in the user's path and that the user is logged in as a user with cluster admin privileges.

⁠[TIP]

⁠If you do not want to manually deploy the agent into OpenShift, the steps are automated in the Makefile. The following will undeploy an old installation of the agent, if available, and deploy a new one:

⁠make openshift-deploy

====

To deploy the agent, you will need to follow the following commands:

⁠[source,shell]

⁠oc adm policy add-cluster-role-to-user cluster-reader system:serviceaccount:openshift-infra:hawkular-agent oc create -f deploy/openshift/hawkular-openshift-agent-configmap.yaml -n openshift-infra oc process -f deploy/openshift/hawkular-openshift-agent.yaml | oc create -n openshift-infra -f -

==== Undeploying the Agent

If you want to remove the agent from your OpenShift environment, you can do so by running the following command:

⁠[source,shell]

⁠oc delete all,secrets,sa,templates,configmaps --selector=metrics-infra=agent -n openshift-infra

Alternatively, the following will also perform the same task from the Makefile:

⁠[source,shell]

⁠make openshift-undeploy

=== Running Outside OpenShift

[NOTE] You must customize Hawkular OpenShift Agent's configuration file so it can be told things like your Hawkular Metrics server endpoint. If you want the agent to connect to an OpenShift master, you need the OpenShift CA cert file which can be found in your OpenShift installation at openshift.local.config/master/ca.crt. If you installed OpenShift in a VM via vagrant, you can use vagrant ssh to find this at /var/lib/origin/openshift.local.config/master/ca.crt. If you wish to configure the agent with environment variables as opposed to the config file, see link:#environment-variables[below] for the environment variables that the agent looks for.

⁠[source,shell]

⁠cd ${GOPATH}/src/github.com/hawkular/hawkular-openshift-agent make install make run

The "install" target installs the Hawkular OpenShift Agent executable in your GOPATH /bin directory so you can run it outside of the Makefile:

⁠[source,shell]

⁠cd ${GOPATH}/src/github.com/hawkular/hawkular-openshift-agent make install ${GOPATH}/bin/hawkular-openshift-agent -config

If you don't want to store your token in the YAML file, you can pass it via an environment variable:

⁠[source,shell]

⁠K8S_TOKEN=oc whoami -t ${GOPATH}/bin/hawkular-openshift-agent -config config.yaml

== Configuring OpenShift

When Hawkular OpenShift Agent is monitoring resources running on an OpenShift node, it looks at custom annotations and config maps found in OpenShift to know what to monitor. In effect, the pods tell Hawkular OpenShift Agent what to monitor, and Hawkular OpenShift Agent does it. (Note that where "OpenShift" is mentioned, it is normally synonymous with "Kubernetes" because Hawkular OpenShift Agent is really interfacing with the underlying Kubernetes software that is running in OpenShift)

One caveat must be mentioned up front. Hawkular OpenShift Agent will only monitor a single OpenShift node. If you want to monitor multiple OpenShift nodes, you must run one Hawkular OpenShift Agent process per node.

There are two features in OpenShift that Hawkular OpenShift Agent takes advantage of when it comes to configuring what Hawkular OpenShift Agent should be monitoring - one is pod annotations and the second is project config maps.

=== Pod Annotations

Each pod running on the node has a set of annotations. An annotation is simply a name/value pair. Hawkular OpenShift Agent expects to see an annotation named "hawkular-openshift-agent" on a pod that is to be monitored. If this annotation is missing, it is assumed you do not want Hawkular OpenShift Agent to monitor that pod. The value of this annotation named "hawkular-openshift-agent" is the name of a config map within the pod's project. If the config map is not found in the pod's project, again Hawkular OpenShift Agent will not monitor the pod.

=== Project Config Map

Pods are grouped in what are called "projects" in OpenShift (Kubernetes calls these "namespaces" - if you see "namespace" in the Hawkular OpenShift Agent configuration settings and log messages, realize it is talking about an OpenShift project). Each project has what is called a "config map". Similiar to annotations, config maps contain name/value pairs. The values can be as simple as short strings or as complex as complete YAML or JSON blobs. Because config maps are on projects, they are associated with multiple pods (the pods within the project).

Hawkular OpenShift Agent takes advantage of a project's config maps by using them as places to put YAML configuration for each monitored pod that belongs to the project. Each pod configuration is found in one config map. The config map that Hawkular OpenShift Agent will look for must be named the same as the value found in a pod's "hawkular-openshift-agent" annotation.

=== Config Map Entry Schema

Each Hawkular OpenShift Agent config map must have one and only one entry which must be named "hawkular-openshift-agent". A config map entry is a YAML configuration. The Go representation of the YAML schema is found link:https://github.com/hawkular/hawkular-openshift-agent/blob/master/k8s/configmap_entry.go[here⁠].

So, in short, each OpenShift project (aka Kubernetes namespace) will have multiple config maps each with an entry named "hawkular-openshift-agent" where those entries contain YAML configuration containing information about what should be monitored on a pod. A named config map is referenced by a pod's annotation also called "hawkular-openshift-agent".

Hawkular OpenShift Agent examines each pod on the node and by cross-referencing the pod annotations with the project config maps, Hawkular OpenShift Agent knows what it should manage.

=== Example

Suppose you have a node running a project called "my-project" that consists of 3 pods (named "web-pod", "app-pod", and "db-pod"). Suppose you do not want Hawkular OpenShift Agent to monitor the "db-pod" but you do want it to monitor the other two pods in your project.

First create two config maps on your "my-project" that each contain a config map entry that indicate what you want to monitor on your two pods. One way you can do this is create a YAML file that represents your config maps and via the "oc" OpenShift command line tool create the config maps. A sample YAML configuration for the web-pod config map could look like this (the schema of this YAML will change in the future, this is just an example).

⁠[source,yaml]

⁠kind: ConfigMap apiVersion: v1 metadata: name: my-web-pod-config namespace: my-project data: hawkular-openshift-agent: | collection_interval_secs: 60 endpoints: - type: prometheus protocol: "http" port: 8080 path: /metrics

Notice the name given to this config map - "my-web-pod-config". This is the name of the config map, and it is this name that should appear as a value to the "hawkular-openshift-agent" annotation found on the "web-pod" pod. It identifies this config map to Hawkular OpenShift Agent as the one that should be used by that pod. Notice also that the name of the config map entry is fixed and must always be "hawkular-openshift-agent". Next, notice the config map entry here. This defines what are to be monitored. Here you see there is a single endpoint for this pod that will expose Prometheus metrics over http and port 8080 at /metrics. The IP address used will be that of the pod itself and thus need not be specified.

To create this config map, save that YAML to a file and use "oc":

⁠[source,shell]

⁠oc create -f my-web-pod-config-map.yaml

If you have already created a "my-web-pod-config" config map on your project, you can update it via the "oc replace" command:

⁠[source,shell]

⁠oc replace -f my-web-pod-config-map.yaml

Now that the config map has been created on your project, you can now add the annotation to the pods that you want to be monitored with the information in that config map. Let's tell Hawkular OpenShift Agent to monitor pod "web-pod" using the configuration named "my-web-pod-config" found in the config map we just created above. We could do something similar for the app-pod (that is, create a config map named, say, "my-app-pod-config" and annotate the app-pod to point to that config map). This can be done with the "oc" command as well.

⁠[source,shell]

⁠oc annotate --overwrite pods web-pod hawkular-openshift-agent=my-web-pod-config oc annotate --overwrite pods app-pod hawkular-openshift-agent=my-app-pod-config

Because we do not want to monitor the db-pod, we do not create that annotation on it. This tells Hawkular OpenShift Agent to ignore that pod.

If you want Hawkular OpenShift Agent to stop monitoring a pod, it is as simple as removing the pod's "hawkular-openshift-agent" annotation:

⁠[source,shell]

⁠oc annotate pods app-pod hawkular-openshift-agent-

=== Try It!

There is a example Docker image you can deploy in your OpenShift environment to see this all work together. The example Docker image will provide you with a WildFly application server that has a Jolokia endpoint installed. You can configure the agent to collect metrics from that Jolokia-enabld WildFly application server such as the "ThreadCount" metric from the MBean "java.lang:type=Threading" and the "used" metric from the composite "HeapMemoryUsage" attribute from the MBean "java.lang:type=Memory".

Assuming you already have your OpenShift environment up and running and you have the Hawkular OpenShift Agent deployed within that OpenShift environment, you can use the link:hack/jolokia-wildfly-example/deploy-to-openshift.sh[example deploy-to-openshift.sh] script to deploy this Jolokia-enabled WildFly application server into your OpenShift environment.

⁠[source,shell]

⁠cd ${GOPATH}/src/github.com/hawkular/hawkular-openshift-agent/hack/jolokia-wildfly-example ./deploy-to-openshift.sh

⁠[NOTE]

⁠The deploy-to-openshift.sh script will install the Jolokia-enabled WildFly application server into your OpenShift's openshift-infra project by default. If you want to install it in another project, simply pass the name of the project to the deploy script. If you want to deploy the example in a new project, create the project first, then pass the name of the new project to the deploy script, like this: [source,shell]

⁠cd ${GOPATH}/src/github.com/hawkular/hawkular-openshift-agent/hack/jolokia-wildfly-example oc login oc new-project hawkular-example ./deploy-to-openshift.sh hawkular-example

====

The deploy script will do the following:

• Installs the Jolokia-enabled WildFly application server in a pod.
• Installs a config map on the project to inform the agent what metrics are to be collected from the pod.
• Installs a "hawkular-openshift-agent" annotation on the pod so it refers to the config map thus turning on metric collection for the pod.

Once the deploy script finishes, within moments the agent will begin collecting metrics and storing them to the Hawkular Metrics server. You can go to the OpenShift console and edit the config map to try things like adding new metric definitions, adding tags to the metrics, and changing the collection interval.

== Configuring External Endpoints To Monitor

Hawkular OpenShift Agent is being developed primarily for running within an OpenShift environment. However, strictly speaking, it does not need to run in or monitor OpenShift. You can run Hawkular OpenShift Agent within your own VM, container, or bare metal and configure it to collect metrics from external endpoints you define in the main config.yaml configuration file.

As an example, suppose you want Hawkular OpenShift Agent to scrape metrics from your Prometheus endpoint running at "http://yourcorp.com:9090/metrics⁠" and store those metrics in Hawkular Metrics. You can add an endpoints section to your Hawkular OpenShift Agent's configuration file pointing to that endpoint which enables Hawkular OpenShift Agent to begin monitoring that endpoint as soon as Hawkular OpenShift Agent starts. The endpoints section of your YAML configuration file could look like this:

⁠[source,yaml]

• type: "prometheus" url: "http://yourcorp.com:9090/metrics⁠" collection_interval_secs: 300

== Prometheus Endpoints

A full Prometheus endpoint configuration can look like this:

⁠[source,yaml]

• type: "prometheus"

  ⁠If this is an endpoint within an OpenShift pod:

  protocol: https port: 9090 path: /metrics

  ⁠If this is an endpoint running outside of OpenShift:

  #url: "https://yourcorp.com:9090/metrics⁠" credentials: token: your-bearer-token-here #username: your-user #password: your-pass collection_interval_secs: 300 metrics:
  • name: go_memstats_last_gc_time_seconds id: gc_time_secs type: gauge
  • name: go_memstats_frees_total type: counter

Some things to note about configuring your Prometheus endpoints:

• Prometheus endpoints can serve metric data in either text or binary form. The agent automatically supports both - there is no special configuration needed. The agent will detect what form the data is in when the endpoint returns it and parses the data accordingly.
• If this is an endpoint running in an OpenShift pod (and thus this endpoint configuration is found in a config map), you do not specify a full URL; instead you specify the protocol, port, and path and the pod's IP will be used for the hostname. URLs are only specified for those endpoints running outside of OpenShift.
• The agent supports either http or https endpoints. If the Prometheus endpoint is over the https protocol, you must configure the agent with a certificate and private key. This is done by either starting the agent with the two environment variables HAWKULAR_OPENSHIFT_AGENT_CERT_FILE and HAWKULAR_OPENSHIFT_AGENT_PRIVATE_KEY_FILE or via the Indentity section of the agent's configuration file: [source,yaml]

⁠identity: cert_file: /path/to/file.crt private_key_file: /path/to/file.key

• The credentials are optional. If the Prometheus endpoint does require authorization, you can specify the credentials as either a bearer token or a basic username/password.
• A metric "id" is used when storing the metric to Hawkular Metrics. If you do not specify an "id" for a metric, its "name" will be used as the default. This metric ID will be prefixed with the "metric_id_prefix" if one is defined in the collector section of the agent's global configuration file.

== Jolokia Endpoints

A full Jolokia endpoint configuration can look like this:

⁠[source,yaml]

• type: "jolokia"

  ⁠If this is an endpoint within an OpenShift pod:

  protocol: https port: 8080 path: /jolokia

  ⁠If this is an endpoint running outside of OpenShift:

  #url: "https://yourcorp.com:8080/jolokia⁠" credentials: token: your-bearer-token-here #username: your-user #password: your-pass collection_interval_secs: 300 metrics:
  • name: java.lang:type=Threading#ThreadCount type: counter id: VM Thread Count
  • name: java.lang:type=Memory#HeapMemoryUsage#used type: gauge id: VM Heap Memory Used

Some things to note about configuring your Jolokia endpoints:

• If this is an endpoint running in an OpenShift pod (and thus this endpoint configuration is found in a config map), you do not specify a full URL; instead you specify the protocol, port, and path and the pod's IP will be used for the hostname. URLs are only specified for those endpoints running outside of OpenShift.
• The agent supports either http or https endpoints. If the Jolokia endpoint is over the https protocol, you must configure the agent with a certificate and private key. This is done by either starting the agent with the two environment variables HAWKULAR_OPENSHIFT_AGENT_CERT_FILE and HAWKULAR_OPENSHIFT_AGENT_PRIVATE_KEY_FILE or via the Indentity section of the agent's configuration file: [source,yaml]

⁠identity: cert_file: /path/to/file.crt private_key_file: /path/to/file.key

• The credentials are optional. If the Jolokia endpoint does require authorization, you can specify the credentials as either a bearer token or a basic username/password.
• A metric "id" is used when storing the metric to Hawkular Metrics. If you do not specify an "id" for a metric, its "name" will be used as the default.
• You must specify a metric's "type" as either "counter" or "gauge".
• A metric "id" is used when storing the metric to Hawkular Metrics. If you do not specify an "id" for a metric, its "name" will be used as the default. This metric ID will be prefixed with the "metric_id_prefix" if one is defined in the collector section of the agent's global configuration file.
• A metric "name" follows a strict format. First is the full MBean name (e.g. java.lang:type=Threading) followed by a hash (#) followed by the attribute that contains the metric data (e.g. ThreadCount). If the attribute is a composite attribute, then you must append a second hash followed by the composite attribute's subpath name which contains the actual metric value. For example, java.lang:type=Memory#HeapMemoryUsage#used will collect the used value of the composite attribute HeapMemoryUsage from the MBean java.lang:type=Memory.

== Environment Variables

Many of the agent's configuration settings can optionally be set via environment variables. If one of the environment variables below are set, they serve as the default value for its associated YAML configuration setting. The following are currently supported:

[cols="1,1a,1"] |=== |Environment Variable Name|YAML Setting|Comments

⁠|HAWKULAR_SERVER_URL | [source,yaml]

⁠hawkular_server: url: VALUE

|This is the Hawkuar Metrics server where all metric data will be stored

⁠|HAWKULAR_SERVER_TENANT | [source,yaml]

⁠hawkular_server: tenant: VALUE

|The default tenant ID to be used if external endpoints do not define their own. Note that OpenShift endpoints always have a tenant which is the same as its pod namespace and thus this setting is not used in that case.

⁠|HAWKULAR_SERVER_CA_CERT_FILE | [source,yaml]

⁠hawkular_server: ca_cert_file: VALUE

|File that contains the certificate that is required to connect to Hawkular Metrics

⁠|HAWKULAR_SERVER_USERNAME | [source,yaml]

⁠hawkular_server: credentials: username: VALUE

|Username used when connecting to Hawkular Metrics

⁠|HAWKULAR_SERVER_PASSWORD | [source,yaml]

⁠hawkular_server: credentials: password: VALUE

|Password used when connecting to Hawkular Metrics

⁠|HAWKULAR_SERVER_TOKEN | [source,yaml]

⁠hawkular_server: credentials: token: VALUE

|Bearer token used when connecting to Hawkular Metrics. If specified, username and password are ignored.

⁠|HAWKULAR_OPENSHIFT_AGENT_CERT_FILE | [source,yaml]

⁠identity: cert_file: VALUE

|File that contains the certificate that identifies this agent.

⁠|HAWKULAR_OPENSHIFT_AGENT_PRIVATE_KEY_FILE | [source,yaml]

⁠identity: private_key_file: VALUE

|File that contains the private key that identifies this agent.

⁠|K8S_MASTER_URL | [source,yaml]

⁠kubernetes: master_url: VALUE

|The location of the OpenShift master. If left blank, it is assumed this agent is running within OpenShift and thus does not need a URL to connect to the master.

⁠|K8S_POD_NAMESPACE | [source,yaml]

⁠kubernetes: pod_namespace: VALUE

|The namespace of the pod where this agent is running. If this is left blank, it is assumed this agent is not running within OpenShift.

⁠|K8S_POD_NAME | [source,yaml]

⁠kubernetes: pod_name: VALUE

|The name of the pod where this agent is running. Only required if the agent is running within OpenShift.

⁠|K8S_TOKEN | [source,yaml]

⁠kubernetes: token: VALUE

|The bearer token required to connect to the OpenShift master.

⁠|K8S_CA_CERT_FILE | [source,yaml]

⁠kubernetes: ca_cert_file: VALUE

|File that contains the certificate required to connect to the OpenShift master. |===

== Metric Tags

Metric data can be tagged with additional metadata called