wurstmeister/mesos-jenkins

By wurstmeister

Updated over 8 years ago

Jenkins with mesos support

Image
1

258

- Jenkins -

An Ubuntu based container built for running a Jenkins Continuous Integration server. Comes packaged with Logstash-Forwarder for log shipping, redpill - a bash script service monitor, and a few Jenkins groovy scripts meant to aid integration with Mesos.

Version Information:
  • Container Release: 1.2.0
  • Mesos: 0.26.0-0.2.145.ubuntu1404
  • Jenkins: 1.642.1
  • Jenkins Mesos Plugin: 0.9.0
Services Include:
  • Jenkins - A well known java based Continuous Integration / Continuous Deployment service.
  • Consul-Template - An application that can populate configs from a consul service.
  • Logrotate - A script and application that aid in pruning log files.
  • Logstash-Forwarder - A lightweight log collector and shipper for use with Logstash.
  • Redpill - A bash script and healthcheck for supervisord managed services. It is capable of running cleanup scripts that should be executed upon container termination.
  • Rsyslog - The system logging daemon.


Index


Usage

In a local environment without Mesos integration, nothing should have to be passed to the container to get going. A simple docker run -d jenkins should spawn Jenkins listening on port 8080.

For a production deployment, a bit more should be considered. Namely, storing the Jenkins configuration long term. Jenkins stores it's configuration information in /var/lib/jenkins ($JENKINS_HOME), this volume should either be mounted to the host or preconfigured via a script supplied by setting the ENVIRONMENT_INIT variable.

Note: If a volume is to be mounted; any Jenkins scripts or configs that are stored in /usr/share/jenkins/ref will not be run. If they should be; copy them to $JENKINS_HOME via script specified in ENVRIONMENT_INIT.

The script method is ideal to use in conjuction with something along the lines of the SCM Sync Configuration Plugin that will allow you to save your configs in git or svn.

Other than that, the only true minimum requirements for getting going in a production setting is to specify ENVIRONMENT as production, and set JENKINS_JNLP_PORT if you do not wish to use the default 8090. All other Jenkins related settings should be tuned to your environment. For further information regarding how to pass other Jenkins settings, please see the Jenkins service section.

Mesos Integration

To run Jenkins with Mesos integration, the container requires several other environment variables to defined. This can be done either at startup and passing them manually with a normal run command, or seeding them via a script and passing it through with the ENVIRONMENT_INIT variable (done when launching via marathon).

  • LIBPROCESS_IP - The ip in which libprocess will bind to. (defaults to 0.0.0.0)

  • LIBPROCESS_PORT - The port used for libprocess communication (defaults to 9000)

  • LIBPROCESS_ADVERTISE_IP - If set, this will be the 'advertised' or 'externalized' ip used for libprocess communication. Relevant when running an application that uses libprocess within a container, and should be set to the host IP in which you wish to use for Mesos communication.

  • LIBPROCESS_ADVERTISE_PORT - If set, this will be the 'advertised' or 'externalized' port used for libprocess communication. Relevant when running an application that uses libprocess within a container, and should be set to the host port you wish to use for Mesos communication.

A supplied sample seed script is available at /opt/scripts/marathon_env_init.sh. This will assign 1 to 1 mappings of the 3 exposed ports needed for Jenkins + Mesos to their associated variables.

  • PORT0 - The Jenkins WebUI
  • PORT1 - The Jenkins JNLP Port
  • PORT3 - port used for both LIBPROCESS_PORT and LIBPROCESS_ADVERTISED_PORT.

To enable Jenkins-Mesos Autoconfiguration set JENKINS_MESOS_AUTOCONF to enabled. This will trigger a Jenkins init groovy script to disable any executors on the master and will either add a NEW cloud to the Jenkins server, or modify a few specific variables of every Mesos Cluster already defined. It CANNOT discern between different Mesos clusters.

For a full list of available options and their descriptions, please see the Jenkins Mesos Autoconfiguration Options.

Example Run Command
docker run -d              \
--name jenkins             \
-e ENVIRONMENT=production  \
-e PARENT_HOST=$(hostname) \
-e JAVA_OPTS="-Xmx1024m"   \
-e LIBPROCESS_PORT=9400    \
-e LIBPROCESS_ADVERTISE_PORT=9400       \
-e LIBPROCESS_ADVERTISE_IP=10.10.0.101  \
-e JENKINS_LOG_FILE_THRESHOLD=WARNING   \
-e JENKINS_LOG_STDOUT_THRESHOLD=WARNING \
-e JENKINS_MESOS_AUTOCONF=enabled       \
-e JENKINS_MESOS_MASTER="zk://10.10.0.11:2181,10.10.0.12:2181,10.10.0.13:2181/mesos"  \
-e JENKINS_MESOS_SLAVE_1_LABEL=mesos-docker                                           \
-e JENKINS_MESOS_SLAVE_1_DOCK_IMG=jenkins-build-base                                  \
-e JENKINS_MESOS_SLAVE_1_VOL_1=/usr/bin/docker::/usr/bin/docker::ro                   \
-e JENKINS_MESOS_SLAVE_1_VOL_2=/var/run/docker.sock::/var/run/docker.sock::rw         \
-e JENKINS_MESOS_SLAVE_1_ADD_URIS_1=file:///docker.tar.gz::false::false               \
-e JENKINS_MESOS_SLAVE_2_LABEL=mesos                  \
-e JENKINS_MESOS_SLAVE_2_DOCK_IMG=jenkins-build-base  \
-p 8080:8080  \
-p 8090:8090  \
-p 9400:9400  \
jenkins
Example Marathon App Definition
{
    "id": "/jenkins",
    "instances": 1,
    "cpus": 1,
    "mem": 512,
    "container": {
        "type": "DOCKER",
        "docker": {
            "image": "registry.address/mesos/jenkins",
            "network": "BRIDGE",
            "portMappings": [
                {
                    "containerPort": 31111,
                    "hostPort": 31111,
                    "protocol": "tcp"
                },
                {
                    "containerPort": 31112,
                    "hostPort": 31112,
                    "protocol": "tcp"
                },
                {
                    "containerPort": 31113,
                    "hostPort": 31113,
                    "protocol": "tcp"
                }
            ]
        }
    },
    "env": {
        "ENVIRONMENT": "production",
        "ENVIRONMENT_INIT": "/opt/scripts/marathon_env_init.sh",
        "JAVA_OPTS": "-Xmx1024m",
        "JENKINS_LOG_FILE_THRESHOLD": "WARNING",
        "JENKINS_LOG_STDOUT_THRESHOLD": "WARNING",
        "JENKINS_MESOS_AUTOCONF": "enabled",
        "JENKINS_MESOS_MASTER": "zk://10.10.0.11:2181,10.10.0.12:2181,10.10.0.13:2181/mesos",
        "JENKINS_MESOS_SLAVE_1_LABEL": "mesos-docker",
        "JENKINS_MESOS_SLAVE_1_DOCK_IMG": "jenkins-build-base",
        "JENKINS_MESOS_SLAVE_1_VOL_1": "/usr/bin/docker::/usr/bin/docker::ro",
        "JENKINS_MESOS_SLAVE_1_VOL_2": "/var/run/docker.sock::/var/run/docker.sock::rw",
        "JENKINS_MESOS_SLAVE_1_ADD_URIS_1": "file:///docker.tar.gz::false::false",
        "JENKINS_MESOS_SLAVE_2_LABEL": "mesos",
        "JENKINS_MESOS_SLAVE_2_DOCK_IMG": "jenkins-build-base"
    },
    "healthChecks": [
        {
            "protocol": "HTTP",
            "portIndex": 0,
            "path": "/",
            "gracePeriodSeconds": 30,
            "intervalSeconds": 20,
            "maxConsecutiveFailures": 3
        }
    ],
    "uris": [
        "file:///docker.tar.gz"
    ]
}
Example ENVIRONMENT_INIT script
#!/bin/bash

##### Sample environment init script #####
# PORT0 = Jenkins HTTP Web Port
# PORT1 = Jenkins JNLP Port
# PORT2 = libprocess bind port
##########################################


#seed $JENKINS_HOME
cp -Ru /usr/share/jenkins/ref/* "$JENKINS_HOME"

if [[ ! -d "$JENKINS_HOME/plugins" ]]; then
  /opt/scripts/fetch-jenkins-plugins.sh "$PLUGIN_DEFS"
fi

chown -R jenkins:jenkins "$JENKINS_HOME"
chown -R jenkins:jenkins /usr/share/jenkins
chown -R jenkins:jenkins /var/log/jenkins


local_ip="$(ip addr show eth0 | grep -m 1 -P -o '(?<=inet )[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}')"

export JENKINS_HTTP_PORT="$PORT0"
export JENKINS_JNLP_PORT="$PORT1"
export LIBPROCESS_IP="$local_ip"
export LIBPROCESS_PORT="$PORT2"
export LIBPROCESS_ADVERTISE_IP="$HOST"
export LIBPROCESS_ADVERTISE_PORT="$PORT2"


echo "[$(date)][env-init][JENKINS_HTTP_PORT] $PORT0"
echo "[$(date)][env-init][JENKINS_JNLP_PORT] $PORT1"
echo "[$(date)][env-init][LIBPROCESS_IP] $local_ip"
echo "[$(date)][env-init][LIBPROCESS_PORT] $PORT2"
echo "[$(date)][env-init][LIBPROCESS_ADVERTISE_IP] $HOST"
echo "[$(date)][env-init][LIBPROCESS_ADVERTISE_PORT] $PORT2"


Modification and Anatomy of the Project

File Structure The directory skel in the project root maps to the root of the file system once the container is built. Files and folders placed there will map to their corresponding location within the container.

Init The init script (./init.sh) found at the root of the directory is the entry process for the container. It's role is to simply set specific environment variables and modify any subsequently required configuration files.

Jenkins Jenkins configs are stored in two locations. An 'initial seed' of configs stored in /usr/share/jenkins/ref are then copied to their final location in /var/lib/jenkins.

Supervisord All supervisord configs can be found in /etc/supervisor/conf.d/. Services by default will redirect their stdout to /dev/fd/1 and stderr to /dev/fd/2 allowing for service's console output to be displayed. Most applications can log to both stdout and their respectively specified log file.

In some cases (such as with zookeeper), it is possible to specify different logging levels and formats for each location.

Logstash-Forwarder The Logstash-Forwarder binary and default configuration file can be found in /skel/opt/logstash-forwarder. It is ideal to bake the Logstash Server certificate into the base container at this location. If the certificate is called logstash-forwarder.crt, the default supplied Logstash-Forwarder config should not need to be modified, and the server setting may be passed through the SERVICE_LOGSTASH_FORWARDER_ADDRESS environment variable.

In practice, the supplied Logstash-Forwarder config should be used as an example to produce one tailored to each deployment.



Important Environment Variables

Below is the minimum list of variables to be aware of when deploying the Jenkins container.

Defaults

VariableDefault
ENVIRONMENT_INIT
APP_NAMEjenkins
ENVIRONMENTlocal
PARENT_HOSTunknown
LIBPROCESS_IP0.0.0.0
LIBPROCESS_PORT9000
LIBPROCESS_ADVERTISE_IP
LIBPROCESS_ADVERTISE_PORT
JAVA_OPTS
JENKINS_HTTP_LISTEN_ADDRESS0.0.0.0
JENKINS_HTTP_PORT8080
JENKINS_JNLP_PORT8090
JENKINS_LOG_FILE_PATTERN/var/log/jenkins/jenkins.log
JENKINS_LOG_FILE_THRESHOLD
JENKINS_LOG_STDOUT_THRESHOLD
SERVICE_CONSUL_TEMPLATEdisabled
SERVICE_LOGROTATEdisabled
SERVICE_LOGSTASH_FORWARDER
SERVICE_LOGSTASH_FORWARDER_CONF/opt/logstash-forwarer/jenkins.conf
SERVICE_REDPILL
SERVICE_REDPILL_MONITORjenkins
SERVICE_RSYSLOGdisabled
Description
  • ENVIRONMENT_INIT - If set, and the file path is valid. This will be sourced and executed before ANYTHING else. Useful if supplying an environment file or need to query a service such as consul to populate other variables.

  • APP_NAME - A brief description of the container. If Logstash-Forwarder is enabled, this will populate the app_name field in the Logstash-Forwarder configuration file.

  • ENVIRONMENT - Sets defaults for several other variables based on the current running environment. Please see the environment section for further information. If logstash-forwarder is enabled, this value will populate the environment field in the logstash-forwarder configuration file.

  • PARENT_HOST - The name of the parent host. If HOST is found as an environment variable, and PARENT_HOST is not set. Init will automatically set PARENT_HOST equal to host (this would be the case by default in marathon). If Logstash-Forwarder is enabled, this will populate the parent_host field in the Logstash-Forwarder configuration file.

  • LIBPROCESS_IP - The ip in which libprocess will bind to.

  • LIBPROCESS_PORT - The port used for libprocess communication.

  • LIBPROCESS_ADVERTISE_IP - If set, this will be the 'advertised' or 'externalized' ip used for libprocess communication. Relevant when running an application that uses libprocess within a container, and should be set to the host IP in which you wish to use for Mesos communication.

  • LIBPROCESS_ADVERTISE_PORT - If set, this will be the 'advertised' or 'externalized' port used for libprocess communication. Relevant when running an application that uses libprocess within a container, and should be set to the host port you wish to use for Mesos communication.

  • JAVA_OPTS - The Java environment variables that will be passed to Jenkins at runtime. Generally used for adjusting memory allocation (-Xms and -Xmx).

  • SERVICE_CONSUL_TEMPLATE - * SERVICE_CONSUL_TEMPLATE- Enables or disables the consul-template service. If enabled, it will also enableSERVICE_LOGROTATEandSERVICE_RSYSLOGto handle logging. (**Options:**enabledordisabled`)

  • SERVICE_LOGROTATE - Enables or disabled the Logrotate service. This is managed by SERVICE_CONSUL_TEMPLATE, but can be enabled/disabled manually. (Options:enabled or disabled)

  • SERVICE_LOGSTASH_FORWARDER - Enables or disables the Logstash-Forwarder service. Set automatically depending on the ENVIRONMENT. See the Environment section below. (Options:enabled or disabled)

  • SERVICE_LOGSTASH_FORWARDER_CONF - The path to the logstash-forwarder configuration.

  • SERVICE_REDPILL - Enables or disables the Redpill service. Set automatically depending on the ENVIRONMENT. See the Environment section below. (Options:enabled or disabled)

  • SERVICE_REDPILL_MONITOR - The name of the supervisord service(s) that the Redpill service check script should monitor.

  • SERVICE_RSYSLOG - Enables of disables the rsyslog service. This is managed by SERVICE_CONSUL_TEMPLATE, but can be enabled/disabled manually. (Options:enabled or disabled)


Environment

  • local (default)
VariableDefault
JAVA_OPTS-Xmx256m
JENKINS_LOG_FILE_THRESHOLDWARNING
JENKINS_LOG_STDOUT_THRESHOLDWARNING
SERVICE_LOGSTASH_FORWARDERdisabled
SERVICE_REDPILLenabled
  • prod|production|dev|development
VariableDefault
JENKINS_LOG_FILE_THRESHOLDWARNING
JENKINS_LOG_STDOUT_THRESHOLDWARNING
SERVICE_LOGSTASH_FORWARDERenabled
SERVICE_REDPILLenabled
  • debug
VariableDefault
JENKINS_LOG_FILE_THRESHOLDFINEST
JENKINS_LOG_STDOUT_THRESHOLDFINEST
SERVICE_LOGSTASH_FORWARDERdisabled
SERVICE_REDPILLdisabled
CONSUL_TEMPLATE_LOG_LEVELdebug *

* Only set if SERVICE_CONSUL_TEMPLATE is set to enabled.



Service Configuration

Jenkins

Jenkins Command Line Parameters

Any of the Jenkins commandline argument may be passed as an environment variable. The init script will pick up on anything prefixed with JENKINS_ and assume it is to be interpreted as a jenkins command line parameter. In addition to starting with the prefix, an _ should be used between any instance where there would be any word change or use of a capital letter. e.g. --httpListenAddress would become JENKINS_HTTP_LISTEN_ADDRESS. For an accurate list of available options; execute the following: docker run --rm jenkins java -jar /usr/share/jenkins/jenkins.war --help.

Note: There are a few exclusions to the above rule, these are things that the script itself interprets or expects to have an groovy init script handle. These options include: JENKINS_ARGS, JENKINS_HOME, JENKINS_JNLP, JENKINS_LOG_FILE_*, JENKINS_LOG_STDOUT_*, and JENKINS_MESOS.

In addition to the above, there are two additional JAVA_OPTS that will always be passed during startup:

  • -Djava.awt.headless=true
  • -Djava.util.logging.config.file=$JENKINS_HOME/logging.properties

These enable headless mode, and tells java to use the logging settings found in $JENKINS_HOME/logging.properties.

Jenkins Environment Variables

Defaults
VariableDefault
JAVA_OPTS
JENKINS_ARGS
JENKINS_HTTP_LISTEN_ADDRESS0.0.0.0
JENKINS_HTTP_PORT8080
JENKINS_JNLP_PORT8090
JENKINS_LOG_FILE_PATTERN/var/log/jenkins/jenkins.log
JENKINS_LOG_FILE_THRESHOLD
JENKINS_LOG_STDOUT_THRESHOLD
SERVICE_JENKINS_CMD
Description
  • JAVA_OPTS - The Java environment variables that will be passed to Jenkins at runtime. Generally used for adjusting memory allocation (-Xms and -Xmx).

  • JENKINS_ARGS - Additional commandline options that will be passed to jenkins at startup (alternative to supplying them as environment variables).

  • JENKINS_HTTP_PORT - The port Jenkins will listen in on for incoming http connections.

  • JENKINS_JNLP_PORT - The port that Jenkins will listen in on for JNLP connections.

  • JENKINS_LOG_FILE_PATTERN - The full path to the log file name to be used. Note: Can use patterns in file path. For more information see the java docs: java.util.logging.FileHandler.

  • JENKINS_LOG_FILE_THRESHOLD - The log level to be used with the file logger. Jenkins uses java.util.logging for it's logging system. The available log levels include: ALL, CONFIG, FINE, FINER, FINEST, INFO, OFF, SEVERE, WARNING.

  • JENKINS_LOG_STDOUT_THRESHOLD - The log level to be used with stdout/stderr. Jenkins uses java.util.logging for it's logging system. The available log levels include: ALL, CONFIG, FINE, FINER, FINEST, INFO, OFF, SEVERE, WARNING.

  • SERVICE_JENKINS_CMD - The command that is passed to supervisor. If overriding, must be an escaped python string expression. Please see the Supervisord Command Documentation for further information.


Jenkins Mesos Autoconfiguration Options

WARNING:

If you take the container as is, and only supply the Mesos settings; then any jobs added will fail to run. It will NOT schedule and a Null Pointer Exception Error will be thrown. The solution to this is simple. Start the Jenkins container then go to Manage Jenkins -> Configure System, change any setting, and save the configuration. The act of saving the configuration will fix the issue, and it should be ready to run jobs.

Note: This will not occur if Jenkins has already been configured (e.g. loading configs from SCM). It is simply a part of a vanilla bootstrap process.

Variables with '###' in their name

Multiple Mesos slave images may be defined. Settings for them are grouped together based on the first ### in the variable name. e.g. JENKINS_MESOS_SLAVE_1_LABEL=mesos and JENKINS_MESOS_SLAVE_1_DOCK_IMG=jenkins-build-base will apply to the same configuration. In instances where there is a ### at the end of the variable name, multiple instances of that variable may be supplied. e.g. JENKINS_MESOS_SLAVE_2_VOL_1=/usr/bin/docker::/usr/bin/docker:ro and JENKINS_MESOS_SLAVE_2_VOL_2=/var/run/docker.sock::/var/run/docker.sock::rw will be two volumes mounted to the same container.

Note: Add/Modify applies to variable availability either when adding a new Mesos Cloud or Modifying one in place.

Defaults

VariableAdd/ModifyDefault
JENKINS_MESOS_AUTOCONFBothenabled
JENKINS_MESOS_MASTERBoth
JENKINS_MESOS_DESCRIPTIONBoth
JENKINS_MESOS_FRAMEWORK_NAMEBothJenkins Scheduler
JENKINS_MESOS_SLAVES_USERBothjenkins
JENKINS_MESOS_PRINCIPALBothjenkins
JENKINS_MESOS_SECRETBoth
JENKINS_MESOS_CHECKPOINTAddfalse
JENKINS_MESOS_ON_DEMANDBothtrue
JENKINS_MESOS_URLBoth
JENKINS_MESOS_SLAVE_###_LABELAddmesos-${###}
`JENKINS_MESO

Docker Pull Command

docker pull wurstmeister/mesos-jenkins