Public Repository

Last pushed: 3 months ago
Short Description
Short description is empty for this repo.
Full Description

monasca/storm Dockerfile

This image runs an instance of Apache Storm optimized for Docker and
Kubernetes environments.

Sources: monasca/storm · Dockerfile · monasca-docker

Tags

Images in this repository are tagged as follows:

  • 1.0.3: storm version
  • latest: the latest version recommended for use with other Monasca
    components. Currently, this is 1.0.3
  • master: a development / testing build not necessarily intended for
    production use

Usage

Storm requires several components. First a nimbus instance should be started:

docker run --name storm-nimbus monasca/storm:1.0.3 storm nimbus

Then one or more supervisor instances:

docker run \
  --name storm-supervisor \
  --link storm-nimbus \
  monasca/storm:1.0.3 storm supervisor

The cluster then requires a topology. To use monasca-thresh:

docker run --rm=true --link storm-nimbus monasca/thresh:master

The submission takes around a minute to complete, after which the container is
no longer needed. Storm will automatically transfer the topology to new workers
that join, restart, etc, and as such the topology only needs to be submitted
once unless the cluster is completely reset.

Configuration

Several parameters can be specified using environment variables:

Variable Default Description
ZOOKEEPER_SERVERS zookeeper Comma-separated list of ZK hostnames
ZOOKEEPER_PORT 2181 Port to use for hostnames
ZOOKEEPER_WAIT true If true, wait for ZK before startup
SUPERVISOR_SLOTS_PORTS 6701,6702 List of supervisor ports, 1 per process
NIMBUS_SEEDS storm-nimbus Comma-separated list of Nimbus hosts
STORM_LOCAL_HOSTNAME If set, use as storm hostname, if not set, use container IP
LOG_LEVEL warn log4j2 console logging level
WORKER_LOGS_TO_STDOUT false if true, workers log to container stdout

Note that the configuration template also supports a number of less common
environment variables. See the storm.yaml.j2 template for a full list.

Memory

Note that all Storm JVM processes are configured to scale their max heap size
based on a configured container limit (using Docker's --memory flag or
similar). This behavior can be tuned as follows:

Variable Default Description
JVM_MAX_HEAP_RATIO 0.75 The max % of allocatable memory to pass to -Xmx
JVM_MAX_HEAP_MB unset Value to use for allocatable memory, autodetect if unset
HEAP_OVERRIDE_MB unset Ignore scaling, always return -Xmx$HEAP_OVERRIDE_MB
SUPERVISOR_MAX_HEAP_MB 256 Max heap size in MiB for supervisor
WORKER_MAX_HEAP_MB 784 Max head size in MiB for workers
NIMBUS_MAX_HEAP_MB 256 Max heap in MiB for nimbus
UI_MAX_HEAP_MB 768 Max heap in MiB for UI

Memory scaling is done based off of an automatically determined "effective"
memory limit. The ultimate limit is either the true system memory limit or, if
set, a limited value set via Docker's --memory flag (whichever value is
smallest). This value is then scaled by JVM_MAX_HEAP_RATIO to ensure some
memory is left for other use.

The final max heap size for each component is then the smallest of:

  • The component-specific memory request, e.g. WORKER_MAX_HEAP_MB
  • The scaled allocatable memory (as described above)
  • If set, JVM_MAX_HEAP_MB

monasca-thresh

Configuration for monasca-thresh is included by default and supports a
number of additional configuration options:

Variable Default Description
METRIC_SPOUT_THREADS 2 Metric Spout threads
METRIC_SPOUT_TASKS 2 Metric Spout tasks
KAFKA_URI kafka:9092 Kafka host
MYSQL_DB_HOST mysql MySQL host
MYSQL_DB_PORT 3306 MySQL port
MYSQL_DB_DATABASE mon Monasca database name
MYSQL_DB_USERNAME thresh MySQL user
MYSQL_DB_PASSWORD password MySQL password
EVENT_SPOUT_THREADS 2 Event Spout Threads
EVENT_SPOUT_TASKS 2 Event Spout Tasks
EVENT_BOLT_THREADS 2 Event Bolt Threads
EVENT_BOLT_TASKS 2 Event Bolt Tasks
FILTERING_BOLT_THREADS 2 Filtering Bolt Threads
FILTERING_BOLT_TASKS 2 Filtering Bolt Tasks
ALARM_CREATION_BOLT_THREADS 2 Alarm Creation Bolt Threads
ALARM_CREATION_BOLT_TASKS 2 Alarm Creation Bolt Tasks
AGGREGATION_BOLT_THREADS 2 Aggregation Bolt Threads
AGGREGATION_BOLT_TASKS 2 Aggregation Bolt Tasks
THRESHOLDING_BOLT_THREADS 2 Thresholding Bolt Threads
THRESHOLDING_BOLT_TASKS 2 Thresholding Bolt Tasks

Note that monasca-thresh configs can be removed by overwriting or removing
/templates/thresh-config.yml.j2.

Docker Pull Command
Owner
monasca

Comments (0)