monasca/storm repository overview

⁠monasca/storm Dockerfile

This image runs an instance of Apache Storm⁠ optimized for Docker and Kubernetes environments.

Sources: monasca/storm⁠ · Dockerfile⁠ · monasca-docker⁠

⁠Tags

Images in this repository are tagged as follows:

• 1.0.3: storm version
• latest: the latest version recommended for use with other Monasca components. Currently, this is 1.0.3
• master: a development / testing build not necessarily intended for production use

⁠Usage

Storm requires several components. First a nimbus instance should be started:

docker run --name storm-nimbus monasca/storm:1.0.3 storm nimbus

Then one or more supervisor instances:

docker run \
  --name storm-supervisor \
  --link storm-nimbus \
  monasca/storm:1.0.3 storm supervisor

The cluster then requires a topology. To use monasca-thresh⁠:

docker run --rm=true --link storm-nimbus monasca/thresh:master

The submission takes around a minute to complete, after which the container is no longer needed. Storm will automatically transfer the topology to new workers that join, restart, etc, and as such the topology only needs to be submitted once unless the cluster is completely reset.

⁠Configuration

Several parameters can be specified using environment variables:

Note that the configuration template also supports a number of less common environment variables. See the storm.yaml.j2⁠ template for a full list.

⁠Memory

Note that all Storm JVM processes are configured to scale their max heap size based on a configured container limit (using Docker's --memory flag or similar). This behavior can be tuned as follows:

Memory scaling is done based off of an automatically determined "effective" memory limit. The ultimate limit is either the true system memory limit or, if set, a limited value set via Docker's --memory flag (whichever value is smallest). This value is then scaled by JVM_MAX_RATIO to ensure some memory is left for other use.

The final max ram size for each component is then the smallest of:

• The component-specific memory request, e.g. WORKER_MAX_MB
• The scaled allocatable memory (as described above)
• If set, JVM_MAX_MB

⁠monasca-thresh

Configuration for monasca-thresh⁠ is included by default and supports a number of additional configuration options:

Note that monasca-thresh configs can be removed by overwriting or removing /templates/thresh-config.yml.j2.