Storm for PromoJam Analytics
Operational Quickstart with Docker
The included Dockerfile creates a self-contained artifact with all ability to
control the Storm cluster. It is a recommended way to manage Analytics
topologies. It is not intended for development, however.
You will need to install Docker to get
Running the Docker Container
PromoJam doesn't have a private docker repository, so download the image .tar
from Github Releases.
Load the image archive.
docker load -i <archive.tar>
- Run the image as a container:
docker run --rm -it promojam/analytics:0.10.12
- Now you can manage the topology using regular commands. The topology uberjars are stored in the
When not using Docker:
Follow these steps on how to setup Storm on your local machine and prepare for deployment.
Overview: Build and Deploy
- Building a release
- Deploy the Storm topology
- Verify the topology
1) Build a release
Compile the topology using the build script:
The build script will make a build of the project and print the command that
to upload the project.
Under the hood the script is taking these steps:
1a) Attach to a Storm cluster
You must change your
~/.storm/storm.yaml file anytime you switch environments.
storm CLI is used to start and stop topologies and is described here. The
storm CLI connects to a Storm Nimbus - the master server that controls Storm for any given environment. Your
~/.storm/storm.yaml file controls which Nimbus server, your local machine will communicate with.
nimbus.host: nimbus.cobrajam.com storm.local.dir: '/mnt/storm' storm.zookeeper.servers: - zookeeper.int.cobrajam.com supervisor.slots.ports: - 6700 - 6701 - 6702 - 6703
nimbus.host: nimbus.promojam.com storm.local.dir: '/mnt/storm' storm.zookeeper.servers: - zookeeper.int.promojam.com supervisor.slots.ports: - 6700 - 6701 - 6702 - 6703
1b) Compile the Storm Topology
Storm topologies written in Clojure are compiled with Leiningen. You must compile an uberjar that contains all the toplogy files.
/storm directory, run:
lein with-profile stage uberjar
This will output an uberjar file in the target directory:
2) Deploy the Storm topology
- Be sure that you have your local
storm.yamlconfigured for the environment to which you intend to deploy.
- Be sure that you're working with a topology compiled for the environment to which you intend to deploy.
2a) List current topologies
2b) Kill currently running topologies
storm kill <topology-name>
2c) Submit new topology
Storm jar command format:
storm jar <uberjar file> <namespace for topology> <name> <environment> <num workers>
storm jar promojam.storm-0.1.0-standalone.jar promojam.storm.analytics_topology analytics prod 4
PromoJam's Topology Parameters
-main function in
src/promojam/analytics_topology.clj expects 3 parameters for remote deployments. These are passed in via the storm cli jar command.
- name: the name of the topology
- environment: the environment deployed to (stage or prod , defaults to stage)
- num-workers: the number of workers for the topology (optional - defaults to 1)
2d) Restarting the topology
There isn't a command to restart, instead you have to kill the currently running topology and submit it again.
storm kill <name> storm jar <uberjar file> <namespace for topology> <name> <environment> <num workers>
3) Verify the topology
Submitting remote topologies can take a few minutes to complete. You can verify that the topology is running with
storm list or the web UI.