Public | Automated Build

Last pushed: a year ago
Short Description
analytics
Full Description

Storm for PromoJam Analytics

Operational Quickstart with Docker

The included Dockerfile creates a self-contained artifact with all ability to
control the Storm cluster. It is a recommended way to manage Analytics
topologies. It is not intended for development, however.

You will need to install Docker to get
started.

Running the Docker Container

  1. PromoJam doesn't have a private docker repository, so download the image .tar
    from Github Releases.

  2. Load the image archive.

docker load -i <archive.tar>
  1. Run the image as a container:
docker run --rm -it promojam/analytics:0.10.12
  1. Now you can manage the topology using regular commands. The topology uberjars are stored in the ./bin directory.

System Requirements

When not using Docker:

Follow these steps on how to setup Storm on your local machine and prepare for deployment.

Overview: Build and Deploy

  1. Building a release
  2. Deploy the Storm topology
  3. Verify the topology

Configuration

1) Build a release

Compile the topology using the build script:

./build [stage|prod]

The build script will make a build of the project and print the command that
to upload the project.

Under the hood the script is taking these steps:

1a) Attach to a Storm cluster

You must change your ~/.storm/storm.yaml file anytime you switch environments.

The storm CLI is used to start and stop topologies and is described here. The storm CLI connects to a Storm Nimbus - the master server that controls Storm for any given environment. Your ~/.storm/storm.yaml file controls which Nimbus server, your local machine will communicate with.

Example staging:

nimbus.host: nimbus.cobrajam.com
storm.local.dir: '/mnt/storm'
storm.zookeeper.servers:
    - zookeeper.int.cobrajam.com
supervisor.slots.ports:
    - 6700
    - 6701
    - 6702
    - 6703

Example production:

nimbus.host: nimbus.promojam.com
storm.local.dir: '/mnt/storm'
storm.zookeeper.servers:
    - zookeeper.int.promojam.com
supervisor.slots.ports:
    - 6700
    - 6701
    - 6702
    - 6703

1b) Compile the Storm Topology

Storm topologies written in Clojure are compiled with Leiningen. You must compile an uberjar that contains all the toplogy files.

From the /storm directory, run:

lein with-profile stage uberjar

This will output an uberjar file in the target directory:

 <project-name>-<version>-standalone.jar

2) Deploy the Storm topology

Important:

  • Be sure that you have your local storm.yaml configured for the environment to which you intend to deploy.
  • Be sure that you're working with a topology compiled for the environment to which you intend to deploy.

2a) List current topologies

storm list

2b) Kill currently running topologies

storm kill <topology-name>

2c) Submit new topology

Storm jar command format:

storm jar <uberjar file> <namespace for topology> <name> <environment> <num workers>

Example:

storm jar promojam.storm-0.1.0-standalone.jar promojam.storm.analytics_topology analytics prod 4
PromoJam's Topology Parameters

The -main function in src/promojam/analytics_topology.clj expects 3 parameters for remote deployments. These are passed in via the storm cli jar command.

  • name: the name of the topology
  • environment: the environment deployed to (stage or prod , defaults to stage)
  • num-workers: the number of workers for the topology (optional - defaults to 1)

2d) Restarting the topology

There isn't a command to restart, instead you have to kill the currently running topology and submit it again.

storm kill <name>
storm jar <uberjar file> <namespace for topology> <name> <environment> <num workers>

3) Verify the topology

Submitting remote topologies can take a few minutes to complete. You can verify that the topology is running with storm list or the web UI.

Environment Nimbus Host
Staging http://storm-nimbus.cobrajam.com:8080
Production http://storm-nimbus.promojam.com:8080
Docker Pull Command
Owner
promojam
Source Repository

Comments (0)