Public | Automated Build

Last pushed: 5 days ago
Short Description
A murder of crows
Full Description

A Murder of Crows

A Murder of Crows, aka amoc, is simple tool for running massively parallel XMPP tests. It can be used to load test ESL's MongooseIM.

It uses escalus, the Erlang XMPP client library.

Developing a scenario

All files related to scenarios should be placed in the scenarios directory.

A scenario specification is just an Erlang module that
exposes the necessary callback functions.

A typical scenario file will look like this:



init() ->
    %% initialize some metrics

start(Id) ->
    %% connect user
    %% fetch user's history
    %% send some messages
    %% wait a little bit
    %% send some messages again

The init/0 function will be called only once per test run, at the very beginning.
It can be used for setting up necessary (global) state: metrics, database
connections, etc.

The start/1 function describes the actual scenario and will be executed for
each user, in the context of that user's process.

Id is the given user's unique integer id.
After this function returns, it will be executed again following some delay (60
seconds by default).

For developing XMPP scenarios, we recommend the
esl/escalus library.

If additional dependencies are required by your scenario, a rebar.config file
can be created inside the scenario dir and deps from that file will be
merged with amoc's deps.

Running a scenario


Everything you need to do is to create the release. To achieve that run:
make rel. Now you are ready to test our scenario locally with one amoc node, to
do this run _build/default/rel/amoc/bin/amoc console.

amoc_local:do(my_scenario, 1, 10).

Start my_scenario spawning 10 amoc users with IDs from range (1,10) inclusive.


Add 10 more user sessions.

amoc_local:remove(10, [{force,true}]).

Remove 10 users.

NOTE: We advise using this mode for the scenario debugging purposes - everything you
log using lager is going to be visible in the erlang shell. In this mode the annotations
are not working, however this may change in the future.

In a distributed environment

Starting a scenario on multiple nodes is slightly more difficult.
You need a number of machines that will be actually running the test
(slaves) and one controller machine (master, which might be one of the test nodes).

Moreover, you need a machine with both Erlang and ansible installed in
order to do the following:

  1. Edit the hosts file and provide all required information - more information in the
    file itself.
  2. Run make prepare to configure the slave nodes, you don't need to do that each
    time you want to run a load test.
  3. Run make deploy in order to build and deploy the release across all nodes.
    Then go to the master's node and start amoc by executing ~/amoc_master/bin/amoc console, that
    will run the amoc shell, wait a couple of seconds till all slave nodes are started.

Now instead of amoc_local use amoc_dist - this will tell amoc to distribute
and start scenarios on all known nodes (except master).

amoc_dist:do(my_scenario, 1, 100, Opts).

Start my_scenario spawning 100 amoc users with IDs from range (1,100) inclusive.
In this case sessions are going to be distributed across all nodes except master.
At this moment users are distributed in round-robin fashion - user1 goes to slave1,
user2 goes to slave2, ...., user10 goes to slave10, user11 goes to slave1 etc
(with the assumption that there are 10 slave nodes configured).
We are planning to make the strategy configurable.

Opts may contain:

  • {nodes, Nodes} - list of nodes on which scenario should be started, by default all available slaves
  • {comment, String} - comment displayed used in the graphite annotations

Add 50 more users to the currently started scenario.

amoc_dist:remove(50, Opts).

Remove 50 sessions.

Where Opts may contain:

  • {force, true} - immediately kill the processes responsible for user connections
  • {force, false} (default) - stop user processes gently - do not start them again

Don't stop scenario on exit

There is one problem with the bin/amoc console command. When you exit the Erlang
shell the scenario is stopped (in fact the erlang nodes are killed).
To prevent that start the amoc node in the background using bin/amoc start.
Now you can run commands by attaching to the amoc's node with bin/amoc attach,
typing a command and then pressing Ctrl+D to exit the shell.
After that the scenario will keep running.


amoc is configured through OTP application environment variables that
are loaded from the configuration file, operating system environment variables
(with prefix AMOC_) and Erlang application environment variables

Internally, amoc is using the following settings:

  • interarrival - a delay in ms, for each node, between creating process
    for two consecutive users. Defaults to 50 ms.
  • repeat_interval - a delay in ms each user process waits
    before starting the same scenario again. Defaults to 1 minute.
  • repeat_num - number of scenario repetitions for each process.
    Defaults to infinity.

You can also define your own entries that you might later use in your

The amoc_config is a module for getting configuration. Every time we ask
for a config value, amoc_config looks into OS environment variables first
(with AMOC_ prefix; e.g. if we want to set interarrival by this mechanism
we should set OS environment variable AMOC_interarrival), and if it doesn't
find it there, it tries to get it from the Erlang application environment variables.
If it cannot find it there either and the default value was not supplied, an error
it thrown. What's more, we can set variables dynamically (by setting OS variable or
application:set_env(amoc, VARIABLE_NAME, VARIABLE_VALUE) in Erlang).

amoc_config provides the following function:

  • get(Name) and get(Name, Default) - return the value for the
    given config entry according to the aforementioned rules.

Locally (without ansible)

If you are not using ansible, just modify the priv/app.config file.
It will later be added to your local release.

Distributed (with ansible)

During the ansible deployment the following file is used as a template:

As long as you need to set only the amoc application variables (including
your scenario-specific settings), you can do
it in the ansible/group_vars/amoc file. You can also do it later in Erlang
console (application:set_env/3) or set OS environment variable (with prefix
AMOC_). Remember that OS environment variable will be taken if present even if you set it by application:set_env/3.

A separate file is required for this
since dictionaries are not supported in the inventory file.


There is available REST API for AMOC where we can:

  • retrieve test status from node
  • check whether AMOC cluster is up or down
  • upload source code of scenario to run
  • list available scenarios
  • run scenario
  • check status of nodes in AMOC cluster

API is described here. You can also get an access to REST API by running AMOC and go to (address:port)/api-docs.



Docker Pull Command
Source Repository