senseyeio/cray-runner repository overview

⁠Batch Processing System

The batch processing system provides compute for large analytical jobs.

To interact with the system, install the CLI by calling pip install cray.

⁠Authentication

The CLI uses boto3, see https://boto3.amazonaws.com/v1/documentation/api/latest/guide/configuration.html#configuring-credentials⁠ for configuration instructions.

When submitting a job, the code's dependencies will be fetched. This may include fetching from private repositories. See README-ssh.md for details about how to set this up.

⁠Job

A job defines a set of targets and some python code to run against each target.

A definition of the job should be present within a file called config.json. This supports the following attributes:

• user : Your email address
• service : (optional) The AWS compute service to make use of, see the service section.
• targets : The targets, see the targets section
• exe : The python code to run
• env : Environmental variables to set when running the python code

⁠Target

A target can be anything, essentially it is the 'thing' which you are running your analysis against. Your python code is executed in isolation against each target.

Typical use cases would be to target sensors, assets, organisations or parts of an organisation.

Currently targets can be specified in the following way:

    "targets": {
        "type": "raw",
        "raw": [<TARGETS>]
    }

Where each target should be a dictionary of values, the provided attributes will be provided as environmental variables to the python code. For example:

    "targets": {
        "type": "raw",
        "raw": [
            {
                "ASSET": "4a1850dc-0e09-4c43-9a7e-60fb9471e8b1"
            },
            {
                "ASSET": "6e81f2e7-f7cf-4e59-a499-2f560434f90b"
            }
        ]
    }

Would run the job's python code twice. Gathering the ASSET env var from the code will result in 4a1850dc-0e09-4c43-9a7e-60fb9471e8b1 on the first run and 6e81f2e7-f7cf-4e59-a499-2f560434f90b on the second run.

⁠Code

Your code will be executed with python. It will be called using: python <config.exe>, where config.exe is the exe attribute within the job's config.

Within the code's environment will be all the env vars defined within the config's env attribute, the target attributes and the following variables:

• OUTPUT: any outputs of your job must be put into this directory
• CODE_DIR : the location of your code, useful if you need to read data from static files

⁠Service

By default, cray will make use of AWS Lambda. Lambda is fast, cheap and extremely scaleable. However, it has a 15 minute task limit.

If your individual tasks will take more than 15 minutes, you can make use of AWS Batch. This uses EC2 servers to process longer running tasks. This is more expensive and doesn't scale to the same degree as Lambda. If you would like to use Batch, provide a service value of batch.

⁠Results

Any files outputted by your code will be collected to S3: s3::ds-analysis-jobs/jobs/<JOB>/output where JOB is the job ID provided when you submitted your job.

⁠Submitting

Navigate to the job's directory and call cray submit -t <TICKET> -d <DESC> where TICKET should be the Jira ticket ID and DESC is a description of the run.

⁠Example

See example-job. This runs across two targets. For each, it will create a txt file named after the asset target, its contents will be gathered from the job's defined environment variable PARAM-A.