Public | Automated Build

Last pushed: a year ago
Short Description
Provides resources to describe a Big Data pipeline in mu.semte.ch
Full Description

Mu Pipeline service

Provides and documents pipeline resources to include in a mu.semte.ch project.

Model

Namespaces

Resources

Pipeline

Big Data pipeline describing a sequence of steps.

Class: pwo:Workflow

Properties:

Property Predicate Description
title dcterms:title Name of the pipeline
description dcterms:description Description of the pipeline

Step

Building blocks of a pipeline.

Class: pwd:Step

Properties:

Property Predicate Description
title dcterms:title Name of the step
description dcterms:description Description of the step
code pip:code Human readable identifier of the step
order pip:order Sequence number of the step in a pipeline
status pip:status Current status of the step. Value must be one of: not_started, starting, running, done, ready, failed.

Integrate pipeline service in a mu.semte.ch project

Add the following snippet to your docker-compose.yml to include the pipeline service in your project.

pipeline:
  image: bde2020/mu-pipeline-service
  links:
    - database:database

database must be another service defined in your docker-compose.yml running a triple store (e.g. Virtuoso)

Add rules to the dispatcher.ex to dispatch requests to the pipeline service. E.g.

  match "/pipelines/*path" do
    Proxy.forward conn, path, "http://pipeline/pipelines/"
  end

  match "/steps/*path" do
    Proxy.forward conn, path, "http://pipeline/steps/"
  end

More information how to setup a mu.semte.ch project can be found in mu-project.

Docker Pull Command
Owner
bde2020