Sign inSign up

crossbario/crossbarfx-workbench

By crossbario

•Updated over 4 years ago

Data scientist and application developers workbench for CrossbarFX and XBR

Image
0

2.8K

crossbario/crossbarfx-workbench repository overview

CrossbarFX

⁠CrossbarFX Workbench for Docker

CrossbarFX Workbench is a Jupyter⁠-based data scientist and application developer environment for CrossbarFX⁠ that allows to analyze and process data directly on CrossbarFX edge nodes.

CrossbarFX includes CFXDB⁠, an embedded database able to persist real-time WAMP events streams (published/subscribed topics) and call queues (registered/called procedures) onto disk, and CrossbarFX Workbench allows to interactively process and analyze this data.

The event persistence available in CrossbarFX allows the WAMP event history API provided by a node to scale beyond RAM limits and survive node restarts. The embedded event stream databases of CrossbarFX nodes also allows to analyze massive sets of historic event data and process historic data right on edge nodes without moving the data first to a data processing cluster.

CFXDB enables high-performance processing of data records with up to 5 million records/second per CPU core (using crossbarfx-workbench:pypy) building on:

  • LMDB⁠: high-performance ACID-compliant in-memory database
  • Flatbuffers⁠: native accelerated data serialization format and
  • zLMDB⁠: object-relational Python database layer on top of LMDB/Flatbuffers
  • NumPy⁠: high-performance array processing and number crunching

⁠Image tags

Images with the following tags are published to DockerHub in this repository⁠:

  • crossbario/crossbarfx-workbench:pypy
  • crossbario/crossbarfx-workbench:cpy

⁠Packages included

CrossbarFX Workbench comes with a complete and pre-configured data science software stack:

The workbench of course also includes the necessary CrossbarFX packages:

  • database access packages for CrossbarFX:
  • asynchronous client package for WAMP:
  • microtransactions and data monetization with XBR:

⁠Running locally

To run the workbench locally (outside JupyterHub, as a single user):

mkdir -p ${HOME}/nodes/workbench1

docker run -p 8888:8888 -it --rm \
    -v ${HOME}/nodes/workbench1:/home/jovyan/notebooks \
    --entrypoint jupyter \
    crossbario/crossbarfx-workbench:cpy \
    notebook --ip=0.0.0.0 --port=8888

The HOME directory inside the container is /home/jovyan. To persist user notebooks, mount a host volume/directy over into this path (or a subpath like we did in above).

To create a persistent database in CrossbarFX for events matching some topics, add the following store configuration item to your CrossbarFX node configuration:


{
    "workers": [
        {
            "type": "router",
            "realms": [
                {
                    "store": {
                        "type": "cfxdb",
                        "path": "../eventdb1",
                        "maxsize": 104857600,
                        "buffer-flush": 250,
                        "event-history": [
                            {
                                "uri": "com.example.on_event1.",
                                "match": "exact"
                            },
                            {
                                "uri": "debug.",
                                "match": "prefix"
                            }
                        ]
                    }
                }
            ]
        }
    ]
}

Then, to analyze the data, start a CrossbarFX Workbench mounting the same host directory that contains the database files (../eventdb1):

docker run -p 8888:8888 -it --rm \
    -v ${HOME}/nodes/workbench1:/home/jovyan/notebooks \
    -v ${HOME}/nodes/edge1/eventdb1:/home/jovyan/eventdb1 \
    --entrypoint jupyter \
    crossbario/crossbarfx-workbench:pypy \
    notebook --ip=0.0.0.0 --port=8888

Then, in a notebook cell run:

DBPATH = '/home/jovyan/eventdb1'

import sys
import zlmdb
from cfxdb.schema import Schema

print('Running on Python {}'.format(sys.version))

db = zlmdb.Database(DBPATH, maxsize=2**30, readonly=True)
schema = Schema.attach(db)

with db.begin() as txn:
    total = schema.publications.count(txn)
        
print('{} publications in database'.format(total))

Tag summary

Content type

Image

Digest

Size

567.6 MB

Last updated

over 6 years ago

docker pull crossbario/crossbarfx-workbench