boilinginsights/wsbackend repository overview

⁠Boiling Insights

Boiling Insights⁠ is a local and S3 first DataWareHouse application (DuckDB) with ETL/ELT data pipeline (DLT) capabilities and charting (Apache ECharts). You can download and run it on your laptop as a standalone application. With Boiling Insights GUI == IaC == configurations on disk and git. No tedious deployment projects needed to make data pipelines work. All configurations are stored on disk as Projects and version controlled with git (along with GitHub).

⁠Boiling Insights Backend

Running the Backend separately turns Boiling Insights from single user local app into multi-user distributed application. Or multi-user centralised server, depending on how you want to use it or see it.

In addition to running Boiling Insights locally, you can run as many remote backends as you like, e.g. on the cloud, on-premises, or on other laptops/machines. You can use these docker images to run Backends. The Backend listens by default on port 8081 (WEBSOCKET_PORT) for incoming WebSocket connections. Then on the GUI, switch to Admin role and create connections to the backends.

Start the Backend with docker. You can pass the AWS credentials via the environment variables.

`aws configure export-credentials --format env` && \
     docker run -p 8081:8081 \
                -e AWS_ACCESS_KEY_ID=${AWS_ACCESS_KEY_ID} \
                -e AWS_SECRET_ACCESS_KEY=${AWS_SECRET_ACCESS_KEY} \
                -e WEBSOCKET_PORT=8081 \
                -e BD_USERNAME=${BD_USERNAME} \
                -e BD_PASSWORD=${BD_PASSWORD} \
             boilinginsights/wsbackend:latest

Or with docker compose, e.g. on an EC2 instance.

services:
  wsbackend:
    image: boilinginsights/wsbackend:latest
    ports:
      - "8081:8081"
    restart: unless-stopped
    environment:
      WEBSOCKET_PORT: 8081 # default
      BD_USERNAME: ${BD_USERNAME} # optional
      BD_PASSWORD: ${BD_PASSWORD} # optional

⁠EC2 Metadata Service (IMDSv2) access from inside the container

NOTE: To allow the container to access the EC2 metadata, you need to configure the EC2 instance metadata to be accessible from 2 hops away:

# NOTE: The instance IAM Role must have ec2:ModifyInstanceMetadataOptions capability over the instance

TOKEN=`curl -s -X PUT "http://169.254.169.254/latest/api/token" -H "X-aws-ec2-metadata-token-ttl-seconds: 21600"`
INSTANCE_ID=`curl -s -H "X-aws-ec2-metadata-token: $TOKEN" http://169.254.169.254/latest/meta-data/instance-id`
aws ec2 modify-instance-metadata-options --instance-id $INSTANCE_ID --http-put-response-hop-limit 2 --http-endpoint enabled

NOTE: You should also set AWS_REGION.

EC2_AVAIL_ZONE=`curl -s -H "X-aws-ec2-metadata-token: $TOKEN" http://169.254.169.254/latest/dynamic/instance-identity/document | jq -r '.availabilityZone'`
AWS_REGION="`echo \"$EC2_AVAIL_ZONE\" | sed 's/"//g' | sed 's/[a-z]$//'`"

⁠Running locally with mocked EC2 Metadata service

You can run it locally without passing AWS creds in the environment variables if you run a mock EC2 Metadata service like this one: https://github.com/slimm609/mock-instance-profile⁠

For example if you want to run multiple Boiling Insights instances (multiple windows) on your laptop against the same local backend.

On OSX, you can run another instance using open with -n switch.

open -n /Applications/BoilingInsights.app

This is essentially a multi-user setup against the same Backend. Multiple users can also connect to remote Backend, there is no difference.