codedstreams/strlabstudio repository overview

⁠Str:::lab Studio

⁠⭐ Support This Project

If you find Str:::lab Studio useful, please consider:

• Starring this repository on GitHub⁠
• Starring it on Docker Hub⁠
• Sharing it with your colleagues and team

Your support helps drive continued development and maintenance. Thank you! 🙏

⁠What is it?

Str:::lab Studio is a self-hosted web IDE that connects to the Flink SQL Gateway and lets you:

• Write and run Flink SQL in a multi-tab editor with live streaming results
• Upload Java, Python, or Scala JARs and register custom UDFs (ScalarFunction, TableFunction, AggregateFunction) directly from the browser — no SSH, no CLI
• Build SQL Views with computed columns and CASE WHEN expressions using the visual View Builder — no JAR needed
• Build streaming pipelines visually using the ◈ Pipeline Manager — drag operators onto a canvas, connect them, and submit to Flink without writing SQL by hand
• Register and test external system connections using the ⊙ Systems Manager — configure Kafka, PostgreSQL, Elasticsearch, MinIO, Hive Metastore, and more with a ⊙ Test Connectivity check before saving
• Register persistent external catalogs using the ⊕ Catalog Manager — JDBC, Hive, Iceberg (Hive/REST/Glue), and Delta Lake catalogs with live connectivity testing
• Visualise running job DAGs with live operator metrics, backpressure indicators, and throughput charts
• Plot streaming results as bar, line, area, scatter, pie, donut, histogram, or heatmap charts with explicit X/Y axis selection
• Highlight result rows in real time using Colour Describe — a rules engine that applies colour to matching rows as they stream in
• Upload and Submit packaged Flink application written and compiled as jar
• Manage multiple sessions — each with its own isolated workspace (tabs, logs, history, jobs, UDFs)
• Monitor cluster health: backpressure, checkpoints, slot utilisation, JVM heap, records/s per operator
• Use Admin Session for full cluster visibility, cross-session oversight, and pipeline inspection
• Generate PDF reports — standard session reports, or admin-grade Technical / Business reports
• Organise work as named Projects — save, load, run, and export your pipelines
• Compatible with Flink 1.16 through 2.x — including the Flink 2.0 release

⁠Quickstart Cluster Setup

Need a Flink cluster to test with? We provide a ready-to-use Docker Compose setup that includes:

• Flink JobManager and TaskManagers (2 Task managers(10 slots per manager), 1 Jobmanager) (v1.19). customizable
• SQL Gateway with pre-downloaded connectors (Kafka, JDBC, Elasticsearch, etc.)
• Nginx for CORS handling
• All connectors used by Str:::lab Studio pre-installed

git clone https://github.com/coded-streams/FLINK-CLUSTER.git
cd FLINK-CLUSTER
docker compose up -d

⁠What do I need to run this?

The only hard requirement is a running Flink SQL Gateway. The Studio talks to the Gateway REST API exclusively — it does not connect directly to Kafka, ZooKeeper, or any other infrastructure.

⁠Quickstart

⁠Option 1 — Start everything from scratch

git clone https://github.com/coded-streams/strlabstudio
cd strlabstudio
docker compose up -d
open http://localhost:3030

Starts: JobManager, TaskManagers, SQL Gateway, Studio, and an optional CORS proxy. Select Via Studio on the connect screen — done.

⁠Option 2 — I already have a Flink cluster and SQL Gateway

Add Studio to your existing docker-compose.yml. Point it at your gateway and jobmanager. No CORS proxy needed. Via Studio mode routes through the Studio nginx which handles CORS.

services:

  flink-studio:
    image: codedstreams/strlabstudio:latest
    container_name: flink-studio
    restart: unless-stopped
    ports:
      - "3030:80"
    environment:
      FLINK_GATEWAY_HOST: flink-sql-gateway      # your gateway container name
      FLINK_GATEWAY_PORT: "8083"
      JOBMANAGER_HOST:    your-jobmanager         # your jobmanager container name
      JOBMANAGER_PORT:    "8081"
    volumes:
      - udf-jars:/var/www/udf-jars               # required for UDF JAR upload
    networks:
      - your-network                             # same network as your cluster

volumes:
  udf-jars:

networks:
  your-network:
    external: true                               # joins your existing network

For UDF JAR upload also add - udf-jars:/var/www/udf-jars to your flink-sql-gateway volumes. See the UDF JAR Upload⁠ section.

docker compose up -d flink-studio
open http://localhost:3030

⁠Option 3 — I have a Flink cluster but no SQL Gateway

The SQL Gateway is the API layer between the Studio and your Flink cluster. Add it alongside the Studio:

services:

  flink-sql-gateway:
    image: flink:1.19.1-scala_2.12-java11
    container_name: flink-sql-gateway
    depends_on:
      your-jobmanager:
        condition: service_healthy
    command: >
      /bin/bash -c "
        cp /opt/flink/plugins/connectors/*.jar /opt/flink/lib/ 2>/dev/null || true &&
        exec /opt/flink/bin/sql-gateway.sh start-foreground
      "
    environment:
      FLINK_PROPERTIES: |
        jobmanager.rpc.address: your-jobmanager
        sql-gateway.endpoint.rest.address: 0.0.0.0
        sql-gateway.endpoint.rest.port: 8083
    volumes:
      - ./connectors:/opt/flink/plugins/connectors
      - udf-jars:/var/www/udf-jars
    networks:
      - your-network
    healthcheck:
      test: ["CMD-SHELL", "wget -qO- http://localhost:8083/v1/info || exit 1"]
      interval: 15s
      timeout: 10s
      retries: 20
      start_period: 45s

  flink-studio:
    image: codedstreams/strlabstudio:latest
    container_name: flink-studio
    depends_on:
      flink-sql-gateway:
        condition: service_healthy
    ports:
      - "3030:80"
    environment:
      FLINK_GATEWAY_HOST: flink-sql-gateway
      FLINK_GATEWAY_PORT: "8083"
      JOBMANAGER_HOST:    your-jobmanager
      JOBMANAGER_PORT:    "8081"
    volumes:
      - udf-jars:/var/www/udf-jars
    networks:
      - your-network

volumes:
  udf-jars:

Put your Kafka/Flink connector JARs in ./connectors/. They are copied to /opt/flink/lib/ at Gateway startup.

⁠Option 4 — Standalone docker run

docker run -d -p 3030:80 \
  --name flink-studio \
  --network <your-flink-network> \
  -e FLINK_GATEWAY_HOST=flink-sql-gateway \
  -e FLINK_GATEWAY_PORT=8083 \
  -e JOBMANAGER_HOST=your-jobmanager \
  -e JOBMANAGER_PORT=8081 \
  codedstreams/strlabstudio:latest

--network is required for Via Studio mode. Without it, nginx cannot resolve the gateway hostname by container name and returns HTTP 500. Use Direct Gateway mode if you cannot join the network.

⁠Option 5 — Cloud / Remote Flink

Run Studio anywhere and point it at your remote gateway:

docker run -d -p 3030:80 codedstreams/strlabstudio:latest
open http://localhost:3030

On the connect screen select Remote / Cloud, enter your gateway URL (e.g. https://your-gateway.example.com), and provide your Bearer token or Basic auth credentials.

Works with: Confluent Cloud, Ververica Platform, Amazon Managed Service for Apache Flink, or any self-hosted gateway with a public endpoint.

⁠Option 6 — Kubernetes

helm repo add strlabstudio https://coded-streams.github.io/strlabstudio-operator/charts
helm install strlabstudio-operator strlabstudio/strlabstudio-operator \
  --namespace flinksql-system --create-namespace

Apply a StrlabStudio custom resource and the operator manages the Deployment, Service, and PVC:

apiVersion: codedstreams.io/v1alpha1
kind: StrlabStudio
metadata:
  name: my-studio
  namespace: flink
spec:
  image: codedstreams/strlabstudio:latest
  gateway:
    host: flink-sql-gateway
    port: 8083
  jobmanager:
    host: flink-jobmanager
    port: 8081
  service:
    type: ClusterIP
    port: 80

⁠How the Studio connects — architecture

Browser (http://localhost:3030)
         │
         ▼
┌─────────────────────────────────────────────────────────────────┐
│  flink-studio  (codedstreams/strlabstudio:latest)               │
│                                                                 │
│  nginx serves the IDE at port 80                                │
│  docker-entrypoint.sh writes nginx.conf at container startup    │
│                                                                 │
│  /flink-api/*       → proxy → FLINK_GATEWAY_HOST:PORT           │
│                               + CORS headers added by nginx     │
│  /jobmanager-api/*  → proxy → JOBMANAGER_HOST:PORT              │
│  /udf-jars/         → WebDAV PUT → /var/www/udf-jars/           │
└─────────────────────────────────────────────────────────────────┘
         │
         ├─── flink-sql-gateway:8083      SQL sessions, statements, results
         │           └─── your-jobmanager:8081   job submission
         │                     ├── taskmanager-1
         │                     └── taskmanager-2
         │
         └─── [optional] flink-gateway-cors-proxy:8084
                    Only for Direct Gateway mode — not required for Via Studio

⁠Connection Modes

⁠Environment Variables

Substituted into nginx.conf at container startup by docker-entrypoint.sh.

⁠What is the CORS proxy and do I need it?

You do NOT need it if you use Via Studio connection mode (the default). The Studio nginx proxies all requests to the gateway and adds CORS headers itself. This covers the vast majority of use cases.

You need it only if you want a browser tab to call the SQL Gateway directly on port 8084 without going through the Studio (Direct Gateway mode).

On Kubernetes — CORS is not relevant. In-cluster traffic is same-origin. The proxy is never deployed by the operator.

⁠Pipeline Manager

The Pipeline Manager is a visual Flink SQL pipeline builder built into the Studio. Open it with ◈ Pipeline in the topbar.

What it does:

• Drag operators from a categorised palette onto an infinite canvas
• Connect operators with typed edges (FORWARD, HASH, BROADCAST, REBALANCE, RESCALE)
• Configure each operator with a modal — the config generates the exact SQL WITH (...) clause
• Animate data flow with particle physics running along Bézier edges
• Validate the pipeline before submission — detects unconfigured nodes, disconnected transforms, and sourceless sinks
• Generate and submit the full CREATE TABLE + INSERT INTO SQL in one click
• Save, load, export (JSON), and import pipelines across sessions

Operator groups:

Key behaviours:

• Operators that require connector JARs are flagged with a ⚡ badge and trigger a Systems Manager warning when dropped
• Print and Blackhole sinks use CREATE TABLE ... WITH (...) LIKE source (EXCLUDING ALL) — they inherit the exact schema from the upstream source automatically, avoiding column mismatch errors
• Filter and Project operators have no table_name — they inject a WHERE clause and SELECT column list respectively into the generated INSERT INTO
• Multiple sinks generate EXECUTE STATEMENT SET BEGIN ... END with one INSERT INTO per sink
• Auto-layout uses Kahn's topological sort to arrange nodes left-to-right by data flow layer

⁠Systems Manager

The Systems Manager centralises connector JAR management and external system integration. Open it with ⊙ Systems in the topbar.

Tabs:

• 📦 Connector JARs — browse all supported connectors with Maven coordinates, version notes, and SQL examples
• ⬆ Upload JAR — drag-and-drop connector JARs; Studio saves them via WebDAV PUT to /opt/flink/lib/ (requires shared volume)
• ⊙ Integrations — configure named connections to Kafka, PostgreSQL, MySQL, Elasticsearch, MinIO/S3, Schema Registry, and Hive Metastore. Each form has a ⊙ Test Connectivity button that probes the service before saving
• 💾 Saved — browse saved integrations; load them back into the form or insert their generated SQL directly into the editor
• ? Guide — deployment cheatsheet and connectivity test explanation

⊙ Test Connectivity behaviour by system type:

Saved integrations are stored in localStorage and appear as a prefill banner in Pipeline Manager JDBC Sink nodes — one click fills all connection fields.

⁠Catalog Manager

The Catalog Manager registers persistent external catalogs in the active Flink SQL Gateway session. Open it with ⊕ Catalogs in the topbar.

Supported catalog types:

Tabs:

• ⊕ Create Catalog — select type, fill form, run ⊙ Test Connectivity, click ⚡ Create Catalog. Generated CREATE CATALOG SQL is previewed in real time as you type.
• ◎ Active Catalogs — lists all catalogs in the current session; click USE to switch active catalog or Drop to remove
• 🕑 History — last 20 catalogs created; Insert or Copy their SQL
• 📖 Setup Guide — JAR placement instructions and connectivity test explanation

After creation, the catalog appears in the Studio sidebar catalog tree. Tables and columns are live — click a column name to insert it at the cursor in the SQL editor.

⁠UDF JAR Upload

You are not limited to what Flink SQL ships with. Upload your own Java, Python, or Scala functions directly from the browser.

How it works:

1. Open ⨍ UDFs → ⬆ Upload JAR — drag your shaded JAR and click Upload
2. Studio nginx saves the file to /var/www/udf-jars/ via WebDAV PUT
3. Studio runs ADD JAR '/var/www/udf-jars/yourjar.jar' in the active Gateway session
4. Go to ＋ Register UDF → fill in class path, method, parameter types, scope → Execute Registration
5. Call your function in any SELECT query against a live streaming source

Supported UDF types: ScalarFunction · TableFunction · AggregateFunction

Requirement: the udf-jars named volume must be mounted on both flink-studio and flink-sql-gateway at the same path:

flink-studio:
  volumes:
    - udf-jars:/var/www/udf-jars

flink-sql-gateway:
  volumes:
    - udf-jars:/var/www/udf-jars

volumes:
  udf-jars:

⁠UDF deployment paths

⁠View Builder

No JAR? No problem. Open ⨍ UDFs → View Builder to create a TEMPORARY VIEW with computed columns, CASE WHEN expressions, and optional WHERE filters — entirely in SQL. The Studio generates and runs the DDL automatically.

⁠Chart Report

Plot your streaming results without leaving the IDE.

1. Click 📊 Chart Report in the results toolbar
2. Select the query slot to visualise
3. Choose your X axis (categories, timestamps, asset names — any column)
4. Add one or more Y axis fields (scores, counts, aggregates)
5. Pick a chart type: Bar · Line · Area · Scatter · Pie · Donut · Histogram · Heatmap
6. Charts refresh live every 2 seconds as new rows arrive
7. Export as PDF in one click

⁠Colour Describe

A rules engine for your result table. Rows highlight in real time as they stream in — no code required.

1. Click 🎨 Colour Describe in the results toolbar
2. Select the live query slot to apply highlighting to
3. Build rules: pick a field, operator, and value
   • Operators: == != > >= < <= contains starts with ends with regex
4. Choose a highlight colour and style: row background · left border accent · text colour
5. Click ⚡ Apply & Activate — matching rows highlight immediately and continue as rows stream in

Rules are evaluated top-to-bottom. First match per row wins. A colour legend appears below the table. Toggle off at any time to clear all highlighting.

⁠Job Graph & Resource Monitoring

Open the Job Graph tab while a pipeline is running to see:

• Live operator DAG with SOURCE / PROCESS / SINK node classification
• Records in/out per second, backpressure %, and parallelism per node
• Animated edges showing data flow direction and shipping strategy
• Fault highlighting with error message overlay on failed vertices
• Zoom, pan, and drag to navigate large graphs

Double-click any operator node to open a drill-down modal:

• Metrics grid: records/s, bytes/s, backpressure, duration, subtask count
• Subtask table: per-subtask status, host, and record counts
• All Metrics tab: full live metric list fetched from the JobManager API
• Live Events stream: continuous throughput polling with a mini throughput chart

⁠Performance Benchmarking

The Performance tab tracks every query and pipeline submitted in your session:

• Execution time (ms) per query
• Row throughput across the session
• Job comparison chart — plot multiple jobs side by side on any metric
• Per-job checkbox toggles to show/hide individual jobs from the comparison

⁠Admin Session

Connect using the 🛡 Admin button. Default passcode: admin1234 — change it immediately via the 🛡 badge in the topbar after first login.

⁠Report Generation

Generate a formatted PDF report from any result set directly from the results toolbar.

Options:

• Row range filter (e.g. rows 1–500)
• Value filter — search across all columns
• Custom report title
• Automatic field descriptions for known column patterns
• Session metadata: catalog, database, session ID, parallelism, job name
• Colour Describe rules included in the report output

Reports are designed to be shared with both business stakeholders (Business/Management PDF) and technical reviewers (Technical PDF via Admin Session).

⁠Flink 2.0 Compatibility

Str:::lab Studio is compatible with Flink 1.16 through 2.x — including the Flink 2.0.0 release (March 2025).

The Studio connects exclusively via the SQL Gateway REST API, which is preserved and enhanced in Flink 2.0. All session management, ADD JAR, CREATE FUNCTION, and streaming SQL features work identically.

Things to update when moving to Flink 2.0:

⁠Connector JARs

Connector JARs must match your Flink version exactly. Wrong version → `NoClassDef