Sign inSign up

niradler/boxy-operator

By niradler

Updated 4 months ago

Kubernetes-native sandbox runtime controller operator for bin-packing and StatefulSet scaling

Image
0

617

niradler/boxy-operator repository overview

boxy

Kubernetes-native sandbox runtime. Run isolated shell commands inside ephemeral Linux environments via a clean HTTP API or MCP — no VMs, no hypervisors, no hardware dependencies.

Go License: MIT Helm chart

Each sandbox is an nsjail process jail: isolated filesystem, network namespace, and resource limits backed by a shared read-only Ubuntu 24.04 rootfs. No kernel modules, no container runtimes, no /dev/kvm — just a standard Linux node.

Architecture

ComponentLanguageRole
boxy-routerGoStateless HTTP frontend — auth, API, MCP server, sandbox CR management
boxy-operatorGoKubernetes controller — bin-packing, StatefulSet auto-scaling, TTL expiry, ControllerPool status
boxy-controllerGoPer-node nsjail daemon — runs actual sandboxes, exposes mTLS HTTP API
Client --(Bearer)--> [boxy-router  Deployment × N]
                          |  Sandbox CR create/read
                          |  mTLS → ctrl-pod-dns:port/v1/...
                          v
                   [boxy-operator  Deployment]
                          |  Scales StatefulSet, assigns pods
                          v
                   [boxy-controller  StatefulSet pod]
                          |  nsjail --chroot /rootfs/ubuntu-24.04
                          v
                   [nsjail sandbox 1…N]
                      /workspace  (per-sandbox, persistent)
                      /tmp        (per-exec tmpfs, ephemeral)
                      /           (shared read-only Ubuntu 24.04)

For a deep dive into request flows, scaling algorithms, storage, failure modes, and security trade-offs, see docs/architecture.md.

Sandbox isolation

Every sandbox gets:

  • Read-only base OS — Ubuntu 24.04 rootfs mounted read-only. Sandboxes cannot modify system files or contaminate each other.
  • Private /workspace — writable directory bind-mounted per sandbox. Persists across multiple execs within the same sandbox lifetime.
  • Ephemeral /tmp — fresh tmpfs per exec, discarded when the command exits.
  • Network namespace isolation — sandboxes have no external network access by default. network.allowInternetAccess: true opts out.
  • cgroup memory capvm.memoryMb enforced via nsjail --cgroup_mem_max.
  • POSIX rlimitsas, core, cpu, fsize, nofile, nproc, stack configurable per sandbox.

Quick start

Prerequisites
  • Kubernetes cluster (kind works fine)
  • kubectl and helm ≥ 3.x
  • Docker for building images
  • Go ≥ 1.26 for local development

Images are published to Docker Hub and the Helm chart to GHCR OCI on every release.

helm upgrade --install boxy oci://ghcr.io/niradler/charts/boxy \
  --version 0.0.2 \
  --namespace boxy --create-namespace \
  --set router.auth.staticToken="$(openssl rand -hex 16)"
Install with Helm (from source)
helm upgrade --install boxy ./deploy/helm/boxy \
  --namespace boxy --create-namespace \
  --set router.auth.staticToken="$(openssl rand -hex 16)"
First API calls

The router accepts any valid Kubernetes ServiceAccount token. Get one for your client SA:

export TOKEN=$(kubectl create token <your-sa> -n <namespace> --duration=3600s)
export BASE=http://<router-service>:8080

# Create a sandbox
curl -sS -H "Authorization: Bearer $TOKEN" -H 'Content-Type: application/json' \
  -d '{"sessionId":"demo","sandboxId":"demo-1","owner":"you","ttlSeconds":3600}' \
  $BASE/v1/sandboxes | jq .

# Execute a command
curl -sS -H "Authorization: Bearer $TOKEN" -H 'Content-Type: application/json' \
  -d '{"sessionId":"demo","sandboxId":"demo-1","command":"sh","args":["-c","echo hello from nsjail"],"timeoutSeconds":30}' \
  $BASE/v1/exec | jq .

# Delete the sandbox
curl -sS -X DELETE -H "Authorization: Bearer $TOKEN" $BASE/v1/sandboxes/demo-1

API reference

GET /healthz

Returns 200 OK with body ok.

POST /v1/sandboxes

Create a sandbox. Returns 201 when the sandbox is Running.

Required fields: sessionId, sandboxId, owner.

FieldTypeDescription
ttlSecondsintSandbox TTL (sliding window, refreshed on each exec). 0 = no expiry.
envmapEnvironment variables explicitly passed to the sandbox. KUBERNETES_* and BOXY_* prefixes are blocked. Max 64 keys. Sandboxes receive only these keys plus PATH and HOME=/workspace — no host environment leaks through.
allowedBinariesstring[]Binaries (e.g. "curl", "python3") bind-mounted read-only from BOXY_NSJAIL_BINARIES_DIR (/usr/local/bin) on the controller into the sandbox. Only listed binaries are accessible; an empty list mounts nothing. Binaries must exist on the controller image — use Dockerfile.controller.dev (or extend it) to pre-bake tools. See docs/architecture.md §4.3 for how to extend the dev image.
vmobjectResource and identity config — see below.
networkobjectNetwork policy — see below.
volumesarrayExtra mounts inside the sandbox.

vm fields:

FieldTypeDescription
userstringRun as this user.
hostnamestringSandbox hostname. Default: sandbox ID.
workdirstringWorking directory inside the sandbox.
memoryMbintMemory cap via cgroup.
rlimitsarray[{ "resource": "nofile", "soft": 1024 }]. Supported: as, core, cpu, fsize, nofile, nproc, stack.
imagestringOverride rootfs path (must be pre-baked on the controller node). Default: /rootfs/ubuntu-24.04.
seccompStringstringKafel/seccomp policy string passed to nsjail.
cloneNewTimeboolIsolate the sandbox in a TIME namespace (Linux ≥ 5.6).

network fields:

FieldTypeDescription
enabledboolDefault true.
allowInternetAccessboolWhen true, disables network namespace isolation (sandbox shares the pod's network).
macvlanobjectClone a MACVLAN interface into the sandbox: { "interface": "eth0", "ip": "...", "netmask": "...", "gateway": "...", "mac": "..." }.
usePastaboolUse pasta userland networking instead of a network namespace.

Response: { "sandboxId", "sessionId", "owner", "runtime", "phase", "ready" }. runtime is always "nsjail".

POST /v1/exec

Execute a command inside an existing sandbox.

FieldTypeDescription
sandboxIdstringRequired.
sessionIdstringRequired.
commandstringRequired. Resolved against rootfs PATH if relative.
argsstring[]Command arguments.
envmapPer-exec env overrides.
timeoutSecondsintRequired. Hard wall-clock timeout enforced by nsjail (SIGKILL).

Response: { "exitCode", "stdout", "stderr", "timedOut" }.

GET /v1/sandboxes/{sandboxId}

Returns sandbox status.

DELETE /v1/sandboxes/{sandboxId}

Deletes the sandbox and removes its CR.

HTTP status codes
CodeMeaning
400Bad request (missing required field, invalid JSON, blocked env prefix, body exceeds 6 MB limit).
401Missing, expired, or invalid bearer token (SA token rejected by TokenReview, or static token mismatch).
404Sandbox not found.
409Sandbox already exists.
429Concurrency limit hit (router or controller semaphore full).
502Controller pod unreachable or returned an error.
503Controller pod not ready.

Output truncation: exec responses that exceed BOXY_MAX_OUTPUT_BYTES (default 6 MB) are truncated at the controller with a \n[output truncated] suffix. The response still returns 200 OK.

MCP server

POST /mcpModel Context Protocol endpoint using Streamable HTTP transport (JSON-RPC 2.0), built with the official Go SDK. Stateless, no session management required.

Required headers:

Authorization: Bearer <BOXY_ROUTER_TOKEN>
Content-Type: application/json
Accept: application/json, text/event-stream

Sandbox routing: set X-Sandbox-Id to target a specific sandbox. Omit to use the default sandbox (when BOXY_DEFAULT_SANDBOX_ENABLED=true).

Available tools:

ToolParameters
bashcommand (string, required), timeoutSeconds (int, default 60)
# MCP initialize
curl -sS -H "Authorization: Bearer $TOKEN" \
  -H 'Content-Type: application/json' -H 'Accept: application/json, text/event-stream' \
  -d '{"jsonrpc":"2.0","id":1,"method":"initialize","params":{"protocolVersion":"2025-03-26","clientInfo":{"name":"cli","version":"1.0"},"capabilities":{}}}' \
  $BASE/mcp | jq .

# MCP bash call
curl -sS -H "Authorization: Bearer $TOKEN" \
  -H 'Content-Type: application/json' -H 'Accept: application/json, text/event-stream' \
  -H 'X-Sandbox-Id: demo-1' \
  -d '{"jsonrpc":"2.0","id":2,"method":"tools/call","params":{"name":"bash","arguments":{"command":"echo hello from nsjail"}}}' \
  $BASE/mcp | jq .

Security model

  • Router auth: Authorization: Bearer <token> required on every route. Two modes:
    • SA token (production): Any valid Kubernetes ServiceAccount token, validated via the TokenReview API. Caller identity is the K8s UserInfo (username + groups). No static secret to manage.
    • Static token (dev/e2e): Set BOXY_ROUTER_TOKEN; requests presenting this token are accepted as dev-token without a TokenReview call. Omit in production. No per-caller RBAC — all authenticated callers have equal access.
  • mTLS: Router and operator dial controllers using a CA with mutual cert verification. No hostname verification; identity is CA membership. Disable with BOXY_MTLS_DISABLED=true for local dev. Generate production certs with bash local/gen-mtls-certs.sh [output-dir] [validity-days] (default: 3 years).
  • NetworkPolicy: Default-deny egress on controller pods (DNS only). Per-sandbox isolation enforced by nsjail network namespaces, not Kubernetes policy.
  • Controller pod capabilities: SYS_ADMIN, SETUID, SETGID, NET_ADMIN, SYS_CHROOT, MKNOD, SETPCAP. All others dropped. allowPrivilegeEscalation: false.

Configuration

Router
VariableDefaultDescription
BOXY_ROUTER_TOKENOptional static dev/e2e bypass token. If set, accepted without TokenReview. Omit in production — use SA tokens instead.
BOXY_SANDBOX_NAMESPACEdefaultNamespace where Sandbox CRs and controller pods live.
BOXY_LISTEN_ADDR:8080
BOXY_CONTROLLER_PORT8080Port the controller pods listen on.
BOXY_MTLS_DISABLEDfalseSet true for local dev.
BOXY_TLS_CA_PATH/tls/ca.crt
BOXY_TLS_CLIENT_CERT_PATH/tls/tls.crt
BOXY_TLS_CLIENT_KEY_PATH/tls/tls.key
BOXY_MAX_BODY_BYTES6291456Max inbound request body (6 MB). Requests over this limit return HTTP 400.
BOXY_MAX_OUTPUT_BYTES6291456Max output per exec forwarded to the caller (6 MB). Oversized output is truncated at the router with a \n[output truncated] suffix.
BOXY_MAX_TIMEOUT_SECONDS3600
BOXY_MAX_CONCURRENCY100Concurrent execs per router replica.
BOXY_MAX_ARGS256
BOXY_MAX_ENV_KEYS64
BOXY_CONTROLLER_TOKENShared secret injected as X-Boxy-Controller-Token on every router→controller request. Must match the value set on the controller. Auto-generated by the Helm chart; override to rotate.
BOXY_DEFAULT_SANDBOX_ENABLEDfalseEnable default sandbox for stateless MCP clients.
BOXY_DEFAULT_SANDBOX_CONFIGRequired when enabled. JSON sandbox create body.
Operator
VariableDefaultDescription
BOXY_NAMESPACERequired. Namespace to watch.
BOXY_CONTROLLER_STATEFULSET_NAMEName of the controller StatefulSet.
BOXY_CONTROLLER_HEADLESS_SERVICEHeadless service for pod DNS.
BOXY_CONTROLLER_PORT8080
BOXY_CONTROLLER_POOL_NAME<statefulset-name>Name of the ControllerPool CR to keep in sync. Defaults to the StatefulSet name.
BOXY_MAX_SANDBOXES_PER_CONTROLLER20Sandboxes per controller pod (bin-packing cap).
BOXY_MAX_CONTROLLER_REPLICAS50StatefulSet scale-out ceiling.
BOXY_MIN_CONTROLLER_REPLICAS1StatefulSet scale-in floor.
BOXY_TERMINATED_RETENTION_SECONDS3600How long to retain Terminated Sandbox CRs before deletion.
BOXY_SCALE_DOWN_COOLDOWN_SECONDS300Minimum seconds between successive scale-down events. Prevents flapping when sandbox count fluctuates near the boundary.
BOXY_MTLS_DISABLEDfalse
Controller
VariableDefaultDescription
BOXY_CONTROLLER_PORT8080
BOXY_CONTROLLER_TOKENToken callers must send in X-Boxy-Controller-Token. Auto-generated and injected by the Helm chart. Enforced on all non-healthz endpoints.
BOXY_MAX_SANDBOXES20Max concurrent sandboxes on this pod.
BOXY_MTLS_DISABLEDfalse
BOXY_TLS_CERT_PATH/tls/tls.crt
BOXY_TLS_KEY_PATH/tls/tls.key
BOXY_TLS_CA_PATH/tls/ca.crt
BOXY_NSJAIL_PATH/usr/sbin/nsjailPath to the nsjail binary.
BOXY_NSJAIL_ROOTFS/rootfs/ubuntu-24.04Default read-only rootfs.
BOXY_NSJAIL_SANDBOX_ROOT/var/lib/boxy/sandboxesHost path for per-sandbox workspace directories.
BOXY_NSJAIL_BINARIES_DIR/usr/local/binHost directory for allowedBinaries.
BOXY_MAX_EXEC_CONCURRENCY50Max parallel exec calls handled concurrently per controller pod. Returns HTTP 429 when full.
BOXY_MAX_OUTPUT_BYTES6291456Max combined stdout/stderr per exec (6 MB). Output over this limit is truncated with a \n[output truncated] suffix.

Development

# Unit tests
make test

# Lint
make lint

# Format
make fmt

Requires Go ≥ 1.26.

E2E tests

Build and load images into a kind cluster, then run the test suites against a live deployment:

# Build and load into kind (production image)
make kind-load

# Build and load dev controller image (includes jq, yq, curl, git, python3, node in rootfs)
make kind-load-dev

# Go e2e suite
BOXY_E2E_BASE_URL=http://127.0.0.1:18080 \
BOXY_E2E_ROUTER_TOKEN=<token> \
make e2e-go

# Shell e2e suite (infra, security, config, api, isolation, operator, network, controllerpool, allowed-binaries)
BASE_URL=http://127.0.0.1:18080 ROUTER_TOKEN=<token> NAMESPACE=boxy \
make e2e-scripts

A kind config and full setup script is included for spinning up a local cluster.

Project layout

cmd/boxy-router/         Router entry point (Go)
cmd/boxy-operator/       Operator entry point (Go)
cmd/boxy-controller/     Controller entry point (Go)
  main.go                Config, mTLS server setup, graceful shutdown
  server.go              HTTP handlers for /v1/sandboxes, /v1/exec, /healthz
local/
  gen-mtls-certs.sh      Generate CA + server + client mTLS certs for production deployment
  setup-mcp-dev.sh       Wire Claude Code (MCP) to a local boxy deployment
internal/
  api/                   Shared types and validation
  nsjail/                nsjail adapter and proto-format config builder
    adapter.go           Sandbox create/exec/delete lifecycle
    nsjail_config.go     NsjailConfig struct + ToTextProto() serializer
  kube/                  Sandbox CR client, controller pod DNS
  operator/              Reconciler, StatefulSet scaling, TTL expiry
  router/                HTTP server, MCP server, mTLS client
deploy/
  helm/boxy/             Helm chart (router, operator, controller, RBAC, mTLS, NetworkPolicy)
  manifests/             Raw RBAC manifests
test/e2e/                Go and shell end-to-end test suites
docs/
  architecture.md        Full system design and architecture reference
Dockerfile.router
Dockerfile.controller      Multi-stage: nsjail build + Ubuntu 24.04 rootfs + Go binary
Dockerfile.controller.dev  Same as above but with jq, yq, curl, git, python3, node baked into rootfs and /usr/local/bin
Dockerfile.operator

License

MIT

Tag summary

Content type

Image

Digest

sha256:c93e16cc4

Size

14.9 MB

Last updated

4 months ago

docker pull niradler/boxy-operator