psyb0t/hybrids3 repository overview

⁠HybridS3

Lightweight object storage that speaks S3 (boto3/AWS SDK compatible), plain HTTP, and MCP. SQLite for metadata, flat files on disk.

⁠Table of Contents

• Why⁠
• Quick Start⁠
• Running⁠
• Configuration⁠
• Keys and Access⁠
  • Key pair model⁠
  • Master key⁠
  • Bucket visibility⁠
• HTTP API⁠
  • Authentication⁠
  • Endpoints⁠
  • Object keys⁠
  • Response format⁠
• S3 Interface⁠
  • Authentication⁠
  • Operations⁠
• MCP Interface⁠
  • Connecting⁠
  • Authentication⁠
  • Tools⁠
• Agent integrations⁠
• Presigned URLs⁠
• MIME Type Detection⁠
• TTL and Expiry⁠
• Locking⁠
• Logging⁠
• Behind a Reverse Proxy⁠
• Security⁠
• Testing⁠
• Development⁠
• Architecture⁠
• Limitations⁠
• License⁠

⁠Why

Most self-hosted S3-compatible storage is designed for large-scale deployments. Distributed erasure coding, IAM policies, WORM compliance, full web consoles — useful if you're running a cloud, overkill if you just want a place to put files that various services and AI agents can read and write.

AWS Sig V4 breaks behind reverse proxy path prefixes. Most implementations verify signatures using the full original upstream path. Put them behind nginx at /storage/, nginx strips the prefix, the server sees /bucket/key instead of /storage/bucket/key, the signature check fails. HybridS3 has a path_prefix config option — set it to /storage and all routes move under that prefix. No path stripping, no special proxy headers. boto3's signed path matches what the server sees.

Three interfaces, one service. boto3 works out of the box. Plain HTTP with curl works. AI agents connect via MCP and get structured tool definitions. No separate services for different clients.

Buckets are configuration, not state. There is no API to create or delete buckets. They live in the YAML config file. You always know exactly what exists, it's version-controlled, and there are no surprise buckets accumulating garbage.

TTL expiry is built in. Set ttl: 24h on a bucket and objects expire automatically after their last write. No lifecycle policies, no cron jobs, no separate process.

Readable and modifiable. Small enough to understand in an afternoon.

⁠Quick Start

# get the example config
wget -O config.yaml https://raw.githubusercontent.com/psyb0t/docker-hybrids3/master/config.example.yaml

# edit it — set your keys and define your buckets
vi config.yaml

# run
docker run -d --name hybrids3 \
    -p 8080:8080 \
    -v ./config.yaml:/config/config.yaml:ro \
    -v hybrids3-data:/data \
    psyb0t/hybrids3

# verify
curl http://localhost:8080/health

⁠Running

The container expects:

• Config file at /config/config.yaml
• Data directory at /data
• Port 8080

Runs as UID 1000.

docker run:

docker run -d --name hybrids3 \
    -p 8080:8080 \
    -v ./config.yaml:/config/config.yaml:ro \
    -v hybrids3-data:/data \
    psyb0t/hybrids3

docker-compose:

services:
  hybrids3:
    image: psyb0t/hybrids3
    ports:
      - "8080:8080"
    volumes:
      - ./config.yaml:/config/config.yaml:ro
      - hybrids3-data:/data
    restart: unless-stopped

volumes:
  hybrids3-data:

⁠Configuration

# Cross-bucket god key. Works on every bucket for every operation.
# Only credential that can list all buckets. Keep secret.
master_key: "change-me-to-something-secret"

# Non-secret identifier paired with master_key in S3 auth (aws_access_key_id).
# Safe to share — it grants nothing without the master_key.
master_public_key: "master"

# How often the background loop runs to delete expired objects and orphan files.
# Accepts human-readable durations: 30s, 5m, 1h, or raw seconds: 60
cleanup_interval: 1m

# How long a request may wait to acquire a lock on an object before giving up with 503.
# Accepts seconds as a float. Applies per object key, globally across all buckets.
lock_acquire_timeout: 30

# How long a request may hold a lock before it is forcibly released and the request gets 503.
# Protects against slow or stalled uploads blocking writes on the same key indefinitely.
lock_hold_timeout: 300

# Maximum number of requests that may queue for the same object key at once.
# Once the queue is full, new requests are rejected immediately with 503.
lock_max_waiters: 100

# Serve all routes under this path prefix.
# Set this when running behind a reverse proxy at a subpath (e.g. /storage).
# Accepts "/storage" or "/storage/" — both are normalized.
# Leave empty or omit to serve at the root.
# path_prefix: /storage

# TTL formats:   30s  5m  1h  1h30m12s  1d  2d12h  0 (never expire)
# Size formats:  500B  50KB  10MB  1GB  0 (no limit)

buckets:
  uploads:
    # true  — anyone can read (GET/HEAD/LIST) without authentication
    # false — authentication required for all operations
    public: true

    # Private key for this bucket.
    # Used as the Bearer token value and as aws_secret_access_key in S3 auth.
    # Never transmitted — only used to sign or verify request signatures.
    # Keep secret.
    key: "uploads-secret"

    # Public identifier for this bucket in S3 auth (aws_access_key_id).
    # Also appears in presigned URL Credential= fields.
    # Safe to share — it identifies the bucket but grants no access.
    # Defaults to the bucket name if not set.
    public_key: "uploads-id"

    # Objects expire this long after their last write. Overwriting resets the clock.
    # 0 means objects never expire.
    ttl: 24h

    # Maximum upload size. Requests over this limit are rejected with 413.
    # 0 means no limit.
    max_file_size: 50MB

  permanent:
    public: false
    key: "perm-secret"
    public_key: "permanent-id"
    ttl: 0
    max_file_size: 100MB

⁠Keys and Access

⁠Key pair model

Each bucket has two keys defined in config:

The split is what makes presigned URLs work safely. A presigned URL must embed an identifier in the Credential= field so the server knows which key to verify against — that identifier is the public_key. Since it is non-secret, having it in the URL is fine. The private key signs the URL on the server and never appears in it.

⁠Master key

The master_key is a cross-bucket credential that works on every bucket for every operation, without needing individual bucket keys. Two situations call for it:

• You need to operate across multiple buckets from a single client
• You need to list all configured buckets — bucket keys only see their own bucket in list_buckets

The master_public_key is the non-secret identifier that pairs with master_key in S3 auth (used as aws_access_key_id).

Do not embed the master key in client-facing code. Use per-bucket keys for that — they limit access to exactly one bucket.

⁠Bucket visibility

⁠HTTP API

⁠Authentication

HTTP requests authenticate using a Bearer token in the Authorization header. Pass the bucket's private key, or the master key for cross-bucket operations.

Authorization: Bearer <private_key>

• Public bucket reads (GET, HEAD, LIST) — no authentication required.
• All writes (PUT, DELETE) and reads on private buckets — require the bucket key or master key.
• Listing buckets (GET /) — master key lists all buckets; bucket key lists only its own bucket.
• Presign endpoint (POST /presign/...) — requires the bucket key or master key.

# write to a bucket
curl -X PUT http://localhost:8080/uploads/file.txt \
  -H "Authorization: Bearer uploads-secret" \
  -d "hello"

# read from a public bucket — no auth needed
curl http://localhost:8080/uploads/file.txt

# read from a private bucket
curl http://localhost:8080/permanent/doc.pdf \
  -H "Authorization: Bearer perm-secret"

# list all buckets — master key sees all
curl http://localhost:8080/ \
  -H "Authorization: Bearer your-master-key"

# list buckets — bucket key sees only its own bucket
curl http://localhost:8080/ \
  -H "Authorization: Bearer uploads-secret"

⁠Endpoints

PUT /{bucket} exists purely for S3 client compatibility. boto3 sends a create_bucket call before any operation, which maps to this endpoint. HybridS3 treats it as a no-op — no buckets are created or modified.

Listing objects accepts prefix and max-keys query parameters:

curl "http://localhost:8080/uploads?prefix=images/&max-keys=50" \
  -H "Authorization: Bearer uploads-secret"

Upload returns an ETag header (MD5 of the file content). GET and HEAD also return ETag, Last-Modified, and Content-Length.

⁠Object keys

Keys support nested paths using /. Parent directories are created automatically on write and pruned when empty on delete.

curl -X PUT http://localhost:8080/uploads/reports/2024/january.pdf \
  -H "Authorization: Bearer uploads-secret" \
  --data-binary @january.pdf

⁠Response format

Requests that include an AWS Sig V4 Authorization header receive S3-compatible XML responses. All other requests receive JSON. Error responses include an "error" field and a "request_id" field.

Every response includes X-Request-Id for log correlation and X-Content-Type-Options: nosniff.

⁠S3 Interface

⁠Authentication

S3 clients authenticate using AWS Signature V4. The client signs each request using the bucket's public_key as aws_access_key_id and the bucket's private key as aws_secret_access_key. The resulting Authorization header contains the access key ID in plaintext in the Credential= field, and the computed HMAC in Signature=. The private key is never transmitted — it is only used locally to compute the signature.

Authorization: AWS4-HMAC-SHA256 Credential=uploads-id/20240101/us-east-1/s3/aws4_request, SignedHeaders=host;x-amz-date, Signature=<hmac>

HybridS3 reads the access key ID from Credential=, finds the bucket with that public_key, then re-derives the expected signature using that bucket's private key and compares it to the one in the header. The access key ID alone grants nothing — the signature must match.

import boto3
from botocore.config import Config

# per-bucket client — access is limited to the "uploads" bucket
s3 = boto3.client(
    "s3",
    endpoint_url="http://localhost:8080",
    aws_access_key_id="uploads-id",         # public_key from config
    aws_secret_access_key="uploads-secret",  # key from config
    region_name="us-east-1",
    config=Config(signature_version="s3v4"),
)

# master key client — access to all buckets, can list them
s3_admin = boto3.client(
    "s3",
    endpoint_url="http://localhost:8080",
    aws_access_key_id="master",              # master_public_key from config
    aws_secret_access_key="your-master-key", # master_key from config
    region_name="us-east-1",
    config=Config(signature_version="s3v4"),
)

⁠Operations

s3.put_object(Bucket="uploads", Key="file.txt", Body=b"hello")
s3.get_object(Bucket="uploads", Key="file.txt")
s3.head_object(Bucket="uploads", Key="file.txt")
s3.delete_object(Bucket="uploads", Key="file.txt")
s3.list_objects_v2(Bucket="uploads", Prefix="images/")
s3_admin.list_buckets()  # master key: returns all buckets
s3.list_buckets()        # bucket key: returns only the "uploads" bucket

# generate a presigned URL for a private bucket object
url = s3.generate_presigned_url(
    "get_object",
    Params={"Bucket": "permanent", "Key": "doc.pdf"},
    ExpiresIn=3600,
)

# presigned URL someone else can use to upload a single key
put_url = s3.generate_presigned_url(
    "put_object",
    Params={"Bucket": "uploads", "Key": "inbox/report.pdf"},
    ExpiresIn=600,
)

⁠MCP Interface

An MCP⁠ server runs at /mcp/ using the Streamable HTTP transport. AI agents connect via any MCP-compatible client and receive structured tool definitions with typed inputs and outputs.

⁠Connecting

{
  "mcpServers": {
    "hybrids3": {
      "type": "streamable-http",
      "url": "http://localhost:8080/mcp/"
    }
  }
}

⁠Authentication

⁠Endpoint-level auth

The /mcp/ endpoint accepts an optional token to authenticate the connection before any tool is invoked. Use the master key for full access, or a bucket key to limit the connection to that bucket's scope. The token is validated against the master key and all bucket keys.

Two methods are accepted:

Authorization header — for clients that support custom headers (e.g. Claude Code):

Authorization: Bearer <master_key_or_bucket_key>

Query parameter — for clients that cannot set custom headers (e.g. ChatGPT):

http://localhost:8080/mcp/?auth=<master_key_or_bucket_key>

If a token is provided and does not match any known key, the request is rejected with 401. If no token is provided at all, the request passes through and per-tool auth applies.

⁠Per-tool auth

Each tool that operates on a bucket accepts an auth_key parameter — the bucket's private key or the master key. This is checked independently of endpoint-level auth and controls what each individual tool call is allowed to do.

# public bucket read — no auth_key
download_object(bucket="uploads", key="file.txt")

# public bucket write — auth_key required
upload_object(bucket="uploads", key="file.txt", content="hello", auth_key="uploads-secret")

# private bucket — auth_key required for all operations
download_object(bucket="permanent", key="doc.pdf", auth_key="perm-secret")
upload_object(bucket="permanent", key="doc.pdf", content="...", auth_key="perm-secret")

# master key lists all buckets; bucket key lists only that bucket
list_buckets(auth_key="your-master-key")
list_buckets(auth_key="uploads-secret")  # returns only the uploads bucket

⁠Tools

All tools return structured output (structuredContent) for clients that support it, with a plain text fallback.

⁠Agent integrations

The skill⁠ works in any agent that reads .agents/skills/, and installs natively in the clients below.

⁠Claude Code

claude plugin marketplace add psyb0t/agents
claude plugin install hybrids3@psyb0t

Claude Code prompts for the HybridS3 URL and, if the endpoint requires connection-level auth, the bucket/master key — the key is stored in your OS keychain.

⁠Codex

codex plugin marketplace add psyb0t/agents
codex plugin add hybrids3@psyb0t

Installed via the marketplace, the skill invokes as $hybrids3:hybrids3. Codex also picks the skill up automatically, no install required, in any repo containing .agents/skills/ — there it invokes as plain $hybrids3.

⁠OpenClaw

The skill is published to ClawHub on every release:

openclaw skills install @psyb0t/hybrids3

For MCP clients that speak local stdio, the @psyb0t/hybrids3⁠ plugin bridges to the service's /mcp/ endpoint:

openclaw plugins install clawhub:@psyb0t/hybrids3

Then set HYBRIDS3_URL (and HYBRIDS3_KEY if your endpoint requires connection-level auth).

⁠Presigned URLs

Presigned URLs allow anyone with the link to read or write a specific object for a limited time, without sending an Authorization header. The /presign/{bucket}/{key} endpoint supports two methods via the method query parameter:

• method=GET (default) — recipient can download the object.
• method=PUT — recipient can upload (overwrite) the object.

A presigned URL is bound to its HTTP verb. A GET URL cannot be used to PUT and vice versa — the signature includes the method in its canonical request.

Expiry range: 1 second to 604800 seconds (7 days). Default: 3600. Expired or tampered URLs return 403.

⁠Presigned GET

Private bucket — generates an AWS Sig V4 presigned URL. The server signs the URL using the bucket's private key, which never appears in the URL. The URL contains the public_key in the Credential= field and the HMAC in X-Amz-Signature.

# generate via HTTP endpoint
curl -X POST "http://localhost:8080/presign/permanent/doc.pdf?expires=3600" \
  -H "Authorization: Bearer perm-secret"

# response
{"url": "http://localhost:8080/permanent/doc.pdf?X-Amz-Algorithm=AWS4-HMAC-SHA256&...&X-Amz-Signature=...", "method": "GET", "expires": 3600}

Public bucket — returns a plain URL with no signature and no expiry, since GET on public buckets requires no auth anyway.

curl -X POST "http://localhost:8080/presign/uploads/photo.jpg" \
  -H "Authorization: Bearer uploads-secret"

# response
{"url": "http://localhost:8080/uploads/photo.jpg", "method": "GET", "expires": null}

⁠Presigned PUT

Use method=PUT to hand someone a URL that lets them upload a specific key without seeing your bucket key. Public buckets are not a shortcut here — anonymous reads are allowed, anonymous writes never are — so a presigned PUT URL is always signed, even for public buckets.

# generate a presigned upload URL
curl -X POST "http://localhost:8080/presign/uploads/inbox/report.pdf?method=PUT&expires=600" \
  -H "Authorization: Bearer uploads-secret"

# response
{"url": "http://localhost:8080/uploads/inbox/report.pdf?X-Amz-Algorithm=AWS4-HMAC-SHA256&...&X-Amz-Signature=...", "method": "PUT", "expires": 600}

# upload using only the URL — no Authorization header
curl -X PUT "<url>" --data-binary @report.pdf

The bucket's max_file_size is enforced server-side during the upload; oversized bodies are rejected with 413 regardless of how the request was authenticated.

With boto3:

url = s3.generate_presigned_url(
    "put_object",
    Params={"Bucket": "uploads", "Key": "inbox/report.pdf"},
    ExpiresIn=600,
)
# hand `url` to a client; they PUT their bytes to it

Generating a presigned URL requires the bucket's private key or the master key. The resulting URL grants exactly one action (GET or PUT) on exactly one key.

Both the /presign/ HTTP endpoint and the presign_url MCP tool follow this same logic — pass method="PUT" to the MCP tool to get an upload URL.

⁠MIME Type Detection

Content type is detected automatically on every upload. No Content-Type header is required. Detection uses libmagic⁠ to inspect the first 8 KB of the file content, with a filename extension fallback when libmagic returns a generic type. The detected type is stored in metadata and returned in Content-Type on GET and HEAD responses.

To override auto-detection, set Content-Type explicitly on the upload request.

⁠TTL and Expiry

Set ttl on a bucket and objects expire automatically after that duration from the last write. Overwriting an object resets its expiry clock. Setting ttl: 0 means objects never expire.

buckets:
  staging: