crowdb/crowdb-s3 repository overview

⁠CROWDB S3

Run S3-compatible object storage locally with one Docker command. Use boto3 to create buckets, upload objects, and read them back without setting up a separate storage cluster.

CROWDB is an open-source distributed storage platform. This image packages its storage services and S3 endpoint into a single-node development environment.

⁠Supported operations

• Bucket creation, listing, existence checks, and deletion.
• Object upload, overwrite, download, metadata retrieval, and deletion.
• Prefix and delimiter listing with ListObjectsV2 and continuation tokens.
• Range and conditional reads.
• Multipart upload, part replacement, part listing, completion, and abort.
• AWS Signature Version 4 authentication and presigned request verification.
• Content-MD5 validation and S3-style ETags.

The repository includes boto3 end-to-end tests for core operations, multipart uploads, interrupted requests, and recovery after service restart.

⁠Quick start

Requires Docker on Linux amd64, or a Docker environment capable of running Linux amd64 containers.

docker run -d --name crowdb-s3 \
  --mount type=volume,source=crowdb-s3-data,target=/opt/crowdb/data \
  -p 127.0.0.1:9091:9091 \
  crowdb/crowdb-s3:latest

Check startup and wait until the health status is healthy:

docker inspect --format '{{.State.Health.Status}}' crowdb-s3
docker logs crowdb-s3

Display the generated client credentials:

docker exec crowdb-s3 crowdb-monitor credentials show --format env

Set AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY, and AWS_DEFAULT_REGION in your client environment using the displayed values. Keep credentials private. The S3 endpoint is http://127.0.0.1:9091 and uses path-style addressing.

⁠Upload and download with boto3

Install the client in your Python environment:

python -m pip install boto3

Run after setting the AWS environment variables above:

import boto3
from botocore.config import Config

s3 = boto3.client(
    "s3",
    endpoint_url="http://127.0.0.1:9091",
    config=Config(
        s3={"addressing_style": "path"},
        request_checksum_calculation="when_required",
        response_checksum_validation="when_required",
    ),
)

s3.create_bucket(Bucket="example")
s3.put_object(Bucket="example", Key="hello.txt", Body=b"Hello from CROWDB!")
print(s3.get_object(Bucket="example", Key="hello.txt")["Body"].read())
print(s3.list_objects_v2(Bucket="example")["Contents"])

The checksum configuration matches the retained container boto3 example. Compatibility with every SDK default or S3 tool is not implied.

⁠Data and tags

• Mount /opt/crowdb/data to retain data and generated credentials across container recreation. Reuse the same volume when recreating the container.
• latest is the moving release tag; select a version tag for an explicit release. Version tags can be replaced when that release is republished; pin an image digest for exact image identity.
• An optional console is available by also mapping container port 9090.

⁠Development preview scope

This image runs a single-node profile with one voting replica and one copy of stored data. It provides no redundancy. Use disposable data for development, testing, and public evaluation. Production use and on-disk upgrade compatibility are not supported.

The supported S3 API is a subset of Amazon S3. Server-side copy, batch deletion, versioning, lifecycle policies, bucket policies/ACLs, server-side encryption, object tags, replication, and event notifications are outside the current advertised surface. See the project documentation for updates.

⁠Related image

crowdb/crowdb-iceberg⁠ provides an Iceberg-focused quick start. Both repositories contain the same runtime: one Access Server listens on S3 port 9091 and Iceberg port 9092. Map both ports if you need both interfaces in one container. Their namespaces and authorization remain separate; general S3 buckets do not expose Iceberg table files.

⁠Links and license

• Project website⁠
• Source and issues⁠
• Container documentation⁠
• License: Apache-2.0⁠