mmezei/s3proxy repository overview

s3proxy⁠ with the additional Middleware overlay as forked from switchmedia/s3proxy overlay branch. Adds support for retrieval of blobs that was not working in overlay module for at least some s3 providers.

⁠Usage

$ docker pull mmezei/s3proxy

$ docker run --publish 80:80
--env S3PROXY_AUTHORIZATION=none
--env JCLOUDS_IDENTITY=S3_IDENTITY
--env JCLOUDS_CREDENTIAL=S3_CREDENTIAL
--env JCLOUDS_PROVIDER=s3
--env S3PROXY_OVERLAY_BLOBSTORE=true
mmezei/s3proxy

⁠How the Overlay works:

A local BlobStore backed by the filesystem implementation is created, and all writes are sent to this container.

⁠Reads

If a Blob exists in the Overlay BlobStore, it will be returned. If a Blob does not exist in the Overlay Blobstore, the request is passed to the Upstream provider.

⁠Writes

Writes are all done in the Overlay BlobStore Delete's are simulated by storing a .__deleted file in the Overlay BlobStore

⁠Listing Container

When listing a container contents, the Overlay and Upstream responses are merged together. Any files that have a .__deleted file present in the Overlay BlobStore is removed from the list.

⁠Warning from switchmedia/s3proxy (from their docker hub readme on 02/13/2022):

This is not production-ready. It was designed for development and testing purposes only. A non-exhaustive list of things that would be needed to be Production Ready are: Use a more robust mechanism to generate "Not Found" errors other than trying to load a non-existant file. The local BlobStore needs a mechanism to limit the amount of space it can consume, and how to handle the case where that space is exhausted.