magsol/distributed-dask repository overview

⁠Distributed dask on Docker

I was working on this as a research alternative to fighting the Spark / MapReduce workflows in highly iterative algorithms run on a cluster.

Of course, I quickly discovered Oliver Grisel⁠ had already done something very similar and arguably more robust⁠. Still, different use-cases and simple stubborn curiosity compelled me to keep working on my own.

⁠Overview

This is a [HIGHLY EXPERIMENTAL] containerized implementation of distributed⁠ dask⁠. If you're really curious, check out Matthew Rocklin's blog posts on the topic⁠.

More details to come as the implementation progresses.