Public | Automated Build

Last pushed: 2 years ago
Short Description
Short description is empty for this repo.
Full Description


.. image::
:alt: Join the chat at

|pypi status| |travis-ci status|

DPark is a Python clone of Spark, MapReduce(R) alike computing framework
supporting iterative computation.

Example for word counting (

.. code:: python

 from dpark import DparkContext
 ctx = DparkContext()
 file = ctx.textFile("/tmp/words.txt")
 words = file.flatMap(lambda x:x.split()).map(lambda x:(x,1))
 wc = words.reduceByKey(lambda x,y:x+y).collectAsMap()
 print wc

This script can run locally or on a Mesos cluster without any
modification, just using different command-line arguments:

.. code:: bash

$ python
$ python -m process
$ python -m host[:port]

See examples/ for more use cases.

Some more docs (in Chinese):\_pro/wiki

DPark can run with Mesos 0.9 or higher.

If a $MESOS_MASTER environment variable is set, you can use a
shortcut and run DPark with Mesos just by typing

.. code:: bash

$ python -m mesos

$MESOS_MASTER can be any scheme of Mesos master, such as

.. code:: bash

$ export MESOS_MASTER=zk://zk1:2181,zk2:2181,zk3:2181/mesos_master

In order to speed up shuffling, you should deploy Nginx at port 5055 for
accessing data in DPARK_WORK_DIR (default is /tmp/dpark), such

.. code:: bash

        server {
                listen 5055;
                server_name localhost;
                root /tmp/dpark/;

Mailing list:

.. |travis-ci status| image::

.. |pypi status| image::

Docker Pull Command
Source Repository