Docker image with spark integrated with pyspark-mongo
629
PySpark + Mongo Hadoop:
To see how it work, you can run a mongo instance with my image:
$ docker-compose run spark-mongo
And then check if it works:
import pymongo
import pymongo_spark
mongo_url = 'mongodb://mongo:27017/'
client = pymongo.MongoClient(mongo_url)
client.foo.bar.insert_many([
{"x": 1.0, "y": -1.0}, {"x": 0.0, "y": 4.0}])
client.close()
pymongo_spark.activate()
rdd = (sc.mongoRDD('{0}foo.bar'.format(mongo_url))
.map(lambda doc: (doc.get('x'), doc.get('y'))))
rdd.collect()
## [(1.0, -1.0), (0.0, 4.0)]
Fork from zero323 with reference to stackoverflow. Fixed some problems with enviroment variables and java.
Reference: mongo-hadoop
Content type
Image
Digest
Size
785.5 MB
Last updated
about 7 years ago
docker pull josemyd/docker-spark-mongo