boechat107/ext-spark-notebook
Jupyter notebook to run Spark. Some extensions are included.
Image
0
446
boechat107/ext-spark-notebook repository overview
Extended Jupyter Notebook
Dockerfile based on jupyter/all-spark-notebook extended with jupyter-vim-binding.
Some features:
- Spark 1.6.0 with Python, Scala or R.
- Edit notebooks using Vim keybindings.
- SQLAlchemy and psycopg2 to connect to Postgres with Pandas.
- JDBC driver for Postgres 9.4.
- The default amount of memory for Driver is 4G.
- Some small style modifications:
- Smaller font size for the code blocks.
- Different style for inline code in Markdown blocks.
SQLAlchemy for Postgres
import pandas as pd
from sqlalchemy import create_engine
from pyspark import SparkContext
from pyspark.sql import SQLContext
sc = SparkContext("local[*]")
sqlsc = SQLContext(sc)
## Connection to a local PostgreSQL.
pgeng = create_engine('postgresql://postgres:postgres@localhost:5432/postgres')
pd.read_sql("select * from SOME_TABLE;", pgeng)
Postgres JDBC driver
An example of usage in Python:
from pyspark import SparkContext
from pyspark.sql import SQLContext
sc = SparkContext("local[*]")
sqlsc = SQLContext(sc)
jdbcOpts = {'url' : 'jdbc:postgresql://localhost:5432/postgres',
'driver' : 'org.postgresql.Driver',
'user' : 'postgres',
'password' : 'postgres'
}
df = sqlsc.read.jdbc(
url = jdbcOpts["url"],
## This query is executed in the database.
table = '(select * from SOME_TABLE limit 1000) as table',
properties = jdbcOpts
)
df.describe().show()
Tag summary
Content type
Image
Digest
Size
2.1 GB
Last updated
over 9 years ago
docker pull boechat107/ext-spark-notebook