Dockerfile
FROM aether/ubuntu-spark:14.04-1.4.2-attributes
ADD target/spark-app.jar /spark-app.jar
ENTRYPOINT ["/bin/sh", "-c", "$SPARK_HOME/bin/spark-submit --class com.kmalik.sample.SparkApp spark-app.jar $*"]
Usage
sudo docker run --net=host -it kmalik/spark-app [--app={app}] ... (app specific options)
Options :
- --app={app} : Run app, default "pi"
pi :
Computes value of pi by looping over an RDD of points, split into slices.
Options :
- --points={points} : number of points to consider, default 400000
- --slices={slices} : number of slices to split, default 4
sample :
Samples (without replacement) lines from an input text file.
Options :
- --input={input} : Input file path (can be local / hdfs / s3)
- --output={output} : Output file path (default = {input}.out.timestamp)
- --ratio={ratio} : ratio of input lines to sample (default = 0.1)
feature-selection :
Selects columns from every line of a delimited text file
Options :
- --input={input} : Input file path (can be local / hdfs / s3)
- --output={output} : Output file path (default = {input}.out.timestamp)
- --features={features} : 0 based indices of columns to select
- --sep={sep} : column delimiter (default = comma)
train :
Train a logistic regression model from an input training file
Options :
- --train={train} : Training data file path (can be local / hdfs / s3)
- --modelPath={modelPath} : Output model file path (default = {train}.model.timestamp)
- --features={features} : 0 based indices of columns to consider for training
- --label={label} : 0 based index of column to treat as label (default = 0)
- --sep={sep} : column delimiter (default = comma)
- --iterations={iterations} : number of iterations (default = 10)
test :
Test a logistic regression model from an input test file
Options :
- --test={test} : Test data file path (can be local / hdfs / s3)
- --modelPath={modelPath} : LR model file path (can be local / hdfs / s3)
- --output={output} : Output file path (default = {test}.out.timestamp)
- --features={features} : 0 based indices of columns to consider for using the model
- --label={label} : 0 based index of column to treat as label (default = 0)
- --sep={sep} : column delimiter (default = comma)
lr :
Train and test a logistic regression model from input train / test files
Options :
- --train={train} : Train data file path (can be local / hdfs / s3)
- --test={test} : Test data file path (default : {train})
- --output={output} : Output file path (default = {test}.out.timestamp)
- --features={features} : 0 based indices of columns to consider for using the model
- --label={label} : 0 based index of column to treat as label (default = 0)
- --sep={sep} : column delimiter (default = comma)
- --iterations={iterations} : number of iterations (default = 10)
Source
https://github.com/kapil-malik/sample-apps/tree/master/spark-app