Sign inSign up

upperroomtechnology/algorithm

By upperroomtechnology

•Updated over 7 years ago

Docker image of algorithm repo

Image
0

605

upperroomtechnology/algorithm repository overview

⁠Upperroom Algorithm

⁠Project Organization

├── bin                     <- Script to submit application to spark master
├── data
│   └── raw                 <- The original, immutable bond and portfolio data.
|
|── docs                    <- A default Sphinx project; see sphinx-doc.org for details
|
├── notebooks               <- Jupyter notebooks. Naming convention is a number (for ordering),
│                           the creator's initials, and a short `-` delimited description, e.g.
│                           `1.0-jqp-initial-data-exploration`.
|
├── requirements            <- The requirements file for reproducing the analysis environment: local, production, test
|
├── src                     <- Source code for use in this project.
│   │
│   ├── data                <- Scripts to download or generate raw bond and portfolio data.
│   │   └── make_dataset.py
│   │
│   └── features            <- Scripts and functions to calculate and generate bond trading recommendation
|
├── Makefile                <- Makefile with command like `make build`
|
└── README.md               <- The top-level README for developers using this project.

⁠Development Environment setup for local

⁠Pre-requirements

- Git
- Python 3.5.0
- Spark 2.1.0 (http://spark.apache.org/downloads.html)

⁠Get Started

The easiest way to work with the backend application is to run it as a standalone server. You would need to install Virtual Environment Wrapper⁠.

To set up a new virtual environment

⁠Step 1: Setup virtual environment
$ mkvirtualenv --python=`which python3` p-raisin-algorithm

To install all python dependencies

$ workon p-raisin-algorithm
$ pip install -r requirements/local.txt
⁠Step 2: Install Spark

Download and install spark from Spark 2.1.0⁠

⁠Step 3: Config Spark to work with python environment

Append the following statements to the ~/.bash_profile or any start up script in your system.

export SPARK_HOME={{ Append the spark folder path you have installed on step 2}}
export PATH=$SPARK_HOME/bin:$PATH
export PYSPARK_PYTHON=~/.virtualenvs/p-raisin-algorithm/bin/python
export PYSPARK_DRIVER_PYTHON=ipython
export PYSPARK_DRIVER_PYTHON_OPTS='notebook'
# Set python path for local importing pyspark and unittest
export PYTHONPATH="${SPARK_HOME}/python/:$PYTHONPATH"

then execute the bash profile.

$ source ~/.bash_profile

⁠Scripts

⁠Start jupyter notebook
$ pyspark
⁠Build the project
$ make build
⁠Submit spark application to local
$ ./bin/submit-local.sh

⁠Development Environment setup with Docker

Make sure you have the project structure like this

├── algorithm   
├── devops   
├── docker-img-jupyter   
└── docker-img-spark
⁠Step 1: Build algorithm image
$ cd algorithm
$ ./bin/build.sh
⁠Step 2: Build jupyter image
$ cd docker-img-jupyter
$ ./bin/build.sh
⁠Step 3: Build spark image
$ cd docker-img-spark
$ ./bin/build.sh
⁠Step 4: Start the container
$ cd devops
$ ./bin/run-algorithm.sh
⁠Step 5: Access the jupyter web:

From the console log we can find out the token to access the jupyter web:

jupyter_1  |         http://localhost:8888/?token=fddab70879c44962c8024f2b9f2b82d827510cfa9d4852b2
⁠Start scheduler

You can run the scheduler by using the following command:

python src/main.py

As default the scheduler will run daily at 5:00 AM.

To run with the debug you can use:

DEBUG=true DEBUG_CRON_TIME=1 python src/main.py

The DEBUG_CRON_TIME is the time in minute that the cron will run. Default is 5.

Tag summary

Content type

Image

Digest

Size

871.5 MB

Last updated

over 7 years ago

docker pull upperroomtechnology/algorithm:develop