rc42/dataprocessor repository overview

⁠dataProcessor

A Data Processor using Flask on Ubuntu & Gunicorn inside Docker
by Collective Acuity

⁠Components

• Ubuntu 14.04 LTS (OS)
• Python 3.5.1 (Environment)
• Gunicorn 19.4.5 (Server)
• Flask 0.10.1 (Framework)
• Bootstrap (CSS Template)
• awsDB (Data Model Map)
• cassandraModel (Data Model Map)

⁠Dev Env

• Docker (Provisioning)
• BitBucket (Version Control)
• Ngrok (Secure Tunnel)
• Cassandra (Accounts Database)
• AWS (Web Services)
• • EC2 (Webhosting)
• • S3 (Records Database)
• • Lambda (Scheduled Task Trigger)
• PyCharm (IDE)
• Google Drive(Sync, Backup)

⁠Languages

• Python
• Regex
• Shell Script
• Javascript

⁠Features

• Flask in a Container
• Local Credential Controls
• Python 2.7 Enabled
• NodeJS Enabled
• Tunnel Ready for Testing
• EC2 Ready for Deployment

⁠Setup DevEnv

• Install Docker Toolbox on Local Device
• Install Git on Local Device
• Clone/Fork Repository from Version Control service
• Create a /cred Folder in Root to Store Tokens
• [Optional] Create a New Private Remote Repository

⁠Server Sub-Folders

• assets/ sub-folder for non-python code and project resources
• data/ sub-folder for ephemeral data stored on image
• imports/ sub-folder for unpublished python modules
• methods/ sub-folder for application specific python classes
• models/ sub-folder for data object model declarations
• static/ sub-folder for public accessible application content
• templates/ sub-folder for html templates

⁠Launch Commands

startProcessor.sh
Creates container with required volumes and starts flask on a gunicorn server
Requires:

• Container Alias
• Container Ports
• Mapped Volumes
• Initial Command
• Container Root Folder Name (if AWS EC2 deployment with awsDocker module)
• Virtualbox Name (if Windows or Mac localhost)

rebuildDocker.sh
Initiates an automated build command on Docker to update base image
Requires:

• Container Alias
• Token from Docker Build Settings
• Environment Variable File (in cred/)

tunnelNgrok.sh
Initiates a secure tunnel from local device to endpoint on ngrok.io
Requires:

• Container Alias
• AuthToken from Ngrok
• Environment Variable File (in cred/)

tunnelMe.sh
Initiates a secure tunnel from local device to endpoint on localtunnel.me
Requires:

• Container Alias

⁠Collaboration Notes

The Git and Docker repos contain all the configuration for deployment to AWS.
Google Drive can be used to synchronize local copies across multiple dev devices.
Use AWS IAM to assign user permissions and create keys for each collaborator.
Collaborators are required to install dependencies on their local device.
Repo should be FORKED by collaborators so reality is controlled by one admin.
New dependencies should be added to the Dockerfile, NOT to the repo files.
Collaborators should test changes to Dockerfile locally before merging to remote:

docker build -t test-image .

.gitignore and .dockerignore have already been installed in key locations.
To prevent unintended file proliferation through version control & provisioning,
add/edit .gitignore and .dockerignore to include all new:

1. local environments folders
2. localhost dependencies
3. configuration files with credentials and local variables