Public | Automated Build

Last pushed: 2 years ago
Short Description
Short description is empty for this repo.
Full Description

Fission

Fission is the home of Fusion's (very modest) data warehousing efforts. See the rough plans on O2 for more information.

Local development

To make sure everybody works in the same environment, development happens in a Docker container. You can think of Docker as a lightweight virtual machine.

  • Note: more documentation to come on how to gain access to the various services involved (Heroku, Travis, Docker Hub, GitHub.)

Requirements

  • Recent-ish versions of both Ruby and Python as well as their respective pip and gem package installers.
    • Ubuntu: apt-get install python python-pip ruby-full
    • OS X: brew install python ruby
  • Docker
  • Fig
    • pip install fig
  • Heroku Toolbelt
    • Ubuntu: wget -qO- https://toolbelt.heroku.com/install-ubuntu.sh | sh
    • OS X: brew install heroku-toolbelt
  • Travis command line client
    • gem install travis
  • fusioneng/fission-app Docker image
    • docker pull fusioneng/fission-app

The Heroku toolbelt and Travis command line client will not be needed during development, but might come in handy e.g. to synchronize environment variables.

Getting started

The application uses a .env file to store configuration parameters. There are two options for setting up a local .env file. If you have access to a working Heroku instance of the Fission App, then you can set it up as the Git remote like this:

$ heroku git:remote -a fission-heroku-app-name

This should give you access to the environment settings being used on the Heroku app, which you can override with the contents of the development.env file.

If you don't have access to a Heroku app, you can insert configuration options directly into the development.env file. Some of the options that you would need to set:

FACEBOOK_PAGE_TOKEN=...
GOOGLE_API_KEY=...
GOOGLE_CLIENT_ID=...
GOOGLE_CLIENT_SECRET=...
HTTP_PASSWORD=...
INSTAGRAM_CLIENT_ID=...
TWITTER_CONSUMER_KEY=...
TWITTER_CONSUMER_SECRET=...

After performing one or the other of these steps, you should be able to populate the .env file by running:

$ make environment

To see if things are working, try make test in the fission-app repository root directory. You should see output like the following:

running build_ext
test_followers_quick (fission.tests.DatabaseTests) ... ok
test_followers (fission.tests.FollowersTests) ... ok
test_followers_quick (fission.tests.FollowersTests) ... ok

----------------------------------------------------------------------
Ran 3 tests in 46.460s

Useful Commands

  • make environment to update your local environment variables
  • make shell for interactive development inside of a Python shell
  • make test to run the tests
  • fig up web to run the REST API
    • if on OS X, run boot2docker ip to figure out on which IP the local server is running
  • fig up followers to run the data fetcher (follower counts on various social media is currently the only thing we poll for and store)

Testing the REST API

First you'll need some followers data. You can collect this data by running fig up followers. You should see some output like this:

$ fig up followers
Recreating fissionapp_dynamodb_1...
Recreating fissionapp_followers_1...
Attaching to fissionapp_followers_1
followers_1 | 14:32:44 system             | worker_followers.1 started (pid=6)
followers_1 | 14:32:45 worker_followers.1 | starting followers
followers_1 | 14:33:33 worker_followers.1 | followers exited successfully
followers_1 | 14:33:33 system             | worker_followers.1 stopped (rc=0)
fissionapp_followers_1 exited with code 0
Gracefully stopping... (press Ctrl+C again to force)

Now start up the server with fig up web. In a separate terminal you should be able to access the REST API with a call to cURL like this:

$ curl --user username:<password> localhost:5000/followers/latest/

<password> should match the HTTP_PASSWORD setting in your .env file. You should get some JSON results like this:

{
  "followers": {
    "ABC": {
      "facebook": 6268530, 
      "instagram": 44064, 
      "twitter": 4071135, 
      "youtube": 1050896
    }, 
    "Azteca Am\u00e9rica": {
      "youtube": 274638
    }, 
    "CNN": {
      "facebook": 16778029,

To filter by organization and platform:

$  curl --user user:<password> 'http://0.0.0.0:5000/followers/latest/?orgs=Fusion,Univision&platforms=facebook,twitter'
{
  "followers": {
    "Fusion": {
      "facebook": 230370, 
      "twitter": 163093
    }, 
    "Univision": {
      "facebook": 2939138, 
      "twitter": 2081517
    }
  }
}

To fetch results before a certain date, you can use the date__lte filter with a Unix timestamp:

$ curl --user user:<password> 'http://0.0.0.0:5000/followers/latest/?date__lte=1429016700'

Architecture

Directory structure

  • /fission is where all of the code lives
  • /tools houses tools for management and deployment

Managing processes and making sense of the various configuration files

The big three configuration files you need to worry yourself about are the Dockerfile, the Procfile and fig.yml.

  • Dockerfile: the operating system and development environment; any dependencies like Python packages should be declared here; this file is used to create a Docker image that can be distributed among the team and will provide a uniform dev environment.
  • Procfile: processes that should run in this environment, defined in a way that Heroku can understand.
  • fig.yml: link together processes and Docker images and specify dependencies between environments, like spinning up a database when needed.

In addition, in the repository you will find:

  • .env*: environment variables, e.g. to interact with various APIs; except for development.env these are not in version control but are instead fetched from Heroku (the source of truth) using make environment.
  • requirements.txt: Python dependencies in a format Travis and Heroku understand. Should not be updated directly, instead edit the Dockerfile and then run make requirements.
  • setup.py exists only to make fission-app available as a Python package you can import and to specify the version number.
  • Makefile: shortcuts to various administrative commands (e.g. build steps); not essential, but it makes life easier.

Deployment

When you push a commit to GitHub, this will trigger a Travis CI test build. If this build succeeds, it will in turn trigger the generation of a new private Docker image on Docker Hub (which ensures the development environment stays up to date) as well as trigger deployment of the latest code to Heroku.

Docker Pull Command
Owner
fusioneng
Source Repository

Comments (0)