placement/python3.7-nodejs12-base repository overview

⁠Getting up and running

Note commands below assume Mac OS and homebrew⁠ is installed.

1. Install python. (You may have to set your path⁠ to override default python2):

   brew install python
   

   Copy

2. Install Openssl for psycopg dependency:

   brew link openssl --force
   export LDFLAGS="-L/usr/local/opt/openssl/lib"
   export CPPFLAGS="-I/usr/local/opt/openssl/include"
   

   Copy

3. Install Postgres:

   brew install postgresql
   brew install postgis
   brew services start postgresql
   psql -d postgres -c "CREATE EXTENSION postgis;"
   

   Copy

4. Install Redis

   brew install redis
   brew services start redis
   

   Copy

5. Install Elasticsearch

   brew install elasticsearch
   brew services start elasticsearch
   

   Copy

6. Create a new virtual environment

   python3 -m venv .venv
   

   Copy

7. Activate your virtual environment. Or, automate venv activation⁠ so you never forget

   source .venv/bin/activate
   

   Copy

8. Install python packages:

   pip install -r requirements/local.txt
   

   Copy

9. Install pre-commit hooks

   pre-commit install
   

   Copy

10. Install the spaCy language model

    python -m spacy download en_core_web_sm
    

    Copy

⁠Setup Database

After installing Postgres, you will need to create and migrate the database. Run from the shell:

psql postgres -c "CREATE DATABASE hyped"
python manage.py migrate

⁠Editor Configuration

We use eslint⁠, prettier⁠, black⁠, isort⁠ and flake8⁠ in our codebase. Please enable these formatters and linters in your code editor. If you're using vim, try ale⁠.

⁠Install mjml

1. Install nvm⁠:

   curl -o- https://raw.githubusercontent.com/nvm-sh/nvm/v0.35.0/install.sh | bash
   

   Copy

2. Use it:

   nvm use
   

   Copy

3. Install yarn⁠:

   brew install yarn
   

   Copy

4. Install npm modules:

   yarn install
   

   Copy

⁠Web app development

For web app development on the node server, see the README file in frontend folder⁠.

⁠Testing secrets in development

1. Set var in your shell:

   export DJANGO_READ_DOT_ENV_FILE=1
   

   Copy

2. Create a .env in the project root and add secret env var. For example:

   PLAID_SECRET="foo"
   

   Copy

⁠Building the quarterly COLI Data

1. Download the quarterly data, format it (follow existing format) and place it in

   hypetrain/main/data/coli/quartery_data/year-quarter.csv
   

   Copy

2. Run script from the shell

   python manage.py runscript build_county_to_coli_map
   

   Copy

3. Spot check build data using the newly created log in hypetrain/main/data/coli/logs

4. Commit log and regenerated file: hypetrain/main/data/county-to-coli.json

5. Deploy

6. Connect to production database and run

   python manage.py runscript update_coli
   

   Copy

⁠Adding a cron job

We use celery beat⁠ to manage periodic tasks.

1. Add your code to tasks
2. Add your schedule to celery.py in the appropriate section

⁠Running Celery in the background

1. In your .env file, add:

   CELERY_TASK_ALWAYS_EAGER=False
   

   Copy

2. In a new terminal window, run the celery worker:

   celery -A hypetrain worker --loglevel=info -n worker1
   

   Copy

3. Make sure to restart your server

⁠Basic Commands

⁠Running Dev Server

python manage.py runserver

⁠Running Tests

python manage.py test

⁠Creating indexes in Elasticsearch

python manage.py search_index --delete
python manage.py search_index --create

⁠Populating Elasticsearch with data from the database

python manage.py search_index --populate --parallel

⁠Delete the indices and then recreate and populate them

python manage.py search_index --rebuild --parallel

⁠Sync new records from the database into Elasticsearch

python manage.py search_index --sync --parallel