novolabs/nginx repository overview

⁠unum (n)

definition: One

⁠Download unum

The first thing you should do is clone unum into your local directory. This can be done in the standard way using git:

$ git clone [email protected]:NovoLabs/unum

All of the instructions that follow will assume you are working out of the root of the unum repository. If a task requires navigating to a sub-project of unum (surprise! unum is a mono-repo), the instructions will be explicit about the change.

⁠Dependencies

unum has a few external dependencies which need to be set up before you can run locally

⁠nexmo-cli

unum uses Nexmo as its telecommunications provider. Nexmo has an API as well as a CLI tool for working with the service. You can install the nexmo CLI⁠ with the following command:

$ npm install nexmo-cli -g

Test that the installation worked properly:

$ nexmo --version
0.3.13

Once you have the nexmo CLI installed, you will need to configure it. This (hopefully) is one of the only steps that will require outside assistence. One of NovoLabs Nexmo account administrators will need to provide you with login credentials to the Nexmo website⁠. Once you have credentials, your API key and secret should be displayed on the main dashboard after you login. You will need both of those to configure the nexmo CLI using the following command:

$ nexmo setup <api_key> <api_secret>

This command should write those values to ~/.nexmorc for use any time you execute the nexmo command. For more information on the nexmo CLI, check out the documentation⁠.

⁠brew

NovoLabs defaults to using brew⁠ to install external packages. If you do not already have brew installed on your machine, you can do so with the following command:

$ /usr/bin/ruby -e "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/master/install)"

Test that brew has successfully installed:

$ brew --version
Homebrew 1.7.5
Homebrew/homebrew-core (git revision c89a; last commit 2018-09-21)

⁠ngrok

The next thing that you will need to install is ngrok⁠. ngrok allows your localhost to be accessible from an external connection over the internet. unum requires ngrok for local development as it allows you to call into the instance running on your machine from an outside line.

You can install ngrok using brew with the following command:

$ brew cask install ngrok

Test that ngrok was successfully installed:

$ ngrok --version
ngrok version 2.2.8

⁠sbt

One of the two languages used to write unum is Scala. As such, you will need to install sbt⁠ in order to build that portion of unum. You can install sbt using brew:

$ brew install sbt

Check that the install was successful:

$ sbt --version
[info] Loading settings for project global-plugins from plugins.sbt,build.sbt ...
[info] Loading global plugins from /Users/marcusoladell/.sbt/1.0/plugins
[info] Loading project definition from /Users/marcusoladell/Projects/clojure/unum/project
[info] Set current project to unum (in build file:/Users/marcusoladell/Projects/clojure/unum/)
[error] Expected symbol
[error] Not a valid command: --
[error] Expected 'debug'
[error] Expected 'info'
[error] Expected 'warn'
[error] Expected 'error'
[error] Expected 'addPluginSbtFile'
[error] --version
[error]   ^

Despite reporting an error, this message is okay. It simply means that --version is not a valid sbt command. However, for our purposes, it is clear that sbt attempted to process the command and is installed and working correctly.

⁠Leiningen

The other primary language used to write unum is Clojure. Building Clojure projects requires that you install Leiningen⁠ which, again, can be done with brew:

$ brew install leiningen

As before, you should test that the install was successful. Note, that the command to execute Leinginen is lein:

$ lein --version
Leiningen 2.8.1 on Java 1.8.0_181 Java HotSpot(TM) 64-Bit Server VM

Note the Java version above. unum has an explicit dependency on Java 8⁠ so make sure Java 8 is the version of Java you are running on your machine. Some day unum may migrate to Java 10, but for now it is on 8.

⁠Docker

unum makes extensive use of Docker⁠ and, specifically, docker-compose⁠. You will need to install docker compose using this link⁠. Once you have installed docker-compose, verify that it is working correctly using the following command:

$ docker-compose --version
docker-compose version 1.22.0, build f46880f

At this point, you should have the foundation for running a local instance of unum

⁠Running Locally

unum is comprised of a couple of different services that work together to provide an NLP-based user interface for ordering food.

⁠Building and Seeding the Database

The first thing we need to do is make sure we have a functioning database. unum has a sub-project called tern which can be used to create, migrate and populate the database used by unum. Execute the following set of commands to bring up unum's database:

$ cd tern
$ docker-compose up -d
$ lein db:migrate:up
$ lein db:seed

First we cd into the tern sub-project. Next, we bring up the Postgres database using docker-compose up -d. The -d parameter tells docker-compose to run the containers in the background (Side note: if you wish to see the log files from the daemonized containers, you can run docker-compose logs). Next, we migrate the database to the most current structural form using lein db:migrate:up. If this is the first time running lein db:migrate:up, Leiningen will pull all of the necessary dependencies before executing the migration. Lastly, we run lein db:seed to populate the database with seed data.

At this point, we have an up-to-date, populated database that is ready to use. You can connect to it (using your favorite SQL Client) with the following credentials:

database: "novolabs"
host: "localhost"
port: 5432
user: "docker"
password: "opensesame"

⁠Running ngrok

Next, we need to run ngrok so that our local machine will be accessible by outside services. The easiest way to do this is with the following command:

$ ngrok http 4567

The command will run in-process (i.e. the terminal window you launched ngrok from will no longer be usable until the ngrok is terminated). You should see output similar to the following:

$ ngrok http 4567
ngrok by @inconshreveable (Ctrl+C to quit)

Session Status                online
Account                       [email protected] (Plan: Pro)
Version                       2.2.8
Region                        United States (us)
Web Interface                 http://127.0.0.1:4040
Forwarding                    http://01345aef.ngrok.io -> localhost:4567
Forwarding                    https://01345aef.ngrok.io -> localhost:4567

Connections                   ttl     opn     rt1     rt5     p50     p90
                              0       0       0.00    0.00    0.00    0.00

This is functional and will work. However, without a paid account, ngrok will assign a random URI to access your local machine. This will become tedious over time, as you will need to update various configurations with the newly assigned URI. No worries! This can be resolved by asking the NovoLabs ngrok admin for an access token. Once you have your token, you can create the following file:

$ mkdir ~/.ngrok2
$ echo "authtoken: <api_key>" > ~/.ngrok2/ngrok.yml

Once ngrok.yml is set up with your configuration file, ngrok will allow you to setup an ingress to your local machine with a user-defined URI:

$ ngrok http -subdomain=my-machine.canal 4567
ngrok by @inconshreveable (Ctrl+C to quit)

Session Status                online
Account                       [email protected] (Plan: Pro)
Version                       2.2.8
Region                        United States (us)
Web Interface                 http://127.0.0.1:4040
Forwarding                    http://my-machine.canal.ngrok.io -> localhost:4567
Forwarding                    https://my-machine.canal.ngrok.io -> localhost:4567

Connections                   ttl     opn     rt1     rt5     p50     p90
                              0       0       0.00    0.00    0.00    0.00

Note that ngrok assigned us a well defined URI, based on the value of the -subdomain flag that we passed into the ngrok command. Now, instead of getting something like http://01345aef.ngrok.io (and having to update our configuration every time we restart ngrok), we get a nice clean URI like http://my-machine.canal.ngrok.io, which is stable and can be used throughout the rest of unum's local dev configuration.

Some other things that are worth noting:

• The use of canal in the URI is by convention. canal is a sub-service that is part of unum which relies on the ngrok tunnel being available
• The port number 4567 is the default port that canal runs on. If you want to run on a different port, you can change it when you open the ngrok tunnel. However, make sure you remember to also update the port number that canal is running on.

⁠Running vox

We need to start vox next:

$ cd vox/vox
$ sbt "run -admin.port=:9991 -http.port=:8080"

⁠Running canal

Once we have vox up and running, the last service we need to start is canal. This requires a bit of configuration. Below is a Leiningen profile that contains all of the configuration options that canal requires. Copy this file into vox/canal/profiles.clj. You will need to update :events-url and :ws-uri to use the base URL that ngrok created. This will allow Nexmo to tunnel into your local environment and communicate with canal.

{:local-dev {:aot [canal.sentry.InsertConvoPostgresFailure
                   canal.sentry.InvalidNexmoAnswer
                   canal.sentry.UnknownNexmoNumber
                   canal.sentry.InsertNexmoRecordingPostgresFailure
                   canal.sentry.UnknownNexmoPhoneCall
                   canal.sentry.InsertNexmoCallFailure]
             :env   {:events-url                     "http://<ngrok_uri>/events"
                     :ws-uri                         "http://<ngrok_uri>/ws"
                     :vox-host                       "localhost"
                     :vox-port                       8081
                     :initial-silence-buffer         600
                     :datadog-prefix                 "canal"
                     :datadog-host                   "localhost"
                     :datadog-port                   8125
                     :datadog-version-group-id       "codes.novolabs"
                     :datadog-version-artifact-id    "canal"
                     :datadog-constant-tags          "environment:dev"
                     :google-storage-keystore-path   "/Users/<username>/.gcloud/NovoLabs-convo-svc-gs.p12"
                     :google-application-credentials "/Users/<username>/.gcloud/NovoLabs-fullstack-dev.json"
                     :google-storage-issuer          "[email protected]"
                     :nexmo-application-id           "a7a07a32-ffb7-4208-85c4-505d17a3a24f"
                     :nexmo-private-key-path         "/Users/jeffdavis/.nexmo/dev.key"
                     :database-host                  "localhost"
                     :database-port                  5432
                     :database-name                  "novolabs"
                     :database-user                  "docker"
                     :database-passwd                "opensesame"
                     :sentry-public-key              "f11309658d964ca9a705297d32532121"
                     :sentry-private-key             "2edef826edbf4a9b982d14fbdbe3061e"
                     :sentry-host                    "sentry.io"
                     :sentry-project-id              "272027"
                     :sentry-environment             "dev"
                     :sentry-stacktrace-app-packages "codes.novolabs.canal"
                     :sentry-servername              "penland365"
                     :slack-token                    "xoxb-214382688562-380839243363-41CvipznOUrJIPoMMVPRwuYa"
                     :unknown-numbers-channel        "DB72AK813"}
             :dependencies [[ring/ring-mock "0.3.2"]]
             :nrepl {:start? true
                     :port   10071}}}

Once you have created and updated vox/canal/profiles.clj, you should be ready to run canal. You can do so with the following commands:

$ cd vox/canal/
$ lein with-profile local-dev run