jkingyens/sessions-io repository overview

⁠supported dev platforms:

mac OSX 10.11.x mac OSX server 5.0.15+ (for http/https apache)

markdown editor (macdown, mou, etc) latest non-beta xcode with iOS SDK latest cocoapods

many of these can be brew installed:

letsencrypt gcloud 0.9.x docker 1.8.x rethinkdb 2.2 redis 3.x ngrok 2.0.19 node.js 5.0.0 logstash 2.1.1 elasticsearch 2.0 nginx 1.8.0

⁠manual setup

cp the nginx congiuration into your own setup ps ax -o pid,ppid,%cpu,vsz,wchan,command|egrep '(nginx|PID)'

/usr/local/Cellar/logstash/1.5.4/libexec/bin/plugin install logstash-input-rethinkdb

test logstash with rethinkb:

logstash -e ' input {rethinkdb {host => "localhost" port => 28015 auth_key => "" watch_dbs => ["test" ] watch_tables => ["users"] }} output {stdout {codec => json_lines}}'

this will wipe elasticsearch clean:

elastic search data is stored here: /usr/local/var/elasticsearch rm -rf everything in that directory to remove the search data log stash keeps rethinkdb in sync with elasticsearch pipes document updates into the searchable/query index

this will route localhost:80 to the service actually running on port 9000

/etc/pf.conf:

scrub-anchor "com.apple/" nat-anchor "com.apple/" rdr-anchor "com.apple/" rdr pass on lo0 inet proto tcp from any to 192.168.1.0/24 port 80 -> 127.0.0.1 port 9000 dummynet-anchor "com.apple/" anchor "com.apple/*" load anchor "com.apple" from "/etc/pf.anchors/com.apple"

sudo pfctl -ef /etc/pf.conf

start pfctrl at system boot:

---

Label com.substrait.pf.plist Program /sbin/pfctl ProgramArguments /sbin/pfctl -e -f /etc/pf.conf RunAtLoad ServiceDescription FreeBSD Packet Filter (pf) daemon StandardErrorPath /var/log/pf.log StandardOutPath /var/log/pf.log ******************

⁠web server setup

think up a subdomain of substrait.com for your local services ie) local get your public ip address reachable from the internet in google domains, set the value of your subdomain to your ip if you move around a lot, the above 3 steps can be autoamted with DDNS client/server if you are behind a router, hole-punch the router on 3 ports: 80, 443, 3000 if you are on ipv6, might be able to route straight to machine? sessions web service will run on port 80/443 sessions api server will run on port 3000 open OSX Server.app and enable apache web service set your hostname to be the same subdomain you chose above (should be green) use the letsencrypt client to fetch ssl certs for your ie) local.substrait.com install these free certificates using the OSX server app configure apache to forward decrypted traffic to privateip:9000 dont worry about SSL on api server for now, we'll set this later.

⁠developers need accounts here:

proto.io - mobile prototyping ngrok.com - webhook tunnel google compute engine sendgrid - GCE outboudn email service docker hub account apple developer account stripe account (with merchant setup for tesitng) layer.com - mobile messaging PaaS launchkit.io - app landing page mailchimp for mail campaigns mixpanel for analytics optimizely - A/B testing trello for project management github + repository access (jkingyens/sessions-io) slack for team chat (domain: sessionsio) circleCI for testing containers new relic for tracking mobile and server error/perf metrics pingdom for monitoring basic domain acccess

⁠running & testing components development

run component-wise mocha tests (with mocks) perhaps setup an environment where people can run multi container testings

⁠get configuration for the local services

add ngrok to stripe dashboard webhook configure a dns record that looks up the local machine (127.0.0.1? or LAN ip) update development.env with redirect URIs based on this domain name if SSL works here, then prefer https:// for all redirects to echo prod update development.env with values for stripe, layer, for your own accounts

⁠dev environment

entire stack is built from single repo: git clone https://github.com/jkingyens/sessions-io⁠

backend: source ./development.env scripts/redis.sh scripts/rethink.sh scripts/elastic.sh scripts/logstash.sh scripts/ngrok.sh scripts/nginx.sh << you now have a public webhook address https://XXXX.ngrok.io⁠ from ngrok >> visit stripe.com and add a development webhook for https://XXXX.ngrok.io/webhooks/stripe⁠ visit dev.fitbit.com and add a development webhook for https://XXXX.ngrok.io/webhooks/fitbit⁠ npm install -g grunt-cli cd email && npm install cd email & gcsupload cd api && npm install cd api && nodemon cd web && npm install cd web && nodemon endpoints: http://localX.substrait.com:8080⁠ (rethinkdb) http://localX.substrait.com:9000⁠ (sessions web) http://localX.substrait.com:3000⁠ (sessions api) http://stripe.com⁠ (stripe dashboard) http://layer.com⁠ (layer dashboard) node.js logs for web and api

ios: cp Configuration.plist update Configuration.plist with values from accounts (stripe, layer, etc) open up xcode load QuickStart workspace in ./ios select "Development" configuration select simulatated device (iphone 6, ios 8.4+) run the simulator, verify signup run through some user stories to ensure correct setup endpoints: device logs in xcode debug output in xcode ide

⁠General Debugging Development Tips

browse stripe.com test dashboard for payments debugging browse layer.com for layer messaging logs/debug browse sendgrid for email debugging browse google cloud platform for openid/connect debugging interact with api server at http://local.substrait.com:3000⁠ interact with web server at http://local.substrait.com⁠ browse rethinkdb at http://local.substrait.com:8080⁠ browse local redis instance using redis-cli browse rethinkdb logs at ?????? browse redis logs at ?????? browse ios app logs by building 'development' with xcode

⁠Transactional email workflows

add email/secrets.json for email workflow keys run 'grunt serve' to spin up iterative email design flow CI/CD will automatically build and host new email email workflow lives in 'email' directory emails are built into production on commits to master this means new emails sent after each master commit will be based on new asset to setup email workflow locally use: npm install npm install -g grunt-cli the workflow commands are: grunt serve (for iterative devleopment) grunt sendgrid template=.html

⁠manually pushing to production

manually push ios builds to production using prodbot

• connect to XCode Server at address: osx.sessions.io happy with changes then do a submit to prod (should be staging) run ./all.sh + monitor results with circle CI + dashboards hit staging server and run all your interactive tests, etc. show off feature / submittion / bug-fix, etc. when happy, close pull-request, live on master verify push to real production and evrything is okay.

⁠submitting ios app to the app store

ensure your using GM or release tool chain (os, xcode, the whole stack!) ensure you bump the build number in the general settings of the app in xcode ie) 8 -> 9 (itunes rejects otherwise) if your OSX is on beta, follow these instructions: https://georgegarside.com/blog/ios/submit-apps-built-beta-xcode/⁠ in xcode select production build with generic iOS device as the target then go product -> archive tap the button to upload the app to itunes connect be sure to UNCHECK bitcode on the following screen (something to do with layerkit framework) wait for the binary to be processed in ituens connect (could take a long time, an hour?) when binary is processed, add binary to iOS app, and click "submit for review"

⁠automated CI/CD to production

when new commits are pushed to master, we automatically trigger build processes our xcode server will build, test and publish a new binary to internal distribution channels our linux servers will build, test, publish and orchastrate a server-side upgrade process if any of our testing fails then this automated process will roll back

⁠adding/removing services in production

product services are built natively in google compute engine infrastructure the archicture is microservice-based and the frontend/entrypoint is an http/https load balanacer SSL is supported natively within the load balancer itself (anything behind the load balancer is non-tls) if you are building a new service or removing an existing one you need to understand this process google imposes fundamental resource limits on the entites required to create services, there remove ones that are not used from the main google cloud console:

• tap http load balancer
• create load balancer
• add new global forwarding rule
• create a new static global external ip address
• create for http, https, or both (depending on service needs)
• copy the new global ip and tap on the cloud DNS section
• tap add record set, and paste in the global ip address and subdomain
• go to compute engine -> instance groups
• tap "edit instance groups"
• tap "add item" in the port name mapping and give your backend servie a name + port
• tap "compute" and then "health checks" -> create a health check
• use the same port as your servicein the previous step
• use the correct path for your health check such that service returns 200 OK
• go back to load balancer, and configure backend
• select the backend you created, select the healthcheck you created
• go to firewall rules, create a new rule for the backend port of the service
• set the source to be ANY (this is not ideal, but required for now, we should use the load balancer)
• set the right ports, your backend service port and protocol (tcp probably) to remove a service, we need to do the opposite. we should turn this into a script using google cloud platform

⁠upgrading discourse in production

login to [email protected]⁠ go to ~/discourse_docker and run "./launcher rebuild app" this will rebuild the base docker image. once completed run "" see "https://github.com/discourse/discourse_docker⁠" for further instructions this sould hopefully get easier with a docker-compsoe build + deploy script

⁠upgrading ghost blog in production

go to blog/ and run docker-compose build && docker-compose up -d

⁠Debugging Production

make sure your ip is added to the ssh firewall or nothing works. browse stripe.com production dashboard for payments debugging browse layer.com for layer messaging logs/debug browse sendgrid for email debugging browse google cloud platform for openid/connect debugging interact with api server at https://api.sessions.io⁠ interact with web server at https://sessions.io⁠ browse rethinkdb using local SSH tunnel:

• ssh -fNTL localhost:8081:$(ssh [email protected]⁠ "docker inspect --format '{{ .NetworkSettings.IPAddress }}' layertest_userstore_1"):8080 [email protected]⁠
• browse to localhost:8081 on local machine browse local redis instance using redis-cli (??) browse redis logs with docker logs -f X browse rethinkdb logs with docker logs -f Y sysdig for machine level debugging in production. run the following on a CoreOS instance:
• docker run -i -t --name sysdig --privileged -v /var/run/docker.sock:/host/var/run/docker.sock -v /dev:/host/dev -v /proc:/host/proc:ro -v /boot:/host/boot:ro -v /lib/modules:/host/lib/modules:ro -v /usr:/host/usr:ro sysdig/sysdig browse ios app logs by building 'production' with xcode

⁠Production Crash Reporting, service status, etc.

pingdom for main site availbility google compute engine health checks for microservice avail custom daemon for emails when health chekcs are failing?

⁠Product Anlaytics

overview:

browse trello to look at upcoming, active and past experiments

new experiment:

each card holds an individual experiment and is tagged as value and/or growth the experiment includes a hypothesis along with target changes to the KPI we are too small to be running A/B tests, so most changes are with respect to time for growth hypothesis, metrics will be around week/week, month/month growth of users, trainers, etc. for value hypothesis, metrics will be around retention, usage metrics, exit surveys, revenue, etc. cards will also hold references to the software features reuqired to support hypothesis card also must have post-experiment, actionable steps. why are we running this?

upcoming -> active:

ensure the required software featuers are complete decide on a timebox/window for the experiment look at the anlaytics required on the card browse mixpanel.com to get production analytics. write script to collect raw data and perform computations capture the results over time and plot on a time graph when timebox completes, capture all data in easy to consume format make conclusions from the results, reference results in card make recommendation based on the conlusion (review action steps) move card to completed

active -> completed

make sure the recommendation is recorded somewhere else move the card to completed, ensure the time box is complete