haymomeran/brommo repository overview

⁠Brommo

⁠Pre requisits

Check out the repository and change into the directory.

Install global packages

npm install -g yarn gulp

Install dependencies:

npm install

Install Homebrew:

/usr/bin/ruby -e "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/master/install)"

Install mongoDb:

brew install mongodb
mkdir -p /data/db # create data dir
sudo chown -R `id -un` /data/db # set permissions
sudo launchctl limit maxfiles 1000000 1000000 # increase open files limit
ulimit -n 2560 # increase open files limit
mongod & # start mongo in background

http://treehouse.github.io/installation-guides/mac/mongo-mac.html⁠

⁠Development

Before developing you have to configure all services

cp aws.secret.sample aws.secret

Setup the neccessary information in aws.secret then load the environment.

source aws.secret
source aws.env

Start server:

# Start server
yarn start

You can test the API under http://localhost:4040/api-doc/⁠

Add authentication to your requests by sending the header:

Authorization: Bearer {jwtToken}

⁠Linting

We use Airbnb lint rules.

# Lint code with ESLint
yarn lint

# Run lint on any file change while developing
yarn lint:watch

⁠Testing

You need to set the AWS env variables in order to run the integration tests. See below in the AWS section on how to set up the env.

# Run tests written in ES6
yarn test

# Run specific tests
yarn test:${filename}

# Run test along with code coverage
yarn test:coverage

# Run tests on file change
yarn test:watch

⁠Test coverage

You can inspect the test coverage after runnig yarn test:coverage in the file open coverage/lcov-report/index.html.

⁠Facebook

We use a facebook test user to test our fb api. Facebook extends only tokens that are valid for a max period of 60days. If out test token expires the tests will fail and we have to update the test token. You can get a new the test token here https://developers.facebook.com/apps/431941600471444/roles/test-users/⁠ and then extend the token here https://developers.facebook.com/tools/debug/accesstoken/⁠ The test token needs to be updated in ./server/tests/auth.test.js

⁠Data extraction

In order to use the tor network to crawl the data use the tor docker container we use on the production system.

docker run -it -p 8118:8118 -p 9050:9050 -d dperson/torproxy
export TOR_SOCKS_PROXY=socks5://localhost:9050

You can test the exractors on the commandline after building with

node ./dist/bin/extract.js http://localhost:4040/test/willhaben.at.html

To test the price crawler for immopreise.at use:

node ./dist/server/lib/immopreise.js

⁠Docker container for testing

Build an image with mongoDb and send it to http://hub.docker.com⁠

# create the prod image with pre cached node_modules
docker build -f Dockerfile.prod -t haymomeran/brommo-prod .
docker tag haymomeran/brommo-prod:latest haymomeran/brommo-prod:0.0.1
docker push haymomeran/brommo-prod

# create a test image with mongodb installed
docker build -f Dockerfile.test -t haymomeran/brommo-test .
docker tag haymomeran/brommo-test:latest haymomeran/brommo-test:0.0.1
docker push haymomeran/brommo-test

# Run the tests locally (see https://bitbucket.org/spittet/node-mongodb)
docker run -it --rm --memory=4g --entrypoint=/bin/sh haymomeran/brommo-test

How to clean up your docker environment https://www.digitalocean.com/community/tutorials/how-to-remove-docker-images-containers-and-volumes⁠

docker rm $(docker ps -a -f status=exited -f status=created -q)
docker rmi $(docker images -f dangling=true -q)

⁠Production

You should manage all you changes before, but this command will bump the version, commit all changes, tag the commit and push tag the branch and all tags to the server. Bitbucket will build and deploy the new version through elastic beanstalk. Docker hub.docker.com is connected with Bitbucket. When ever a tag is pushed automatically a docker build is triggered.

npm version patch -m 'Commit message'

⁠Build the production container

Manually build a production container

docker build -t haymomeran/brommo .

⁠Debug the production container

# Run the contianer with local db settings and forward port
docker run --name brommo --rm -d -i --entrypoint /bin/sh -p 4040:4040 -e BROMMO_MONGODB='mongodb://192.168.1.4/brommo-prod' haymomeran/brommo

# List container and connect interactively to the container id
docker exec -it brommo /bin/sh

# monitor the process manager in the docker container
pm2 list
pm2 monit

# stop the container
docker stop brommo

⁠Push production container to dub.docker.com

# Tag and push the container to hub.docker.com
docker tag haymomeran/brommo:latest haymomeran/brommo:1.0.0
docker push haymomeran/brommo:1.0.0

⁠Other deployment types

In case you want to deploy on a node image on AWS consider npm install eb-fix-npm

⁠AWS Webservices

Setup AWS infrastructure:

• setup VPC
• setup a security group with inbound HTTP, HTTPS (0.0.0.0/0) and SSH (service IP)
• setup S3 for file storage and for config file

⁠Configure Beanstalk

Instgall elastic beanstalk

brew install awsebcli

Initialize the directory with the EB CLI. Create the application.

eb init brommo-api

Only Create the application if it note exists yet or you want another eg. 'brommo-test'.

eb create brommo -c brommo --cfg brommo-docker

⁠Manual Deployment

eb deploy brommo

Connect ECS with private repository from hub.docker.com http://dotnetliberty.com/index.php/2015/12/03/asp-net-5-on-aws-ec2-using-docker-hub-private-repository/⁠

⁠Database

We use the mongo database service at https://mlab.com⁠.

⁠Logging

Universal logging library winston⁠ is used for logging. We use https://logentries.com/⁠ to access the logs.

⁠Image processing and cdn

We use the image service at http://cloudimg.io⁠.

⁠Code Coverage

yarn test:check-coverage # collect code coverage
yarn report-coverage # genereate report in ./coverage/lcov-report/index.html

Reference: Express Boilerplate⁠