icalialabs/codeclimate repository overview

⁠Foresight Services

This is the main repository of the "Foresight" project.

This repository contains the following applications:

• "Artanis"⁠, in which all contact information for organizations, individuals (and teams?) will be managed. It also provides authentication services for individuals & applications and grants them access to the rest of the system.

• "Vulcan"⁠, in which all the information regarding applications in development will be tracked, categorized and managed.

• "Sibyl"⁠, in which the real analysis work is done.

• "Web UI"⁠, which will present the usable functionality to the users of the system.

• "Cthulhu"⁠, which is our main integration service with Github, propagating github events through the Foresight Message Bus, and pushes data, perfors actions back at Github.

⁠Table of contents

• Environment URLS⁠
• The Team⁠
• Management tools⁠
• Project Setup⁠
• Restoring the databases⁠
• Starting the stack in development mode⁠
• Debugging⁠
• Stopping the stack⁠
• Running the Tests⁠
• Restoring the database⁠
• Running tests⁠
• Detecting code issues⁠
• Project peculiarities⁠

⁠Environment URLS

• Production - https://foresight.icalialabs.com⁠

⁠The Team

⁠Management tools

You must ask for access to this tools if you don't have them already:

• Github repo⁠
• Team Slack⁠

⁠Development

⁠Project Setup

1. Be sure to have Docker installed - Visit https://www.docker.com/community-edition⁠ for details.

2. Install plis⁠ - It's particularly important to use it in this project, due to the high service count, and auxiliary/test/benchmark services in it.

3. Clone this repository into your local machine into the suggested path icalialabs/foresight:

$ git clone [email protected]:IcaliaLabs/foresight.git ~/icalialabs/foresight && cd ~/icalialabs/foresight

⁠Environment variables

Since this project requires some access information (API keys, etc), and these are not checked-in on Git, you should ask your Dev lead for a dotenv file. This file must be placed at the root of this project, with the .env filename. An example on how this file should look is provided at example.env.

⁠Restoring the databases

Although the project is setup in such a way that databases are initialized automatically, sometimes it may be more convenient to restore the databases from database dump files. Since each app uses it's own database, the process is a bit more involved:

1. Get the dumps from your project lead. Place them in the db/dumps folder.
2. Ensure the postgres service is up and running: plis start postgres. Wait a few seconds for the server to start accepting connections.
3. Restore the artanis database: plis run postgres restoredb artanis_development [path-to-the-artanis-dump]
4. Restore the vulcan database: plis run postgres restoredb vulcan_development [path-to-the-vulcan-dump]
5. Restore the sibyl database: plis run postgres restoredb sibyl_development [path-to-the-sibyl-dump]
6. Restore the cthulhu database: plis run postgres restoredb cthulhu_development [path-to-the-cthulhu-dump]

⁠Starting the stack in development mode

To bring up the whole application (backend and frontend), open up a terminal window and run:

$ plis start backend frontend

It will take a while when starting this the first time, as it will build the project containers.

Once started, you can follow the logs of all the project containers with:

$ plis logs -f

Once you see the web containers giving outputs like these:

artanis_web_1 | Puma starting in single mode...
artanis_web_1 | * Listening on tcp://0.0.0.0:32102
web_ui_1      | Build successful (15980ms) – Serving on http://localhost:4200/

This means the stack is up and running, and you should be able to see the website at http://localhost:4200⁠

⁠Working with the UI without Docker

You might find out that working in the Web UI using Docker is slow - only on Mac and Windows. If this is the case, you can alternatively:

• Install nvm⁠
• Install the latest nodejs LTS version using nvm⁠
• Make sure you follow the Ember CLI prerequisites guide⁠, as we will use watchman, and we need to have ember-cli, bower, and phantomjs npm packages installed globally.
• Make sure you got a dotenv file web-ui/.env (ask your dev lead for it)
• Remove the web-ui/node_modules and web-ui/bower_components folders, if they are present.
• Start only the frontend_deps containers with plis start frontend_deps.
• In another terminal window, move into the web-ui directory.
• Run yarn to install dependencies.
• Run ember serve to fire up the web app.

⁠Debugging

Most of the services are Rails applications. Since these are running inside containers, using tools like byebug or pry to follow breakpoints in the code for debugging and interact with the app is not straightforward when using Docker, but easy when using plis:

• Place debugger whenever you need it in the code.
• Figure out which service pauses when hitting the breakpoint - you'll notice it if your'e looking at the logs when using plis logs -f
• In another command line window, use plis attach [service_name] to attach your terminal window to the paused process - plis will figure out which container to attach by using the service name.
• Once attached, you can interact with the paused process as usual.
• If you want to quit debugging, type CTRL + P and then CTRL + Q to detach your terminal window from the process. Using CTRL + C will effectively kill the container, and you'll need to start the container again.

⁠Stopping the stack

To stop all of the running services in the stack at once:

$ plis stop

If you need to stop one in particular - or a list of services - you can:

% plis stop artanis_web artanis_jobs

artanis_web and artanis_jobs are the name of services located on the docker-compose.yml file. You can get the names of all the available services/containers there, and stop any of them if you need to.

⁠Running Tests

Every application should have a test daemon that executes the test suite automatically whenever a change is made in the code. You can start all of the test containers:

plis start test && plis logs -f

Alternatively, you can fire up the test container to run the test suite manually:

plis run artanis_test rspec

And to run a specific test:

plis run artanis_test rspec spec/models/user_spec.rb

⁠Detecting code issues

Icalia Foresight currently can't check reliably the code in a "mono-repository", so it's imperative that you do this manually - at least for the time being:

1. Install CodeClimate CLI⁠
2. Run codeclimate analyze

Remember, any PR must not add new code issues to the project. See CONTRIBUTING.md⁠ for more collaboration guidelines.

⁠Project peculiarities

• We replicate data (rather than tables) across all of the databases via AMQP, either on the same tables (as it's the case with OAuth tokens), or merged into tables (as it's the case with Organizations and Organization Units of Directory Services getting 'merged' as DirectoryEntries in Inventory Services). This allows each µService to replicate the information using the data structures that are a better fit for it's role in the system.

• For data objects to be able to merge, it's imperative that they don't use a serial (i.e. autoincremental) primary key, but rather an UUID key. This keeps the ID collision probability to a minimum.