lucasbasquerotto/cloud repository overview

⁠(Under Construction) Cloud Layer

This repository corresponds to the cloud layer and is used to deploy projects. This layer is responsible to deploy a specific project, using the Cloud Input Vars⁠ as the input values that contain the data needed to deploy the project⁠.

It's recommended to use a controller layer, like defined at http://github.com/lucasbasquerotto/ctl⁠, to manage projects and generate those variables, instead of using this layer to deploy a project directly.

⁠Demo

Before start using this layer, it's easier to see it in action. Below is a simple demo used to deploy a project. The demo uses pre-defined input variables⁠, and then execute this layer to deploy a project.

To execute the demo you will need a container engine (like docker or podman).

1. Create an empty directory somewhere in your filesystem, let's say, /var/demo.

2. Create 2 directories in it: env and data (the names could be different, just remember to use these directories when mapping volumes to the container).

3. Create a demo.yml file inside env with the data needed to deploy the project:

# Enter the data here (see the demo examples)

1. Deploy the project:

docker run -it --rm -v /var/demo/env:/env -v /var/demo/data:/lrd local/demo

The above commands in a shell script:

mkdir -p /var/demo/env /var/demo/data

cat <<'SHELL' > /var/demo/env/demo.yml
# Enter the data here (see the demo examples)
SHELL

docker run -it --rm -v /var/demo/env:/env -v /var/demo/data:/lrd local/demo

That's it. The project was deployed.

🚀 You can see examples of project deployment demos here⁠.

The demos are great for what they are meant to be: demos, prototypes. They shouldn't be used for development (bad DX if you need real time changes without having to push and pull newer versions of repositories, furthermore you are unable to clone repositories in specific locations defined by you in the project folder). They also shouldn't be used in production environments due to bad security (the vault value used for decryption is 123456, and changes to the project environment repository⁠ may be lost if you forget to push them).

⁠Table of Contents

• Deploying a Project⁠

  • Project Base Directory⁠

  • Cloud Input Vars⁠

  • Cloud Preparation Step⁠

  • Cloud Context Input Vars⁠

  • Cloud Context Preparation Step⁠

  • Cloud Context Main Step⁠

    • Main Step - Load Environment⁠

    • Main Step - Initial Services⁠

    • Main Step - Nodes - Create⁠

    • Main Step - Nodes - Setup⁠

    • Main Step - Nodes - Prepare⁠

    • Main Step - Prepare the Pods⁠

    • Main Step - Run Stages⁠

    • Main Step - Nodes - Define the cron jobs⁠

    • Main Step - Nodes - Finish⁠

    • Main Step - Final Services⁠

    • Main Step - Delete Temporary Cloud Instances⁠

    • Main Step - Delete everything that was created previously⁠

• Project Environment⁠

  • Project Environment File⁠

  • Project Environment Base File⁠

• Useful Information⁠

  • Cloud Next Parameters⁠

  • Mergeable Parameters⁠

  • Credentials⁠

  • Contents⁠

    • Content Type⁠

    • Content Origin⁠

    • Transfer Content⁠

    • Content Full Example⁠

  • Schemas⁠

  • Validators⁠

  • Extra Repositories⁠

  • Pod Context⁠

    • Pod Context Example⁠

    • Pod Context Example Notes⁠

  • Services⁠

  • Run Stages⁠

  • Encrypt and Decrypt⁠

⁠Deploying a Project

The deployment of a project in this layer is done, by default, in 3 steps.

1. Cloud Preparation Step⁠

2. Cloud Context Preparation Step⁠

3. Cloud Context Main Step⁠

⁠Project Base Directory

The project base directory is the directory (project_base_dir) in which the files generated by the project deployment and used to deploy the project are located. The project_base_dir should be in the path <base_path>/projects/<project_name>/ where <base_path> is a base folder for the relative paths of repositories defined in path_params (used in development).

When running with the dev input variable⁠ equal true, there should be a relative symlink <project_base_dir>/dev/link that points to <base_path> to map repositories (so that you can share repositories across projects).

When running this step in a container, <project_base_dir>/dev should map to <base_path> in the host, and a symlink <base_path>/link should be created pointing to itself (.) so that the relative symlinks work both inside and outside the container.

If using the controller layer at http://github.com/lucasbasquerotto/ctl⁠ to deploy the project, the project base directory will be at <root_dir>/projects/<project_name>/ and the symlinks and volume mappings will be already handled.

⁠Cloud Input Vars

The Cloud Preparation Step⁠ needs a file in the following format located at <project_base_dir>/files/ctl/vars.yml to deploy a project:

ctxs: []
dev: 'true'
env_params:
  env_dir: common
init:
  allow_container_type: false
  container: lucasbasquerotto/cloud:1.4.9
  container_type: docker
  root: true
  run_file: /usr/local/bin/run
key: demo
lax: true
no_log: false
migration: ''
path_params:
  path_env: repos/env
project_dir_relpath: projects/demo
repo:
  env_file: common/demo.yml
  src: https://github.com/lucasbasquerotto/env-base.git
  ssh_file: ''
  version: master
repo_vault:
  file: ''
  force: false

(The values above are the output after running ./run launch --dev demo with the vars.yml in the main environment repository being the same as this example⁠.)

⁠Cloud Preparation Step

This step receives a project-dir parameter with the project base directory⁠, then use the Cloud Input Vars⁠ at <project_base_dir>/files/ctl/vars.yml to load the Project Environment⁠, and, finally, for each context defined in the input vars (ctxs), clone the cloud repository for that context, as well as the repositories that will act as extensions (ext_repos) for the cloud repository for the given context.

This preparation step is commonly executted inside a container, runs only once for the project and is the same even if the cloud repositories for the contexts are different, so it's expected that all the contexts in a project are compatible with this preparation step, and any specific stuff related to the context is run in the Cloud Context Preparation Step⁠.

When loading the environment variables defined in the env_file⁠, if repo_vault.file is defined, the vault file there is used to decrypt the encrypted values⁠. The env_file⁠ can access use jinja2 expressions and has access to the following variables:

• project_name (string): the project name, the value of key in the Cloud Input Vars⁠.
• project_ctxs (list of string): the contexts that will run in the project, the value of ctxs in the Cloud Input Vars⁠. The value ctxs is optional in the Cloud Input Vars⁠, and if not defined there, should be defined in the env_file (or env_base_file)
• params (dict): any parameters defined at env_params in the Cloud Input Vars⁠.

Aside from project-dir, the file that runs this preparation step⁠ also accepts the following options:

In this step, when the dev input var is true, the path_params value in the Cloud Input Vars⁠ file will be included in a new file at <project_base_dir>/files/cloud/path-map.yml so that the next steps can use it to map repositories to other locations and skip pulling already cloned repositories.

The value of env_params is written to the file <project_base_dir>/files/cloud/env-params.yml so that the next steps can use it to load the env_file⁠.

For each context in the project, 2 files with the same content in a different format will be created at <project_base_dir>/files/cloud/ctxs/<ctx>/vars.yml and <project_base_dir>/files/cloud/ctxs/<ctx>/vars.sh to be used in the Cloud Context Preparation Step⁠ and Cloud Context Main Step⁠. The contents of those files are defined in the Cloud Context Input Vars⁠ section.

This steps generate a file <project_base_dir>/files/cloud/run-ctxs to run each context passing as the first parameter the location of context directory (<project_base_dir>/files/cloud/ctxs/<ctx>/). Each context is run entirely (Cloud Context Preparation Step⁠ and Cloud Context Main Step⁠) before starting the next context.

⁠Cloud Context Input Vars

These are the input variables used by the Cloud Context Preparation Step⁠ and Cloud Context Main Step⁠. They are generated by the Cloud Preparation Step⁠. There are 2 files generated with the same content, but in a different format:

<project_base_dir>/files/cloud/ctxs/<ctx>/vars.sh

export commit=24c74d6130bc3602388769aad14cbca8092a20b5
export ctx=demo
export ctx_dev_dir=/main/dev/link/projects/demo/files/cloud/ctxs/demo
export ctx_dir=/main/files/cloud/ctxs/demo
export dev_repos_dir=/main/dev/link
export env_dev=true
export env_dir=/main/dev/link/repos/env
export env_file=/main/dev/link/repos/env/common/demo.yml
export env_lax=true
export env_no_log=false
export env_params_file=/main/files/cloud/env-params.yml
export path_map_file=/main/files/cloud/path-map.yml
export project=demo
export repo_dir=/main/files/cloud/ctxs/demo/repo
export repo_run_file=/main/files/cloud/ctxs/demo/repo/run
export secrets_cloud_dir=/main/secrets/cloud
export secrets_ctx_dir=/main/secrets/cloud/ctxs/demo
export vault_file=/main/secrets/ctl/vault

<project_base_dir>/files/cloud/ctxs/<ctx>/vars.yml

commit: 24c74d6130bc3602388769aad14cbca8092a20b5
ctx: demo
ctx_dev_dir: /main/dev/link/projects/demo/files/cloud/ctxs/demo
ctx_dir: /main/files/cloud/ctxs/demo
dev_repos_dir: /main/dev/link
env_dev: 'true'
env_dir: /main/dev/link/repos/env
env_file: /main/dev/link/repos/env/common/demo.yml
env_lax: 'true'
env_no_log: 'true'
env_params_file: /main/files/cloud/env-params.yml
path_map_file: /main/files/cloud/path-map.yml
project: demo
repo_dir: /main/files/cloud/ctxs/demo/repo
repo_run_file: /main/files/cloud/ctxs/demo/repo/run
secrets_cloud_dir: /main/secrets/cloud
secrets_ctx_dir: /main/secrets/cloud/ctxs/demo
vault_file: /main/secrets/ctl/vault

The values of the files above are the output after running the Cloud Preparation Step⁠ with the input variables in the example⁠ above.

⁠Cloud Context Preparation Step

This step as defined in this repository⁠ does the following tasks:

1. Loads (from the environment repository) and validates the environment (env) variable schema⁠ (as defined in the corresponding schema file⁠).

2. Prepare the repositories defined in the extra_repos defined for the context in the environment file (main.<ctx>.extra_repos), which could be used, for example, to setup all the required repositories of a development environment to setup the workspace. It also clones the repositories of the pods defined for the nodes of the context (used when transfering templates of the pod to the actual pod repository in remote hosts, because Ansible requires that templates should be in the local machine, as well as some validations). This step doen't run when the --prepare and --fast flags are specified.

3. Defines and validates the context (ctx_data) variable, merging and overriding parameters⁠, defining the context ansible fact to be used for the next steps, so that those steps don't need to do it again. Validates schemas⁠ for services, nodes, tasks and pods, and do several other types of validations, like ensuring the existence of some files that will be transfered.

4. Creates the hosts file to be used by Ansible when connecting to hosts (when new hosts are created dynamically, this file is updated) as well as the (optional) configuration file (ansible.cfg), that by default is the file ansible/ansible.cfg⁠, but can be overridden using the cfg property in the context object (in the environment file):

# ...
main:
  my_context:
    repo: "cloud"
    cfg: |
      [defaults]
      interpreter_python=/usr/bin/python3
      stdout_callback = default
      collections_paths = collections
    hosts: |
      [main]
      localhost ansible_connection=local
      [host]
    # ...
  # ...
# ...

1. Creates the playbook to execute instructions in the hosts (from files/run.tpl.yml to plays/run.yml). This is needed because the instructions, and hosts to run the instructions, as well the order in which they are run, are dynamically defined in the project environment file, but Ansible expects that the playbook is already created and the hosts and plays to be statically defined when it starts to run the Cloud Context Main Step⁠.

⁠Cloud Context Main Step

The main step of the cloud context is the actual deployment of the project. It consists of the following internal steps:

⁠Main Step - Load Environment

This step loads the environment file and defines and validates the context (ctx_data) variable, just like the steps 1 and 3 of the Cloud Context Preparation Step⁠, except that it doesn't validate the schemas. These variables are used by the next steps in the local host (ansible group main).

⁠Main Step - Initial Services

This step create the initial services declared for the context in the environment file (in the property initial_services). For example, the code below would create the services service_1, service_2 and service_3 for the context my_ctx.

# ...
main:
  my_ctx:
    repo: "cloud"
    # ...
    initial_services:
      - "service_1"
      - "service_2"
      - "service_3"
  # ...
# ...
services:
  service_1:
    #...
  servi