lucasbasquerotto/ctl repository overview

⁠(Under Construction) Controller Layer

This repository corresponds to the controller layer and is used to deploy projects. This is the top layer and is responsible to manage and deploy projects. After setting up this layer, you can start deploying projects, according to the projects defined in the main environment repository⁠.

The deployment of a specific project is done by the cloud layer, which uses the variables⁠ generated in the preparation step⁠ of this layer (controller).

A default implementation of the cloud layer is located at http://github.com/lucasbasquerotto/cloud⁠.

⁠Demo

Before start using this layer, it's easier to see it in action. Below is a simple demo used to deploy a project. The demo uses pre-defined variables generated by the controller layer⁠ so that this layer is not needed to execute, and then execute the cloud layer⁠ directly to deploy a project.

To execute the demo you will need a container engine (like docker or podman).

1. Create an empty directory somewhere in your filesystem, let's say, /var/demo.

2. Create 2 directories in it: env and data (the names could be different, just remember to use these directories when mapping volumes to the container).

3. Create a demo.yml file inside env with the data needed to deploy the project:

# Enter the data here (see the demo examples)

1. Deploy the project:

docker run -it --rm -v /var/demo/env:/env -v /var/demo/data:/lrd local/demo

The above commands in a shell script:

mkdir -p /var/demo/env /var/demo/data

cat <<'SHELL' > /var/demo/env/demo.yml
# Enter the data here (see the demo examples)
SHELL

docker run -it --rm -v /var/demo/env:/env -v /var/demo/data:/lrd local/demo

That's it. The project was deployed.

🚀 You can see examples of project deployment demos here⁠.

The demos are great for what they are meant to be: demos, prototypes. They shouldn't be used for development (bad DX if you need real time changes without having to push and pull newer versions of repositories, furthermore you are unable to clone repositories in specific locations defined by you in the project folder). They also shouldn't be used in production environments due to bad security (the vault value used for decryption is 123456, and changes to the project environment repository⁠ may be lost if you forget to push them).

⁠Root Directory

The root folder is the base directory in which the projects managed by the controller are defined. It's the parent of the controller repository (ctl) and contains:

• The secrets directory: has the vault files to decrypt the ssh files to clone the projects environment repositories⁠).

• The projects directory: has the project files, secrets and repositories.

• The users directory: has the users home directories (when use_subuser⁠ is true).

• The data directory: used mainly in local development environments to contain project deployment data (like logs, databases and uploaded files).

The following instructions assume that they are being run from the <root> folder, which will contain the data generated from all projects.

⁠Setup

The machine that will deploy the projects should have the following tools:

• Bash 4+

• Git

• A container engine (like docker⁠ or podman⁠)

To be able to deploy the projects, the controller will need to know which projects to deploy. That information is declared in the main environment repository⁠. You can manually clone the repository with git at the folder ctl/env-main (or point a symlink located at ctl/env-main to another location on you machine) or run the following command that will do that for you:

./ctl/run setup

The above command will ask you to enter the repository which willl be cloned (git clone <git_env_main_repository>). An alternative is to enter the repository directly like the following:

./ctl/run setup <git_env_main_repository>

If you have a symlink at ctl/env-main pointing to an empty directory, the git repository will be cloned at that target repository.

To make things more practical, you can create a repository to become the root folder of your environment and then make an instruction that runs the entire setup step in a more straightforward way, with just a simple command like ./setup.

This repository⁠ does that, and you can fork it and change only the env.sh file, defining in it the controller repository and branch, your main environment repository, and optionally the location of this repository relatively to the root directory (a symlink will be created at ctl/env-main).

⁠Main Environment Repository

The main environment repository will be a hub containing information about the projects to deploy. It will be located at ctl/env-main and must have the following files:

• Main Environment Options File: env.sh

File that will be sourced during launch⁠ to know which container engine should run the projects and also which container image it will run. It also has other useful options⁠ and examples⁠ explained below.

• Main Environment Vars File: vars.yml

File that contains the specifications for all projects. It will be used during launch⁠ to deploy the project specified in the launch command. See its structure⁠ and the example⁠ below.

⁠Main Environment Options

⁠Main Environment Options File - Examples

An example of the file for a development environment is as follows:

export container=lucasbasquerotto/ansible:0.0.2
export container_type=podman

Another example:

export container=lucasbasquerotto/ansible:0.0.2
export root=true

The above example will use docker as the container engine (container_type).

An example of the file for a production environment is as follows:

export container=lucasbasquerotto/ansible:0.0.2
export container_type=podman
export use_subuser=true
export subuser_prefix=project-

⁠Main Environment Vars

⁠Main Env Vars - Init

This object contains data about the container in which the step after the Controller Preparation Step⁠ will run.

⁠Main Env Vars - Repo

This object contains data about the project environment repository⁠.

⁠Main Env Vars - Repo Vault

This object contains the location to the project vault file⁠.

⁠Main Env Vars - Env Params

This object contains project specific variables. It's expected that the next layer is able to handle this option, merging the options defined here with the options defined in the project environment file⁠. Specifying options here can be convenient to deploy multiple similar projects without having to create a different project environment file⁠ for each case, achieving a more DRY approach. To make staging and production environments more predictable, it's advisable to not use this option for such projects.

There are no pre-established properties for this option (they are project specific).

Using the cloud layer defined at http://github.com/lucasbasquerotto/cloud⁠, the variables defined in this option will be accessible through the params object in the project environment file⁠.

⁠Main Env Vars - Path Params

This object could be considered a special case of the Env Params⁠ section. It contains paths to repositories, to be mapped to different locations than they would be otherwise. The paths are relative to the root repository.

Repositories mapped this way will be cloned the first time (when the directory is empty) and ignored from then on, allowing the deployment of projects along with real-time changes to the deployment code, if the code is inside one of the mapped repositories.

Mapped repositories can be shared accross different projects as long as the mapped locations are the same.

Using the cloud layer defined at http://github.com/lucasbasquerotto/cloud⁠, this option will be ignored when not running in a development environment (dev: false⁠). In this case, it accepts the following options:

⁠Main Environment Vars File - Example

dev: true

init:
  default:
    container: "lucasbasquerotto/cloud:1.3.6"
    root: true
  other:
    container: "lucasbasquerotto/cloud:1.3.6"
    container_type: "podman"
  other2:
    container: "lucasbasquerotto/cloud"
    container_type: "podman"
    root: true

repo:
  default:
    src: "ssh://[email protected]/lucasbasquerotto/project-env-demo.git"
    version: "master"
    ssh_file: "ssh/repo.encrypted.key"
  other:
    src: "ssh://[email protected]/lucasbasquerotto/other-project-env-demo.git"
    version: "master"
    ssh_file: "ssh/repo.encrypted.key"

repo_vault:
  default:
    file: "vault/main"
  other:
    force: true

env_params:
  local:
    pod_custom_dir_sync: true
    named_volumes: true
  custom:
    name: "my-custom-project"

path_params:
  default:
    path_env: "repos/env"
    path_env_base: "repos/env-base"
    path_map_repos:
      env_base: "repos/env-base"
      cloud: "repos/cloud"
      ext_cloud: "repos/ext-cloud"
      pod: "repos/pod"
      ext_pod: "repos/ext-pod"
      app: "repos/app"

# TODO

⁠Controller Preparation Step

This step is the first and only step executed in the controller layer to launch the project (deployment). The main environment repository⁠ must be present at ctl/env-main so that this step can be run.

This step generates the Controller Output Vars⁠, as well as the project ssh key file⁠ and the project vault file⁠ to be used by the next layer⁠.

⁠Main Vault File

The main vault file for a project is located at secrets/projects/<project_name>/vault and contains the value to decrypt:

1. The project ssh key file⁠ to clone the project environment repository⁠.

2. The project vault file⁠ to decrypt the contents of the project environment file⁠.

⁠Project SSH Key File

The SSH file used to clone the project environment repository⁠. This file is optional and isn't needed for public repositories, but it's very important that this ssh file is specified if the aforementioned repository has secrets in it (and the repository should be private).

⁠Project Vault File

The vault file used to decrypt the project environment file⁠ in the project environment repository⁠. This file is optional, but recommended to be used if the environment file has secrets in it. It's also recommended to make the repository private. The encryption should be done with ansible-vault⁠.

⁠Launch

The launch command deploys a project. It is executed like ctl/launch ... (or alternatively ctl/run launch ..., or even ctl/run l ...) and is responsible to run the Controller Preparation Step⁠ and call the command to run the subsequent steps (in the next layers). Below you can see the options⁠ that can be used with it as well as examples⁠ of running this step.

⁠Launch options

Below are the options that can be used to launch⁠ a project: