sellpy/ansible-deploy repository overview

⁠Ansible Deploy

Repository containing deployment functionality for automation integrations, using the brilliant Ansible toolset⁠. All the deployment scripts and functionality in this repository are used to deploy the various automation systems to our bare-metal on-site hosts in our warehouses.

⁠Prerequisites

• Server should have Ubuntu server installed
• Server needs to be accessible through the local network
• Your local machine may (depending on what application is being provisioned/deployed) need to have Docker installed and be logged in to user sellpydeploy

⁠Provision (Vault) pass

The Vault pass is stored in lastpass. It is the same pass as pass for Linux robots.

⁠Setup

⁠Generating PAT (personal access token) for ghcr authentication

Go to settings > developer settings in the Github web client. Open the menu for creating personal acccess tokens, and create a new (classic) token. Give it the correct permissions - if you'll only be using it for this purpose, the only permission it needs is read:packages. Keep track of it however you want (just store it locally on your machine or in Keeper). Should you store it locally on your machine and then need to create a new token after losing it or moving to a new machine, you can always revoke your old token.

⁠Save GHCR authentication credentials on local filesystem

Create a file called ghcr_credentials and place it in the root of the repository. The first line of file should be your Github username, and the second one should be the access token that you generated in the Github web client. It's ignored by version control so don't worry about your token getting into Git history.

⁠🏎️

Start bash in the docker container.

With image from GHCR (may take a while to download):

$ bash ./run-ansible.sh

With local Dockerfile:

$ docker-compose run local bash 

⁠Test connection

Once inside the docker container, you can test the connection to any linux hosts by running ping.sh⁠, that will test the SSH connection. It will prompt for the SSH password.

# cd automation/
# ./ping.sh printerlion_hosts_site_J 

Now you are ready to provision new servers or deploy photo robots.

⁠❗️The Deploy/Provision process is not as smooth as e.g. Parsoku and will cause some downtime❗️

You need to make sure you have communicated with the affected teams. It's usually smart to deploy one server at a time (for non-photo station servers) for the ones that are used in production to minimise downtime.

⁠SSH

When you want to ssh into a server, you can do so with the network identifier from the host list. Keep in mind that some servers are not configured with a hostname.

⁠Provision a new Linux server

Run provision_server.sh script in automation/ with one parameter which is the dns name or ip of the robot which should be provisioned. E.g:

$ ./provision_server.sh photoserver-boone.hq.sellpy.net

⁠Provision a server with AWS CLI for secrets management

Secrets for the package-sorter server (for example) are stored in the AWS SSM parameter store, and fetched at build time using the AWS command line tool. In order to be able to fetch the secrets, the AWS CLI need to be configured with the correct credentials beforehand. This is done using the provision_with_aws_cli.sh script.

⁠Provision existing servers with a kiosk

Each photo server should be provisioned with a kiosk, i.e. a screen in which the pictures that are taken by the robot are displayed.

⁠Provision kiosk on all existing hosts in a site

• Make sure all hosts are present in the photo_server_hosts_site_{{site}} file.
• Run $ ./provision_kiosks.sh hosts {{site}} with the site you want.

⁠Provision kiosk to specific server(s)

• Add the host(s) that you want to provision to to the new_hosts file and the photo_server_hosts_site_{{site}} file. (Use the photo_server_hosts_site_{{site}} file for format reference)
• Run $ ./provision_kiosks.sh new_hosts
• The new_hosts file will be wiped after the script is run.

⁠Deploy photo robots

Before deploying a new version be sure to check that the docker image has finished building for the corresponding release on Github actions⁠.

If you're deploying to the prod environment you should start by deploying to a specific server so that you can monitor it and check that everything works as it's supposed to before deploying to the rest of the hosts.

⁠Pre-download new Docker image ℹ️

There is an optional fourth parameter available in the deploy_photo_servers.sh deployment script - the option to only download the newest Docker image, without tearing down the current container (or starting the new one for that matter). This is important in order to reduce downtime in the sites where we have a lot of active photo stations, since the image download step is the step that takes the longest time. This can be done ahead of time, as long as the new image has been built on Github - the user won't notice any changes until you run the full deployment (by omitting only-download as a fourth parameter). Even if the on-site operator restarts the station, the new image won't be used to spin up the container until the full deployment has been performed. Note: this step is not at all necessary, per se - the download step is always included in the full workflow anyway - but it will simply be skipped in the full workflow if it has been downloaded before, since the host has confirmed that it already has the newest build that's available centrally in GHCR.

⁠Deploy to all existing hosts in a site

• Make sure all hosts are present in the photo_server_hosts_site_{{site}} file.
• Run $ ./deploy_photo_servers.sh {{env}} hosts {{site}}, where {{env}} should be prod, canary or dev and {{site}} should be J, K or L

⁠Deploy to a specific server(s)

• Add the host(s) that you want to deploy to to the new_hosts file. (Use the photo_server_hosts_site_{{site}} file for format reference)
• Run $ ./deploy_photo_servers.sh.sh {{env}} new_hosts {{site}}, where {{env}} should be prod, canary or dev and {{site}} should be J, K or L
• The new_hosts file will be wiped after the script is run.

⁠Deploy package-sorter/printerlion servers

Deploying the package-sorter/printerlion application to hosts on a site is performed with the (you guessed it!) deploy_package_sorter.sh/deploy_printerlion.sh script, respectively. The hosts are listed in the respective hosts file in /hosts subdirectory. Both of the scripts, just as the other deployment scripts, also accept specification of what hosts to deploy on using the new_hosts file.

⁠Caveats

When deploying the printerservers the printers that are "installed" to that printerserver will be removed for the new deployment. When a printer is used for the first time on that printerserver it will need to be installed, which takes some extra time. Therefore it is important to wait between deployments of the different printerservers.

• AWS: Monitor the SQS for the site you are deploying to. Search after sellpy-printing-prod-{{site}}.fifo and check the monitor tab to see that there are no queue building up before deploying the next printerserver.
• BetterStack: Check the dashboard for the printerservers⁠ for the "avg_received_after" and "Events by hostname" metrics for the site you are deploying to. If they are increasing, wait until they are back to normal levels before deploying another host.

⁠Troubleshooting

⁠Debugging critical Docker application errors on hosts

If a critical error is encountered from inside a containerized application, the final error stacktrace is probably not logged to Papertrail. In that case, ssh to host that encountered the error and find the Docker container that exited early (using docker ps -a command). Then the logs can be retrieved (with timestamps) using docker logs -t {containerId}.

⁠General tips

If you encounter problems, make sure your local docker image of this repo is up-to-date with the latest image in DockerHub, possibly try to delete the latest local images and pull again. In one case, I encountered issues where community.windows was not installed due to the fact that I wasn't logged into Docker and it defaulted to a downloaded image that hadn't been updated for 2 years and therefore didn't include the install in the Dockerfile.

⁠Errors with resolving python modules

Since ansible and python are currently not locked to a specific version when building a new docker image, it's possible for these dependencies to get out of sync. One example could be this error:

fatal: [photoserver-20-l-1.hq.sellpy.net]: FAILED! => {"ansible_facts": {}, "changed": false, "failed_modules": {"setup": {"failed": true, "module_stderr": "Shared connection to photoserver-20-l-1.hq.sellpy.net closed.\r\n", "module_stdout": "\r\nTraceback (most recent call last):\r\n  File \"/home/sellpy/.ansible/tmp/ansible-tmp-1759763078.2755835-26931684306291/AnsiballZ_setup.py\", line 102, in <module>\r\n    _ansiballz_main()\r\n  File \"/home/sellpy/.ansible/tmp/ansible-tmp-1759763078.2755835-26931684306291/AnsiballZ_setup.py\", line 94, in _ansiballz_main\r\n    invoke_module(zipped_mod, temp_path, ANSIBALLZ_PARAMS)\r\n  File \"/home/sellpy/.ansible/tmp/ansible-tmp-1759763078.2755835-26931684306291/AnsiballZ_setup.py\", line 37, in invoke_module\r\n    from ansible.module_utils import basic\r\n  File \"/tmp/ansible_setup_payload_psiefugu/ansible_setup_payload.zip/ansible/module_utils/basic.py\", line 171, in <module>\r\nModuleNotFoundError: No module named 'ansible.module_utils.six.moves'\r\n", "msg": "MODULE FAILURE\nSee stdout/stderr for the exact error", "rc": 1}}, "msg": "The following modules failed to execute: setup\n"}

which indicates that a connection to a host was closed, but in fact is caused by ansible not being able to resolve a python module correctly. In this situation it is good to check versions of both ansible and python, which can be done with ansible --version and python3 --version. The last known combination of these dependencies that works is ansible 2.16.3 and python 3.12.3