splashblot/tileo-swarm repository overview

DANGER!!! WIP docker-sync adaptation, seems to work well ... but breaks sometime. See FAQ and add to it, please.

⁠tileo-swarm

This docker container provides a fully working Tileo development environment without the installation hassle by wrapping docker-dronedb into a docker-compose.yml with all other required services (mostly PostgreSQL, Redis and NGINX ... plus other goodies!)

It is an evolution of tileo-beta and it attemps to make it simple to deploy Tileo to a production (no quotes this time) environment by taking advantage of Docker Swarm, in particular the secrets⁠ functionality.

Pre-requisites are a working Docker⁠ installation in Docker Swarm⁠ mode or simply via Docker Compose⁠, on a good Unix-like system.

[WIP/danger] If you want to develop using platform-native tools (meaning, a way to access the files that are otherwise inside of containerized filesystems, both in MacOS and Linux), you will also need to:

1. (first time only) sudo gem install docker-sync

For Docker Compose mode, just run the commands below, add dev.localhost.lan to your /etc/hosts and then connect to http://dev.localhost.lan/⁠ with your browser after you prime the DB schema with some data (first time only steps):

1. git clone https://github.com/splashblot/tileo-swarm.git
2. cd tileo-swarm
3. (first time only) docker-compose up postgis dronedb redis - only those three for now to prime the DB
4. (first time only) docker exec -ti tileo-swarm_dronedb_1 /bin/bash -c "export TERM=xterm-256color; exec bash"
5. (first time only - inside that shell) cd /dronedb/script/ && ./create_db_only.sh && ./setup_tileo_feature_flags.sh && ./create_tileo_dev_user.sh && exit
6. (first time only) docker-compose down - to cleanup those containers
7. Ongoing operation
   1. (normal) docker-compose up
   2. ([WIP/danger] docker-sync) docker-sync-stack start - Tends to time out, just re-issue the command the first time around

The default login is dev/pass1234.

⁠How to run the containers:

docker-compose up

Last but not least, you need to add hostname aliases to your /etc/hosts file. For example:

sudo sh -c 'echo 127.0.0.1 dev.localhost.lan >> /etc/hosts'

... more might be needed if you want to configure other organizations or subdomains:

sudo sh -c 'echo 127.0.0.1 localhost.lan >> /etc/hosts'
sudo sh -c 'echo 127.0.0.1 agroguia.localhost.lan >> /etc/hosts'
sudo sh -c 'echo 127.0.0.1 splashblot.localhost.lan >> /etc/hosts'
sudo sh -c 'echo 127.0.0.1 smartrural.localhost.lan >> /etc/hosts'

You should now be able to access the install at http://dev.localhost.lan/⁠

⁠Configuring ml-py-inferer:

(WARNING: outdated!) In order to be able to perform object detection with ml-py-inferer, you need to follow these steps:

1. Put a configuration file like this one⁠ in /data/ml-py-inferer
2. Upload your models data to /data/ml-py-inferer/models and configure them properly in the config file

Example tree of /data/ml-py-inferer for a particular configuration

.
├── config.py
└── models
   └── nectarines
       ├── cfgs
       │   └── faster_rcnn_end2end.yml
       ├── VGG16
       │   └── faster_rcnn_end2end
       │       └── test.prototxt
       ├── vgg16_faster_rcnn_iter_18000.caffemodel
       └── vgg16_faster_rcnn_iter_6000.caffemodel

Once done and when started you will be able to add a procesing node inferer at port 5000 (port can be changed in configuration file) into WebODM so that you can perform object detection.

⁠FAQ:

- My docker-sync is acting up ... and unison keeps bugging me about something or other, what should I do?

Start fresh. Run docker-sync clean and remove your ./dronedb folder. Help us debug it and fix it via PR here ...

⁠USEFUL DOCKER COMMANDS:

- To connect to a remote Swarm:

docker run --rm -ti -v /var/run/docker.sock:/var/run/docker.sock -e DOCKER_HOST dockercloud/client <stack_name>

- To update a service running in the stack:

To avoid downtime, we make sure there are 2 replicas running first, make the rolling update, then go back to 1:

docker service scale <service_name>=2
docker service update --force --image splashblot/docker-dronedb:tileo <service_name>
docker service scale <service_name>=1

example:
docker service scale prod_dronedb=2
docker service update --force --image splashblot/docker-dronedb:tileo prod_dronedb
docker service scale prod_dronedb=1

- To roll a service update back which was just updated in the stack to the previous image:

docker service update --rollback <service_name>

example:
docker service update --rollback prod_dronedb

- To access a container you need ssh access to:

docker ps

docker exec -ti <container-ID> bash

- To create a new Organization:

1. See sample script here⁠
2. Create a similar script, look for a 128x128 logo of the organization (see here⁠) and push them to the repo after testing the script works locally.
3. Access the dronedb container via ssh (see above)
4. cd /dronedb/script/tileo/sampletada/
5. mkdir <your-new-org-ID>
6. Run the script you created (notice the script won't be there and git pull won't make it magically appear as it is put in there by the docker-dronedb build process, you decide how you want to do it (creating a new image is probably overkill, but works)

- To add a new hostname to the Let's Encrypt SSL certificate:

1. Edit the docker-entrypoint.sh file like this⁠
2. Create a new Docker image for tileo-swarm (once you git push it should happen automagically)
3. docker service scale prod_nginx=0 (yes, that means downtime! :scream:)
4. docker volume rm prod_data-volume-www-ssl-certs so the SSL certs get regenerated when you start nginx again
5. docker service scale prod_nginx=1 (this will take a while, as SSL challenges take a few minutes each