splashblot/tiles-swarm repository overview

DANGER!!! WIP docker-sync adaptation, seems to work well ... but breaks sometime. See FAQ and add to it, please.

⁠tiles-swarm

This docker container provides a fully working environment without the installation hassle by wrapping docker-dronedb into a docker-compose.yml with all other required services (mostly PostgreSQL, Redis and NGINX ... plus other goodies!)

While Kubernetes is what we should probably do, we first attempted to make it simple to deploy a production (no quotes this time) environment by taking advantage of Docker Swarm, in particular the secrets⁠ functionality.

Pre-requisites are a working Docker⁠ installation in Docker Swarm⁠ mode or simply via Docker Compose⁠, on a good Unix-like system.

[WIP/danger] If you want to develop using platform-native tools (meaning, a way to access the files that are otherwise inside of containerized filesystems, both in MacOS and Linux), you will also need to:

1. (first time only) sudo gem install docker-sync

For Docker Compose mode, just run the commands below, add dev.localhost.lan to your /etc/hosts and then connect to http://dev.localhost.lan/⁠ with your browser after you prime the DB schema with some data (first time only steps):

1. git clone https://github.com/splashblot/tiles-swarm.git
2. cd tiles-swarm
3. (first time only) docker-compose up postgis dronedb redis - only those three for now to prime the DB
4. (first time only) docker exec -ti tiles-swarm_dronedb_1 /bin/bash -c "export TERM=xterm-256color; exec bash"
5. (first time only - inside that shell) cd /dronedb/script/ && ./create_db_only.sh && ./setup_tiles_feature_flags.sh && ./create_tiles_dev_user.sh && exit
6. (first time only) docker-compose down - to cleanup those containers
7. Ongoing operation
   1. (normal) docker-compose up
   2. ([WIP/danger] docker-sync) docker-sync-stack start - Tends to time out, just re-issue the command the first time around

The default login is dev/pass1234.

⁠How to run the containers:

docker-compose up

Last but not least, you need to add hostname aliases to your /etc/hosts file. For example:

sudo sh -c 'echo 127.0.0.1 dev.localhost.lan >> /etc/hosts'

... more might be needed if you want to configure other organizations or subdomains:

sudo sh -c 'echo 127.0.0.1 localhost.lan >> /etc/hosts'
sudo sh -c 'echo 127.0.0.1 agroguia.localhost.lan >> /etc/hosts'
sudo sh -c 'echo 127.0.0.1 splashblot.localhost.lan >> /etc/hosts'
sudo sh -c 'echo 127.0.0.1 smartrural.localhost.lan >> /etc/hosts'

You should now be able to access the install at http://dev.localhost.lan/⁠

⁠Configuring ml-py-inferer:

(WARNING: outdated!) In order to be able to perform object detection with ml-py-inferer, you need to follow these steps:

1. Put a configuration file like this one⁠ in /data/ml-py-inferer
2. Upload your models data to /data/ml-py-inferer/models and configure them properly in the config file

Example tree of /data/ml-py-inferer for a particular configuration

.
├── config.py
└── models
   └── nectarines
       ├── cfgs
       │   └── faster_rcnn_end2end.yml
       ├── VGG16
       │   └── faster_rcnn_end2end
       │       └── test.prototxt
       ├── vgg16_faster_rcnn_iter_18000.caffemodel
       └── vgg16_faster_rcnn_iter_6000.caffemodel

Once done and when started you will be able to add a procesing node inferer at port 5000 (port can be changed in configuration file) into WebODM so that you can perform object detection.

⁠FAQ:

- My docker-sync is acting up ... and unison keeps bugging me about something or other, what should I do?

Start fresh. Run docker-sync clean and remove your ./dronedb folder. Help us debug it and fix it via PR here ...

⁠USEFUL DOCKER COMMANDS:

- To connect to a remote Swarm:

docker run --rm -ti -v /var/run/docker.sock:/var/run/docker.sock -e DOCKER_HOST dockercloud/client <stack_name>

- To update a service running in the stack:

To avoid downtime, we make sure there are 2 replicas running first, make the rolling update, then go back to 1:

docker service scale <service_name>=2
docker service update --force --image splashblot/docker-dronedb:master <service_name>
docker service scale <service_name>=1

example:
docker service scale prod_dronedb=2
docker service update --force --image splashblot/docker-dronedb:master prod_dronedb
docker service scale prod_dronedb=1

- To roll a service update back which was just updated in the stack to the previous image:

docker service update --rollback <service_name>

example:
docker service update --rollback prod_dronedb

- To access a container you need ssh access to:

docker ps

docker exec -ti <container-ID> bash

- To create a new Organization:

1. See sample script here⁠
2. Create a similar script, look for a 128x128 logo of the organization (see here⁠) and push them to the repo after testing the script works locally.
3. Access the dronedb container via ssh (see above)
4. cd /dronedb/script/tiles/sampletada/
5. mkdir <your-new-org-ID>
6. Run the script you created (notice the script won't be there and git pull won't make it magically appear as it is put in there by the docker-dronedb build process, you decide how you want to do it (creating a new image is probably overkill, but works)

- To add a new hostname to the Let's Encrypt SSL certificate:

1. Edit the docker-entrypoint.sh file like this⁠
2. Create a new Docker image for tiles-swarm (once you git push it should happen automagically)
3. docker service scale prod_nginx=0 (yes, that means downtime! :scream:)
4. docker volume rm prod_data-volume-www-ssl-certs so the SSL certs get regenerated when you start nginx again
5. docker service scale prod_nginx=1 (this will take a while, as SSL challenges take a few minutes each