Magic sync for docker
This is currently a proof of concept.
An issue with OSXFS prevents
inotify eventsto get triggered in a mounted volume if the number of files is high. This prevents this to run as soon as the number of files to synced exceed a rather very low limit.
While waiting that Docker fixes the issue with OSXFS, you could check
docker-syncwhere this concept has been succesfully deployed.
Magic sync for docker is a simple tool that watches local directory trees and syncs in real time the modification in a
This image provides:
- real time bi-directionnal syncing between your host and a container
- full performance for your docker mounts (OSXFS, NFS, Vboxfs... are really slow especially for large code base)
- simple configuration
- docker only as a requirement
The mounted folders in the container must be abled to emit
inotify events, currently this was tested only with OSXFS
(Docker for Mac).
How to use
A full example is provided in the example folder, it runs
composer install of a Drupal project (large code base). To run it:
cd example docker-compose -f docker-compose.yml -f docker-compose.development.yml up
Then, spawn in a new shell:
docker exec -it example_php_1 bash -c 'composer --working-dir=/src/example/src install'
Here is an example on how to use it with a docker-compose file. The scenario illustrates a way to sync source code in real-time from the host inside
the container and get the output generated by the execution of the code back to the host.
app container inits the volume
/src where the code will reside.
magic-sync container maps the local folder
./src to the folder
/src.magic with native mounts (osxfs)
version: '2' services: app: image: alpine:latest command: /bin/true volumes: - /src php: build: ./fpm volumes_from: - app magic-sync: image: mickaelperrin/docker-magic-sync:latest environment: - SYNC_USER=www-data - SYNC_UID=33 - SYNC_IGNORE=.idea:.git #- MAXIMUM_INOTIFY_WATCHES=524288 volumes: # Configure here the mappings between your host and the container. # Simply add a `.magic` extension to the volume path in the container. - ./src:/src.magic # this is needed for automatic discovery of mounted volumes in this container. - /var/run/docker.sock:/tmp/docker.sock:ro # If you want to configure USER / UID / IGNORE by volumes, you can use a simple YAML configuration file # - ./docker-syncs.yml:/sync-entrypoint.d/docker-syncs.yml volumes_from: - app # Privilegied mode is needed if you want to increase the maximum inotify watches for very large synced folders # by setting the ENV var MAXIMUM_INOTIFY_WATCHES. #privileged: true
Folders to sync
The mapping between the volumes to sync is done magically by looking for volumes mounted in the container that ends
.magic extension. They will be synced with a folder of the same name without the
Configuration is done through environment variables and/or YAML config file(s).
Configuration set in environment variables is used globally for all synced folders, except a specific value is defined
in a YAML config file. All yaml files provided in the
/sync-entrypoint.d will be parsed as configuration files.
You can configure the user mapping by providing the user name and the uid of the user, through the ENV variables
SYNC_UID, or the
uid entries in the YAML file.
You can configure the files / folders to ignore by providing the
-ignore options that need to be passed to unison.
See the official documentation of unison on how to configure excludes.
If you need to sync very large code base, you can run out of
inotify_watchers, you can increase this amount by
providing the number you want in the ENV variable
MAXIMUM_INOTIFY_WATCHES. Be aware that you need to set the
option for the container in this case.
Behind the scenes
This container uses:
unisonfor bi-directionnal syncing and file watching
docker-genfor volumes discovery
Special thanks for Eugen Mayer which makes me think about this implementation with OSXFS.
Besides the usual disclaimer in the license, we want to specifically emphasize that the authors, and any organizations the authors are associated with, can not be held responsible for data-loss caused by possible malfunctions of Docker Magic Sync.
GPLv2 or any later GPL version.