Public | Automated Build

Last pushed: a year ago
Short Description
sbex/duply:master : Duply for backup sbex/duply:tahoe-arm : Tahoe LAFS for raspberry pi (raspbian)
Full Description


Duply run on zeus, uses duplicity/rsync to incrementally backup remote folder, crypt and put on Tahoe LA FS storage grid.

1. Get all files from bitbucket sbex/duply and build the container

cd /root/ && git clone && cd duply
docker build -t duply .

2. Tahoe storage grid

2.1 Introducer: (backendpublic)

Connector/hub/directory which will list all available storages. Where our client will get the list of all available storage.


docker run --rm -v /root/.introducer:/root/.introducer/ dockertahoe/tahoe-lafs tahoe create-introducer --basedir=/root/.introducer --port=tcp:41464 --location=tcp:<YOUR_LOCAL_IP>:41464

Run container:

docker run -d --name tahoe-introducer -p 41464:41464 -v /root/.introducer:/root/.introducer/ tdockertahoe/tahoe-lafs tahoe start /root/.introducer --nodaemon --logfile=-

Get the introducer url and replace --introducer in next commands!!

docker exec introducer cat /root/.introducer/private/introducer.furl

2.2a Storage server (tdengine)

Node all the actual data will be. Can be raspberry, server, homepc.

For linux box use:


docker run --rm -v /root/.tahoe:/root/.tahoe/ dockertahoe/tahoe-lafs tahoe create-node --nickname=`hostname` --port=tcp:3457 --location=tcp:<YOUR_LOCAL_IP>:3457 --introducer=pb://zyadrwufzm34fwquu6oz6ktqu2e4phlg@tcp:

Run container:

docker run -d --name tahoe-storage -p 3457:3457 -v /root/.tahoe:/root/.tahoe/ dockertahoe/tahoe-lafs tahoe start /root/.tahoe --nodaemon --logfile=-

2.2b Storage server on raspberry pi (raspian)

What you need:

  • Ubuntu laptop to load the pi os
  • Access to your home router configuration (for port forwarding)
  • Network cable (prefered) or wifi access

Download raspian image:

cd Downloads && wget

Unmount all SD card drive

df -h
/dev/sdb1                       40862   22414     18448  55% /media/gg/resin-boot
/dev/sdb2                      174392  134814     30362  82% /media/gg/resin-root
/dev/sdb5                       20422       2     20420   1% /media/gg/resin-conf
/dev/sdb6                    15115776  919240  13766776   7% /media/gg/resin-data
sudo umount /dev/sdb1  (2,5,6)

(Not needed: if you want to format your partition: sudo mkdosfs -F 32 -v /dev/sdb)

Install OS (

cd Downloads 
sudo dd bs=4M if=2016-09-23-raspbian-jessie-lite.img of=/dev/sdb

If wifi only: edit your local wifi networks:

sudo nano /media/gg/3598ef8e-09be-47ef-9d01-f24cf61dff1d/etc/wpa_supplicant/wpa_supplicant.conf

Edit keyboard
sudo nano /media/gg/3598ef8e-09be-47ef-9d01-f24cf61dff1d/etc/default/keyboard

Edit hostname with your nickname/location

sudo nano /media/gg/3598ef8e-09be-47ef-9d01-f24cf61dff1d/etc/hostname

sudo nano /media/gg/3598ef8e-09be-47ef-9d01-f24cf61dff1d/etc/hosts greghome-pi

sudo umount /dev/sdb1  (2,5,6)

At home: put sd card in rasberry and start it. Wait and scan (before and after) the network to find his ip:

nmap -sP

Or you can connect to your home router and check the dhcp.

Connect to it

ssh pi@YOUR_RASPBERRY_IP  (password: raspberry)

Install docker

sudo apt-get update -y
sudo apt-get dist-upgrade -y
sudo shutdown -r
sudo curl -sSL | sh


sudo nano /home/pi/Dockerfile
FROM resin/rpi-raspbian:jessie

RUN apt-get update && \
    apt-get install -y git python-pip build-essential python-dev libffi-dev libssl-dev python-virtualenv

RUN git clone
ADD . /tahoe-lafs
  cd /tahoe-lafs && \
  git pull --depth=100 && \
  pip install . && \
  rm -rf ~/.cache/

RUN tahoe --version


Build images

sudo docker build -t tahoe-arm .

Add settings:
YOUR_PUBLIC_IP --> at home find your public IP here:

sudo docker run --rm -v /home/pi/.tahoe:/root/.tahoe/ tahoe-arm tahoe create-node --nickname=`hostname` --port=tcp:3457 --location=tcp:<YOUR_PUBLIC_IP>:3457 --introducer=pb://zyadrwufzm34fwquu6oz6ktqu2e4phlg@tcp:

Run tahoe storage

sudo docker run -d --name tahoe-storage -p 3457:3457 -v /home/pi/.tahoe:/root/.tahoe/ tahoe-arm tahoe start /root/.tahoe --nodaemon --logfile=-
sudo docker logs tahoe-storage

Activate your home router redirection:

Name            : tahoe
Protocal        : TCP
IP source       : *              <- access from internet
Port source     : 3457           <- port of the router to be redirected internally
IP destination  :   <- your raspberry local IP
Port destination: 3457

Test with: (should get a filtered/open state at leat)


2.3 Storage client (zeus)

No backup here, but all secrets here. We will run backup and restore from here. So this container got tahoe-client and duply.

Initialize tahoe-client (if first time) and use the introducer url from your grid:

docker run --rm -v /root/.tahoe:/root/.tahoe/ duply \
    tahoe create-client \
        --nickname=zeus --webport=tcp:3456:interface= \

nano /root/.tahoe/tahoe.cfg  <-- change happy=2 depending of your number of storage server

Create(/import) tahoe alias+subfolder: we want a structure like backup/test

docker exec duply tahoe create-alias backup
URI:DIR2:pmhosio65ugrucuvyt4uminbnq:7msac6ypv6nl5z33ionhaenzcrqutpycatuzgix7462nafgqtegq   <-- KEEP THIS URI REF, ONLY WAY TO RECOVER YOUR BACKUPS!

And a subfolder:

docker exec duply tahoe mkdir backup:test

To connect to an already existing tahoe alias:

docker exec duply tahoe add-alias backup URI:DIR2:pmhosio65ugrucuvyt4uminbnq:7msac6ypv6nl5z33ionhaenzcrqutpycatuzgix7462nafgqtegq

docker exec duply tahoe ls backup:test

nano /root/.tahoe/private/aliases  <-- name of tahoe backup folder

If empty add existing like:

backup: URI:DIR2:pmhosio65ugrucuvyt4uminbnq:7msac6ypv6nl5z33ionhaenzcrqutpycatuzgix7462nafgqtegq

or create new one:

docker exec duply tahoe create-alias backup     <-- and save the URI/privatekey somewhere save.

Run duply + tahoe-client container:

docker run -d --name duply \
    -v ~/.ssh:/ssh -v /root/.gnupg:/root/.gnupg -v /root/duply/.duply:/root/.duply -v /root/backup:/root/backup \
    --cap-add SYS_ADMIN --device /dev/fuse --privileged \
    -p 3456:3456 \
    -v /root/.tahoe:/root/.tahoe \

Note: always edit config on the host, and use docker restart duply to refresh container.

3. Duply

Create a test config for only tahoe encryption
Copy/paste an existing one or use new default config :

docker run --rm -v /root/duply:/root duply default create

nano .duply/test/conf
# tahoe
# base directory to backup

Run a backup to tahoe

docker exec duply duply test backup --disable-encryption     ( --allow-source-mismatch could be needed if error)

Check backup via Duply

docker exec duply duply test list --disable-encryption

Check file are present in Tahoe
docker exec duply tahoe ls backup:

Or via gui and entering the URI of backup:

4. All backup profiles

Quick run:

Syntax for running commands will always be the following:


docker run --rm -v ~/.ssh:/ssh -v /root/.gnupg:/root/.gnupg -v /root/duply/.duply:/root/.duply --cap-add SYS_ADMIN --device /dev/fuse --privileged duply YOUR_BCK_PROFILE backup --allow-source-mismatch


docker run --rm -v /root/duply/.duply:/root/.duply duply YOUR_BCK_PROFILE list


docker run --rm -v /root/restore:/root/restore -v /root/duply/.duply:/root/.duply duply YOUR_BCK_PROFILE restore /root/restore/YOUR_CHOICE

4a. Ethereum

Edit conf:

nano /root/duply/.duply/ethereum/conf

# gpg encryption:
GPG_KEY='DC581D8A'   <-- pgp key used
GPG_PW='geneva2016'  <-- pgp password

# s3
TARGET='s3://'  <-- s3 storage
TARGET_USER='w_Y9G_G22bldi9qYhPPLJ7TAEJjwXB9h8H5-CzoxcF-eX41wL0HgEWpm_KjU9Nv7eW6loWRWHJXsfe6gF_zQhQ'      <-- s3 credentials

# base directory to backup
SOURCE='/mnt/backup'     <-- for a docker, always the same

## Backup retention: need to use purge to remove old backup. Help:
# MAX_AGE=              is how far back in time you want backups to be kept
# MAX_FULL_BACKUPS=     is how many full backups we want to keep
# MAX_FULLS_WITH_INCRS= is how many full backups with incremental backups we want to keep
# MAX_FULLBKP_AGE=      is how long we should to incremental backups before we need to do a full backup



Will be executed before/pre backup (please change with ip_of_ethereum_vm)

nano /root/duply/.duply/ethereum/pre
mkdir -p /mnt/backup
sshfs -oIdentityFile=/ssh/id_rsa_sbexx root@ /mnt/backup
ls -la /mnt/backup

Will be executed after/post backup

nano /root/duply/.duply/ethereum/post
umount /mnt/backup

Run a Backup (ssh,--cap-add, --device, --priviledged: is to be able to map ethereum folder to vm zeus, .gnupg: share the host key to container duply )

docker run --rm -v ~/.ssh:/ssh -v /root/.gnupg:/root/.gnupg -v /root/duply/.duply:/root/.duply --cap-add SYS_ADMIN --device /dev/fuse --privileged duply ethereum backup --allow-source-mismatch

List backup

docker run --rm -v /root/.gnupg:/root/.gnupg -v /root/duply/.duply:/root/.duply duply ethereum list

Edit a file and make another backup to see the incremental:

ansible ethereum -a "docker exec ethereum /bin/bash -c 'echo test1 >> /root/.ethereum/keystore/testgreg1'"
ansible ethereum -a "docker exec ethereum cat /root/.ethereum/keystore/testgreg1"
docker run --rm -v ~/.ssh:/ssh -v /root/.gnupg:/root/.gnupg -v /root/duply/.duply:/root/.duply --cap-add SYS_ADMIN --device /dev/fuse --privileged duply ethereum backup --allow-source-mismatch

list backup of x time ago: s, m, h, D, W, M, or Y (indicating seconds, minutes, hours, days, weeks, months, or years respectively)

docker run --rm -v /root/duply/.duply:/root/.duply duply ethereum list --time 10m
docker run --rm -v /root/duply/.duply:/root/.duply duply ethereum list --time 2W  

List backup collection in tahoe

docker exec duply duplicity collection-status tahoe://backup

restore last version

docker run --rm -v /root/duply/.duply:/root/.duply -v /root/restore:/root/restore duply ethereum restore /root/restore/ethereum

restore version of 10 min ago.

docker run --rm -v /root/duply/.duply:/root/.duply -v /root/restore:/root/restore duply ethereum restore /root/restore/ethereum --time 10m

4.b Bitcoin:


docker run --rm -v ~/.ssh:/ssh -v /root/.gnupg:/root/.gnupg -v /root/duply/.duply:/root/.duply --cap-add SYS_ADMIN --device /dev/fuse --privileged duply bitcoin backup --allow-source-mismatch

list backup

docker run --rm -v /root/duply/.duply:/root/.duply duply bitcoin list


docker run --rm -v /root/restore:/root/restore -v /root/duply/.duply:/root/.duply duply bitcoin restore /root/restore/bitcoin



docker run --rm -v ~/.ssh:/ssh -v /root/.gnupg:/root/.gnupg -v /root/duply/.duply:/root/.duply --cap-add SYS_ADMIN --device /dev/fuse --privileged duply dbslave backup --allow-source-mismatch

Restore granular: file dbslave-pgdump-2016-06-15.gz from 2DAY ago to /root/restore

docker run --rm -v /root/restore:/root/restore -v /root/duply/.duply:/root/.duply duply dbslave fetch dbslave-pgdump-2016-06-15.gz /root/restore/dbslave-pgdump-2016-06-15.gz --time 2D

Check file:

gunzip < /root/restore/dbslave-pgdump-2016-06-15.GZ > /root/restore/dbslave-pgdump-2016-06-15
tail -n 100 /root/restore/dbslave-pgdump-2016-06-15

4.d MARIADB (piwik, chat)

4.e Bitbucket

Here we need a location to store the code, and only push updates to s3. Otherwise the container always clone the repo and send all to s3 --> no diff.

mkdir /root/backup/bitbucket
docker run --rm -v /root/backup/bitbucket-backup:/mnt/backup -v ~/.ssh:/ssh -v /root/.gnupg:/root/.gnupg -v /root/duply/.duply:/root/.duply --cap-add SYS_ADMIN --device /dev/fuse --pr ivileged duply bitbucket backup --allow-source-mismatch

4. f kyc

We will sync s3 kyc butcket to local (only new files), and send the to s3. Need local too in order to send only the diff.

nano /root/duply/.duply/exo.s3cfg   <-- add bucket credential
nano /root/duply/.duply/kyc/pre     <-- edit the source butcket
mkdir /root/backup/bitbucket
docker run --rm -v /root/backup/kyc:/mnt/backup -v ~/.ssh:/ssh -v /root/.gnupg:/root/.gnupg -v /root/duply/.duply:/root/.duply --cap-add SYS_ADMIN --device /dev/fuse --privileged dupl y kyc backup --allow-source-mismatch

4.f Create new profile:

Copy one profile and edit the vars.

cp -r /root/duply/.duply/ethereum /root/duply/.duply/mynewprofile

4.g Amazon s3 profile:

# s3

5. Maintenance


To reconstruct the backup, if one node goes offline, or one new node get added: (run every day by cron)

docker exec duply tahoe deep-check --repair backup:

5.2 Crontab

#duply: backups
00 01 * * * /usr/bin/docker run --rm -v ~/.ssh:/ssh -v /root/.gnupg:/root/.gnupg -v /root/duply:/root --cap-add SYS_ADMIN --device /dev/fuse --privileged duply ethereum backup --allow-source-mismatch | tee --append /root/script/duply-ethereum.log > /dev/null 2>/dev/null
02 01 * * * /usr/bin/docker run --rm -v ~/.ssh:/ssh -v /root/.gnupg:/root/.gnupg -v /root/duply:/root --cap-add SYS_ADMIN --device /dev/fuse --privileged duply bitcoin backup --allow-source-mismatch | tee --append /root/script/duply-bitcoin.log > /dev/null 2>/dev/null
05 01 * * * /usr/bin/docker run --rm -v ~/.ssh:/ssh -v /root/.gnupg:/root/.gnupg -v /root/duply:/root --cap-add SYS_ADMIN --device /dev/fuse --privileged duply media backup --allow-source-mismatch | tee --append /root/script/duply-media.log > /dev/null 2>/dev/null
10 01 * * * /usr/bin/docker run --rm -v ~/.ssh:/ssh -v /root/.gnupg:/root/.gnupg -v /root/duply:/root --cap-add SYS_ADMIN --device /dev/fuse --privileged duply mariadb backup --allow-source-mismatch | tee --append /root/script/duply-mariadb.log > /dev/null 2>/dev/null
15 01 * * * /usr/bin/docker run --rm -v ~/.ssh:/ssh -v /root/.gnupg:/root/.gnupg -v /root/duply:/root --cap-add SYS_ADMIN --device /dev/fuse --privileged duply blog backup --allow-source-mismatch | tee --append /root/script/duply-blog.log > /dev/null 2>/dev/null
20 01 * * * /usr/bin/docker run --rm -v ~/.ssh:/ssh -v /root/.gnupg:/root/.gnupg -v /root/duply:/root --cap-add SYS_ADMIN --device /dev/fuse --privileged duply dbslave backup --allow-source-mismatch | tee --append /root/script/duply-dbslave.log > /dev/null 2>/dev/null
30 01 * * * /usr/bin/docker run --rm -v ~/.ssh:/ssh -v /root/.gnupg:/root/.gnupg -v /root/duply:/root --cap-add SYS_ADMIN --device /dev/fuse --privileged -v /root/backup/bitbucket-backup:/mnt/backup duply bitbucket backup --allow-source-mismatch | tee --append /root/script/duply-bitbucket.log > /dev/null 2>/dev/null
40 01 * * * /usr/bin/docker run --rm -v ~/.ssh:/ssh -v /root/.gnupg:/root/.gnupg -v /root/duply:/root --cap-add SYS_ADMIN --device /dev/fuse --privileged -v /root/backup/kyc:/mnt/backup duply kyc backup --allow-source-mismatch | tee --append /root/script/duply-kyc.log > /dev/null 2>/dev/null

01 02 * * * /usr/bin/docker run --rm -v ~/.ssh:/ssh -v /root/.gnupg:/root/.gnupg -v /root/duply:/root --cap-add SYS_ADMIN --device /dev/fuse --privileged duply ethereum-aws backup --allow-source-mismatch | tee --append /root/script/duply-ethereum-aws.log > /dev/null 2>/dev/null

A. To encrypt with pgp keys the backup: (not needed as tahoe already encrypt)

Use the provided one for test:

gpg --import /root/duply/
gpg --import /root/duply/gpgtest.DC581D8A.pri.asc.export

Or create new one for prod:

gpg --gen-key
for exemple: name:backuptest,, pw: geneva2016

Backup your keys: save in a save place these 2 files and folder:

gpg --export-secret-keys -a keyid > backup_private_key.asc
gpg --export -a keyid > backup_public_key.asc

trust the key

gpg --edit-key DC581D8A


5 (select 5 if you ultimately trust the key)


Check the key

gpg --list-key

pub   2048R/DC581D8A 2016-03-29
uid                  backuptest <>
sub   2048R/B380730B 2016-03-29

Need to backup

  • DBslave(pg_dump) ok
  • bitcoin ok
  • ethereum ok
  • Backend Media folder : backendadmin:/root/media ok
  • chat, piwik --> mariadb ok
  • bitbucket : ok

  • elk

  • KYC
Docker Pull Command
Source Repository