Public | Automated Build

Last pushed: 10 months ago
Short Description
sbex/duply:master : Duply for backup sbex/duply:tahoe-arm : Tahoe LAFS for raspberry pi (raspbian)
Full Description

DUPLY TO TAHOE: BACKUP INFRA

Duply run on zeus, uses duplicity/rsync to incrementally backup remote folder, crypt and put on Tahoe LA FS storage grid.

1. Get all files from bitbucket sbex/duply and build the container

cd /root/ && git clone https://bitbucket.org/sbex/duply.git && cd duply
docker build -t duply .

2. Tahoe storage grid

2.1 Introducer: (backendpublic)

Connector/hub/directory which will list all available storages. Where our client will get the list of all available storage.

Initialize:

docker run --rm -v /root/.introducer:/root/.introducer/ dockertahoe/tahoe-lafs tahoe create-introducer --basedir=/root/.introducer --port=tcp:41464 --location=tcp:<YOUR_LOCAL_IP>:41464

Run container:

docker run -d --name tahoe-introducer -p 41464:41464 -v /root/.introducer:/root/.introducer/ tdockertahoe/tahoe-lafs tahoe start /root/.introducer --nodaemon --logfile=-

Get the introducer url and replace --introducer in next commands!!

docker exec introducer cat /root/.introducer/private/introducer.furl

2.2a Storage server (tdengine)

Node all the actual data will be. Can be raspberry, server, homepc.

For linux box use:

Initialize:

docker run --rm -v /root/.tahoe:/root/.tahoe/ dockertahoe/tahoe-lafs tahoe create-node --nickname=`hostname` --port=tcp:3457 --location=tcp:<YOUR_LOCAL_IP>:3457 --introducer=pb://zyadrwufzm34fwquu6oz6ktqu2e4phlg@tcp:185.19.30.78:41464/uqrzlcn5etmnrb5x7rzbhkgq6ctoakrb

Run container:

docker run -d --name tahoe-storage -p 3457:3457 -v /root/.tahoe:/root/.tahoe/ dockertahoe/tahoe-lafs tahoe start /root/.tahoe --nodaemon --logfile=-

2.2b Storage server on raspberry pi (raspian)

What you need:

  • Ubuntu laptop to load the pi os
  • Access to your home router configuration (for port forwarding)
  • Network cable (prefered) or wifi access

Download raspian image:

cd Downloads && wget https://downloads.raspberrypi.org/raspbian_lite_latest

Unmount all SD card drive

df -h
/dev/sdb1                       40862   22414     18448  55% /media/gg/resin-boot
/dev/sdb2                      174392  134814     30362  82% /media/gg/resin-root
/dev/sdb5                       20422       2     20420   1% /media/gg/resin-conf
/dev/sdb6                    15115776  919240  13766776   7% /media/gg/resin-data
sudo umount /dev/sdb1  (2,5,6)

(Not needed: if you want to format your partition: sudo mkdosfs -F 32 -v /dev/sdb)

Install OS (https://www.raspberrypi.org/documentation/installation/installing-images/linux.md)

cd Downloads 
sudo dd bs=4M if=2016-09-23-raspbian-jessie-lite.img of=/dev/sdb

If wifi only: edit your local wifi networks:

sudo nano /media/gg/3598ef8e-09be-47ef-9d01-f24cf61dff1d/etc/wpa_supplicant/wpa_supplicant.conf
country=CH
network={
    ssid="KSYS_bdc7"
    psk="moS2QEUa8t2fM"
}

Edit keyboard
sudo nano /media/gg/3598ef8e-09be-47ef-9d01-f24cf61dff1d/etc/default/keyboard
XKBLAYOUT="ch"

Edit hostname with your nickname/location

sudo nano /media/gg/3598ef8e-09be-47ef-9d01-f24cf61dff1d/etc/hostname
greghome-pi

sudo nano /media/gg/3598ef8e-09be-47ef-9d01-f24cf61dff1d/etc/hosts
127.0.0.1 greghome-pi

sudo umount /dev/sdb1  (2,5,6)

At home: put sd card in rasberry and start it. Wait and scan (before and after) the network to find his ip:

nmap -sP 192.168.1.0/24

Or you can connect to your home router and check the dhcp.

Connect to it

ssh pi@YOUR_RASPBERRY_IP  (password: raspberry)

Install docker

sudo apt-get update -y
sudo apt-get dist-upgrade -y
sudo shutdown -r
sudo curl -sSL https://get.docker.com | sh

Dockerfile:

sudo nano /home/pi/Dockerfile
FROM resin/rpi-raspbian:jessie
MAINTAINER gregbkr@outlook.com

RUN apt-get update && \
    apt-get install -y git python-pip build-essential python-dev libffi-dev libssl-dev python-virtualenv

RUN git clone https://github.com/tahoe-lafs/tahoe-lafs
ADD . /tahoe-lafs
RUN \
  cd /tahoe-lafs && \
  git pull --depth=100 && \
  pip install . && \
  rm -rf ~/.cache/

RUN tahoe --version

WORKDIR /root

Build images

sudo docker build -t tahoe-arm .

Add settings:
YOUR_PUBLIC_IP --> at home find your public IP here: https://www.whatismyip.com/what-is-my-public-ip-address/
(mine: 185.31.100.144)

sudo docker run --rm -v /home/pi/.tahoe:/root/.tahoe/ tahoe-arm tahoe create-node --nickname=`hostname` --port=tcp:3457 --location=tcp:<YOUR_PUBLIC_IP>:3457 --introducer=pb://zyadrwufzm34fwquu6oz6ktqu2e4phlg@tcp:185.19.30.78:41464/uqrzlcn5etmnrb5x7rzbhkgq6ctoakrb

Run tahoe storage

sudo docker run -d --name tahoe-storage -p 3457:3457 -v /home/pi/.tahoe:/root/.tahoe/ tahoe-arm tahoe start /root/.tahoe --nodaemon --logfile=-
sudo docker logs tahoe-storage

Activate your home router redirection:

Name            : tahoe
Protocal        : TCP
IP source       : *              <- access from internet
Port source     : 3457           <- port of the router to be redirected internally
IP destination  : 192.168.1.33   <- your raspberry local IP
Port destination: 3457

Test with: (should get a filtered/open state at leat)

nmap YOURPUBLIC_IP_ADDRESS -p 3456 -Pn

2.3 Storage client (zeus)

No backup here, but all secrets here. We will run backup and restore from here. So this container got tahoe-client and duply.

Initialize tahoe-client (if first time) and use the introducer url from your grid:

docker run --rm -v /root/.tahoe:/root/.tahoe/ duply \
    tahoe create-client \
        --nickname=zeus --webport=tcp:3456:interface=0.0.0.0 \
        --introducer=pb://zyadrwufzm34fwquu6oz6ktqu2e4phlg@tcp:185.19.30.78:41464/uqrzlcn5etmnrb5x7rzbhkgq6ctoakrb

nano /root/.tahoe/tahoe.cfg  <-- change happy=2 depending of your number of storage server

Create(/import) tahoe alias+subfolder: we want a structure like backup/test

docker exec duply tahoe create-alias backup
URI:DIR2:pmhosio65ugrucuvyt4uminbnq:7msac6ypv6nl5z33ionhaenzcrqutpycatuzgix7462nafgqtegq   <-- KEEP THIS URI REF, ONLY WAY TO RECOVER YOUR BACKUPS!

And a subfolder:

docker exec duply tahoe mkdir backup:test
URI:DIR2:x5azfaectlywtchb2nddagxxwi:mhk3ehghys3t2oahjelolsppfbpf2iu67uvg2yiv36afqncodgna

To connect to an already existing tahoe alias:

docker exec duply tahoe add-alias backup URI:DIR2:pmhosio65ugrucuvyt4uminbnq:7msac6ypv6nl5z33ionhaenzcrqutpycatuzgix7462nafgqtegq

docker exec duply tahoe ls backup:test

nano /root/.tahoe/private/aliases  <-- name of tahoe backup folder

If empty add existing like:

backup: URI:DIR2:pmhosio65ugrucuvyt4uminbnq:7msac6ypv6nl5z33ionhaenzcrqutpycatuzgix7462nafgqtegq

or create new one:

docker exec duply tahoe create-alias backup     <-- and save the URI/privatekey somewhere save.

Run duply + tahoe-client container:

docker run -d --name duply \
    -v ~/.ssh:/ssh -v /root/.gnupg:/root/.gnupg -v /root/duply/.duply:/root/.duply -v /root/backup:/root/backup \
    --cap-add SYS_ADMIN --device /dev/fuse --privileged \
    -p 3456:3456 \
    -v /root/.tahoe:/root/.tahoe \
    duply

Note: always edit config on the host, and use docker restart duply to refresh container.

3. Duply

Create a test config for only tahoe encryption
Copy/paste an existing one or use new default config :

docker run --rm -v /root/duply:/root duply default create

nano .duply/test/conf
# tahoe
TARGET='tahoe://backup/test'
# base directory to backup
SOURCE='/root/backup/test1'

Run a backup to tahoe

docker exec duply duply test backup --disable-encryption     ( --allow-source-mismatch could be needed if error)

Check backup via Duply

docker exec duply duply test list --disable-encryption

Check file are present in Tahoe
docker exec duply tahoe ls backup:

Or via gui and entering the URI of backup: http://185.19.29.253:3456/

4. All backup profiles

Quick run:

Syntax for running commands will always be the following:

Backup

docker run --rm -v ~/.ssh:/ssh -v /root/.gnupg:/root/.gnupg -v /root/duply/.duply:/root/.duply --cap-add SYS_ADMIN --device /dev/fuse --privileged duply YOUR_BCK_PROFILE backup --allow-source-mismatch

List

docker run --rm -v /root/duply/.duply:/root/.duply duply YOUR_BCK_PROFILE list

restore

docker run --rm -v /root/restore:/root/restore -v /root/duply/.duply:/root/.duply duply YOUR_BCK_PROFILE restore /root/restore/YOUR_CHOICE

4a. Ethereum

Edit conf:

nano /root/duply/.duply/ethereum/conf

# gpg encryption:
GPG_KEY='DC581D8A'   <-- pgp key used
GPG_PW='geneva2016'  <-- pgp password

# s3
TARGET='s3://sos.exo.io/backup-duply-staging/ethereum'  <-- s3 storage
TARGET_USER='w_Y9G_G22bldi9qYhPPLJ7TAEJjwXB9h8H5-CzoxcF-eX41wL0HgEWpm_KjU9Nv7eW6loWRWHJXsfe6gF_zQhQ'      <-- s3 credentials
TARGET_PASS='X_haTCVy6q4UQJBSzyHg0ykPZn_MKX_33aYAYOuqy9HQmBeqmXrqux4hku4PgWCzKJ7r4b1FLKGd4B66kUogsA'

# base directory to backup
SOURCE='/mnt/backup'     <-- for a docker, always the same

## Backup retention: need to use purge to remove old backup. Help: https://zetta.io/en/help/articles-tutorials/backup-linux-duply/
# MAX_AGE=              is how far back in time you want backups to be kept
# MAX_FULL_BACKUPS=     is how many full backups we want to keep
# MAX_FULLS_WITH_INCRS= is how many full backups with incremental backups we want to keep
# MAX_FULLBKP_AGE=      is how long we should to incremental backups before we need to do a full backup

MAX_AGE=12M
#MAX_FULL_BACKUPS=6
#MAX_FULLS_WITH_INCRS=1
MAX_FULLBKP_AGE=1W

DUPL_PARAMS="$DUPL_PARAMS --full-if-older-than $MAX_FULLBKP_AGE "

Will be executed before/pre backup (please change with ip_of_ethereum_vm)

nano /root/duply/.duply/ethereum/pre
#!/bin/bash
mkdir -p /mnt/backup
sshfs -oIdentityFile=/ssh/id_rsa_sbexx root@185.19.30.213:/var/lib/docker/volumes/ethereum/_data/keystore /mnt/backup
ls -la /mnt/backup

Will be executed after/post backup

nano /root/duply/.duply/ethereum/post
#!/bin/bash
umount /mnt/backup

Run a Backup (ssh,--cap-add, --device, --priviledged: is to be able to map ethereum folder to vm zeus, .gnupg: share the host key to container duply )

docker run --rm -v ~/.ssh:/ssh -v /root/.gnupg:/root/.gnupg -v /root/duply/.duply:/root/.duply --cap-add SYS_ADMIN --device /dev/fuse --privileged duply ethereum backup --allow-source-mismatch

List backup

docker run --rm -v /root/.gnupg:/root/.gnupg -v /root/duply/.duply:/root/.duply duply ethereum list

Edit a file and make another backup to see the incremental:

ansible ethereum -a "docker exec ethereum /bin/bash -c 'echo test1 >> /root/.ethereum/keystore/testgreg1'"
ansible ethereum -a "docker exec ethereum cat /root/.ethereum/keystore/testgreg1"
docker run --rm -v ~/.ssh:/ssh -v /root/.gnupg:/root/.gnupg -v /root/duply/.duply:/root/.duply --cap-add SYS_ADMIN --device /dev/fuse --privileged duply ethereum backup --allow-source-mismatch

list backup of x time ago: s, m, h, D, W, M, or Y (indicating seconds, minutes, hours, days, weeks, months, or years respectively) http://duplicity.nongnu.org/duplicity.1.html

docker run --rm -v /root/duply/.duply:/root/.duply duply ethereum list --time 10m
docker run --rm -v /root/duply/.duply:/root/.duply duply ethereum list --time 2W  

List backup collection in tahoe

docker exec duply duplicity collection-status tahoe://backup

restore last version

docker run --rm -v /root/duply/.duply:/root/.duply -v /root/restore:/root/restore duply ethereum restore /root/restore/ethereum

restore version of 10 min ago.

docker run --rm -v /root/duply/.duply:/root/.duply -v /root/restore:/root/restore duply ethereum restore /root/restore/ethereum --time 10m

4.b Bitcoin:

Backup

docker run --rm -v ~/.ssh:/ssh -v /root/.gnupg:/root/.gnupg -v /root/duply/.duply:/root/.duply --cap-add SYS_ADMIN --device /dev/fuse --privileged duply bitcoin backup --allow-source-mismatch

list backup

docker run --rm -v /root/duply/.duply:/root/.duply duply bitcoin list

restore

docker run --rm -v /root/restore:/root/restore -v /root/duply/.duply:/root/.duply duply bitcoin restore /root/restore/bitcoin

4.c DBSLAVE

Backup

docker run --rm -v ~/.ssh:/ssh -v /root/.gnupg:/root/.gnupg -v /root/duply/.duply:/root/.duply --cap-add SYS_ADMIN --device /dev/fuse --privileged duply dbslave backup --allow-source-mismatch

Restore granular: file dbslave-pgdump-2016-06-15.gz from 2DAY ago to /root/restore

docker run --rm -v /root/restore:/root/restore -v /root/duply/.duply:/root/.duply duply dbslave fetch dbslave-pgdump-2016-06-15.gz /root/restore/dbslave-pgdump-2016-06-15.gz --time 2D

Check file:

gunzip < /root/restore/dbslave-pgdump-2016-06-15.GZ > /root/restore/dbslave-pgdump-2016-06-15
tail -n 100 /root/restore/dbslave-pgdump-2016-06-15

4.d MARIADB (piwik, chat)

4.e Bitbucket

Here we need a location to store the code, and only push updates to s3. Otherwise the container always clone the repo and send all to s3 --> no diff.

mkdir /root/backup/bitbucket
docker run --rm -v /root/backup/bitbucket-backup:/mnt/backup -v ~/.ssh:/ssh -v /root/.gnupg:/root/.gnupg -v /root/duply/.duply:/root/.duply --cap-add SYS_ADMIN --device /dev/fuse --pr ivileged duply bitbucket backup --allow-source-mismatch

4. f kyc

We will sync s3 kyc butcket to local (only new files), and send the to s3. Need local too in order to send only the diff.

nano /root/duply/.duply/exo.s3cfg   <-- add bucket credential
nano /root/duply/.duply/kyc/pre     <-- edit the source butcket
mkdir /root/backup/bitbucket
docker run --rm -v /root/backup/kyc:/mnt/backup -v ~/.ssh:/ssh -v /root/.gnupg:/root/.gnupg -v /root/duply/.duply:/root/.duply --cap-add SYS_ADMIN --device /dev/fuse --privileged dupl y kyc backup --allow-source-mismatch

4.f Create new profile:

Copy one profile and edit the vars.

cp -r /root/duply/.duply/ethereum /root/duply/.duply/mynewprofile

4.g Amazon s3 profile:

# s3
TARGET='s3://s3-us-west-2.amazonaws.com/backup-duply-staging/ethereum'
TARGET_USER='AKIAJHU6NG3JNP6LR2QA'
TARGET_PASS='KUgmxaZJha6XJmYiE3DnUQEYEZih9dQS3RO1Vclv'

5. Maintenance

5.1 TAHOE

To reconstruct the backup, if one node goes offline, or one new node get added: (run every day by cron)

docker exec duply tahoe deep-check --repair backup:

5.2 Crontab

#duply: backups
00 01 * * * /usr/bin/docker run --rm -v ~/.ssh:/ssh -v /root/.gnupg:/root/.gnupg -v /root/duply:/root --cap-add SYS_ADMIN --device /dev/fuse --privileged duply ethereum backup --allow-source-mismatch | tee --append /root/script/duply-ethereum.log > /dev/null 2>/dev/null
02 01 * * * /usr/bin/docker run --rm -v ~/.ssh:/ssh -v /root/.gnupg:/root/.gnupg -v /root/duply:/root --cap-add SYS_ADMIN --device /dev/fuse --privileged duply bitcoin backup --allow-source-mismatch | tee --append /root/script/duply-bitcoin.log > /dev/null 2>/dev/null
05 01 * * * /usr/bin/docker run --rm -v ~/.ssh:/ssh -v /root/.gnupg:/root/.gnupg -v /root/duply:/root --cap-add SYS_ADMIN --device /dev/fuse --privileged duply media backup --allow-source-mismatch | tee --append /root/script/duply-media.log > /dev/null 2>/dev/null
10 01 * * * /usr/bin/docker run --rm -v ~/.ssh:/ssh -v /root/.gnupg:/root/.gnupg -v /root/duply:/root --cap-add SYS_ADMIN --device /dev/fuse --privileged duply mariadb backup --allow-source-mismatch | tee --append /root/script/duply-mariadb.log > /dev/null 2>/dev/null
15 01 * * * /usr/bin/docker run --rm -v ~/.ssh:/ssh -v /root/.gnupg:/root/.gnupg -v /root/duply:/root --cap-add SYS_ADMIN --device /dev/fuse --privileged duply blog backup --allow-source-mismatch | tee --append /root/script/duply-blog.log > /dev/null 2>/dev/null
20 01 * * * /usr/bin/docker run --rm -v ~/.ssh:/ssh -v /root/.gnupg:/root/.gnupg -v /root/duply:/root --cap-add SYS_ADMIN --device /dev/fuse --privileged duply dbslave backup --allow-source-mismatch | tee --append /root/script/duply-dbslave.log > /dev/null 2>/dev/null
30 01 * * * /usr/bin/docker run --rm -v ~/.ssh:/ssh -v /root/.gnupg:/root/.gnupg -v /root/duply:/root --cap-add SYS_ADMIN --device /dev/fuse --privileged -v /root/backup/bitbucket-backup:/mnt/backup duply bitbucket backup --allow-source-mismatch | tee --append /root/script/duply-bitbucket.log > /dev/null 2>/dev/null
40 01 * * * /usr/bin/docker run --rm -v ~/.ssh:/ssh -v /root/.gnupg:/root/.gnupg -v /root/duply:/root --cap-add SYS_ADMIN --device /dev/fuse --privileged -v /root/backup/kyc:/mnt/backup duply kyc backup --allow-source-mismatch | tee --append /root/script/duply-kyc.log > /dev/null 2>/dev/null

01 02 * * * /usr/bin/docker run --rm -v ~/.ssh:/ssh -v /root/.gnupg:/root/.gnupg -v /root/duply:/root --cap-add SYS_ADMIN --device /dev/fuse --privileged duply ethereum-aws backup --allow-source-mismatch | tee --append /root/script/duply-ethereum-aws.log > /dev/null 2>/dev/null

A. To encrypt with pgp keys the backup: (not needed as tahoe already encrypt)

Use the provided one for test:

gpg --import /root/duply/gpgtest.DC581D8A.pub.asc.export
gpg --import /root/duply/gpgtest.DC581D8A.pri.asc.export

Or create new one for prod:

gpg --gen-key
for exemple: name:backuptest, email:backuptest@bity.com, pw: geneva2016

Backup your keys: save in a save place these 2 files and folder:

gpg --export-secret-keys -a keyid > backup_private_key.asc
gpg --export -a keyid > backup_public_key.asc
~/.gnupg

trust the key

gpg --edit-key DC581D8A

trust

5 (select 5 if you ultimately trust the key)

y

Check the key

gpg --list-key

------------------------------
pub   2048R/DC581D8A 2016-03-29
uid                  backuptest <backuptest@bity.com>
sub   2048R/B380730B 2016-03-29

Need to backup

  • DBslave(pg_dump) ok
  • bitcoin ok
  • ethereum ok
  • Backend Media folder : backendadmin:/root/media ok
  • chat, piwik --> mariadb ok
  • bitbucket : ok

  • elk

  • KYC
Docker Pull Command
Owner
sbex
Source Repository

Comments (0)