Public | Automated Build

Last pushed: 5 months ago
Short Description
The zero knowledge backup for CloudFleet
Full Description

#+TITLE: Parachute: a zero knowledge backup system

  • Installation

Running

#+NAME: Basic installation of parachute in one command

#+BEGIN_SRC shell
bash -x ./setup/install-parachute.bash

#+END_SRC
with the appropiate permissions should enable all the following
subclauses to be met automatically
Historical
Install instructions in detail if ~./setup/install-parachute.bash~ doesn't work
*
Place chute.asd in ASDF3 system registry

#+NAME: Configuration of ASDF3 system registry to find blimp-parachute

#+BEGIN_SRC shell
bash -x ./chute/configure-asdf.bash

#+END_SRC

*** Loading from ASDF

#+BEGIN_SRC lisp
(asdf:load-system :chute)

#+END_SRC

*** Testing from ASDF

#+BEGIN_SRC lisp
(asdf:test-system :chute)

#+END_SRC

*** Loading Quicklisp dependencies

  1. From the shell, download the Quicklisp install shim.
    #+BEGIN_SRC shell
    wget https://beta.quicklisp.org/quicklisp.lisp
    #+END_SRC

  2. From a Common Lisp implementation, execute the shim.
    #+BEGIN_SRC lisp
    (load "~/quicklisp.lisp")
    (quicklisp-quickstart:install)
    (ql:add-to-init-file)
    #+END_SRC

  3. Install the Quicklisp packages needed by chute.
    #+BEGIN_SRC lisp
    (load (asdf:system-relative-pathname
    (asdf:find-system :chute) "quicklisp-setup.lisp"))
    #+END_SRC

  • Dockerification
    To build an image on arm32 with Docker installed:
    #+BEGIN_SRC shell
    bash make-arm32-dockerfile.bash
    sudo docker build --file Dockerfile-arm32 .
    #+END_SRC
  • REST API
    ** Current version 'v1'
    #+NAME: Fundamental Chute REST API
    #+BEGIN_EXAMPLE
    --> POST /chute/blob/
    (index.json)
    <-- 201 Resource Created or [45]xx Error
    ("/new/uri/to/use")
-->  GET /chute/blob/<URI>/index.json              
nil
<--  200 Original or 304 Not Modified or [45]xx Error
(index.json) or nil


-->  PUT /chute/blob/<URI>/0    
(application/octet-bytes)
<--  201 Ok
(json) "true" or "false" 


-->  GET /chute/blob/<URI>/0
<--  200 
(json) SHA256 Hash

#+END_EXAMPLE

Authentication to be handled via HTTP Headers submitted to
spire.marina.io API.

** Basic transfer of opaque binary data

PUT with 'Byte-range' headers n bytes in m windows. Use timing of
previous sessions to dynamically expand/contract number of bytes in
each PUT request.

Each chunk of a blob is adressed as a URI of the form:

.../<domain>/<node>/<mount-path>/<timestamp>/<n-byte-chunks>/<mth-chunk>

MIME type for PUT content is "application/octet-stream".

  • Platform
    ~parachute~ provides both a client and a server implementation. All
    nodes boot can potentially boot either.

A local management REST API is always available bound to ~tcp4:2002~
Client platform
A Common Lisp implementation on ~x86_64~, ~arm32~, or ~arm64~.
*
CCL
Most efficient runtime on arm32 platform
http://trac.clozure.com/ccl/wiki/PlatformNotes
SBCL ABCL
Server platform
A Common Lisp implementation running on ~x86_64~.
*
SBCL
~sbcl-1.3.1~ or later known to work
CCL
~ccl-1.11~
ABCL
~abcl-1.5.0~

  • What is to be DONE
    ** DONE BTRFS volume must have ".snapshot/" directory
    CLOSED: [2015-12-14 Mon 13:55]
    • CLOSING NOTE [2015-12-14 Mon 13:55] \
      Should have been completed in the setup procedure. File issue if
      found to be otherwise.
      The CHUTE:SNAPSHOT command will create snapshots under a the specified
      volume (by default "/opt/cloudfleet/data") in a sub-directory named
      ".snapshot/", i.e. for the default in
      "/opt/cloudfleet/data/.snapshot/<TIMESTAMP>".

Therefore, the "/opt/cloudfleet/data/.snapshot/" must exist and be
owned by root. Currently this is done once per node as part of
installation when "setup/add-subvolume.bash" is executed. It should
be made part of the setup done by setting up the encrypted volumes.

** DONE make a hard link to btrfs, setuid
CLOSED: [2015-10-27 Tue 12:44]

  • CLOSING NOTE [2015-10-27 Tue 12:44] \
    Done as setup/make-suid-btfs.bash

BTRFS/SEND
*
DONE stdout/stderr mixing
CLOSED: [2015-11-14 Sat 10:16]

- CLOSING NOTE [2015-11-14 Sat 10:16] \\
  Fixed by specifying separate error/output arguments to CCL:RUN-PROGAM

SBCL/CCL returning different starts of output stream. CCL contains
"At subvolume" which means it is mixing stdout/stderr?

*** DONE Use octet streams
CLOSED: [2015-11-19 Thu 10:47]

- CLOSING NOTE [2015-11-19 Thu 10:47] \\
  Resolved by use of octets streams directly on MAKE-BLOB

We should pass the streams we wish to read from, rather than having
RUN-PROGAM construct them for us.

Unfortunately, this doesn't seem to work:

#+BEGIN_SRC lisp
(ccl:run-program
"/bin/ls" nil
:wait nil
:output (ironclad:make-octet-input-stream
(make-array 16 :element-type '(unsigned-byte 8)))
:element-type '(unsigned-byte 8))

#+END_SRC

complaining about

#+BEGIN_EXAMPLE
There is no applicable method for the generic function:

#<STANDARD-GENERIC-FUNCTION STREAM-WRITE-VECTOR #x30200006518F>
when called with arguments:
(#<IRONCLAD::OCTET-INPUT-STREAM #x3020023CA75D> "chute
LICENSE
parachute.org
setup
...

#+END_EXAMPLE

Fortunately, CCL:RUN-PROGRAM takes an :ELEMENT-TYPE keyword which
allows us to get octet streams out of the subprocesses.

** DONE parse id for PUT uri from POST
CLOSED: [2015-12-14 Mon 14:09]

  • CLOSING NOTE [2015-12-14 Mon 14:09] \
    Finished with proper implementation of POST followed by subsequent
    PUTs to the unspecified sub-uri namespace.
    TODO Read key from key device
    Propsoal: extract appropiate number of bytes from SHA256 of the LuKS key
    TODO Initialize nonce from random data
    Done. But needs to be verified via a test.
    ** TODO Read domain from /opt/cloudfleet/data/config/blimp-vars.sh

Best implementation: exec a bash process then read its environment table
** DONE MAKE-NEW-DIRECTORY
CLOSED: [2015-12-13 Sun 10:40]

  • CLOSING NOTE [2015-12-13 Sun 10:40] \
    Use CL-FAD routines instead of ASDF.
    Fails until run from the REPL. Unsure what this entails.
    ** TODO Verify basic transfer
    Basic transfer of backups needs to be implemented completely and
    tested:

*** REST Transfer Implementation
Initial mplementation completed. Mocks in place for many other
systems.

*** Tests of transfer integrity

** CHUTE.TEST::TRANSFER.BLOB.1
Transform a given file into a blob

** CHUTE.TEST::TRANSFER.BLOB.2

Use results of BTRFS/SEND into a blob

** TODO Future interface for subaddressing components of a blob
For resumable transfers

#+NAME: REST for resumable transfers

#+BEGIN_SRC

            PUT /<URI>/0/<chunk-bytes>/<nth-chunk>
            ->>   201 on success or [345]00 
              (json) "true" or "false"

            GET /<URI>/0/<chunk-bytes>/<nth-chunk>/hash/sha256  
            ->>   20x [345]xx (does 314 make sense?)
              (json) SHA256 Hash

#+END_SRC

** (at first without byte ranges).
Implementation use HTTP 'Byte-range' header to files attached/detatched via mmap().

TODO Tests for use cases TODO Metadata transcriptions
* URI Scheme for previous link TODO Generalize filesytem snapshoting abstraction
Implement ZFS.

Docker Pull Command
Owner
cloudfleet
Source Repository