jkingyens/builder repository overview

⁠builder

build from a DAG

⁠requirements

• docker cloud account with some nodes
• ssl certificates and at least one domain name
• traefik container running on docker cloud with certs loaded
• forked version of the 'builder' repo on your github account

⁠setup / bootstrap

• connect builder github repo to your docker cloud
• auto build the the builder account
• deploy a stack from the builder image to a particular node
• load your github token and docker cloud token directly onto this node securely
  • github.token and docker.token files
• set environment variable BUILDER_URL_ROOT to: https://domain.com/builder⁠
• setup trafek configuration variables to proxy this HOST into the builder container
• visit https://domain.com/builder⁠ and you should see "okay" to confirm everything is setup

⁠usage

• create and deploy build scripts graphically in the web browser
• load https://domain.com/builder⁠
• click new build
• click add node -> github
• type in https://github.com/jkingyens/slinky⁠
• a node will appear, add a new source to it, that defines something else
• instead of github, choose google cloud credentials
• you will now be redirected to google to sign into your account
• you will grant builder access to furnish json service account files on your behalf
• the builder UI will then, using this API connection, create such files, load them into a docker container for storage, and connect them with the dpeloyment
• the deployment is saved when the node is added and the entire thing is run through, it tests for errors along the way and validates each pipe

this file defines a DAG like thing that connects container jobs together via filesystem entry points. our main use case is:

1. app repo containing the source for a react app -> docker container volume
2. the active builder container, imports the app container volume, runs the build, and produces the output in build/
3. the output container has the resulting build in it which is input to an nginx persistent container.

what is important here?

• proper management of sources and sinks
• the builder is always listening to the source waiting for new data
• the builder respects the dag and invokes containers on new data
• the builder understands the daemon and manages any back pressure

neat things:

• the intermediate container nodes come from the community (dont re-invent the wheel)
• eventually we can handle security the right way, not using environment variables or things like that
• we could build native support for collections that allows sinks to not build up pressure

what is the minimum spec we need?

• it needs to have a concept of a stream? is this like node.js streams? or something else?

• it needs to be recursive out of the box: streams are a first class data type as a volume

  • a filesystem with a single JSON file that defines a valid stream

• it needs to reference volumes outside its own runtime/process (via the docker API)

• it needs to be backed by a database of some kind (rethinkdb)

• it needs to work with github sources and volumes out of the box

• it needs to store and forward intermediate volumes in docker containers

• it needs to support collection types (the notion of an immutable stream sink)

• it needs to support android and ios device sinks (how do we model device instances?)

• it needs to support mounting volumes into URL spaces (map a volume to an HTTP) address

• what does the first test case look like?

• grabs volume from a github repo, mounts it inside a builder github repo, mounts it into the URL space

• is this just continuous integration and deployment?

• is the app itself self aware of the URL space and mapping?