broadinstitute/dsde-toolbox

docker

Hub

By broadinstitute

•Updated 8 months ago

DSDE Toolbox Docker container

Image

500K+

Overview Tags

DSDE Toolbox

Before you begin, ensure:

You can visit https://hub.docker.com/r/broadinstitute/dsde-toolbox/ while logged in.
You install Docker⁠ and start it on your machine
You are logged into dockerhub on your command line via docker login.
You are on Broad-Internal wifi or VPN

Authenticating to Vault

New instructions! See the document here⁠.

We're recommending folks install the vault executable directly, because using it from dsde-toolbox can have performance and compatibility issues. If you're happy using dsde-toolbox, no need to switch! This is mainly for new folks.

Your Vault auth is the same under the hood, so you can use either vault or docker run -it --rm -v $HOME:/root broadinstitute/dsde-toolbox:dev vault interchangeably. You might still want to use dsde-toolbox for the useful vault-edit helper script below.

The old instructions are here:

Generate a GitHub Personal access token⁠ by going to this page in your Github profile⁠ to Authenticate with Vault
Make sure read:org is the only scope which is selected. It's necessary to read the org teams.
Authenticate to vault using the command below (this will create a .vault_token file in you home directory). Once authenticated, you can delete your Github access token but you will need another one next time you need to auth.

docker run -it --rm \
    -v $HOME:/root:rw \
    broadinstitute/dsde-toolbox:dev vault login -method=github token=<<GITHUB TOKEN VALUE>>
Note: The command above requires you to replace <<GITHUB TOKEN VALUE>> with the plaintext value of the token, not a file path. If you've saved the token to a file, or otherwise do not want to write the token in plaintext on the command line, you can instead use:
[...] -method=github token=$(cat <<PATH TO GITHUB TOKEN FILE>>)

Using vault to retrieve secrets

docker run -it --rm \
    -v $HOME:/root \
    broadinstitute/dsde-toolbox:dev vault read secret/path/to/secret

Editing vault secrets

docker run -it --rm \
    -v $HOME:/root \
    broadinstitute/dsde-toolbox:dev vault-edit secret/path/to/secret

Converting a binary file to a format suitable for a Vault secret

Note: Vault token is not required for this, so you don't need to mount your home directory in for this example, but /working directory is a helpful place to put the binary file to be converted.

docker run -it --rm \
    -v "$PWD":/working \
    broadinstitute/dsde-toolbox:dev binary2string.sh /working/foo.binary

The file will be stored in a new file with the .b64 extension after running this command, so in the example above that would be /working/foo.binary.b64.

Creating a set of configs

docker run -it --rm \
    -v $HOME:/root \
    -v "$PWD":/working \
    broadinstitute/dsde-toolbox:dev render-templates.sh local

This will render the configuration templates in ./src/main/config to ./target/config.

Note: Any template file with the extension .p12.ctmpl or .jks.ctmpl will have its contents from Vault base64 decoded before being written to the destination file. This allows us to store binary files in Vault.

Getting info about your current Vault token

docker run -it --rm \
    -v $HOME:/root \
    broadinstitute/dsde-toolbox:dev tokenInfo

This will take the current Vault token that you have authenticated with and passed in via your $HOME directory and query Vault for information about that token.

Getting info about your an arbitrary Vault token

docker run -it --rm \
    -v $HOME:/root \
    broadinstitute/dsde-toolbox:dev tokenInfo some-vault-token-xxx-yyy

This will take the Vault token passed as a command-line parameter and use your Vault token (passed in via your $HOME directory) to query Vault for information about that token.

Note: This requires admin privileges, so not all users can use this function.

Connecting to a database

docker run -it --rm
   -v $HOME:/root \
   broadinstitute/dsde-toolbox:dev mysql-connect.sh -p firecloud -e $ENV -a $SERVICE

Where ENV is a firecloud environment (dev, alpha, staging, prod, qa, perf), and SERVICE is a firecloud service (i.e. rawls, consent, etc). This will connect you to the CloudSQL database in google.

Rebuilding this image

docker build -t broadinstitute/dsde-toolbox:dev .

Viewing Workbench Status / Version Dashboard

In your terminal, minimize the font size by pressing Command+- three to four times. This will help you view the whole dashboard at once.

docker run --rm -it -e TZ="America/New_York" broadinstitute/dsde-toolbox:dev status

To see the status of individual instances in an environment for which you have permissions, run the following command.

docker run --rm -it -v ${HOME}/.vault-token:/root/.vault-token dashboard envStatus <staging | perf | alpha | dev | prod>

Make sure to choose one environment, for example [...] envStatus perf.

Docker Pull Command

docker pull broadinstitute/dsde-toolbox