A toolbox for devops scripts
To build the container, run:
docker build .
Note that the deploy scripts do not actually run deploys, but rather generate metadata *.properties files for Jenkins jobs/other scripts to use to run deploys.
To generate properties files for GCE hosts of a particular environment, configure the environment with:
PROJECT: either "kdux" or "firecloud". Can be configured for other groups.SERVICE: the name of the service to deploy (should be a tag on the host)ENVIRONMENT: environment to deploy toGOOGLE_PROJ: name of google project where SERVICE is hostedSERVICE_ACCT: a service account filepath that has read rights in the Google project. Should be a json file mounted in working directory.TAGS [optional]: path to a json file of tag_key -> tag to filter gce instances onNOT_TAGS [optional]: comma separated string of tags to exclude gce instances onexample tags file
{
"ENV": "dev",
"VERSION": "v1"
}
Then run:
docker run --rm -v $PWD:/app \
-e OUTPUT_DIR=/app/properties \
-e ENVIRONMENT=$ENVIRONMENT \
-e GOOGLE_PROJ=$GOOGLE_PROJ \
-e SERVICE_ACCT=/app/$SERVICE_ACCT \
-e TAGS=$TAGS \
-e NOT_TAGS=$NOT_TAGS \
broadinstitute/dsp-toolbox deploy $PROJECT $SERVICE
Will write out properties to the OUTPUT_DIR directory for every host in GOOGLE_PROJ tagged with SERVICE, ENVIRONMENT, and any other TAGS specified.
Firecloud has its own way of detecting changes to services on the environment branches (dev, alpha, staging, and prod)
and deploying only those changes. You can run the deploy binary specifying only an ENV, not a SERVICE:
docker run --rm -v $FC_DEVELOP:/app \
-e OUTPUT_DIR=/app/properties \
-e ENV=$ENV \
-e GOOGLE_PROJ=$GOOGLE_PROJ \
-e SERVICE_ACCT=/app/$SERVICE_ACCT \
-e TAGS=$TAGS \
broadinstitute/dsp-toolbox deploy firecloud
You can also run a firecloud deploy as detailed above, passing a specific SERVICE name (i.e. deploy firecloud rawls).
To run for all projects regardless of scm changes, pass in the keyword all: deploy firecloud all.
To perform these operations, you will need a valid service account to authenticate with the FiaB allocator, as well as the URL of the allocator.
For dev and qa FiaBs this is https://fiab-allocator.dsp-techops.broadinstitute.org, and a service account in the
broad-dsde-dev project or broad-dsde-qa project with "Project Editor" permissions can be used to authenticate.
With python:
python -m src.main.ops.fiab --command create --user <username> --project <google-proj> --allocator <allocator url> --serviceAcct <service acct filepath> --diskSize <(optional)> --machineType <(optional)>
With docker:
docker run --rm -v $PWD:/app \
-e ALLOCATOR_URL=$ALLOCATOR_URL \
-e SERVICE_ACCT=$SERVICE_ACCT \
-e GOOGLE_PROJ=$GOOGLE_PROJ \
-e USERNAME=$USERNAME \
-e DISK_SIZE=100 \
-e MACHINE_TYPE=n1-standard-1 \
broadinstitute/dsp-toolbox fiab create
(DISK_SIZE and MACHINE_TYPE are optional and default to the above specs)
With python:
python -m src.main.ops.fiab --command stop --allocator <allocator url> --serviceAcct <service acct filepath> --hostName <host to stop> --project <google-proj>
With docker:
docker run --rm -v $PWD:/app \
-e ALLOCATOR_URL=$ALLOCATOR_URL \
-e SERVICE_ACCT=$SERVICE_ACCT \
-e GOOGLE_PROJ=$GOOGLE_PROJ \
-e FIAB_HOST=$FIAB_HOST \
broadinstitute/dsp-toolbox fiab stop
With python:
python -m src.main.ops.fiab --command restart --allocator <allocator url> --serviceAcct <service acct filepath> --hostName <host to restart> --project <google-proj>
With docker:
docker run --rm -v $PWD:/app \
-e ALLOCATOR_URL=$ALLOCATOR_URL \
-e SERVICE_ACCT=$SERVICE_ACCT \
-e GOOGLE_PROJ=$GOOGLE_PROJ \
-e FIAB_HOST=$FIAB_HOST \
broadinstitute/dsp-toolbox fiab restart
With python:
python -m src.main.ops.fiab --command reset --allocator <allocator url> --serviceAcct <service acct filepath> --hostName <host to reset> --project <google-proj>
With docker:
docker run --rm -v $PWD:/app \
-e ALLOCATOR_URL=$ALLOCATOR_URL \
-e SERVICE_ACCT=$SERVICE_ACCT \
-e GOOGLE_PROJ=$GOOGLE_PROJ \
broadinstitute/dsp-toolbox instance reset <instance-name>
With python:
python -m src.main.ops.fiab --command destroy --allocator <allocator url> --serviceAcct <service acct filepath> --hostName <host to destroy> --project <google-proj>
With docker:
docker run --rm -v $PWD:/app \
-e ALLOCATOR_URL=$ALLOCATOR_URL \
-e SERVICE_ACCT=$SERVICE_ACCT \
-e GOOGLE_PROJ=$GOOGLE_PROJ \
-e FIAB_HOST=$FIAB_HOST \
broadinstitute/dsp-toolbox fiab destroy
To list all fiab hosts in a google project and (optionally) filter on a given tag.
Will return a json list of InstanceName and IP for each instance.
With python:
python -m src.main.ops.fiab --command list --allocator <allocator url> --serviceAcct <service acct filepath> --project <(optional) google-proj> --hostName <(optional) host to find>
With docker:
docker run --rm -v $PWD:/app \
-e ALLOCATOR_URL=$ALLOCATOR_URL \
-e SERVICE_ACCT=$SERVICE_ACCT \
-e GOOGLE_PROJ=$GOOGLE_PROJ \
-e FIAB_HOST=$FIAB_HOST \
broadinstitute/dsp-toolbox fiab list
where GOOGLE_PROJ and FIAB_HOST are optional variables.
To list all submodules with a common branch name.
With python:
python -m src.main.github.get_submodule_dependencies $BRANCH $GITHUB_TOKEN
With docker:
docker run --rm -e GITHUB_BRANCH=$GITHUB_BRANCH
-e GITHUB_TOKEN=$GITHUB_TOKEN
broadinstitute/dsp-toolbox github dependencies
This is used in setting up a FiaB for automated tests on PR.
This is used to update the submodule hash in firecloud-develop after a change has been pushed to the develop branch of one of the Firecloud services.
docker run --rm -e GITHUB_TOKEN=$GITHUB_TOKEN \
-e ENV=$ENV \
-e PROJECT=$PROJECT \
-e GIT_SHA=$GIT_SHA \
broadinstitute/dsp-toolbox:latest update-firecloud-service
where PROJECT is the service to update and GIT_SHA is the hash of the commit. Needs a valid GITHUB_TOKEN to authenticate.
Used in the update-submodule-hash Jenkins job.
Content type
Image
Digest
Size
798.2 MB
Last updated
over 4 years ago
docker pull broadinstitute/dsp-toolbox