andrioni/sql-runner repository overview

⁠SQL Runner

[ ![Build Status] travis-image⁠ ] travis⁠ [ ![Release] release-image⁠ ] releases⁠ [ license-image⁠ ] license⁠

⁠Overview

Run playbooks of SQL scripts in series and parallel on Amazon Redshift and PostgreSQL.

Used with Snowplow⁠ snowplow⁠ for scheduled SQL-based transformations of event stream data.

⁠User Quickstart

Assuming you are running on 64bit Linux:

> wget http://dl.bintray.com/snowplow/snowplow-generic/sql_runner_0.1.0_linux_amd64.zip
> unzip sql_runner_0.1.0_linux_amd64.zip
> ./sql-runner -usage

See the User Guide section below for more.

⁠Developer Quickstart

⁠Building

Assuming git, [Vagrant] vagrant-install⁠ and [VirtualBox] virtualbox-install⁠ installed:

 host> git clone https://github.com/snowplow/sql-runner
 host> cd sql-runner
 host> vagrant up && vagrant ssh
guest> cd /opt/gopath/src/github.com/snowplow/sql-runner
guest> godep go build

⁠Testing

Assuming Building complete:

guest> psql -c 'create database sql_runner_tests_1' -U postgres
guest> psql -c 'create database sql_runner_tests_2' -U postgres
guest> ./sql-runner -playbook ./integration-tests/postgres.yml

⁠Publishing

Assuming Travis⁠ travis⁠ is green and versions updated:

guest> godep go build
 host> vagrant push

⁠User guide

⁠Playbooks

A playbook consists of one of more steps, each of which consists of one or more queries. Steps are run in series, queries are run in parallel within the step.

Each query contains the path to a query file. See Query files for details

All steps are applied against all targets. All targets are processed in parallel.

For the playbook template see: [config/config.yml.sample] example-config⁠

⁠Query files

A query file contains one or more SQL statements. These are executed "raw" (i.e. not in a transaction) in series by SQL Runner.

If the query file is flagged as a template in the playbook, then the file is pre-processed as a template before being executed. See Templates for details

⁠Templates

Templates are run through Golang's [text template processor] go-text-template⁠. The template processor can access all variables defined in the playbook.

The following custom functions are also supported:

• nowWithFormat [timeFormat] - where timeFormat is a valid Golang [time format] go-time-format⁠

For an example query file using templating see: [integration-tests/postgres-sql/good/3.sql] example-query⁠

⁠Failure modes

If a statement fails in a query file, the query will terminate and report failure.

If a query fails, its sibling queries will continue running, but no further steps will run.

Failures in one target do not affect other targets in any way.

⁠Copyright and license

SQL Runner is copyright 2015 Snowplow Analytics Ltd.

Licensed under the [Apache License, Version 2.0] license⁠ (the "License"); you may not use this software except in compliance with the License.

Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.