urbica/martin repository overview

⁠Martin

Martin is a PostGIS⁠ vector tiles⁠ server suitable for large databases. Martin is written in Rust⁠ using Actix⁠ web framework.

• Requirements⁠
• Installation⁠
• Usage⁠
• API⁠
• Using with Mapbox GL JS⁠
• Using with Leaflet⁠
• Using with deck.gl⁠
• Table Sources⁠
  • Table Sources List⁠
  • Table Source TileJSON⁠
  • Table Source Tiles⁠
• Composite Sources⁠
  • Composite Source TileJSON⁠
  • Composite Source Tiles⁠
• Function Sources⁠
  • Function Sources List⁠
  • Function Source TileJSON⁠
  • Function Source Tiles⁠
• Command-line Interface⁠
• Environment Variables⁠
• Configuration File⁠
• Using with Docker⁠
• Using with Docker Compose⁠
• Using with Nginx⁠
• Building from Source⁠
• Debugging⁠
• Development⁠

⁠Requirements

Martin requires PostGIS >= 2.4.0.

⁠Installation

You can download martin from Github releases page⁠.

If you are using macOS and Homebrew⁠ you can install martin using Homebrew tap.

brew tap urbica/tap
brew install martin

You can also use official Docker image⁠

docker run -p 3000:3000 -e DATABASE_URL=postgres://postgres@localhost/db urbica/martin

⁠Usage

Martin requires a database connection string. It can be passed as a command-line argument or as a DATABASE_URL environment variable.

martin postgres://postgres@localhost/db

Martin provides TileJSON⁠ endpoint for each geospatial-enabled⁠ table in your database.

⁠API

⁠Using with Mapbox GL JS

Mapbox GL JS⁠ is a JavaScript library for interactive, customizable vector maps on the web. It takes map styles that conform to the Mapbox Style Specification⁠, applies them to vector tiles that conform to the Mapbox Vector Tile Specification⁠, and renders them using WebGL.

You can add a layer to the map and specify martin TileJSON endpoint as a vector source URL. You should also specify a source-layer property. For Table Sources⁠ it is {schema_name}.{table_name} by default.

map.addLayer({
  id: "public.points",
  type: "circle",
  source: {
    type: "vector",
    url: "http://localhost:3000/public.points.json",
  },
  "source-layer": "public.points",
  paint: {
    'circle-color': 'red',
  },
});

You can also combine multiple tables into one source with Composite Sources⁠. Each Table Source⁠ in Composite Source can be accessed with its {schema_name}.{table_name} as a source-layer property.

map.addSource("points", {
  type: "vector",
  url: `http://0.0.0.0:3000/public.points1,public.points2.json`,
});

map.addLayer({
  id: "red_points",
  type: "circle",
  source: "points",
  "source-layer": "public.points1",
  paint: {
    "circle-color": "red",
  },
});

map.addLayer({
  id: "blue_points",
  type: "circle",
  source: "points",
  "source-layer": "public.points2",
  paint: {
    "circle-color": "blue",
  },
});

⁠Using with Leaflet

Leaflet⁠ is the leading open-source JavaScript library for mobile-friendly interactive maps.

You can add vector tiles using Leaflet.VectorGrid⁠ plugin. You must initialize a VectorGrid.Protobuf⁠ with a URL template, just like in L.TileLayers. The difference is that you should define the styling for all the features.

L.vectorGrid
  .protobuf('http://localhost:3000/public.points/{z}/{x}/{y}.pbf', {
    vectorTileLayerStyles: {
      'public.points': {
        color: 'red',
        fill: true,
      },
    },
  })
  .addTo(map);

⁠Using with deck.gl

deck.gl⁠ is a WebGL-powered framework for visual exploratory data analysis of large datasets.

You can add vector tiles using MVTLayer⁠. MVTLayer data property defines the remote data for the MVT layer. It can be

• String: Either a URL template or a TileJSON⁠ URL.
• Array: an array of URL templates. It allows to balance the requests across different tile endpoints. For example, if you define an array with 4 urls and 16 tiles need to be loaded, each endpoint is responsible to server 16/4 tiles.
• JSON: A valid TileJSON object⁠.

const pointsLayer = new MVTLayer({
  data: 'http://localhost:3000/public.points.json', // 'http://localhost:3000/public.table_source/{z}/{x}/{y}.pbf'
  pointRadiusUnits: 'pixels',
  getRadius: 5,
  getFillColor: [230, 0, 0]
});

const deckgl = new DeckGL({
  container: 'map',
  mapStyle: 'https://basemaps.cartocdn.com/gl/dark-matter-gl-style/style.json',
  initialViewState: {
    latitude: 0,
    longitude: 0,
    zoom: 1
  },
  layers: [pointsLayer]
});

⁠Table Sources

Table Source is a database table which can be used to query vector tiles⁠. When started, martin will go through all spatial tables in the database and build a list of table sources. A table should have at least one geometry column with non-zero SRID. All other table columns will be represented as properties of a vector tile feature.

⁠Table Sources List

Table Sources list endpoint is available at /index.json

curl localhost:3000/index.json

Note: if in watch mode, this will rescan database for table sources.

⁠Table Source TileJSON

Table Source TileJSON⁠ endpoint is available at /{schema_name}.{table_name}.json.

For example, points table in public schema will be available at /public.points.json

curl localhost:3000/public.points.json

⁠Table Source Tiles

Table Source tiles endpoint is available at /{schema_name}.{table_name}/{z}/{x}/{y}.pbf

For example, points table in public schema will be available at /public.points/{z}/{x}/{y}.pbf

curl localhost:3000/public.points/0/0/0.pbf

⁠Composite Sources

Composite Sources allows combining multiple Table Sources into one. Composite Source consists of multiple Table Sources separated by comma {schema_name1}.{table_name1},...,{schema_nameN}.{table_nameN}

Each Table Source⁠ in Composite Source can be accessed with its {schema_name}.{table_name} as a source-layer property.

⁠Composite Source TileJSON

Composite Source TileJSON⁠ endpoint is available at /{schema_name1}.{table_name1},...,{schema_nameN}.{table_nameN}.json.

For example, composite source for points and lines tables in public schema will be available at /public.points,public.lines.json

curl localhost:3000/public.points,public.lines.json

⁠Composite Source Tiles

Composite Source tiles endpoint is available at /{schema_name1}.{table_name1},...,{schema_nameN}.{table_nameN}/{z}/{x}/{y}.pbf

For example, composite source for points and lines tables in public schema will be available at /public.points,public.lines/{z}/{x}/{y}.pbf

curl localhost:3000/public.points,public.lines/0/0/0.pbf

⁠Function Sources

Function Source is a database function which can be used to query vector tiles⁠. When started, martin will look for the functions with a suitable signature. A function that takes z integer, x integer, y integer, and query_params json and returns bytea, can be used as a Function Source.

Hint: You may want to use TileBBox⁠ function to generate bounding-box geometry of the area covered by a tile.

For example, if you have a table public.table_source in WGS84 (4326 SRID), then you can use this function as a Function Source:

CREATE OR REPLACE FUNCTION public.function_source(z integer, x integer, y integer, query_params json) RETURNS bytea AS $$
DECLARE
  mvt bytea;
BEGIN
  SELECT INTO mvt ST_AsMVT(tile, 'public.function_source', 4096, 'geom') FROM (
    SELECT
      ST_AsMVTGeom(ST_Transform(geom, 3857), TileBBox(z, x, y, 3857), 4096, 64, true) AS geom
    FROM public.table_source
    WHERE geom && TileBBox(z, x, y, 4326)
  ) as tile WHERE geom IS NOT NULL;

  RETURN mvt;
END
$$ LANGUAGE plpgsql IMMUTABLE STRICT PARALLEL SAFE;

The query_params argument is a JSON representation of the tile request query params. For example, if user requested a tile with urlencoded⁠ params:

curl \
  --data-urlencode 'arrayParam=[1, 2, 3]' \
  --data-urlencode 'numberParam=42' \
  --data-urlencode 'stringParam=value' \
  --data-urlencode 'booleanParam=true' \
  --data-urlencode 'objectParam={"answer" : 42}' \
  --get localhost:3000/rpc/public.function_source/0/0/0.pbf

then query_params will be parsed as:

{
  "arrayParam": [1, 2, 3],
  "numberParam": 42,
  "stringParam": "value",
  "booleanParam": true,
  "objectParam": { "answer": 42 }
}

You can access this params using json operators⁠:

...WHERE answer = (query_params->'objectParam'->>'answer')::int;

⁠Function Sources List

Function Sources list endpoint is available at /rpc/index.json

curl localhost:3000/rpc/index.json

Note: if in watch mode, this will rescan database for function sources.

⁠Function Source TileJSON

Function Source TileJSON⁠ endpoint is available at /rpc/{schema_name}.{function_name}.json

For example, points function in public schema will be available at /rpc/public.points.json

curl localhost:3000/rpc/public.points.json

⁠Function Source Tiles

Function Source tiles endpoint is available at /rpc/{schema_name}.{function_name}/{z}/{x}/{y}.pbf

For example, points function in public schema will be available at /rpc/public.points/{z}/{x}/{y}.pbf

curl localhost:3000/rpc/public.points/0/0/0.pbf

⁠Command-line Interface

You can configure martin using command-line interface

Usage:
  martin [options] [<connection>]
  martin -h | --help
  martin -v | --version

Options:
  -h --help                         Show this screen.
  -v --version                      Show version.
  --config=<path>                   Path to config file.
  --keep-alive=<n>                  Connection keep alive timeout [default: 75].
  --listen-addresses=<n>            The socket address to bind [default: 0.0.0.0:3000].
  --pool-size=<n>                   Maximum connections pool size [default: 20].
  --watch                           Scan for new sources on sources list requests.
  --workers=<n>                     Number of web server workers.
  --danger-accept-invalid-certs     Trust invalid certificates. This introduces significant vulnerabilities, and should only be used as a last resort.

⁠Environment Variables

You can also configure martin using environment variables

⁠Configuration File

If you don't want to expose all of your tables and functions, you can list your sources in a configuration file. To start martin with a configuration file you need to pass a path to a file with a --config argument.

martin --config config.yaml

You can find an example of a configuration file here⁠.

# Database connection string
connection_string: "postgres://postgres@localhost/db"

# Maximum connections pool size [default: 20]
pool_size: 20

# Connection keep alive timeout [default: 75]
keep_alive: 75

# Number of web server workers
worker_processes: 8

# The socket address to bind [default: 0.0.0.0:3000]
listen_addresses: "0.0.0.0:3000"

# Enable watch mode
watch: true

# Trust invalid certificates. This introduces significant vulnerabilities, and should only be used as a last resort.
danger_accept_invalid_certs: false

# associative arrays of table sources
table_sources:
  public.table_source:
    # table source id
    id: public.table_source

    # table schema
    schema: public

    # table name
    table: table_source

    # geometry column name
    geometry_column: geom

    # geometry srid
    srid: 4326

    # tile extent in tile coordinate space
    extent: 4096

    # buffer distance in tile coordinate space to optionally clip geometries
    buffer: 64

    # boolean to control if geometries should be clipped or encoded as is
    clip_geom: true

    # geometry type
    geometry_type: GEOMETRY

    # list of columns, that should be encoded as a tile properties
    properties:
      gid: int4

# associative arrays of function sources
function_sources:
  public.function_source:
    # function source id
    id: public.function_source

    # schema name
    schema: public

    # function name
    function: function_source

⁠Using with Docker

You can use official Docker image urbica/martin⁠

docker run \
  -p 3000:3000 \
  -e DATABASE_URL=postgres://postgres@localhost/db \
  urbica/martin

If you are running PostgreSQL instance on localhost, you have to change network settings to allow the Docker container to access the localhost network.

For Linux, add the --net=host flag to access the localhost PostgreSQL service.

docker run \
  --net=host \
  -p 3000:3000 \
  -e DATABASE_URL=postgres://postgres@localhost/db \
  urbica/martin

For macOS, use host.docker.internal as hostname to access the localhost PostgreSQL service.

docker run \
  -p 3000:3000 \
  -e DATABASE_URL=postgres://[email protected]/db \
  urbica/martin

For Windows, use docker.for.win.localhost as hostname to access the localhost PostgreSQL service.

docker run \
  -p 3000:3000 \
  -e DATABASE_URL=postgres://[email protected]/db \
  urbica/martin

⁠Using with Docker Compose

You can use example docker-compose.yml⁠ file as a reference

version: "3"

services:
  martin:
    image: urbica/martin
    restart: unless-stopped
    ports:
      - 3000:3000
    environment:
      - WATCH_MODE=true
      - DATABASE_URL=postgres://postgres:password@db/db
    depends_on:
      - db

  db:
    image: postgis/postgis:13-3.1-alpine
    restart: unless-stopped
    environment:
      - POSTGRES_DB=db
      - POSTGRES_USER=postgres
      - POSTGRES_PASSWORD=password
    volumes:
      - ./pg_data:/var/lib/postgresql/data

First, you need to start db service

docker-compose up -d db

Then, after db service is ready to accept connections, you can start martin

docker-compose up -d martin

By default, martin will be available at localhost:3000⁠

⁠Using with Nginx

If you are running martin behind nginx proxy, you may want to rewrite request URL, to properly handle tile urls in TileJSON⁠ endpoints⁠.

location ~ /tiles/(?<fwd_path>.*) {
    proxy_set_header  X-Rewrite-URL $request_uri;
    proxy_set_header  X-Forwarded-Host $host:$server_port;
    proxy_set_header  X-Forwarded-Proto $scheme; # or $http_x_forwarded_proto
    proxy_pass        http://martin:3000/$fwd_path$is_args$args;
}

⁠Building from Source

You can clone the repository and build martin using cargo⁠ package manager.

git clone [email protected]:urbica/martin.git
cd martin
cargo build --release

The binary will be available at ./target/release/martin.

cd ./target/release/
./martin postgres://postgres@localhost/db

⁠Debugging

Log levels are controlled on a per-module basis, and by default all logging is disabled except for errors. Logging is controlled via the RUST_LOG environment variable. The value of this environment variable is a comma-separated list of logging directives.

This will enable debug logging for all modules:

export RUST_LOG=debug
martin postgres://postgres@localhost/db

While this will only enable verbose logging for the actix_web module and enable debug logging for the martin and tokio_postgres modules:

export RUST_LOG=actix_web=info,martin=debug,tokio_postgres=debug
martin postgres://postgres@localhost/db

⁠Development

Clone project

git clone [email protected]:urbica/martin.git
cd martin

Start db service using docker-compose⁠

docker-compose up -d db

Then, after db service is ready to accept connections, you can start martin with

DATABASE_URL=postgres://postgres@localhost/db cargo run

By default, martin will be available at localhost:3000⁠

Make your changes, and check if all the tests are running

DATABASE_URL=postgres://postgres@localhost/db cargo test

You can also run benchmarks with

DATABASE_URL=postgres://postgres@localhost/db cargo bench

An HTML report displaying the results of the benchmark will be generated under target/criterion/report/index.html