tor2tor/d2d-directions repository overview

⁠DRT-OSRM-Server Data Preprocessing

This document will guide you through creating routable graphs which will be stored on S3 and used by this application(s).

⁠Goal / Why is this required?

We want to take an OSM file, e.g. europe-latest.osm.pbf remove all the contents which are outside of our operating areas and apply local changes as well.

This will shrink the OSM file size tremendeously, creates a smaller routing graph and OSRM can load the smaller graph much faster. (Also it requires less memory and cpu, which leads to cost-friendly EC2 instances.)

⁠Note

You will need to get the data (if needed) and then trigger the building/updating of the graph. The current process to create a driving and walking graph from scratch takes approximately 4.5 hours (processing), so it's recommended to trigger it on AWS, but you can also do it manually. Both described below.

⁠Steps

1. Get the data
2. Trigger building and uploading of graphs (AWS or Manual)
3. Use the graphs

⁠1. Get the data

(the explanation here is relevant if you need to create graphs for a new region that has not been processed before)

⁠Data preparation

These are the steps to create a small OSRM graph.

⁠1. Download the OSM dataset

First a OSM dataset is required, which covers all the regions we are operating in (including the fake operating area in London) or want to run simulations.

We will need all OSM files which cover our regions. You could download a large area or multiple smaller ones.

The short example above uses daily snapshots of different areas provided by Geofabrik⁠. The whole planet is provided by different soures, an overview can be found here⁠.

This is achieved by a bash-script⁠ inside of the container.

Note: It is recommendent to use the .pbf files as they are much smaller.

If there are new regions to be added, please look at this file⁠ and add the URL.

⁠2. Define the GeoJSON polygons for our regions

The second step defines the .geojson file, to include only the features which are within the described MultiPolygons.

We need to provide the GeoJSON files for all areas we operate in or want to run simulations. For each region a separate file is present here⁠

After making a change to these files or adding a new area, see the section below on how to update the final routing area merged.geojson.

If you don't need to update these files, you can skip this step.

⁠Merge region specific areas into final routing area

Enter the helper-scripts directory⁠ and install geojson-clipping:

yarn install

Merge all region speficic areas into one routing area for the whole planet:

geojson-clipping union ../graph-builder/data/geojson > ../graph-builder/data/geojson/merged.geojson

Now you should have one big MultiPolygon in merged.geojson that will be used to clip our OSM dataset to only include the features of our desired routing area. Be sure to commit your changes to this file.

⁠Websites which provide suitable geojson files:

• http://polygons.openstreetmap.fr/⁠: You require the OSM relation ID to generate the polygon. (The relation ID can be queried via Overpass Turbo⁠ or directly on the OSM website⁠) - London: http://polygons.openstreetmap.fr/?id=65606⁠ - Berlin: http://polygons.openstreetmap.fr/?id=62422⁠

⁠Alternative

You could also create a bounding box around your region on geojson.io⁠.

⁠OSM-Changesets

Some OSM nodes, ways and/or relations may require local changes for a better routing experience. For example the vehicle depot could be in an area with restricted access (Example⁠) and we want to be able to route on that specific street.

It is necessary to modify the OSM data and generate a new dataset before proceeding. For those use-cases we can create OSM-Change files (.osc⁠).

An example can be found here⁠ - we change the access=private key-value to access=yes and keep all other elements in place.

Note: If you ever need to create a new changeset, follow these steps:

• Search for the osm element and get the ID (either via OSM⁠ (activate the data layer) or via Overpass-Turbo⁠).
• Create a new .osc file based on the example: cp ./data/changesets/example.osc /data/changesets/what-you-change.osc
• Copy and paste the element with all its current attributes from the first step into the correct section and remove the others if needed.
• Modify the tags/nodes to your needs.
• Don't forget to increase the version number of the element you are editing. If it is not increased, the changes won't be applied to the dataset.
• Save the file.

⁠[AWS] 2. Trigger graph generation and upload

1. Commit your changes to the data and push them to a branch with the format: build_graph_YYYY_MM_DD (the date part will be used for the graph filename). Tip: also commit the changes to osrm-routing-core (last step) in the same push if you want to use it once merged to master.
2. Create a PR (within 1 hour of pushing) for this data.
3. CircleCI will trigger jobs on AWS to build and upload the graph to all stages (sandbox, dev, staging, production). The builders will update their status via comments on the PR.
4. Once all graphs are built (4 graphs, 4 comments, in all stages), and the PR is approved, merge to master.

Note: Every push to this branch format will trigger a new build in all stages.

⁠[Manual] 2. Trigger graph generation and upload

(Given your data is already present and you only need to update the graph)

⁠Requirements

In order to run the script, you require Docker 1.12+, awscli and eventually a text editor.

Install the missing parts via brew:

brew cask install docker
brew install awscli

After this build the docker container (cd into this folder first):

docker build -t osrm-data -f Dockerfile .

Or pull the build from the built image on Docker Hub (synced with master):

docker pull tor2tor/d2d-directions:preprocessor && docker tag tor2tor/d2d-directions:preprocessor osrm-data

⁠How to do it

Run the docker container to automatically:

• Download OSM data
• Merge OSM files into one
• Clip merged OSM data to generate a smaller one
• Alter clipped OSM data to apply local changes
• Generate a routing graph with a driving profile based on the altered OSM data
• Generate a routing graph with a walking profile based on the altered OSM data
• Generate a small routing graph for test purposes
• Cleanup / remove merged, clipped and altered files

This can be achieved via:

docker run -v $(pwd)/graph-builder/data:/data -it osrm-data

Note: You need to execute this command inside this folder.

If you only want to do specific things run:

docker run -v $(pwd)/graph-builder/data:/data -it osrm-data bash ./bootstrap.sh -h

It will print the usage of the script.

Afterwards, you need to zip the graphs and upload them to AWS S3 to their respective buckets (driving/walking):

# compress the files first before uploading
find graph-builder/data/osrm/driving -name "*.osrm*" -execdir tar -zcvf data/driving_graph.tgz -C data/osrm/driving {} +
find graph-builder/data/osrm/walking -name "*.osrm*" -execdir tar -zcvf data/walking_graph.tgz -C data/osrm/walking {} +

# upload archive to buckets per environment
aws --profile d2d-drt-devel s3 cp graph-builder/data/driving_graph.tgz s3://d2d-directions-driving-dev/v5.21.0/graph_$(date +"%Y_%m_%d").tgz
aws --profile d2d-drt-devel s3 cp graph-builder/data/walking_graph.tgz s3://d2d-directions-walking-dev/v5.21.0/graph_$(date +"%Y_%m_%d").tgz

aws --profile d2d-drt-sandbox s3 cp graph-builder/data/driving_graph.tgz s3://d2d-directions-driving-sandbox/v5.21.0/graph_$(date +"%Y_%m_%d").tgz
aws --profile d2d-drt-sandbox s3 cp graph-builder/data/walking_graph.tgz s3://d2d-directions-walking-sandbox/v5.21.0/graph_$(date +"%Y_%m_%d").tgz

aws --profile d2d-drt-staging s3 cp graph-builder/data/driving_graph.tgz s3://d2d-directions-driving-staging/v5.21.0/graph_$(date +"%Y_%m_%d").tgz
aws --profile d2d-drt-staging s3 cp graph-builder/data/walking_graph.tgz s3://d2d-directions-walking-staging/v5.21.0/graph_$(date +"%Y_%m_%d").tgz

aws --profile d2d-drt-prod s3 cp graph-builder/data/driving_graph.tgz s3://d2d-directions-driving-production/v5.21.0/graph_$(date +"%Y_%m_%d").tgz
aws --profile d2d-drt-prod s3 cp graph-builder/data/walking_graph.tgz s3://d2d-directions-walking-production/v5.21.0/graph_$(date +"%Y_%m_%d").tgz

Phew, done. :)

⁠Other operations that can be done with the script

⁠Merge, clip and modify the OSM data:

Once all *.osm.pbf and *.poly files are downloaded/created, we will merge all osm files into one, clip the areas based on the poly files and then alter some nodes/ways/relations by applying a .osc-file to the whole dataset.

This is achieved by a bash-script⁠ inside of the container.

⁠Generate a routable graph:

As the last step, we generate a routable graph with OSRM and the clipped file for each profile.

This is achieved by a bash-script⁠ inside of the container.

Note: This process will generate a graph based on Contraction Hierachies (CH) which best fits use-cases where query performance is key, especially for large distance matrices.

The other option is Multi-Level Dijkstra (MLD) which best fits use-cases where query performance still needs to be very good and live-updates to the data need to be made e.g. for regular Traffic updates.

As we don't use regular data updates yet, we'll stick with CH for now.

You can have a look at the offical Wiki⁠

⁠3. Use the generated graphs

You can use these graphs in osrm-routing-core locally. Both profiles (walking and driving) will be deployed after changing the dataset and their graphs are stored in S3.

If you uploaded a new dataset, please ensure that the version number and timestamp is configured in osrm-routing-core/.ebextensions⁠