Public | Automated Build

Last pushed: 5 months ago
Short Description
A C++ library and associated command line tools designed to create terrain tiles for use in the Cesium JavaScript library.
Full Description

Cesium Terrain Builder

This is a C++ library and associated command line tools designed to create
terrain tiles for use with the Cesium JavaScript library.

Cesium can create interactive 3D globes (à la Google Earth) in your web browser
whereby imagery is draped over a model of the underlying terrain. Cesium
provides a number of
different sources for
the terrain data, one of which is height map data for use with the
CesiumTerrainProvider
JavaScript class.

Cesium Terrain Builder can be used to create the tilesets that sit behind a
terrain server used by CesiumTerrainProvider. Note that these tools do not
provide a way of serving up those tilesets to the browser: instead
Cesium Terrain Server is
designed to serve terrain tilesets. In particular the Docker
geodata/cesium-terrain-server
image is designed to simplify the visualisation of terrain tilesets.

Command Line Tools

The following tools are built on top of the C++ libctb library:

ctb-tile

This creates gzipped terrain tiles from a GDAL raster representing a
Digital Elevation Model
(DEM), saving the resulting tiles to a directory. It calculates the maximum
zoom level concomitant with the native raster resolution and creates terrain
tiles for all zoom levels between that maximum and zoom level 0 where the
tile extents overlap the raster extents, resampling and subsetting the data as
necessary. E.g.

ctb-tile --output-dir ./terrain-tiles dem.tif

The input raster should contain data representing elevations relative to sea
level. NODATA (null) values are not currently dealt with: these should be
filled using interpolation in a data preprocessing step.

Note that in the case of multiband rasters, only the first band is used as the
input DEM.

As well as creating terrain tiles, the tool can also be used for generating
tiles in GDAL supported formats using the --output-format option. This
provides similar functionality to the
gdal2tiles.py script. Tiles can be
created in either Web Mercator or Global Geodetic projections using the
--profile option. e.g.

ctb-tile --output-format JPEG --profile mercator \
  --output-dir ./jpeg-tiles RGB-image.tif

An interesting variation on this is to specify --output-format VRT in order to
generate GDAL Virtual Rasters: these can be useful for debugging and are easily
modified programatically.

Usage: ctb-tile [options] GDAL_DATASOURCE

Options:

  -V, --version                 output program version
  -h, --help                    output help information
  -o, --output-dir <dir>        specify the output directory for the tiles (defaults to working directory)
  -f, --output-format <format>  specify the output format for the tiles. This is either `Terrain` (the default) or any format listed by `gdalinfo --formats`
  -p, --profile <profile>       specify the TMS profile for the tiles. This is either `geodetic` (the default) or `mercator`
  -c, --thread-count <count>    specify the number of threads to use for tile generation. On multicore machines this defaults to the number of CPUs
  -t, --tile-size <size>        specify the size of the tiles in pixels. This defaults to 65 for terrain tiles and 256 for other GDAL formats
  -s, --start-zoom <zoom>       specify the zoom level to start at. This should be greater than the end zoom level
  -e, --end-zoom <zoom>         specify the zoom level to end at. This should be less than the start zoom level and >= 0
  -r, --resampling-method <algorithm> specify the raster resampling algorithm.  One of: nearest; bilinear; cubic; cubicspline; lanczos; average; mode; max; min; med; q1; q3. Defaults to average.
  -n, --creation-option <option> specify a GDAL creation option for the output dataset in the form NAME=VALUE. Can be specified multiple times. Not valid for Terrain tiles.
  -z, --error-threshold <threshold> specify the error threshold in pixel units for transformation approximation. Larger values should mean faster transforms. Defaults to 0.125
  -m, --warp-memory <bytes>     The memory limit in bytes used for warp operations. Higher settings should be faster. Defaults to a conservative GDAL internal setting.
  -R, --resume                  Do not overwrite existing files
  -q, --quiet                   only output errors
  -v, --verbose                 be more noisy

Recommendations

  • For performance reasons it is recommended that the input raster be in the same
    spatial reference system as the output tile grid in order to bypass the need
    to reproject the data. For terrain data this is
    World Geodetic System
    (WGS 84). If the source data is in another spatial reference system, however,
    the tool will attempt to reproject the data but with an associated performance
    penalty.

  • For large rasters a tile based format (as opposed to scanline based) will
    drastically speed up processing. A block size that is similar to the tile
    output size (i.e. 65x65 for terrain tiles) should be chosen.

  • Adding overviews to the source dataset will speed up tile generation. The
    overview will be chosen whose resolution most closely matches that of the zoom
    level being rendered. Overviews will only be downsampled, never upsampled.
    As such, it is recommended to use rolutions corresponding to the
    Global Geodetic Profile
    in the Tile Mapping Service specification. See the
    gdaladdo tool for creating overviews.

  • DEM datasets composed of multiple files can be composited into a single GDAL
    Virtual Raster (VRT) dataset for use
    as input to ctb-tile and ctb-extents. See the
    gdalbuildvrt tool.

  • Setting
    GDAL runtime configuration
    options will also affect Cesium Terrain Builder. Specifically the
    GDAL_CACHEMAX
    environment variable should be set to a relatively high value, in conjunction
    with the warp memory, if required (see next recommendation).

  • If warping the source dataset then set the warp memory to a relatively high
    value. The correct value is system dependent but try starting your benchmarks
    from a value where the combined value of GDAL_CACHEMAX and the warp memory
    represents about 2/3 of your available RAM.

  • ctb-tile will resample data from the source dataset when generating
    tilesets for the various zoom levels. This can lead to performance issues and
    datatype overflows at lower zoom levels (e.g. level 0) when the source dataset
    is very large. To overcome this the tool can be used on the original dataset
    to only create the tile set at the highest zoom level (e.g. level 18) using
    the --start-zoom and --end-zoom options. Once this tileset is generated
    it can be turned into a GDAL Virtual Raster dataset for creating the next zoom
    level down (e.g. level 17). Repeating this process until the lowest zoom
    level is created means that the resampling is much more efficient (e.g. level
    0 would be created from a VRT representation of level 1). Because terrain
    tiles are not a format supported by VRT datasets you will need to perform this
    process in order to create tiles in a GDAL DEM format as an intermediate step.
    VRT representations of these intermediate tilesets can then be used to create
    the final terrain tile output.

ctb-info

This provides various information on a terrain tile, mainly useful for
debugging purposes.

Usage: ctb-info [options] TERRAIN_FILE

Options:

  -V, --version                 output program version
  -h, --help                    output help information
  -e, --show-heights            show the height information as an ASCII raster
  -c, --no-child                hide information about child tiles
  -t, --no-type                 hide information about the tile type (i.e. water/land)

ctb-export

This exports a terrain tile to GeoTiff
format for use in GIS software. Terrain tiles do not contain information
defining their tile location, so this must be specified through the command
options.

Note that the tool does not normalise the terrain data to sea level but
displays it exactly as it is found in the terrain data.

Usage: ctb-export -i TERRAIN_FILE -z ZOOM_LEVEL -x TILE_X -y TILE_Y -o OUTPUT_FILE

Options:

  -V, --version                 output program version
  -h, --help                    output help information
  -i, --input-filename <filename> the terrain tile file to convert
  -z, --zoom-level <int>        the zoom level represented by the tile
  -x, --tile-x <int>            the tile x coordinate
  -y, --tile-y <int>            the tile y coordinate
  -o, --output-filename <filename> the output file to create

ctb-extents

Sometimes it is useful to see the extent of coverage of terrain tilesets that
would be produced from a raster. This tool does this by outputting each zoom
level as a GeoJSON file containing the tile extents for
that particular zoom level.

Usage: ctb-extents GDAL_DATASET

Options:

  -V, --version                 output program version
  -h, --help                    output help information
  -o, --output-dir <dir>        specify the output directory for the geojson files (defaults to working directory)
  -p, --profile <profile>       specify the TMS profile for the tiles. This is either `geodetic` (the default) or `mercator`
  -t, --tile-size <size>        specify the size of the tiles in pixels. This defaults to 65 for terrain tiles and 256 for other GDAL formats
  -s, --start-zoom <zoom>       specify the zoom level to start at. This should be greater than the end zoom level
  -e, --end-zoom <zoom>         specify the zoom level to end at. This should be less than the start zoom level and >= 0

LibCTB

libctb is a library implemented in standard C++11. It is capable of creating
terrain tiles according to the
heightmap-1.0 terrain format. It
does not provide a way of serving up or storing the resulting tiles: this is
application specific. Instead its aim is simply to take a
GDAL supported raster format representing a Digital
Terrain Model (DTM) and convert this to terrain tiles.

See the source code for the tools provided with the library
(e.g. ctb-tile) for examples on how the library is used to achieve
this.

Documentation

Doxygen based documentation is available for the C++
code: run the doxygen command in the doc/ directory and point your browser
at doc/html/index.html.

Status

Although the software has been used to create a substantial number of terrain
tile sets currently in production use, it should be considered beta quality
software: it needs broader testing, a comprehensive test harness and the API is
liable to change.

The software has primarily been developed and deployed on a Linux OS, and this
is the only officially supported platform. However, it has been reported as
compiling and running on:

  • Windows using Visual Studio 2010 and 2013 (see
    this issue).

  • Mac OS X Mavericks using clang (see
    this issue).

Porting it to other systems should be relatively painless as the library
dependencies have been ported to numerous systems and the code itself is
standard C++11.

Requirements

Runtime requirements

Ensure GDAL >= 2.0.0 is installed. At the time of
writing this is not a stable release so you may need to use a nightly build or
to build the source directly from version control. Specifically, you will need
a version of trunk that has added the min,max,med,q1 and q3 resampling
algorithms. In the subversion repository this is commit 28717 and on the GitHub
mirror this is 0a90a34.

Build requirements

In addition to ensuring the GDAL library is installed, you will need the GDAL
source development header files. You will also need
CMake to be available.

Installation

From Source

  1. Ensure your system meets the requirements above.

  2. Download and
    unpack the source.

  3. In the root package directory, assuming you are on a UNIX system, type
    mkdir build && cd build && cmake .. && make install.

  4. On a UNIX system you may need to run ldconfig to update the shared library
    cache.

Alternatively in step 3 above you can create a debug build by running cmake -DCMAKE_BUILD_TYPE=Debug ... You can also install to a different location by
specifying the CMAKE_INSTALL_PREFIX directive e.g. cmake -DCMAKE_INSTALL_PREFIX=/tmp/terrain ...

Note that if you have GDAL installed in a custom location (e.g under
/home/user/install) it will likely not be found by running cmake ... In this
case you will need to provide the GDAL_LIBRARY_DIR, GDAL_LIBRARY and
GDAL_INCLUDE_DIR directives e.g.

cmake -DGDAL_LIBRARY_DIR=/home/user/install/lib \
      -DGDAL_LIBRARY=/home/user/install/lib/libgdal.so \
      -DGDAL_INCLUDE_DIR=/home/user/install/include \
      ..

Using Docker

homme/cesium-terrain-builder
is a Docker image that bundles the CTB tools and
simplifies their deployment. Follow the link for usage information.

The only requirement to getting up and running with Cesium Terrain Builder is
having docker available on your system: all software dependencies, build and
installation issues are encapsulated in the image.

In addition, the
geodata/cesium-terrain-server
Docker image provides a way of visualising the tilesets created by
homme/cesium-terrain-builder.

Limitations and TODO

  • Create a comprehensive test harness (possibly using
    Bandit, including code coverage and valgrind
    analysis.

  • Better coordination between threads in ctb-tile to enable graceful exits if
    there is a fatal error or other interrupt.

  • Add support for the new
    quantized-mesh-1.0 terrain format.

  • The ctb-tile command currently only outputs files to a directory and
    as such is subjected to filesystem limits (e.g. inode limits): it should be
    able to output tiles in a format that overcomes these limits and which is
    still portable and accessible. SQLite would appear
    to be a strong contender.

  • Provide hooks into the GDAL error handling mechanism to more gracefully
    intercept GDAL errors.

  • Expose tilers using a standard container api (map and/or vector).

  • Enable more options to be passed to the VRT warper by parameterising
    GDALWarpOptions::papszWarpOptions and GDALCreateGenImgProjTransformer2 in
    GDALTiler::createRasterTile.

  • Encapsulate the multithreading tile generation functionality currently
    implemented in ctb-tile within the library to make it more widely
    available.

  • One of the ctb-tile recommendations above illustrates a process for
    efficiently creating tilesets at lower zoom levels by resampling an already
    generated tileset at the next highest zoom level. This could be built
    directly into the ctb-tile tool. An implementation could create a
    read-only GDAL TiledDataset driver (or use a VRT, if it efficiently supports
    the large number of tile files) which accesses the already generated tileset;
    this dataset could then be used as an input to the tiler.

  • Add support for interpolating out NODATA values. This could be done using
    either GDALFillNodata() or GDALGridCreate().

  • Adding support for creating water masks to tiles could be useful: at the
    moment all tiles are flagged as being of type 'land' (see
    this issue).

Issues and Contributing

Please report bugs or issues using the
GitHub issue tracker.

Code and documentation contributions are very welcome, either as GitHub pull
requests or patches. If you cannot do this but would still like to improve the
software, particularly overcoming the limitations listed above, then please
consider funding further development.

License

The Apache License, Version 2.0.

Acknowledgements

Software development funded by the
Maritime Archaeology Trust and the
European Regional Development Fund through the
Interreg IVA 2 Seas Programme.

Software developed by GeoData through the
University of Southampton Open Source Geospatial Laboratory.

Thanks to everyone in the community who has contributed to the code base.

Contact

Homme Zwaagstra hrz@geodata.soton.ac.uk

Docker Pull Command
Owner
homme
Source Repository