Sign inSign up

buzzardpython/ci

By buzzardpython

•Updated about 6 years ago

Image
0

2.3K

buzzardpython/ci repository overview

⁠buzzard

In a nutshell, the buzzard library provides powerful abstractions to manipulate together images and geometries that come from different kind of sources (GeoTIFF, PNG, GeoJSON, Shapefile, numpy array, buzzard pipelines, ...).



licenseCircleCIcodecovreadthedoc

Join us on Slack! ⁠

⁠buzzard is

  • A python library.
  • Primarily designed to hide all cumbersome operations when doing data-science with GIS⁠ files.
  • A Multipurpose computer vision library, it can be used in all kind of situations where images or geometries are involved.
  • A pythonic wrapper for osgeo's gdal/ogr/osr.
  • A solution to work with arbitrary large images by simplifying and automating the manipulation of image slices.

⁠buzzard contains

  • A Dataset⁠ class that oversees all opened raster and vector files in order to share resources.
  • An immutable toolbox class, the Footprint⁠, designed to locate a rectangle in both image space and geometry space.

⁠How to open and read files

This example demonstrates how to visualize a large raster polygon per polygon.

import buzzard as buzz
import numpy as np
import matplotlib.pyplot as plt

# Open the files. Only files' metadata are read so far
r = buzz.open_raster('path/to/rgba-image.tif')
v = buzz.open_vector('path/to/polygons.geojson', driver='GeoJSON')


# Load the polygons from disk one by one as shapely objects
for poly in v.iter_data():

    # Compute the Footprint bounding `poly`
    fp = r.fp.intersection(poly)
    print(fp)

    # Load the image from disk at `fp` to a numpy array
    rgb = r.get_data(fp=fp, channels=(0, 1, 2))
    alpha = r.get_data(fp=fp, channels=3)

    # Create a boolean mask as a numpy array from the shapely polygon
    mask = np.invert(fp.burn_polygons(poly))

    # Darken pixels outside of polygon, set transparent pixels to orange
    rgb[mask] = (rgb[mask] * 0.5).astype(np.uint8)
    rgb[alpha == 0] = [236, 120, 57]

    # Show the result with matplotlib
    plt.imshow(rgb)
    plt.show()

Images from the ISPRS's Potsdam dataset⁠.

Footprint(tl=(3183.600000, -914.550000), br=(3689.700000, -1170.450000), size=(506.100000, 255.900000), rsize=(3374, 1706))



Footprint(tl=(3171.600000, -1321.500000), br=(4553.400000, -2400.000000), size=(1381.800000, 1078.500000), rsize=(9212, 7190))



⁠How to create files and manipulate Footprints

import buzzard as buzz
import numpy as np
import matplotlib.pyplot as plt
import keras

r = buzz.open_raster('path/to/rgba-image.tif')
km = keras.models.load_model('path/to/deep-learning-model.hdf5')

# Chunk the raster's Footprint to Footprints of size
# 1920 x 1080 pixel stored in a 2d numpy array
tiles = r.fp.tile(1920, 1080)

all_roads = []

for i, fp in enumerate(tiles.flat):
    rgb = r.get_data(fp=fp, channels=(0, 1, 2))

    # Perform pixelwise semantic segmentation with a keras model
    predictions_heatmap = km.predict(rgb[np.newaxis, ...])[0]
    predictions_top1 = np.argmax(predictions_heatmap, axis=-1)

    # Save the prediction to a `geotiff`
    with buzz.create_raster(path='predictions_{}.tif'.format(i), fp=fp,
                            dtype='uint8', channel_count=1).close as out:
        out.set_data(predictions_top1)

    # Extract the road polygons by transforming a numpy boolean mask to shapely polygons
    road_polygons = fp.find_polygons(predictions_top1 == 3)
    all_roads += road_polygons

    # Show the result with matplotlib for one tile
    if i == 2:
        plt.imshow(rgb)
        plt.imshow(predictions_top1)
        plt.show()

# Save all roads found to a single `shapefile`
with buzz.create_vector(path='roads.shp', type='polygon').close as out:
    for poly in all_roads:
        out.inser_data(poly)





⁠Advanced examples

Additional examples can be found here:

⁠buzzard allows

  • Opening and creating raster⁠ and vector⁠ files. Supports all GDAL drivers (GTiff, PNG, ...)⁠ and all OGR drivers (GeoJSON, DXF, Shapefile, ...)⁠.
  • Reading⁠ raster files pixels from disk to numpy.ndarray.
    • Options: sub-rectangle reading, rotated and scaled sub-rectangle reading (thanks to on-the-fly remapping with OpenCV), automatic parallelization of read and remapping (soon), async (soon), be the source of an image processing pipeline (soon).
    • Properties: thread-safe
  • Writing⁠ raster files pixels to disk from numpy.ndarray.
    • Options: sub-rectangle writing, rotated and scaled sub-rectangle writing (thanks to on-the-fly remapping with OpenCV), masked writing.
  • Reading⁠ vector files geometries from disk to shapely objects, geojson dict and raw coordinates.
    • Options: masking.
    • Properties: thread-safe
  • Writing⁠ vector files geometries to disk from shapely objects, geojson dict and raw coordinates.
  • Powerful manipulations of raster windows⁠
  • Instantiation⁠ of image processing pipelines where each node is a raster, and each edge is a user defined python function working on numpy.ndarray (beta, partially implemented).
    • Options: automatic parallelization using user defined thread or process pools, disk caching.
    • Properties: lazy evaluation, deterministic, automatic tasks chunking into tiles, fine grain task prioritization, backpressure prevention.
  • Spatial reference homogenization⁠ between opened files like a GIS software does (beta)

⁠Documentation

https://buzzard.readthedocs.io/⁠

⁠Dependencies

The following table lists dependencies along with the minimum version, their status for the project and the related license.

LibraryVersionMandatoryLicenseComment
gdal>=2.3.3YesMIT/X⁠Hard to install. Will be included in buzzard wheels
opencv-python>=3.1.0Yes3-clause BSD⁠Easy to install with opencv-python wheels. Will be optional
shapely>=1.6.1Yes3-clause BSD⁠
affine>=2.0.0Yes3-clause BSD⁠
numpy>=1.15.0Yesnumpy⁠
scipy>=0.19.1Yesscipy⁠
pint>=0.8.1Yes3-clause BSD⁠
six>=1.11.0YesMIT⁠
sortedcontainers>=1.5.9Yesapache⁠
Rtree>=0.8.3YesMIT⁠
scikit-image>=0.14.0Yesscikit-image⁠
chainmap>=1.0.2YesPython 2.7 license⁠Only for python <3.2
pytest>=3.2.2NoMIT⁠Only for tests
attrdict>=2.0.0NoMIT⁠Only for tests

⁠How to install from terminal

⁠Anaconda and pip
# Step 1 - Install Anaconda
# https://www.anaconda.com/download/

# Step 2 - Create env
conda create -n buzz python gdal>=2.3.3 shapely rtree -c 'conda-forge'

# Step 3 - Activate env
conda activate buzz

# Step 4 - Install buzzard
pip install buzzard
⁠Docker
docker build -t buzz --build-arg PYTHON_VERSION=3.7 https://raw.githubusercontent.com/earthcube-lab/buzzard/master/.circleci/images/base-python/Dockerfile
docker run -it --rm buzz bash
pip install buzzard

⁠Package manager and pip
# Step 1 - Install GDAL and rtree ******************************************* **
# Windows
# https://www.lfd.uci.edu/~gohlke/pythonlibs/#gdal
# https://www.lfd.uci.edu/~gohlke/pythonlibs/#rtree

# MacOS
brew install gdal
brew tap osgeo/osgeo4mac
brew tap --repair
brew install gdal2
brew install spatialindex
export PATH="/usr/local/opt/gdal2/bin:$PATH"
python3 -m pip install 'gdal==2.3.3'

# Ubuntu
# Run the commands from the following Dockerfile:
# https://github.com/earthcube-lab/buzzard/blob/master/doc/ubuntu_install/Dockerfile

# Step 2 - Install buzzard ************************************************** **
python3 -m pip install buzzard

⁠Supported Python versions

To enjoy the latest buzzard features, update your python!

⁠Full python support
  • Latest supported version: 3.7 (June 2018)
  • Oldest supported version: 3.4 (March 2014)
⁠Partial python support
  • 2.7: use buzzard version 0.4.4

⁠Slack

You want some help? You have a question? You want to contribute? Join us on Slack!

Join us on Slack! ⁠

⁠How to test

git clone https://github.com/earthcube-lab/buzzard
pip install -r buzzard/requirements-dev.txt
pytest buzzard/buzzard/test

⁠How to build documentation

cd docs
make html
open _build/html/index.html

⁠Contributions and feedback

Welcome to the buzzard project! We appreciate any contribution and feedback, your proposals and pull requests will be considered and responded to. For more information, see the CONTRIBUTING.md⁠ file.

⁠Authors

See AUTHORS⁠

⁠License and Notice

See LICENSE⁠ and NOTICE⁠.

⁠Other pages


Tag summary

Content type

Image

Digest

Size

1.2 GB

Last updated

about 6 years ago

docker pull buzzardpython/ci