Public | Automated Build

Last pushed: a year ago
Short Description
Short description is empty for this repo.
Full Description

Socrata's open data portal doesn't support gzip compression of bulk file
downloads. It also tends to be very slow in serving these large downloads, as
if a large, slow, SELECT * FROM ... was sitting between you and your
download...

The solution

   ___                   ____        _         ____           _
  / _ \ _ __   ___ _ __ |  _ \  __ _| |_ __ _ / ___|__ _  ___| |__   ___
 | | | | '_ \ / _ \ '_ \| | | |/ _` | __/ _` | |   / _` |/ __| '_ \ / _ \
 | |_| | |_) |  __/ | | | |_| | (_| | || (_| | |__| (_| | (__| | | |  __/
  \___/| .__/ \___|_| |_|____/ \__,_|\__\__,_|\____\__,_|\___|_| |_|\___|
       |_|

Basically, we do the compression and caching Socrata's open data portals don't
do.

(Socrata) --> (nginx gzip) --> (AWS S3) --> (you)

Take a look

An Opendatacache is available already at
http://www.opendatacache.com. Some example URLs:

Deploying

Using docker

You should build the image locally, then you can run it:

$ ./build.sh
$ WARM=1 APP_TOKEN=[Your Socrata App Token] ./run.sh

OpenDataCache makes a lot of API requests, so you'll need to sign up for an application token.

A note on cache warming

If you don't specify WARM=1 as above, the image will start & serve, but will
not cache any new datasets. Its listings will be based on prior caching progress
in the log/ folder.

Thanks

Thank you to OpenPrism for its list of
Socrata portals, which is
here.

TODO

  • Licensing
Docker Pull Command
Owner
thegovlab
Source Repository

Comments (0)