Public | Automated Build

Last pushed: a month ago
Short Description
EU Open Data Portal CKAN client. It uses RabbitMQ and a SPARQL enpoint.
Full Description


ODP CKAN - EU Open Data Portal CKAN client

  • read messages from the RabbitMQ service
  • interrogate SDS <>_ and retrieve full data about the specified datasets in JSON format
  • updates the EU Open Data Portal (ODP) <>_ using CKAN API

.. contents::

Base docker image

  • <>_

Source code

  • eea.odpckan <>_

Usage via Docker

Start the odpckan client with the following command::

$ sudo docker run -d \
                  -e RABBITMQ_HOST= \
                  -e RABBITMQ_PORT=5672 \
                  -e RABBITMQ_USERNAME=client \
                  -e RABBITMQ_PASSWORD=secret \
                  -e CKAN_ADDRESS= \
                  -e CKAN_APIKEY=secret-api-key \
                  -e SERVICES_EEA= \
                  -e SERVICES_SDS= \
                  -e SERVICES_ODP= \
                  -e SDS_TIMEOUT=60 \
                  -e CKANCLIENT_INTERVAL="0 */3 * * *" \
                  -e CKANCLIENT_INTERVAL_BULK="0 0 * * 0" \
                  -e  eeacms/odpckan

For docker-compose orchestration see eea.docker.odpckan <>_.

Usage w/o Docker


  • Pika <>_ a python client for RabbitMQ
  • ckanapi <> a python client for CKAN API <> to work with ODP
  • rdflib <>_ a python library for working with RDF
  • rdflib-jsonld <>_ JSON-LD parser and serializer plugins for RDFLib

Clone the repository::

$ git clone
$ cd eea.odpckan

Install all dependencies with pip command::

$ pip install -r requirements.txt

Example usage

ODP CKAN entry point that will start consume all the messages from the queue and stops after. This command can be setup as a cron job.::

$ python app/ -d
$ #debug mode: creates debug files for dataset data from SDS and ODP, before and after the update

$ python app/
$ #default/working mode: reads and process all messages from specified queue

Inject test messages (default howmany = 1)::

$ python app/ howmany

Query SDS (default url = and print result::

$ python app/ -d
$ #debug mode: queries SDS and dumps a dataset and all datasets

$ python app/
$ #default/working mode: initiate the bulk update

EEA main portal use case

Information published on EEA main portal <> is submitted to the EU Open Data Portal <>.

.. image::

The workflow is described below:

  • EEA CMS <>_ (Plone)
    • content is published
    • CMS content rules are triggered and the following operations are performed:
      • a message is added in RabbitMQ message broker <>_ queue, see example below
      • SDS <>_ is pinged to update its harvested content
  • EEA ODP CKAN <>_ client
    • CKAN client is triggered periodically via a cron job
    • CKAN client connect to RabbitMQ message broker <>_ and consumes all the messages from the “odp_queue” queue performing following operations:
      • dataset is identified
      • dataset’s metadata is extracted from SDS <>_
      • using CKAN API, OPD <>_ is updated
      • if issues occur during message processing the message is re queued
  • EEA ODP CKAN <>_ client (bulk update operation)
    • is triggered periodically via a cron job
    • it reads all the datasets from the SDS <>_
    • generates update messages in the RabbitMQ message broker <>_, one message per dataset found

RabbitMQ message example


$ update| |eea-coastline-for-analysis-1

Message structure::

$ action|url|identifier


$ create/update/delete

Copyright and license

The Initial Owner of the Original Code is European Environment Agency (EEA).
All Rights Reserved.

The Original Code is free software;
you can redistribute it and/or modify it under the terms of the GNU
General Public License as published by the Free Software Foundation;
either version 2 of the License, or (at your option) any later


European Environment Agency (EU) <>_

Docker Pull Command
Source Repository