inovexis/csdv repository overview

⁠Containerized SHACL Data Validator

This project is provides a deployable service for pulling SHACL records out of mobi⁠ with which to validate data from various types of semantic data stores. The resulting SHACL validation report data can be configured to be written to a number of data sinks, including mobi datasets.

⁠Architecture

The high-level CSDV architecture is to provide a containerized worker that can load SHACL shapes from a source or record, then validate sets of data against it. The resulting SHACL Validation Reports will then be written out to a reporting system (another triple store or other data sink).

The image itself provides a number of customization options with which to configure running containers (detailed below). The concept behind this image is a reusable compute engine for performing SHACL validation outside of your data repository to offload sometimes expensive operations.

⁠Executing

This application exposes REST APIs for interacting with the CSDV system in order to validate data against SHACL shapes. It hosts a resource-efficient, reactive vertx.io services that wrap an RDF4j framework for evaluating SHACL against input data.

To trigger the systen, you can use Postman to POST at http://localhost:8888/⁠ (port can be overridden) with JSON in the body of the request.

⁠Request Structure

The structure of the JSON in the request to trigger CSDV is described in this section. Overall it can be thought of as a JSON document with 5 parts: SHACL, Data Sources, Reporters, Connections, and Engine Configuration.

⁠Process Configuration

A SHACL, Data Source, and Reporter configuration are all described as a Process Configuration (sharing the same base schema).

⁠SHACL Configuration

This section of the request tells the CSDV system where to pull SHACL Shapes Records from to validate data from requested data sources. Each instance of this class in the shacl array are Process Configurations.

Note that you can specify multiple SHACL records to pull and instrument your validation against.

Currently there is only one supported type, mobi, which requires a shape_record in the config map in order to tell CSDV which SHACL record to pull and validate against. The source key should reference a specified Connection Configuration (described below) in order to describe how to connect and authenticate against the running Mobi system.

⁠Data Sources

This section of the JSON request document describes where to pull data from to validate against configured SHACL shapes. Each data source entry specifies a set of data that should be validated against the SHACL configured in the previous section. You can specify multiple sources of data to validate.

⁠Supported Types

The following tables detail the custom configuration each supported type of Data Source supports. Each Data Source entry in the JSON is of the Process Configuration type, so it should have a type and source key, then a config key with the configuration items specified below.

⁠stardog

StarDog⁠ is an enterprise knowledge platform and RDF graph database that allows for sophisticated virtualization. CSDV can integrate with it to pull data to be validated against SHACL shapes.

⁠mobi

Mobi⁠ is a graph data platform, primarily focused on ontology and vocabulary reference data development.

⁠anzo

Anzo⁠ is a comprehensive knowledge graph platform that can be leveraged as a source of data to validate against SHACL shapes.

⁠graphdb

GraphDB is a RDF graph database, built by OntoText⁠, that can be leveraged as a source of data to validate against your SHACL shapes.

⁠Reporters

A reporter in this context is configuration that tells the CSDV process what to do with the SHACL validation report it generates. Reporters can insert the resulting validation report into a remote system based upon configuration here. Each reporter is a Process Configuration, as specified above, and should describe the type and source, as well as custom type configuration.

⁠Supported Types

The following tables detail the custom type configuration available for each supported type of reporter.

⁠stardog

SHACL validation reports can be inserted into a StarDog database.

⁠anzo

The CSDV system can write validation reports into existing layers of a GraphMart within an Anzo system. One thing to note is that GraphMarts are typically transient, meaning data will be lost when the GraphMart reloads unless otherwise configured.

⁠graphdb

The CSDV system can insert SHACL validation reports into a GraphDB database as well.

⁠mobi

SHACL validation reports can also be inserted into a dataset record in Mobi.

⁠Connections

The Connections section of the request JSON specifies the various repositories that CSDV will need to know how to connect to and operate with. These are referenced by various Process Configurations using the source attribute that should overlap with the name attribute as described here.

A connection should follow a structure similar to:

{
   "name": "mobienterprise",
   "host": "https://mobienterprise.inovexcorp.com",
   "auth": {
     "type": "basic",
     "user": "admin",
     "password": "admin"
  }
}

Where the:

• name uniquely identifies the connection details,
• host identifies the host that is running the remote system.
• auth describes how to authenticate to the remote system.
  • type describes the type of authentication (basic is all that is currently supported)
  • user is username to connect to the system with.
  • password is the password to associate with the given username.

⁠Engine Configuration

The final portion of the configuration is slightly different in that is not an array of documents, but a single document that describes how to instrument the process performing the actual validation.

⁠Example Request

The following is a sample JSON payload that could be sent to CSDV to hook into several different systems to validate data against a SHACL shape against basic SKOS Concepts.

{
   "shacl": [
      {
         "type": "mobi",
         "source": "mobienterprise",
         "config": {
            "shape_record": "https://mobi.com/records#05e033a3-8ef9-49cb-b847-987f1c1d234f"
         }
      }
   ],
   "dataSources": [
      {
         "type": "stardog",
         "source": "local_stardog",
         "config": {
            "database": "local",
            "limit": 1000,
            "types": [
               "http://www.w3.org/2004/02/skos/core#Concept",
               "http://www.w3.org/2004/02/skos/core#ConceptScheme"
            ]
         }
      },
      {
         "type": "mobi",
         "source": "mobienterprise",
         "config": {
            "datasetRecord": "https://mobi.com/records#95fa2559-2bf1-486d-a274-729211bff7b2",
            "types": [
               "http://www.w3.org/2004/02/skos/core#Concept",
               "http://www.w3.org/2004/02/skos/core#ConceptScheme"
            ],
            "limit": 1000
         }
      },
      {
         "type": "anzo",
         "source": "anzodemo",
         "config": {
            "graphmart": "http://openanzo.org/Graphmart/89523810-6d55-45e6-881f-33ef44810e8d",
            "limit": 1000,
            "types": [
               "http://www.w3.org/2004/02/skos/core#Concept",
               "http://www.w3.org/2004/02/skos/core#ConceptScheme"
            ]
         }
      },
      {
         "type": "graphdb",
         "source": "local_graphdb",
         "config": {
            "repository": "local",
            "limit": 100,
            "types": [
               "http://www.w3.org/2004/02/skos/core#Concept",
               "http://www.w3.org/2004/02/skos/core#ConceptScheme"
            ]
         }
      }
   ],
   "reporters": [
      {
         "type": "mobi",
         "source": "mobienterprise",
         "config": {
            "dataset": "https://mobi.com/records#404b513a-820f-4343-88cd-54027b7167a5",
            "overwrite": true
         }
      },
      {
         "type": "anzo",
         "source": "anzodemo",
         "config": {
            "graphmart": "http://openanzo.org/Graphmart/89523810-6d55-45e6-881f-33ef44810e8d",
            "layer": "http://inovexcorp.com/Layer/c2374a8fff974c9e82d39d560d1e0aa4",
            "overwrite": true
         }
      },
      {
         "type": "graphdb",
         "source": "local_graphdb",
         "config": {
            "graph": "urn://report",
            "repository": "local",
            "overwrite": true
         }
      }
   ],
   "connections": [
      {
         "name": "mobienterprise",
         "host": "https://mobienterprise.inovexcorp.com",
         "auth": {
            "type": "basic",
            "user": "username",
            "password": "password"
         }
      },
      {
         "name": "anzodemo",
         "host": "https://anzodemo.inovexcorp.com",
         "auth": {
            "type": "basic",
            "user": "username",
            "password": "password"
         }
      },
      {
         "name": "local_stardog",
         "host": "http://10.5.1.2:5820",
         "auth": {
            "type": "basic",
            "user": "username",
            "password": "password"
         }
      },
      {
         "name": "local_graphdb",
         "host": "http://127.0.0.1:7200",
         "auth": {
            "type": "basic",
            "user": "username",
            "password": "password"
         }
      }
   ],
   "engine": {
      "parallelValidation": true,
      "validationResultsLimit": -1,
      "validationResultsLimitPerConstraint": -1,
      "storeType": "lmdb",
      "index": "spoc,ospc,psoc",
      "performanceLogging": false,
      "transactionalValidationLimit": -1,
      "rdf4jShaclExtensions": false,
      "isolationLevel": "READ_COMMITTED",
      "isolateDataSources": true
   }
}

⁠Container

The csdv-vertx submodule uses a maven plugin to build a container. This container is built locally when you build the submodule.

The docker container spins up a non-clustered Vertx runtime, meaning all the actors in the setup will run on the same compute/container.

You can run the container in docker using a command like:

docker run --name csdv -p 8888:8888 -e CSDV_PORT=8888 inovexis/csdv:latest

The following environment variables are leveraged by the CSDV image to enable customization of the behavior of the running container:

• CSDV_PORT
  • The port the CSDV http server will listen on.
  • Defaults to 8888.
• CSDV_HTTPS
  • Whether or not to enable https (encryption) on the http server.
  • Defaults to false.
• CSDV_KEYSTORE
  • The path to the keystore to use (JKS supported).
  • Required if using HTTPS.
• CSDV_KEYSTORE_PASS
  • The password to use to access the keystore.
  • Required if using HTTPS.
• CSDV_WORKER_INST
  • The number of worker verticle instances. A worker is the core processor for a request to run validation against a set of data.
  • Defaults to 1.
• CSDV_SHACL_FETCHER_INST
  • The number of SHACL fetcher instances. This is the actor who fetches SHACL records.
  • Defaults to 1.
• CSDV_DATA_FETCHER_INST
  • The number of data fetching instances. This is the actor fetching instance data to validate against the SHACL records.
  • Defaults to 1.
• CSDV_REPORTER_INST
  • The number of reporter instances. This is the actor who submits the validation report to an external system.
  • Defaults to 1.
• CSDV_BASE_URL_OVERRIDE
  • Override the host/basename (e.g. 'https://name.com⁠') that will be sent back; helps if you're hosting CSDV behind a proxy. Defaults to none, and we'll use the context that comes into CSDV.
• CSDV_BASE_CONTEXT_OVERRIDE
  • Override the base path that requests will be processed on. Defaults to null, meaning /requests. Helpful if you're behind a proxy and want to centralize CSDV requests on a given path.

You can set these variables using the -e flag (as seen in the example above) to allow customization of your running docker container.

⁠Goals

Longer term, this project would benefit from some of these actions:

• Figure out how to page over data in the source system, currently just limiting page size of types...
• Figure out how to migrate to a horizontally scalable setup where verticles can actually run on separate containers.
• Implement quality unit testing across the various projects.