Search facility for mu-semtech, powered by ElasticSearch
10K+
A component to integrate authorization-aware full-text search into a mu.semte.ch stack using Elasticsearch.
The mu-search service uses Elasticsearch as a backend. Since the Elasticsearch docker image requires a lot of memory, increase the maximum on your system by executing the following command:
sysctl -w vm.max_map_count=262144
Next, add the mu-search and accompanying elasticsearch service to docker-compose.yml
services:
search:
image: semtech/mu-search:0.8.0-beta.3
links:
- db:database
volumes:
- ./config/search:/config
elasticsearch:
image: semtech/mu-search-elastic-backend:1.0.0
volumes:
- ./data/elasticsearch/:/usr/share/elasticsearch/data
environment:
- discovery.type=single-node
The indices will be persisted in ./data/elasticsearch. The search service needs to be linked to an instance of the mu-authorization service.
Create the ./config/search directory and create a config.json with the following contents:
{
"types" : [
{
"type" : "document",
"on_path" : "documents",
"rdf_type" : "http://xmlns.com/foaf/0.1/Document",
"properties" : {
"title" : "http://purl.org/dc/elements/1.1/title",
"description" : "http://purl.org/dc/elements/1.1/description"
}
},
{
"type" : "user",
"on_path" : "users",
"rdf_type" : "http://xmlns.com/foaf/0.1/Person",
"properties" : {
"fullname" : "http://xmlns.com/foaf/0.1/name"
}
}
]
}
Finally, add the following rules to your dispatcher configuration in ./config/dispatcher.ex to make the search endpoint available:
define_accept_types [
json: [ "application/json", "application/vnd.api+json" ]
]
@json %{ accept: %{ json: true } }
get "/search/*path", @json do
Proxy.forward conn, path, "http://search/"
end
Restart the dispatcher service to pick up the new configuration
docker-compose restart dispatcher
Restart the stack using docker-compose up -d. The elasticsearch and search services will be created.
Search queries can now be sent to the /search endpoint. Make sure the user has access to the data according to the authorization rules.
By default search indexes are deleted on (re)start of the mu-search service. This guide describes how to make sure search indexes are persisted on restart. Obviously, this configuration is recommended on production environments.
First, make sure the search indexes are written to a mounted volume my specifying a bind mount to /usr/share/elasticsearch/data on the Elasticsearch container.
services:
elasticsearch:
image: semtech/mu-search-elastic-backend:1.0.0
volumes:
- ./data/elasticsearch/:/usr/share/elasticsearch/data
Recreate the elasticsearch container by executing the following command
docker-compose up -d
Next, enable the persistent indexes flag in the root of the search configuration file ./config/search/config.json of your project.
{
"persist_indexes": true,
"types": [
// index type specifications
]
}
Restart the search service to pick up the new configuration.
docker-compose restart search
Search indexes will be persisted in ./data/elasticsearch folder and not be deleted on restart of the search service.
The search API provided by mu-search is authorization-aware. I.e. search results will only contain resources the user is allowed to access. To this end mu-search organises its search indexes per access right. Based on the user's allowed groups set on the incoming search requests, mu-search determines which indexes to search in.
Indexes that don't exist yet will be created before the search operation is performed. Depending on the number of documents to index this may be a time-consuming operation.
Mu-search allows to configure authorization groups for which the indexes need to be created on startup already. This will save time at the moment the first search query for that profile arrives.
Configuration is done via the eager_indexing_groups in the search configuration file ./config/search/config.json. The eager indexing groups are tightly related to the GroupSpecobjects configured in mu-authorization.
The eager_indexing_groups is an array of group specifications. Each group specification is defined by an array of objects in which each object consists of:
GroupSpec) in mu-authorizationvars in case of an AccessByQuery access rule in the GroupSpec. In case of an AlwaysAccessible access rule, this should be an empty array.Each eager indexing group must always contain { "name": "clean", "variables": [] }.
In case additive search indexes are used, each eager indexing group will be a singleton list. The indexes will be combined to match the user's allowed groups.
If the application only provides public data for unauthenticated users in the graph http://mu.semte.ch/graphs/public, the following eager indexing groups must be configured:
[
[{"name": "clean", "variables": []}, {"name": "public", "variables" : []}]
]
If, next to the public data, data is organized per organization unit in graphs like http://mu.semte.ch/graphs/<unit-name>, the following eager indexing groups must be configured:
[
[{"name": "clean", "variables": []}, {"name": "public", "variables" : []}],
[{"name": "clean", "variables": []}, {"name": "public", "variables" : []}, {"name": "organization-unit", "variables" : ["finance"]}],
[{"name": "clean", "variables": []}, {"name": "public", "variables" : []}, {"name": "organization-unit", "variables" : ["legal"]}]
]
In case non-additive indexes are used, an eager indexing group must be provided for each possible combination (permutation) of groups. For example, if some users have access to the data of the finance department as well as the legal department, the example above must be extended with the following eager indexing group:
[
...,
[{"name": "clean", "variables": []}, {"name": "public", "variables" : []}, {"name": "organization-unit", "variables" : ["finance"]}, {"name": "organization-unit", "variables" : ["legal"]}]
]
This how-to guide explains how to integrate mu-search with the delta-notification in order to automatically update search index entries when data in the triplestore is modified.
This guide assumes the mu-authorization and delta-notifier components have been added to your stack as explained in their respective installation guides.
Open the delta-notifier rules configuration ./config/delta/rules.js and add the following rule:
{
match: {
// listen to all changes
},
callback: {
url: 'http://search/update',
method: 'POST'
},
options: {
resourceFormat: "v0.0.1",
gracePeriod: 10000,
ignoreFromSelf: true
}
}
Enable automatic index updates (not only invalidation) in mu-search by setting the automatic_index_updates flag at the root of ./config/search/config.json.
{
"automatic_index_updates": true,
"types": [
// definition of the indexed types
]
}
Restart the search and delta-notifier service.
docker-compose restart search delta-notifier
Any change you make in your application will now trigger a request to the /update endpoint of mu-search. Depending on the indexed resources and properties, mu-search will update the appropriate search index entries.
This guide explains how to make the content of files attached to a project resource searchable in the index.
This guide assumes you have already integrated mu-search in your application and configured an index for resources of type schema:Project.
For indexing files mu-search requires a Tika server to extract the content. Add the tika service next to the search and elasticsearch services in docker-compose.yml:
services:
search:
...
elasticsearch:
...
tika:
image: apache/tika:1.25-full
Next, add the following mounted volumes to the mu-search service in docker-compose.yml:
/data: folder containing the files to be indexed/cache: folder to persist Tika's search cacheservices:
search:
image: semtech/mu-search:0.8.0-beta.3
volumes:
- ./config/search:/config
- ./data/files:/data
- ./data/search/cache:/cache
Next, add a property files in the project type index configuration. The property files will hold the content and metadata of the files.
{
"types" : [
{
"type" : "project",
"on_path" : "projects",
"rdf_type" : "http://schema.org/Project",
"properties" : {
"name" : "http://schema.org/name",
"files" : {
"via" : [
"http://purl.org/dc/terms/hasPart",
"^http://www.semanticdesktop.org/ontologies/2007/01/19/nie#dataSource"
],
"attachment_pipeline" : "attachment"
}
}
}
]
}
via expresses the path from the indexed resource to the file(s) having a URI like <share://path/to/your/file.pdf>.
Recreate the mu-search service using
docker-compose up -d
After reindex has completed, each indexed project will now contain a property files holding the content and metadata of the files linked to the project via dct:hasPart/^nie:dataSource.
Searching the file's content is done using the nested property content on the defined field name, files in this case:
GET /documents/search?filter[files.content]=open-source"
The content of a search index can be inspected by running a Kibana dashboard on top of Elasticseach.
[To be completed...]
Make sure not to expose the Kibana dashboard in a production environment!
[To be completed...]
Elasticsearch is used as search engine. It indexes documents according to a specified configuration and provides a REST API to search documents. The mu-search service is a layer in front of Elasticsearch that allows to specify the mapping between RDF triples and the Elasticsearch documents/properties. It also integrates with mu-authorization making sure users can only search for documents they're allowed to access.
This section describes how to configure the resources and properties to be indexed and how to pass Elasticsearch specific configurations and mapping in the mu-search configuration file.
This section describes how to mapping between RDF triples and Elasticsearch documents can be specified in the mounted /config/config.json configuration file.
The config.json file contains a JSON object with a property types. This property contains an array of objects, one per document type that must be searchable.
{
"types": [
// object per searchable document type
]
}
Note that these types do not map one-on-one with the search indexes in Elasticsearch. For each document type in the list a search index will be created per authorization group.
Each type object in the types array consists of the following properties:
properties contains a JSON object with a key per property in the resulting Elasticsearch document. These are the properties that will be searchable via the search API for the given resource type. The value of each key defines the mapping to RDF predicates starting from the root resource.
WARNING: there are two protected fields that should not be used as property keys: uuid and uri. Both are used internally by the mu-search service to store the uuid and URI of the root resource.
In the simplest scenario, the properties that need to be searchable map one-by-one on a predicate of the resource.
In the example below, a search index per user group will be created for documents and users. The documetns index contains resources of type foaf:Documents with a title and description. The users index contains foaf:Persons with only fullname as searchable property.
{
"types" : [
{
"type" : "document",
"on_path" : "documents",
"rdf_type" : "http://xmlns.com/foaf/0.1/Document",
"properties" : {
"title" : "http://purl.org/dc/elements/1.1/title",
"description" : "http://purl.org/dc/elements/1.1/description"
}
},
{
"type" : "user",
"on_path" : "users",
"rdf_type" : "http://xmlns.com/foaf/0.1/Person",
"properties" : {
"fullname" : "http://xmlns.com/foaf/0.1/name"
}
}
]
}
If multiple values are found in the triplestore for a given predicate, the resulting value for the property in the search document will be an array of all values.
A property of the search document may also map to an inverse predicate. I.e. resource to be indexed is the object instead of the subject of the triple. An inverse predicate can be indicated in the mapping by prefixing the predicate URI with ^ as done in a SPARQL query.
In the example below the users index contains a property group that maps to the inverse predicate foaf:member relating a group to a user.
{
"types" : [
{
"type" : "user",
"on_path" : "users",
"rdf_type" : "http://xmlns.com/foaf/0.1/Person",
"properties" : {
"fullname" : "http://xmlns.com/foaf/0.1/name",
"group": "^http://xmlns.com/foaf/0.1/member"
}
}
]
}
Properties can also be mapped to lists of predicates, corresponding to a property path in RDF. In this case, the property value is an array of strings. One string per path segment. The array starts from the indexed resource and may also include inverse predicate URIs.
In the example below the documents index contains a property topics that maps to the label of the document's primary topic and a property publishers that maps to the names of the publishers via the inverse foaf:publications predicate.
{
"types" : [
{
"type" : "document",
"on_path" : "documents",
"rdf_type" : "http://xmlns.com/foaf/0.1/Document",
"properties" : {
"title" : "http://purl.org/dc/elements/1.1/title",
"description" : "http://purl.org/dc/elements/1.1/description",
"topics" : [
"http://xmlns.com/foaf/0.1/primaryTopic",
"http://www.w3.org/2004/02/skos/core#prefLabel"
],
"publishers": [
"^http://xmlns.com/foaf/0.1/publications",
"http://xmlns.com/foaf/0.1/name"
]
}
}
]
}
To make the content of a file searchable, it needs to be indexed as a property in a search index. Basic indexing of PDF, Word etc. files is provided using Elasticsearch's Ingest Attachment Processor Plugin and a local Apache Tika instance. The plugin is already installed in the mu-semtech/search-elastic-backend image while the Tika server is running inside the mu-search container. A default ingest pipeline named attachment is created on startup of the mu-search service. Note that this is under development and liable to change.
Defining a property to index the content of a file requires the following keys:
share:// indicating the location of the file. E.g. <share://path/to/your/file.pdf>.attachment to use the default ingest pipeline.The example below adds a property files in the project type index configuration. The property files will hold the contents of the files related to the project via dct:hasPart/^nie:dataSource.
{
"types" : [
{
"type" : "project",
"on_path" : "projects",
"rdf_type" : "http://schema.org/Project",
"properties" : {
"name" : "http://schema.org/name",
"files" : {
"via" : [
"http://purl.org/dc/terms/hasPart",
"^http://www.semanticdesktop.org/ontologies/2007/01/19/nie#dataSource"
],
"attachment_pipeline" : "attachment"
}
}
}
]
}
For each file retrieved through the via-definition, the Tika-processing results in an object containing the extracted text (as content), as well as other extracted metadata (in the future). Such object may look like this:
{
content: "Extracted text here"
}
These objects are structured in the same way as the attachment objects resulting from the Elasticsearch's Ingest Attachment Processor Plugin. Keep in mind that this implies you need to specify the path to a specific property of the attachment object when defining an Elasticsearch mapping. E.g. mapping the file's content for the files field from the example above may look as follows:
{
"types": [
{
"type": "project",
"on_path": "projects",
...
"mappings" : {
"name" : { "type" : "text" },
"files.content" : { "type" : "text" }
}
},
// other type definitions
]
}
Currently, only indexing of local files is supported. The files' logical path as well as other metadata is expected to be in the format specified by the file-service. Files must be present in the Docker volume /data inside the container.
Attachments processed by Tika are cached in the directory /cache (by SHA256 of the file contents). This must be defined as a shared volume for the cache to be persistent.
See also "How to specify a file's content as property".
A search document can contain nested objects up to an arbitrary depth. For example for a person you can nest the address object as a property of the person search document.
A nested object is defined by the following properties:
Objects can be nested to arbitrary depth. The properties object is defined the same way as the properties of the root document, but the properties of a nested object cannot contain file attachments.
Elasticsearch mappings for nested objects must be specified in the mappings object at the root type using a path expression as key.
In the example below the document's creator is nested in the author property of the search document. The nested person object contains properties fullname and the current project's title as project.
{
"types" : [
{
"type" : "document",
"on_path" : "documents",
"rdf_type" : "http://xmlns.com/foaf/0.1/Document",
"properties" : {
"title" : "http://purl.org/dc/elements/1.1/title",
"description" : "http://purl.org/dc/elements/1.1/description",
"author" : {
"via" : "http://purl.org/dc/elements/1.1/creator",
"rdf_type" : "http://xmlns.com/foaf/0.1/Person",
"properties" : {
"fullname" : "http://xmlns.com/foaf/0.1/name",
"project": [
"http://xmlns.com/foaf/0.1/currentProject",
"http://purl.org/dc/elements/1.1/title"
]
}
}
},
"mappings": {
"title" : { "type" : "text" },
"author.fullname": { "type" : "text" }
}
}
]
}
A search index can contain documents of different types. E.g. documents (foaf:Document) as well as creative works (schema:CreativeWork). Currently, each simple type the composite index is constituted of must be defined seperately in the index configuration as well.
A definition of a composite type index consists of the following properties:
In contrast to the properties of a simple index, the properties of a composite index is an array. Each entry in the array is an object with the folliwng properties:
The example below contains 2 simple indexes for documents and creative works, and a composite index dossier containing both simple index types. The composite index contains (1) a property name mapping to the document's title and creative work's name property respectively, and (2) a property description mapping to the description property for both simple types.
{
"types" : [
{
"type" : "document",
"on_path" : "documents",
"rdf_type" : "http://xmlns.com/foaf/0.1/Document",
"properties" : {
"title" : "http://purl.org/dc/elements/1.1/title",
"description" : "http://purl.org/dc/elements/1.1/description"
}
},
{
"type" : "creative-work",
"on_path" : "creative-works",
"rdf_type" : "http://schema.org/CreativeWork",
"properties" : {
"name": "http://schema.org/name",
"description": "http://schema.org/description"
}
},
{
"type" : "dossier",
"composite_types" : ["document", "creative-work"],
"on_path" : "dossiers",
"properties" : [
{
"name" : "name",
"mappings" : {
"document" : "title",
"creative-work" : "name"
}
Content type
Image
Digest
sha256:45795d6de…
Size
183.8 MB
Last updated
22 days ago
docker pull semtech/mu-search