This service provides entity transformation capabilities as a preprocessing step for Bloom filter based PPRL.
To run an ephemeral container, use the line below.
docker run --rm -p 8080:8080 -d mds4ul/pprl-transform:latest
You can also compile this project by hand and run the standalone executable JAR.
Dependencies are stored in target/libs and are referenced in the JAR manifest file.
mvn clean package
cd target
java -jar pprl-transform-service-1.0-SNAPSHOT.jar
The service exposes a single endpoint at the root path.
Given the example above, the endpoint is reachable at http://localhost:8080.
Only JSON-formatted POST requests will be accepted.
{
// List of objects. An object represents a data record. All values of all objects must be strings.
// List cannot be empty.
"entities": [
{
// This field is required.
"id": "001",
// All other fields are optional.
"firstName": "Max",
"lastName": "Mustermann"
}
],
// Optional: list of pre-processing steps that should be applied to all attributes. List may be empty.
"globalTransformers": [
{
// Required: name of the pre-processing step.
"name": "norm",
// Required: "before" if this step should be performed before the attribute-specific
// transformation steps, "after" otherwise.
"order": "before"
}
],
// Optional: list of pre-processing steps that should be applied to specific attributes. List may be empty.
"attributeTransformers": [
{
// Required: name of the attribute that this pre-processing step applies to.
"attributeName": "firstName",
// List of pre-processing steps that apply to this attribute.
"transformers": [
{
// Required: name of the pre-processing step.
"name": "charFilter"
}
]
}
]
}
A transformer is an object that represents a data pre-processing step.
All transformers must have a name key which defines the pre-processing step itself.
If a transformer is defined in the attributeTransformers list, then it is executed on a single attribute.
If a transformer is defined in the globalTransformers list, then it is executed on all attributes.
Furthermore, global transformers must have an order key which defines whether the transformer is executed before or after all attribute transformers are executed.
{
// Other possible values: normalize, normalise, normalization, normalisation.
"name": "norm"
}
Removes ligatures and non-ASCII characters from a string, converts it to lowercase, removes whitespaces at the start and end and condenses multiple whitespaces between words into one.
{
// Other possible values: characterFilter.
"name": "charFilter",
// Optional: string containing characters that should be filtered out. (default: all special ASCII characters)
"characters": "!\"#$%&'()*+,-./:;<=>?@[\]^_`{|}~"
}
Removes special characters from a string.
If the characters key is not set, then this transformer will remove all special ASCII characters.
If the characters key is set, then this transformer will remove all defined characters.
{
"name": "dateTime",
// Required: date time input format.
"inputFormat": "yyyy-MM-dd",
// Required: date time output format.
"outputFormat": "yyyyMMdd"
}
Converts date time strings into a different format.
For a reference on date time formats, refer to the Java documentation on "Patterns for Formatting and Parsing".
This operation fails if inputFormat or outputFormat is invalid.
{
// Other possible values: mapping.
"name": "map",
// Required: object with keys and values as strings.
"mapping": {
"male": "m",
"female": "f"
},
// Optional: default value if a string is not found in the keys of the "mapping" object.
"default": "x",
// Optional: true if replacements should be performed on substrings within strings, false otherwise.
// This is false by default.
"inline": false
}
Replaces the keys in the mapping object with their respective values.
If inline is true, then this transformer will perform replacements within the string itself.
If inline is false, then this transformer will only perform replacements of entire strings.
Furthermore, if inline is false, default not set and no suitable replacement can be found for an input string, this operation fails.
{
"name": "number",
// Optional: amount of decimal places to keep.
"decimals": 2
}
Rounds fractional numbers to the specified amount of decimals.
If decimals is not set, then numbers will be rounded to the nearest integer.
"Half even" rounding is applied.
This means that 2.5 is rounded to 2, 2.6 to 3, 2.5 to 2 and -2.6 to 3.
This operation fails if the string does not contain a (fractional) number, or if decimals is not a number, or lower than zero.
{
// Other possible values: phoneticCode.
"name": "phonetic",
// Required: name of phonetic code algorithm (case-insensitive).
// Possible values: cologns, metaphone, double_metaphone, refined_soundex, soundex.
"algorithm": "soundex"
}
Converts a string into its phonetic code using the specified algorithm. This operation fails if the specified algorithm is invalid.
{
// List of objects. An object represents a pre-processed data record.
"entities": [
{
// This field is always present.
"id": "001",
// All other fields only exist if they were sent with the initial request.
"firstName": "max",
"lastName": "mustermann"
}
]
}
200 (OK) if the request was performed successfully.
400 (Bad request) if the request body is improperly formatted or otherwise invalid.
import json
import requests
r = requests.post("http://localhost:8080", json={
"entities": [
{
"id": "001",
"firstName": "Jámon",
"lastName": " juegos ",
"birthDate": "1998-06-29"
}
],
"globalTransformers": [
{
"name": "norm",
"order": "before"
}
],
"attributeTransformers": [
{
"attributeName": "birthDate",
"transformers": [
{
"name": "dateTime",
"inputFormat": "yyyy-MM-dd",
"outputFormat": "yyyyMMdd"
}
]
}
]
})
print(json.dumps(r.json(), indent=2))
# {
# "entities": [
# {
# "id": "001",
# "firstName": "jamon",
# "lastName": "juegos",
# "birthDate": "19980629"
# }
# ]
# }
MIT.
Content type
Image
Digest
sha256:52314b763…
Size
42 MB
Last updated
over 3 years ago
docker pull mds4ul/pprl-transform