cafdataprocessing/worker-familyhashing repository overview

⁠worker-familyhashing

The Family Hashing Worker is used to identify elements within a document or e-mail. For example, given an e-mail thread the Family Hashing Worker is able to identify the individual e-mails that make up the thread and to mark these up (using xml markup tags). It can recognise individual headers and other elements of the e-mail and similarly mark them up for further processing. In addition to its recognition capabilities the Family Hashing Worker is also able to apply normalisations against data and to generate hash digests, in order to help with comparing the data. Ultimately these facilities can be combined to identify related e-mails, such as replied-to and forwarded e-mails, thereby allowing e-mails to be organised into a conversation tree.

The Family Hashing Worker is a Document Worker implementation that uses the worker-javascript⁠ library to call a JavaScript file that contains multiple hash and output fields configurations for a received document. Conditions are used in the JavaScript to determine which configurations to use based on the fields on the document. This JavaScript then calls the worker-markup-core⁠ library to perform the markup operation on the document with the matched configurations.

⁠E-mail Markup

When used with e-mails, the worker splits e-mails in the e-mail thread, identifies metadata such as headers and body content, and marks up the text with XML. Hash values will be generated based on the user supplied configuration on a per-task basis and added to the marked up document. The worker will extract values from the XML document based on user defined output field names and XPath expressions. It then returns these field names and values as a list of name-value pairs.

For more details about the Markup Core library used by this worker visit the worker-markup⁠ repository.