Public | Automated Build

Last pushed: 21 days ago
Short Description
Smart logic concept transformer
Full Description

smartlogic-concept-transformer

Introduction

This service will listen to Kafka for a notification of a change made in Smartlogic convert the JSON-LD in the message to a normalised UPP view of a concept and send the JSON to the concepts-rw-s3

Installation

Download the source code, dependencies and test dependencies:

    go get -u github.com/kardianos/govendor
    go get -u github.com/Financial-Times/smartlogic-concept-transformer
    cd $GOPATH/src/github.com/Financial-Times/smartlogic-concept-transformer
    govendor sync
    go build .

Running locally

  1. Run the tests and install the binary:

     govendor sync
     govendor test -v -race
     go install
    
  2. Run the binary (using the help flag to see the available optional arguments):

     $GOPATH/bin/smartlogic-concept-transformer [--help]
    

Options:

    --app-system-code="smartlogic-concept-transformer"      System Code of the application ($APP_SYSTEM_CODE)
    --app-name="smartlogic-concept-transformer"             Application name ($APP_NAME)
    --port="8080"                                           Port to listen on ($APP_PORT)
    --kafka-address=""                                      Kafka Address
    --kafka-topic="smartlogic-concept"                      Kafka Topic for Smartlogic concepts
    --kafka-group="smartlogic-concept-transformer"          Kafka Group for this service 
    --writer-endpoint="http://localhost:8080/"              Routing for the RW App                             

Build and deployment

Utility endpoints

See the api/api.yml for the swagger definitions of the endpoints

POST /transform

This endpoint is for testing and help ongoing support. This endpoint only transforms the JSON-LD payload and returns the UPP source representation but doesn’t send it on down the pipeline to the concept-rw-s3

Using curl:

curl -X POST -i https://user:pass@pub-prod-up.ft.com/__smartlogic-concept-transformer/transform/send --d @payload.txt --header "Content-Type:application/json"

Payload.txt:

{
  "@graph": [
    {
      "@id": "http://www.ft.com/thing/018be9f8-22ff-459e-b605-288b101362e3",
      "@type": [
        "http://www.ft.com/ontology/product/Brand"
      ],
      "http://www.ft.com/ontology/description": [
        {
          "@language": "en",
          "@value": "<p><span>This is a concepts description </span><br/></p>"
        }
      ],
      "http://www.ft.com/ontology/subBrandOf": [
        {
          "@id": "http://www.ft.com/thing/d1748734-e917-4217-a2cd-e7c19717dd4b"
        }
      ],
      "http://www.ft.com/thing/_logoURL": [
        {
          "@language": "en",
          "@value": "<p><img src=\"http://im.ft-static.com/content/images/d5ffade2-99ea-11e6-8f9b-70e3cabccfae.png\"/></p>"
        }
      ],
      "sem:guid": [
        {
          "@value": "efd4a777-c2c4-4a04-9c52-1f1485196338"
        }
      ],
      "skosxl:prefLabel": [
        {
          "@id": "http://www.ft.com/thing/efd4a777-c2c4-4a04-9c52-1f1485196338/Lex_en",
          "skosxl:literalForm": [
            {
              "@language": "en",
              "@value": "Lex"
            }
          ]
        }
      ]
    }
  ],
  "@context": {}
  }
}

Based on the following google doc.

POST /transform/send

Transforms smartlogic payload into the upp representation of concordance and sends result to concepts-rw-s3

Using curl:

curl -X POST -i https://user:pass@pub-prod-up.ft.com/__smartlogic-concept-transformer/transform/send --d @payload.txt --header "Content-Type:application/json"

Payload.txt:

{
  "@graph": [
    {
      "@id": "http://www.ft.com/thing/018be9f8-22ff-459e-b605-288b101362e3",
      "@type": [
        "http://www.ft.com/ontology/product/Brand"
      ],
      "http://www.ft.com/ontology/description": [
        {
          "@language": "en",
          "@value": "<p><span>This is a concepts description </span><br/></p>"
        }
      ],
      "http://www.ft.com/ontology/subBrandOf": [
        {
          "@id": "http://www.ft.com/thing/d1748734-e917-4217-a2cd-e7c19717dd4b"
        }
      ],
      "http://www.ft.com/thing/_logoURL": [
        {
          "@language": "en",
          "@value": "<p><img src=\"http://im.ft-static.com/content/images/d5ffade2-99ea-11e6-8f9b-70e3cabccfae.png\"/></p>"
        }
      ],
      "sem:guid": [
        {
          "@value": "efd4a777-c2c4-4a04-9c52-1f1485196338"
        }
      ],
      "skosxl:prefLabel": [
        {
          "@id": "http://www.ft.com/thing/efd4a777-c2c4-4a04-9c52-1f1485196338/Lex_en",
          "skosxl:literalForm": [
            {
              "@language": "en",
              "@value": "Lex"
            }
          ]
        }
      ]
    }
  ],
  "@context": {}
  }
}

The expected response will give us a UPP source system representation of this smart logic concept

e.g

HTTP/1.1 200 OK
Content-Type: application/json
X-Request-Id: transaction ID, e.g. tid_etmIWTJVeA
X
{
  "uuid": "018be9f8-22ff-459e-b605-288b101362e3",
  "prefLabel": "Lex",
  "type": "Brand",
  "authority": "SmartLogic",
  “parentBrandUuids”: [
    "d1748734-e917-4217-a2cd-e7c19717dd4b"
],
  "authorityValue": "efd4a777-c2c4-4a04-9c52-1f1485196338",
  "description": "This is a concepts description",
  "strapline": "Another FT concept managed in Smart Logic",
  "descriptionXML": "<p>This is a concepts description XML</p>",
  "_imageUrl": "http://im.ft-static.com/content/images/d5ffade2-99ea-11e6-8f9b-70e3cabccfae.png"
}

Based on the following google doc.

Healthchecks

Admin endpoints are:

/__gtg

/__health

/__build-info

There are several checks performed:

  • Checks that a connection can be made to the concepts-rw-s3 service

Logging

  • The application uses logrus; the log file is initialised in main.go.
  • Logging requires an env app parameter, for all environments other than local logs are written to file.
  • When running locally, logs are written to console. If you want to log locally to file, you need to pass in an env parameter that is != local.
  • NOTE: /__build-info and /__gtg endpoints are not logged as they are called every second from varnish/vulcand and this information is not needed in logs/splunk.
Docker Pull Command
Owner
coco

Comments (0)