philipssoftware/fluent-bit-out-hsdp repository overview

⁠fluent bit HSDP logging output plugin

This plugin outputs your logs to the HSDP Host Logging service. This is useful when your workloads are not running on Cloud foundry, but you still want to utilize the central logging facilities of HSDP.

Fluent bit supports parser and filter plugin which can convert unstructured data gathered from the log Input interface into a structured one and to alter existing structured data before ingestion.

More on fluent-bit⁠

⁠Cloud and On-Premise

The plugin supports deployment to both Cloud and On-Premise environments. Depending on the deployment type you can either specify the Cloud Region and Environment or the On-Premise IamUrl and IdmUrl values.

⁠API Signing or Service Identities

The plugin supports both the API Signing authorization mechanism or the use of a IAM Service Identity with the LOG.CREATE scope

⁠Configuration options

Your fluent-bit.conf file should include an entry like below to enable the plugin:

[output]
    Name hsdp
    Match *

Configuring the authorization mechanism and HSDP Logging endpoints should ideally be done by setting the right Environment variables:

⁠Settings

⁠General

⁠Signing keys

⁠Service identities

⁠Record field mapping to HSDP logging resource

The plugin supports full pass-through of the native LogEvent JSON message type, example:

{
  "resourceType": "LogEvent",
  "id": "7f4c85a8-e472-479f-b772-2916353d02a4",
  "applicationName": "OPS",
  "eventId": "110114",
  "category": "TRACELOG",
  "component": "TEST",
  "transactionId": "2abd7355-cbdd-43e1-b32a-43ec19cd98f0",
  "serviceName": "OPS",
  "applicationInstance": "INST‐00002",
  "applicationVersion": "1.0.0",
  "originatingUser": "SomeUsr",
  "serverName": "ops-dev.cloud.pcftest.com",
  "logTime": "2017-01-31T08:00:00Z",
  "severity": "INFO",
  "logData": {
    "message": "VGVzdCBNZXNzYWdl"
  },
  "custom": {
    "foo": "bar"
  }
}

⁠field mapping

Alternatively, the plugin maps certain record fields to defined HSDP logging resource fields. The below table shows the mapping, and the default value.

The below filter definition shows an example of assigning fields

[filter]
    Name record_modifier
    Match *
    Record server_name ${HOSTNAME}
    Record service_name Awesome_Tool

[filter]
    Name modify
    Match *
    Copy container_name app_name
    Copy container_name service_name
    Copy component_name component
    Copy container_id app_instance

⁠Building

docker build -t fluent-bit-out-hsdp .

⁠Testing with Docker

docker run --rm \
    -p 127.0.0.1:24224:24224 \
    -e HSDP_PRODUCT_KEY=product-key-here \
    -e HSDP_REGION=us-east \
    -e HSDP_ENVIRONMENT=client-test \
    -e [email protected] \
    -e HSDP_SERVICE_PRIVATE_KEY="$(cat service_private_key.pem)" \
    -it philipssoftware/fluent-bit-out-hsdp:latest

Once the above is running you can start other Docker containers and use fluentd log driver to start logging to HSDP logging:

docker run --rm -it --log-driver fluentd alpine echo "hello world"

⁠Contact / Getting help

Andy Lo-A-Foe [email protected]⁠

⁠License

License is MIT