smokserwis/seq-log-parser repository overview

⁠seq-log-parser

The project is available at GitHub⁠.

Have you ever noticed that standard syslog or gelf entries that Seq extracts with sqelf and squiflog are quite bland, ie. that they don't contain many useful properties, that make Seq such a powerhouse? No need to fear.

seq-log-parser is an intermediate proxy in which sqelf, squiflog and all the other things can relay their logging entries to. Seq-log-parser will then extract properties with the awesome power of regex and relay them straight back to the Seq server, even using the same API keys!

Oh, it's available as a Docker container⁠ as well!

⁠Rationale

Consider such docker-compose file:

version: '3.5'
services:
  seq:
    image: datalust/seq
    environment:
        ACCEPT_EULA: "Y"
  squiflog:
    image: datalust/squiflog
    environment:
      SEQ_ADDRESS: "http://seq:80"
    ports:
      - published: 514
        target: 514
        protocol: udp
        mode: ingress

In this case, say you configure your server to deposit logs there. Example log entry you might get is:

{
  "@t": "2020-05-03T20:41:39.0000000Z",
  "@mt": "kernel: [341690.053545] veth2c1b259: renamed from eth0",
  "@m": "kernel: [341690.053545] veth2c1b259: renamed from eth0",
  "@i": "5dceda5b",
  "facility": "kern",
  "hostname": "hypervisor1",
}

Not too useful, isn't it? Now say you use seq-log-parser in such a way:

version: '3.5'
services:
  seq:
    image: datalust/seq
    environment:
        ACCEPT_EULA: "Y"
  seq-log-parser:
    image: smokserwis/seq-log-parser
    environment:
      SEQ_ADDRESS: "http://seq"
      REGEX: "(?P<source>.*): \\[(?P<uptime(\\d+)\\.(\\d+))\] (?P<interface>.*): (?P<message>.*)"
      OVERWRITE_CONTENTS: "message"
  squiflog:
    image: datalust/squiflog
    environment:
      SEQ_ADDRESS: "http://seq-log-parser"
    ports:
      - published: 514
        target: 514
        protocol: udp
        mode: ingress

And now the log entry will look like this:

{
  "@t": "2020-05-03T20:41:39.0000000Z",
  "@mt": "renamed from eth0",
  "@m": "kernel: [341690.053545] veth2c1b259: renamed from eth0",
  "@i": "5dceda5b",
  "facility": "kern", 
  "Properties": {
    "source": "kernel",
    "uptime": "341690.053545", 
    "interface": "veth2c1b259",
    "message": "renamed from eth0"
  },
  "hostname": "hypervisor1"
}

Now isn't that much nicer?

⁠Usage

The container is configured by following envs:

Take care for your regexes to be valid Python named group regexes⁠. Don't forget about escaping the escape character if you're writing YAML for deployment!

What matches given named group will be added to log's Properties.

If you are using a list of regexes, and you want to add some kind of a property depending on which regex matched, you can specify envs called REGEX_PROPERTYi with i being the number of your regex. You specify them in a format key=value.

If you want to add a custom property but are using only a single regex, just name your env REGEX_PROPERTY and also specify it in form of key=value.

If an entry doesn't match any of the regexes, it will be sent as-is.

OVERWRITE_CONTENTS will update both the FIELD_TO_PARSE, as well as a MessageTemplate field, if it's present.

⁠Multiple regexes

If you input can be matched by multiple regexes, just specify them as environment variables REGEX1, REGEX2, REGEX3 instead of a single REGEX.