[{"_1":2,"_161":-5,"_162":-5},"loaderData",{"_3":4,"_17":18,"_25":26,"_140":141},"root",{"_5":-5,"_6":-7,"_7":8,"_9":-7,"_10":11,"_12":-7,"_13":14,"_15":16},"currentUser","primaryEmail","siteVerification","6LeIxAcTAAAAAJcZVRqyHh71UMIEGNQ_MXjiZKhI","toast","enableDev",false,"incidentStatus","theme","system","contrast","normal","routes/_layout",{"_19":-5,"_20":21,"_22":21,"_23":11,"_24":21},"username","isAskGordonEnabled",true,"isUsingNewTemplate","showMinimalLanding","showHoneypotTrap","routes/_layout.r.$namespace.$reponame",{"_27":28,"_29":-5,"_30":11,"_31":11,"_32":33,"_34":35,"_36":-5,"_37":11,"_38":39,"_88":-5,"_89":90,"_129":-5,"_130":11,"_131":21,"_132":11,"_133":-5,"_134":11,"_135":-5,"_136":137,"_138":91,"_139":-7},"canonicalUrl","https://hub.docker.com/r/3hren/blackhole/tags","mcpServer","canSeeDockerfile","canSeeAutobuilds","namespace","3hren","reponame","blackhole","cloudBuildSource","isAutobuildsAvailable","repo",{"_40":41,"_42":43,"_45":46,"_47":48,"_49":11,"_50":51,"_52":53,"_54":55,"_57":35,"_32":33,"_58":59,"_60":-5,"_61":62,"_63":64,"_65":66,"_67":-5,"_68":62,"_69":70,"_71":11,"_72":33,"_73":74,"_79":21,"_80":81,"_85":33,"_86":87},"categories",[],"content_types",[44],"image","date_registered","2016-07-27T12:17:19.056451Z","description","C++ Logger","is_private","last_updated","2016-08-03T15:45:38.838517Z","last_modified","2024-10-16T13:48:34.145251Z","media_types",[56],"application/vnd.docker.container.image.v1+json","name","pull_count",32443,"repository_type","star_count",0,"status",1,"status_description","active","source","collaborator_count","full_description","Blackhole - eating your logs with pleasure\n==========================================\n[![Build Status](https://travis-ci.org/3Hren/blackhole.svg?branch=master)][CI]\n\nBlackhole is an attribute-based logger with strong focus on gaining maximum performance as possible for such kind of loggers.\n\n# Features\n\n## Attributes\n\nAttributes is the core feature of Blackhole. Technically speaking it's a key-value pairs escorting every logging record.\n\nFor example we have HTTP/1.1 server which produces access logs like:\n\n```\n[::] - esafronov [10/Oct/2000:13:55:36 -0700] 'GET /porn.png HTTP/1.0' 200 2326 - SUCCESS\n```\n\nIt can be splitted into indexes or attributes:\n\n```\nmessage: SUCCESS\nhost: [::]\nuser: esafronov\ntimestamp: 10/Oct/2000:13:55:36 -0700\nmethod: GET\nuri: /porn.png\nprotocol: HTTP/1.0\nstatus: 200\nelapsed: 2326\n```\n\nBlackhole allows to specify any number of attributes you want, providing an ability to work with them before of while\nyou writing them into its final destination. For example, Elasticsearch.\n\n## Shared library\n\nDespite the header-only dark past now Blackhole is developed as a shared library. Such radical change of distributing\nprocess was chosen because of many reasons.\n\nMainly, header-only libraries has one big disadvantage: any code change may (or not) result in recompiling all its dependencies, otherwise having weird runtime errors with symbol loading race.\n\nThe other reason was the personal aim to reduce compile time, because it was fucking huge!\n\nOf course there are disadvantages, such as virtual function call cost and closed doors for inlining, but here my personal benchmark-driven development helped to avoid performance degradation.\n\n## Planning\n\n- [x] Shared library.\n- [x] Inline namespaces.\n- [x] Optional compile-time inline messages transformation (C++14).\n - [ ] Compile-time placeholder type checking.\n - [ ] Compile-time placeholder spec checking (?).\n- [x] Python-like formatting (no printf-like formatting support) both inline and result messages.\n- [x] Attributes.\n- [x] Scoped attributes.\n- [x] Wrappers.\n- [x] Custom verbosity.\n- [x] Custom attributes formatting.\n- [ ] Optional asynchronous pipelining.\n - [x] Queue with block on overload.\n - [x] Queue with drop on overload (count dropped message).\n - [ ] The same but for handlers.\n- [x] Formatters.\n - [x] String by pattern.\n - [ ] Optional placeholders.\n - [x] Configurable leftover placeholder.\n - [x] JSON with tree reconstruction.\n- [x] Sinks.\n - [x] Colored terminal output.\n - [x] Files.\n - [x] Syslog. \n - [x] Socket UDP.\n - [x] Socket TCP.\n - [x] Blocking.\n - [x] Non blocking.\n- [ ] Scatter-gathered IO (?)\n- [x] Logger builder.\n- [ ] Macro with line and filename attributes.\n- [x] Initializer from JSON (filename, string).\n- [ ] Inflector.\n- [ ] Filter category.\n - [x] Category type.\n - [x] For sinks.\n - [ ] For handlers.\n - [ ] For loggers.\n\n## Experimental\nNote, that there are some symbols, that are wrapped into `experimental` namespace. These symbols don't adhere semantic versioning and, well... experimental. Use them with caution and only, where you want to try super-unstable features, which can be changed or even dropped.\n\n## Formatters\n\nFormatters in Blackhole are responsible for converting every log record passing into some byte array representation. It can be either human-readable string, JSON tree or even [protobuf](https://github.com/google/protobuf) packed frame.\n\n### String\n\nString formatter provides an ability to configure your logging output using pattern mechanics with powerful\ncustomization support.\n\nUnlike previous Blackhole versions now string formatter uses python-like syntax for describing patterns with using `{}`\nplaceholders and format specifications inside. Moreover now you can specify timestamp specification directly inside the\ngeneral pattern or even format it as an microseconds number since epoch.\n\nFor example we have the given pattern:\n```\n[{severity:>7}] [{timestamp:{%Y-%m-%d %H:%M:%S.%f}s}] {scope}: {message}\n```\n\nAfter applying some log events we expect to receive something like this:\n```\n[ DEBUG] [2015-11-19 19:02:30.836222] accept: HTTP/1.1 GET - / - 200, 4238\n[ INFO] [2015-11-19 19:02:32.106331] config: server has reload its config in 200 ms\n[WARNING] [2015-11-19 19:03:12.176262] accept: HTTP/1.1 GET - /info - 404, 829\n[ ERROR] [2015-11-19 19:03:12.002127] accept: HTTP/1.1 GET - /info - 503, 829\n```\n\nAs you may notice the severity field is aligned to the right border (see that *>7* spec in pattern), the timestamp is\nformatted using default representation with a microseconds extension and so on. Because Blackhole is all about\nattributes you can place and format every custom attribute you want, as we just done with *scope* attribute.\n\nThe Blackhole supports several predefined attributes, with convenient specifications:\n\n| Placeholder | Description |\n|--------------------------|------------------------------------------------------------------------|\n|{severity:s} | User provided severity string representation |\n|{severity}, {severity:d} | Numeric severity value |\n|{timestamp:d} | Number of microseconds since Unix epoch |\n|{timestamp:{spec}s} | String representation using *strftime* specification in UTC |\n|{timestamp:{spec}l} | String representation using *strftime* specification in local timezone |\n|{timestamp}, {timestamp:s}| The same as *{timestamp:{%Y-%m-%d %H:%M:%S.%f}s}* |\n|{process:s} | Process name |\n|{process}, {process:d} | PID |\n|{thread}, {thread::x} | Thread hex id as an opaque value returned by *pthread_self(3)* |\n|{thread:s} | Thread name or *unnnamed* |\n|{message} | Logging message |\n|{...} | All user declared attributes |\n\nFor more information please read the documentation and visit the following links:\n\n - http://cppformat.github.io/latest/syntax.html - general syntax.\n - http://en.cppreference.com/w/cpp/chrono/c/strftime - timestamp spec extension.\n\nNote, that if you need to include a brace character in the literal text, it can be escaped by doubling: `{{` and `}}`.\n\nThere is a special attribute placeholder - `{...}` - which means to print all non-reserved attributes in a reverse order they were provided in a key-value manner separated by a comma. These kind of attributes can be configured using special syntax, similar with the timestamp attribute with an optional separator.\n\nFor example the following placeholder `{...:{{name}={value}:p}{\\t:x}s}` results in tab separated key-value pairs like `id=42\\tmethod=GET`.\n\nFor pedants there is a full placeholder grammar in EBNF:\n```\nGrammar = Ph\n | OptPh\n | VarPh\nPh = \"{\" Name \"}\"\nOptPh = \"{\" Name \":\" Spec? \"}\"\nVarPh = \"{...}\"\n | \"{...:\" Ext? s \"}\"\nExt = Pat\n | Sep\n | Pat Sep\n | Sep Pat\nName = [a-zA-Z0-9_]\nSpec = Fill? Align? Width? Type\nFill = [a character other than '{' or '}']\nAlign = [>^<]\nWidth = [1-9][0-9]*\nType = [su]\nPat = \"{\" PatSpec \":p}\"\nSep = \"{\" SepLit* \":s}\" (\"}\" SepLit* \":s}\")*\nSepLit = . ! (\":s\" | \"}\" | \"}}\" | \"{\" | \"{{\")\n | LeBrace\n | RiBrace\nLeBrace = \"{{\" -> \"{\"\nRiBrace = \"}}\" -> \"}\"\nPatSpec = (AtName | AtValue | PatLit)*\nAtName = \"{name}\"\n | AtNameSpec\nAtNameSpec = \"{name:\" AtSpec \"}\"\nAtSpec = Align? Width? AtType\nAtType = [sd]\nAtValue = \"{value}\"\n | AtValueSpec\nAtValueSpec = \"{value:\" AtSpec \"}\"\nPatLit = . ! (\"}\" | \"}}\" | \"{\" | \"{{\")\n | LeBrace\n | RiBrace\n```\n\nLet's describe it more precisely. Given a complex leftover placeholder, let's parse it manually to see what Blackhole see.\nGiven: `{...:{{name}={value}:p}{\\t:s}>50s}`.\n\nParameter | Description \n-------------------|--------------------------------------------------------------------------------------------------\n... | Reserved placeholder name indicating for Blackhole that this is a leftover placeholder.\n: | Optional spec marker that is placed after placeholder name where you want to apply one of several extensions. There are pattern, separator, prefix, suffix and format extensions. All of them except format should be surrounded in curly braces.\n{{name}={value}:p} | Pattern extension that describes how each attribute should be formatted using typical Blackhole notation. The suffix **:p**, that is required for extension identification, means pattern. Inside this pattern you can write any pattern you like using two available sub-placeholders for attribute name and value, for each of them a format spec can be applied using cppformat grammar. At last a format spec can be also applied to the entire placeholder, i.e. **:>50p** for example.\n{\\t:s} | Separator extension for configuring each key-value pair separation. Nuff said.\n{[:r} | (Not implemented yet). Prefix extension that is prepended before entire result if it is not empty.\n{]:u} | (Not implemented yet). Suffix extension that is appended after entire result if it is not empty.\n>50s | Entire result format. See cppformat rules for specification.\n\n### JSON.\nJSON formatter provides an ability to format a logging record into a structured JSON tree with attribute handling features, like renaming, routing, mutating and much more.\n\nBriefly using JSON formatter allows to build fully dynamic JSON trees for its further processing with various external tools, like logstash or rsyslog lefting it, however, in a human-readable manner.\n\nBlackhole allows you to control of JSON tree building process using several predefined options.\n\nWithout options it will produce just a plain tree with zero level depth. For example for a log record with a severity of 3, message \"fatal error, please try again\" and a pair of attributes `{\"key\": 42, \"ip\": \"[::]\"}` the result string will look like:\n```json\n{\n \"message\": \"fatal error, please try again\",\n \"severity\": 3,\n \"timestamp\": 1449859055,\n \"process\": 12345,\n \"thread\": 57005,\n \"key\": 42,\n \"ip\": \"[::]\"\n}\n```\nUsing configuration parameters for this formatter you can:\n- Rename parameters.\n- Construct hierarchical tree using a standardized JSON pointer API. For more information please follow \\ref https://tools.ietf.org/html/rfc6901.\n\nAttributes renaming acts so much transparently as it appears: it just renames the given attribute name using the specified alternative.\n\nAttributes routing specifies a location where the listed attributes will be placed at the tree construction. Also you can specify a default location for all attributes, which is \"/\" meaning root otherwise.\n\nFor example with routing `{\"/fields\": [\"message\", \"severity\"]}` and \"/\" as a default pointer the mentioned JSON will look like:\n```json\n{\n \"fields\": {\n \"message\": \"fatal error, please try again\",\n \"severity\": 3\n },\n \"timestamp\": 1449859055,\n \"process\": 12345,\n \"thread\": 57005,\n \"key\": 42,\n \"ip\": \"[::]\"\n}\n```\n\nAttribute renaming occurs after routing, so mapping \"message\" => \"#message\" just replaces the old name with its new alternative.\n\nTo gain maximum speed at the tree construction no filtering occurs, so this formatter by default allows duplicated keys, which means invalid JSON tree (but most of parsers are fine with it). If you are really required to deal with unique keys, you can enable `unique` option, but it involves heap allocation and may slow down formatting.\n\nAlso formatter allows to automatically append a newline character at the end of the tree, which is strangely required by some consumers, like logstash.\n\nNote, that JSON formatter formats the tree using compact style without excess spaces, tabs etc.\n\nFor convenient formatter construction a special builder class is implemented allowing to create and configure instances of this class using streaming API. For example:\n```c++\nauto formatter = blackhole::formatter::json_t::builder_t()\n .route(\"/fields\", {\"message\", \"severity\", \"timestamp\"})\n .route(\"/other\")\n .rename(\"message\", \"#message\")\n .rename(\"timestamp\", \"#timestamp\")\n .newline()\n .unique()\n .build();\n```\n\nThis allows to avoid hundreds of constructors and to make a formatter creation to look eye-candy.\n\nThe full table of options:\n\nOption | Type | Description\n------------|-------------------|---------------\n**/route** | Object of:
[string]
\"\\*\" | Allows to configure nested tree mapping. Each key must satisfy [JSON Pointer](https://tools.ietf.org/html/rfc6901) specification and sets new attributes location in the tree. Values must be either an array of string, meaning list of attributes that are configured with new place or an \"\\*\" literal, meaning all other attributes.\n**/mapping** | Object of: [string] | Simple attribute names renaming from key to value.\n**/newline** | bool | If true, a newline will be appended to the end of the result message. The default is _false_.\n**/unique** | bool | If true removes all backward consecutive duplicate elements from the attribute list range. For example if there are two attributes with name \"name\" and values \"v1\" and \"v2\" inserted, then after filtering there will be only the last inserted, i.e. \"v2\". The default is _false_.\n**/mutate/timestamp** | string | Replaces the timestamp field with new value by transforming it with the given _strftime_ pattern.\n**/mutate/severity** | [string] | Replaces the severity field with the string value at the current severity value.\n\nFor example:\n```json\n\"formatter\": {\n \"type\": \"json\",\n \"newline\": true,\n \"unique\": true,\n \"mapping\": {\n \"message\": \"@message\",\n \"timestamp\": \"@timestamp\"\n },\n \"routing\": {\n \"\": [\"message\", \"timestamp\"],\n \"/fields\": \"*\"\n },\n \"mutate\": {\n \"timestamp\": \"%Y-%m-%dT%H:%M:%S.%fZ\",\n \"severity\": [\"D\", \"I\", \"W\", \"E\"]\n }\n}\n```\n\n## Sinks\n\n### Null\nSometimes we need to just drop all logging events no matter what, for example to benchmarking purposes. For these cases there is null output (or sink), which just ignores all records.\n\nThe common configuration for this sink looks like:\n\n```json\n\"sinks\": [\n {\n \"type\": \"null\"\n }\n]\n```\n\n### Console\nRepresents a console sink which is responsible for writing all incoming log events directly into the terminal using one of the selected standard outputs with an ability to optionally colorize result strings.\n\nThe sink automatically detects whether the destination stream is a TTY disabling colored output otherwise, which makes possible to redirect standard output to file without escaping codes garbage.\n\nNote, that despite of C++ `std::cout` and `std::cerr` thread-safety with no undefined behavior its guarantees is insufficiently for safe working with them from multiple threads, leading to result messages intermixing. To avoid this a global mutex is used internally, which is kinda hack. Any other stdout/stderr usage outside from logger will probably results in character mixing, but no undefined behavior will be invoked.\n\nThe configuration:\n\n```json\n\"sinks\": [\n {\n \"type\": \"console\"\n }\n]\n```\n\nNote, that currently coloring cannot be configured through dynamic factory (i.e through JSON, YAML etc.), but can be through the builder.\n\n```cpp\nenum severity {\n debug = 0,\n info,\n warn,\n error\n};\n\nauto console = blackhole::builder()\n .colorize(severity::debug, blackhole::termcolor_t())\n .colorize(severity::info, blackhole::termcolor_t::blue())\n .colorize(severity::warn, blackhole::termcolor_t::yellow())\n .colorize(severity::error, blackhole::termcolor_t::red())\n .stdout()\n .build();\n```\n\n### File\nRepresents a sink that writes formatted log events to the file or files located at the specified path.\n\nThe path can contain attribute placeholders, meaning that the real destination name will be deduced at runtime using provided log record (not ready yet). No real file will be opened at construction time. All files are opened by default in append mode meaning seek to the end of stream immediately after open.\n\nThis sink supports custom flushing policies, allowing to control hardware write load. There are three implemented policies right now:\n\n- Fully automatic (without configuration), meaning that the sink will decide whether to flush or not after each record consumed.\n- Count of records written - this is the simple counter with meaning of \"flush at least every N records consumed\", but the underlying implementation can decide to do it more often. The value of 1 means that the sink will flush after every logging event, but this results in dramatically performance degradation.\n- By counting of number of bytes written - Blackhole knows about bytes, megabytes, even mibibytes etc.\n\nNote, that it's guaranteed that the sink always flush its buffers at destruction time. This guarantee with conjunction of thread-safe logger reassignment allows to implement common SIGHUP files reopening during log rotation.\n\nBlackhole won't create intermediate directories, because of potential troubles with ACL. Instead an exception will be thrown, which will be anyway caught by the internal logging system notifying through stdout about it.\n\nNote, that associated files will be opened on demand during the first write operation.\n\n```json\n\"sinks\": [\n {\n \"type\": \"file\",\n \"flush\": \"10Mb\",\n \"path\": \"/var/log/blackhole.log\"\n }\n]\n```\n\nBlackhole knows about the following marginal binary units:\n\n- Bytes (B).\n- Kilobytes (kB).\n- Megabytes (MB).\n- Gigabytes (GB).\n- Kibibytes (KiB).\n- Mibibytes (MiB).\n- Gibibytes (GiB).\n\nMore you can read at https://en.wikipedia.org/wiki/Binary_prefix.\n\n### Socket\nThe socket sinks category contains sinks that write their output to a remote destination specified by a host and port. Currently the data can be sent over either TCP or UDP.\n\n#### TCP\nThis appender emits formatted logging events using connected TCP socket.\n\n| Option | Type | Description|\n|--------|:-----:|------------|\n|host |string | **Required**.
The name or address of the system that is listening for log events. |\n|port |u16 | **Required**.
The port on the host that is listening for log events. |\n\n#### UDP\nNuff said.\n\n#### Syslog\n| Option | Type | Description |\n|-----------|:-----:|-----------------------------------------------------------|\n|priorities |[i16] | **Required**.
Priority mapping from severity number. |\n\n## Configuration\nBlackhole can be configured mainly in two ways:\n- Using *experimental* builder.\n- Using abstract factory (GoF, yeah).\n\n### Builder\nThe first way involves using _experimental_ yet builder. For each library component (formatter, sink, etc.) there should be appropriate builder specialization that is used to create instances of associated component in a flow-way.\n\nFor example:\n```c++\n// Here we are going to configure our string/console handler and to build the logger.\nauto log = blackhole::experimental::partial_builder()\n // Add the blocking handler.\n .handler()\n // Configure string formatter.\n //\n // Pattern syntax behaves like as usual substitution for placeholder. For example if\n // the attribute named `severity` has value `2`, then pattern `{severity}` will invoke\n // severity mapping function provided and the result will be `W`.\n .set(\"{severity}, [{timestamp}]: {message}\")\n .mapping(&sevmap)\n .build()\n // Configure console sink to write into stdout (also stderr can be configured).\n .add()\n .build()\n // And build the handler. Multiple handlers can be added to a single logger, but right\n // now we confine ourselves with a single handler.\n .build()\n // Build the logger.\n .build();\n```\n\nThe result is a `std::unique_ptr where C: Component`, sorry for my Rust.\n\nThis is also called *static initialization*, because you must know the configuration of your logging system at compile time. If this isn't suit for you there is another way.\n\n### Factory\nAlso called as *dynamic initialization*, and is the recommended way to configure the Blackhole, because it implements some kind of dependency injection through some external source, like JSON file, XML, or `folly::dynamic`.\n\nBlackhole for now implements only initialization from JSON, but it can be easily extended as a plugin, because all you need is just to implement proper interface to allow tree-like traversing through your config object.\n\nHere there is an example how to configure the library from JSON file.\n\n```c++\n// Here we are going to build the logger using registry. The registry's responsibility is to\n// track registered handlers, formatter and sinks, but for now we're not going to register\n// anything else, since there are predefined types.\nauto log = blackhole::registry::configured()\n // Specify the concrete builder type we want to use. It may be JSON, XML, YAML or whatever\n // else.\n ->builder(std::ifstream(argv[1]))\n // Build the logger named \"root\".\n .build(\"root\");\n```\n\nThe result is a `std::unique_ptr` object.\n\nFor more information see [blackhole::registry_t](https://github.com/3Hren/blackhole/blob/master/include/blackhole/registry.hpp#L27) class and the [include/blackhole/config](include/blackhole/config) where all magic happens. If you look for an example how to implement your own factory, please see [src/config](src/config) directory.\n\n## Facade\nOne can say that the raw logger interface is inconvenient, and this is true, unfortunately, because it must work both in simple cases, where intermediate message formatting is not required, without attributes; and in complex cases, where lazy message formatting occurs, with attributes provided, remaining at the same time as fast as possible, giving a high-performance solution.\n\nLet's take a look on the interface:\n\n```cpp\nclass logger_t {\npublic:\n virtual ~logger_t() = 0;\n virtual auto log(severity_t severity, const message_t& message) -> void = 0;\n virtual auto log(severity_t severity, const message_t& message, attribute_pack& pack) -> void = 0;\n virtual auto log(severity_t severity, const lazy_message_t& message, attribute_pack& pack) -> void = 0;\n\n virtual auto manager() -> scope::manager_t& = 0;\n};\n```\n\nTo avoid manually creating all these structures a special extension is provided: facade. In two words it is a thin template adapter over any given logger which extends its interface, providing methods that makes logging convenient again. We describe all these methods by abusing a random HTTP logging event of success file serve.\n\nFor simple cases there is a thin wrapper that transforms a string into string view and passes it further.\n\n```cpp\nlogger.log(0, \"GET /static/image.png HTTP/1.1 436 200\");\n```\n\nSometimes we want to provide additional attributes. In these cases they can be passed using initializer list.\n\n```cpp\nlogger.log(0, \"GET /static/image.png HTTP/1.1 436 200\", {\n {\"cache\", true},\n {\"elapsed\", 435.72},\n {\"user-agent\", \"Mozilla Firefox\"}\n});\n```\n\nOften we want to format a message using predefined pattern, but with arguments obtained at runtime.\n\n```cpp\nlogger.log(0, \"{} {} HTTP/1.1 {} {}\", \"GET\", \"/static/image.png\", 436, 200);\n```\n\nAt last, we can combine two previous examples to obtain something really useful. Note that attribute list argument must be the last.\n\n```cpp\nlogger.log(0, \"{} {} HTTP/1.1 {} {}\", \"GET\", \"/static/image.png\", 436, 200, attribute_list{\n {\"cache\", true},\n {\"elapsed\", 435.72},\n {\"user-agent\", \"Mozilla Firefox\"}\n});\n```\n\nTo use it all you need is to create a logger, import the facade definition and wrap the logger with it. We show you an improved example:\n\n```cpp\n/// This example demonstrates how to initialize Blackhole from configuration file using JSON\n/// builder.\n/// In this case the entire logging pipeline is initialized from file including severity mapping.\n/// The logging facade is used to allow runtime formatting and attributes provisioning.\n\n#include \n#include

3hren/blackhole

Tags for 3hren/blackhole