siim/fluent-bit-for-operator repository overview
Fluent Bit
This project is a fork of fluentd/fluent-bit. Due to the known issue, the original Fluent Bit doesn't support dynamic configuration. To address that, we incorporate a configuration reloader on top of the original.
Development
The following files have been either modified or added in comparison with the original Fluent Bit:
- main.go: implementes configuration reloader in Golang.
- Dockerfile: launches main.go instead of fluent bit core.
- fluent-bit.conf: links to
/fluent-bit/config/fluent-bit.conf, the location for dynamic configuration
<<<<<<< HEAD Fluent Bit runs on x86_64, x86, arm32v7 and arm64v8 architectures.
Features
- High Performance
- Data Parsing
- Reliability and Data Integrity
- Backpressure Handling
- Data Buffering in memory and file system
- Networking
- Security: built-in TLS/SSL support
- Asynchronous I/O
- Pluggable Architecture and Extensibility: Inputs, Filters and Outputs
- More than 50 built-in plugins available
- Extensibility
- Write any input, filter or output plugin in C language
- Bonus: write Filters in Lua or Output plugins in Golang
- Monitoring: expose internal metrics over HTTP in JSON and Prometheus format
- Stream Processing: Perform data selection and transformation using simple SQL queries
- Create new streams of data using query results
- Aggregation Windows
- Data analysis and prediction: Timeseries forecasting
- Portable: runs on Linux, MacOS, Windows and BSD systems
Fluent Bit in Production
Fluent Bit is used widely, just on 2019 it was deployed more than 62 Million times. The following is a preview of who uses Fluent Bit heavily in production:
If your company uses Fluent Bit and is not listed, feel free to open a Github issue and we will add the logo.
Documentation
Our official project documentation for installation, configuration, deployment and development topics is located here:
Quick Start
Build from Scratch
If you aim to build Fluent Bit from sources, you can go ahead and start with the following commands.
$ cd build
$ cmake ..
$ make
$ bin/fluent-bit -i cpu -o stdout -f 1
If you are interested into more details, please refer to the Build & Install section.
Linux Packages
We provide packages for most common Linux distributions:
Linux / Docker Container Images
Our Linux containers images are the most common deployment model, thousands of new installation happen every day, learn more about the available images and tags here:
Windows Packages
Fluent Bit is fully supported on Windows environments, get started with the following instructions:
Plugins: Inputs, Filters and Outputs
Fluent Bit is based in a pluggable architecture where different plugins plays a major role in the data pipeline:
Input Plugins
| name | title | description |
|---|---|---|
| collectd | Collectd | Listen for UDP packets from Collectd. |
| cpu | CPU Usage | measure total CPU usage of the system. |
| disk | Disk Usage | measure Disk I/Os. |
| dummy | Dummy | generate dummy event. |
| exec | Exec | executes external program and collects event logs. |
| forward | Forward | Fluentd forward protocol. |
| head | Head | read first part of files. |
| health | Health | Check health of TCP services. |
| kmsg | Kernel Log Buffer | read the Linux Kernel log buffer messages. |
| mem | Memory Usage | measure the total amount of memory used on the system. |
| mqtt | MQTT | start a MQTT server and receive publish messages. |
| netif | Network Traffic | measure network traffic. |
| proc | Process | Check health of Process. |
| random | Random | Generate Random samples. |
| serial | Serial Interface | read data information from the serial interface. |
| stdin | Standard Input | read data from the standard input. |
| syslog | Syslog | read syslog messages from a Unix socket. |
| systemd | Systemd | read logs from Systemd/Journald. |
| tail | Tail | Tail log files. |
| tcp | TCP | Listen for JSON messages over TCP. |
| thermal | Thermal | measure system temperature(s). |
Filter Plugins
| name | title | description |
|---|---|---|
| grep | Grep | Match or exclude specific records by patterns. |
| kubernetes | Kubernetes | Enrich logs with Kubernetes Metadata. |
| lua | Lua | Filter records using Lua Scripts. |
| parser | Parser | Parse record. |
| record_modifier | Record Modifier | Modify record. |
| stdout | Stdout | Print records to the standard output interface. |
| throttle | Throttle | Apply rate limit to event flow. |
| nest | Nest | Nest records under a specified key |
| modify | Modify | Modifications to record. |
Output Plugins
| name | title | description |
|---|---|---|
| azure | Azure Log Analytics | Ingest records into Azure Log Analytics |
| bigquery | BigQuery | Ingest records into Google BigQuery |
| counter | Count Records | Simple records counter. |
| datadog | Datadog | Ingest logs into Datadog. |
| es | Elasticsearch | flush records to a Elasticsearch server. |
| file | File | Flush records to a file. |
| flowcounter | FlowCounter | Count records. |
| forward | Forward | Fluentd forward protocol. |
| http | HTTP | Flush records to an HTTP end point. |
| influxdb | InfluxDB | Flush records to InfluxDB time series database. |
| kafka | Apache Kafka | Flush records to Apache Kafka |
| kafka-rest | Kafka REST Proxy | Flush records to a Kafka REST Proxy server. |
| stackdriver | Google Stackdriver Logging | Flush records to Google Stackdriver Logging service. |
| stdout | Standard Output | Flush records to the standard output. |
| splunk | Splunk | Flush records to a Splunk Enterprise service |
| tcp | TCP & TLS | flush records to a TCP server. |
| td | Treasure Data | Flush records to the Treasure Data cloud service for analytics. |
| nats | NATS | flush records to a NATS server. |
| null | NULL | throw away events. |
Contributing
Fluent Bit is an open project, several individuals and companies contribute in different forms like coding, documenting, testing, spreading the word at events within others. If you want to learn more about contributing opportunities please reach out to us through our Community Channels.
If you are interested into contribute with bug fixes, new features or coding in general, please refer to the code CONTRIBUTING guidelines.
Community & Contact
Feel free to join us on our Slack channel, Mailing List or IRC:
- Slack: http://slack.fluentd.org (#fluent-bit channel)
- Mailing List: https://groups.google.com/forum/#!forum/fluent-bit
- Twitter: http://twitter.com/fluentbit
- IRC: irc.freenode.net #fluent-bit
License
This program is under the terms of the Apache License v2.0.
Authors
Fluent Bit is originally made and currently sponsored by Treasure Data among other contributors.
b22b5f39... feat: add config reloader
Tag summary
Content type
Image
Digest
Size
26.8 MB
Last updated
over 6 years ago
docker pull siim/fluent-bit-for-operator