cosmo0920/fluent-bit-go-s3 repository overview
fluent-bit s3 output plugin
Windows binaries are available in release pages.
This plugin works with fluent-bit's go plugin interface. You can use fluent-bit-go-s3 to ship logs into AWS S3.
The configuration typically looks like:
fluent-bit --> AWS S3
Usage
$ fluent-bit -e /path/to/built/out_s3.so -c fluent-bit.conf
Or,
$ make build-image
and then, specify configuration parameters as environment variables:
$ docker run -it -e="FLUENT_BIT_ACCESS_KEY_ID=yourawsaccesskey" \
-e="FLUENT_BIT_SECRET_ACCESS_KEY=yourawsaccesssecret" \
-e="FLUENT_BIT_BUCKET_NAME=yourbucketname" \
-e="FLUENT_BIT_S3_PREFIX=yours3prefix" \
-e="FLUENT_BIT_REGION=awsregion" \
-e="FLUENT_BIT_SUFFIX_ALGORITHM=algorithm" \
cosmo0920/fluent-bit-go-s3[:built-image-version]
Using docker image from docker hub.
$ docker pull cosmo0920/fluent-bit-go-s3[:latest or other tag]
Other released images are available in DockerHub's fluent-bit-go-s3 image tags.
Or, using helm:
helm install [YOURRELEASENAME] ./helm/fluent-bit
Prerequisites
- Go 1.11+
- gcc (for cgo)
- make
Building
$ make
Configuration Options
| Key | Description | Default value | Note |
|---|---|---|---|
| Credential | URI of AWS shared credential | "" | (See Credentials) |
| AccessKeyID | Access key ID of AWS | "" | (See Credentials) |
| SecretAccessKey | Secret access key ID of AWS | "" | (See Credentials) |
| Bucket | Bucket name of S3 storage | - | Mandatory parameter |
| S3Prefix | S3Prefix of S3 key | - | Mandatory parameter |
| SuffixAlgorithm | Algorithm for naming S3 object suffix | "" | sha256 or no suffix("") |
| Region | Region of S3 | - | Mandatory parameter |
| Compress | Choose Compress method | "" | gzip or plainText("") |
| Endpoint | Specify the endpoint URL | "" | URL with port or empty string |
| AutoCreateBucket | Create bucket automatically | false | true/false |
| LogLevel | Specify Log Level | "info" | trace/debug/info/warning/error/fatal/panic |
| TimeFormat | Time format to add to the S3 path | "20060102/15" | Specify in Go's Time Format |
| TimeZone | Specify TimeZone | "" | Specify TZInfo based region. e.g.) Asia/Tokyo |
Example:
Add this section to fluent-bit.conf:
[Output]
Name s3
Match *
# Credential /path/to/sharedcredentialfile
AccessKeyID yourawsaccesskeyid
SecretAccessKey yourawssecretaccesskey
Bucket yourbucketname
S3Prefix yours3prefixname
SuffixAlgorithm sha256
Region us-east-1
Compress gzip
# Endpoint parameter is mainly used for minio.
# Endpoint http://localhost:9000
# TimeFormat 20060102/15
# TimeZone Asia/Tokyo
Credentials
By default AWS credentials are loaded from their usual providers. See AWS CLI Configuration and Credential File Settings.
If the default credentials are not desired, this plugin supports the following credential providers.
Shared Credentials
Create the following file which includes credentials:
[default]
aws_access_key_id = YOUR_AWS_ACCESS_KEY_ID
aws_secret_access_key = YOUR_AWS_SECRET_ACCESS_KEY
Then, specify the following parameter in fluent-bit configuration:
Credential /path/to/sharedcredentialfile
Static Credentials
Specify the following parameters in fluent-bit configuration:
AccessKeyID yourawsaccesskeyid
SecretAccessKey yourawssecretaccesskey
Useful links
Tag summary
Content type
Image
Digest
Size
39.7 MB
Last updated
about 6 years ago
docker pull cosmo0920/fluent-bit-go-s3