Fluentd

What do you like best?

Fluentd provides tones of plugins to collect data from different sources and store in different sinks.

You can collect data from log files, databases, and even Kafka streams.

You can process log contents with Fluentd and store with JSON format schema in files or even NoSQL.

Fluentd is part of the Cloud Native Computing Foundation (CNCF).

You can use Fluentd to collect Kubernetes and Docker logs and store in different cloud storage.

The community is active. You can find help from Fluentd Slack workspace.

If you're interested in collecting data from IoT devices, you can give fluent-bit [1] a try.

[1] https://fluentbit.io/

What do you dislike?

- Difficult to monitor and debug Fluentd internal issues (ex. out of bufffer).

You'll need DataDog [2] or using Prometheus to collect fluentd metrics.

- Even there are high availability configurations [3],

it still needs extra design to ensure service availability.

[2] https://www.datadoghq.com/blog/monitor-fluentd-datadog/

[3] https://docs.fluentd.org/deployment/high-availability

What problems are you solving with the product? What benefits have you realized?

We use Fluentd to collect server logs and Kafka stream, then save to cloud storage.

It reduce the development time to write our own connectors for different data sources and data sinks.