Fluentd

#lang-ruby #web 

Fluentd is an open source data collector for unified logging layer. It allows you to unify data collection and consumption for a better use and understanding of data.

Note

For this guide you should be familiar with the basic concepts of

License

Fluentd is released under the Apache-2.0 License.

All relevant legal information can be found here

Prerequisites

We’re using Ruby in the stable version 2.6:

[isabell@stardust ~]$ uberspace tools version show ruby
Using 'Ruby' version: '2.6'
[isabell@stardust ~]$

Installation

Fluentd can be installed as a ruby gem:

[isabell@stardust ~]$ gem install fluentd --no-doc
Fetching http_parser.rb-0.6.0.gem
Fetching concurrent-ruby-1.1.7.gem
Fetching tzinfo-2.0.2.gem
Fetching serverengine-2.2.1.gem
Fetching sigdump-0.2.4.gem
Fetching yajl-ruby-1.4.1.gem
Fetching msgpack-1.3.3.gem
Fetching cool.io-1.6.0.gem
Fetching fluentd-1.11.2.gem
Fetching tzinfo-data-1.2020.1.gem
Fetching strptime-0.2.4.gem
WARNING:  You don't have /home/isabell/.gem/ruby/2.6.0/bin in your PATH, gem executables will not run.
Building native extensions. This could take a while...
Successfully installed msgpack-1.3.3
Building native extensions. This could take a while...
Successfully installed yajl-ruby-1.4.1
Building native extensions. This could take a while...
Successfully installed cool.io-1.6.0
Successfully installed sigdump-0.2.4
Successfully installed serverengine-2.2.1
Building native extensions. This could take a while...
Successfully installed http_parser.rb-0.6.0
Successfully installed concurrent-ruby-1.1.7
Successfully installed tzinfo-2.0.2
Successfully installed tzinfo-data-1.2020.1
Building native extensions. This could take a while...
Successfully installed strptime-0.2.4
Successfully installed fluentd-1.11.2
11 gems installed
[isabell@stardust ~]$

Note

You can ignore the warning about the executables not being in the PATH, Uberspace already took care of that.

To verify the installation, check the version of fluentd:

[isabell@stardust ~]$ fluentd --version
fluentd 1.11.2
[isabell@stardust ~]$

Configuration

Create configuration

Create the default configuration in the ~/fluentd directory.

[isabell@stardust ~]$ fluentd --setup ~/fluentd
Installed /home/isabell/fluentd/fluent.conf.
[isabell@stardust ~]$

Edit the config file ~/fluentd/fluent.conf and comment out the following lines:

## match tag=system.** and forward to another fluent server
#<match system.**>
#  @type forward
#  @id forward_output
#
#  <server>
#    host 192.168.0.11
#  </server>
#  <secondary>
#    <server>
#      host 192.168.0.12
#    </server>
#  </secondary>
#</match>

Setup daemon

Create ~/etc/services.d/fluentd.ini with the following content:

[program:fluentd]
directory=%(ENV_HOME)s/fluentd
command=fluentd --config fluent.conf --no-supervisor

After creating the configuration, tell supervisord to refresh its configuration and start the service:

[isabell@stardust ~]$ supervisorctl reread
SERVICE: available
[isabell@stardust ~]$ supervisorctl update
SERVICE: added process group
[isabell@stardust ~]$ supervisorctl status
SERVICE                            RUNNING   pid 26020, uptime 0:03:14
[isabell@stardust ~]$

If it’s not in state RUNNING, check your configuration.

Finishing installation

Further adjust the configuration in ~/fluentd/fluent.conf according to your needs (see configuration).

Optional: Expose the HTTP Endpoint

To expose the fluentd http endpoint (default port 8888), configure a web backend:

[isabell@stardust ~]$ uberspace web backend set /fluentd --http --port 8888 --remove-prefix
Set backend for /fluentd to port 8888; please make sure something is listening!
You can always check the status of your backend using "uberspace web backend list".
[isabell@stardust ~]$

You can then test this endpoint by sending a GET request to https://isabell.uber.space/fluentd/debug.test?json={%22test%22:%22message%22}. The json payload should then appear in the fluentd logs:

[isabell@stardust ~]$ supervisorctl tail fluentd
...
2020-09-09 12:10:46.630093423 +0200 debug.test: {"test":"message"}
[isabell@stardust ~]$

Updates

Note

Check the update feed regularly to stay informed about the newest version.

Tested with Fluentd 1.11.2, Uberspace 7.7.6

Written by: Andreas Pabst