Fluent - Data Collector

Monitoring
Page content

Fluentd is an open-source data collector for a unified logging layer. Fluentd allows you to unify data collection and consumption for better use and understanding of data.

Docu

Install Ruby

pkg_add ruby-3.1.2

ln -sf /usr/local/bin/ruby31 /usr/local/bin/ruby
ln -sf /usr/local/bin/bundle31 /usr/local/bin/bundle
ln -sf /usr/local/bin/bundler31 /usr/local/bin/bundler
ln -sf /usr/local/bin/erb31 /usr/local/bin/erb
ln -sf /usr/local/bin/gem31 /usr/local/bin/gem
ln -sf /usr/local/bin/irb31 /usr/local/bin/irb
ln -sf /usr/local/bin/rdoc31 /usr/local/bin/racc
ln -sf /usr/local/bin/rake31 /usr/local/bin/rake
ln -sf /usr/local/bin/rdoc31 /usr/local/bin/rbs
ln -sf /usr/local/bin/rdoc31 /usr/local/bin/rdbg
ln -sf /usr/local/bin/rdoc31 /usr/local/bin/rdoc
ln -sf /usr/local/bin/ri31 /usr/local/bin/ri
ln -sf /usr/local/bin/typeprof31 /usr/local/bin/typeprof

Install Fluentd

gem install fluentd

ln -sf /usr/local/bin/fluentd31 /usr/local/bin/fluentd

Create Config

fluentd --setup /etc/fluent

Edit config

cat << 'EOF ' > /etc/fluent/fluent.conf
# fluent.conf

# Nginx Log
<source>
  @type tail
  format nginx
  path /var/log/nginx/blog.stoege.net.log
  pos_file /var/log/nginx-pos/blog.stoege.net.log
  time_key fluentd_time
  time_format %d/%b/%Y:%H:%M:%S %z
  tag nginx.access
</source>

# Output
<match nginx.access>
  @type file
  path /var/log/fluent/blog.stoege.net
</match>
EOF

Run

fluentd -c /etc/fluent/fluent.conf

Sample Data

tail -f /var/log/fluent/blog.stoege.net/buffer.b5e6e8e8ee14db5e3a401bb38f14a6de5.log
2022-08-23T15:54:58+02:00	nginx.access	{"remote":"114.119.135.215","host":"-","user":"-","time":"23/Aug/2022:15:54:58 +0200","method":"GET","path":"/tags/monitoring/","code":"200","size":"33684","referer":"https://blog.stoege.net/tags/vpn","agent":"Mozilla/5.0 (Linux; Android 7.0;) AppleWebKit/537.36 (KHTML, like Gecko) Mobile Safari/537.36 (compatible; PetalBot;+https://webmaster.petalsearch.com/site/petalbot)","http_x_forwarded_for":"-"}
2022-08-23T15:55:42+02:00	nginx.access	{"remote":"147.182.168.107","host":"-","user":"-","time":"23/Aug/2022:15:55:41 +0200","method":"GET","path":"/index.xml","code":"304","size":"0","referer":"-","agent":"NewsBlur Feed Fetcher - 2 subscribers - https://www.newsblur.com/site/8313934/blog-stoege-net (\\x22Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/14.0.1 Safari/605.1.15\\x22)","http_x_forwarded_for":"-"}
2022-08-23T15:55:42+02:00	nginx.access	{"remote":"147.182.168.107","host":"-","user":"-","time":"23/Aug/2022:15:55:42 +0200","method":"GET","path":"/","code":"200","size":"56891","referer":"-","agent":"NewsBlur Page Fetcher - 2 subscribers - https://www.newsblur.com/site/8313934/blog-stoege-net (\\x22Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/14.0.1 Safari/605.1.15\\x22)","http_x_forwarded_for":"-"}
Any Comments ?

sha256: fb02e126f5e34e4be8630c5190e9af68d2c36e91e5c4d54a11f6b70a3be6563b