How to create your own Fluentd container to send logs to VMware Log Intelligence

I will walk through how to build your own fluentd container which can be used to send logs to VMware Log Intelligence
Earlier blogs had procedure listed how to install fluentd on linux and windows using rpms.
Linux – Click Here
Windows – Click Here

Use Cases

This container helps with  following use cases
  • Centralized Syslog Receiver on standard syslog port (514) and forwards to VMware Log Intelligence
  • Centralized App Logs Receiver on HTTP port 9880 and forwards to VMware Log Intelligence
  • Centralized Server to receive events from other fluentd agents over tcp port 24224 and forwards to VMware Log Intelligence








VMware Log Intelligence is a Cloud based service which means you will need to send logs over WAN and users may not want to send it from all the Machines hence having centralized server which sends out to Cloud would be a good idea 

Steps

We will be using https://github.com/fluent/fluentd-docker-image and modify to install http-out-ext plugin which is needed to send logs to VMware Log Intelligence
Execute the following command to clone the fluentd-docker-image repo

git clone https://github.com/fluent/fluentd-docker-image.git

You will notice it will create a folder \”fluentd-docker-image\”
I will be using fluentd version 1.2 which latest with Debian image as it is recommended one
Navigate to cd fluentd-docker-image/v1.2/Debian 
You will see Dockerfile 
The only part which we will be adding is to install the http-out-ext plugin in the docker image
Add following line in the Docker file

&&gem install fluent-plugin-out-http-ext -v 0.1.10 \\

Next, we will run docker build to build the container image

docker build -t docker.io/mmakhija/fluentdcontainer:v0 ./

A couple of points to note
  • The reason I am choosing this name is to that I can push the container name to public docker hub. You don\’t need to do it unless you want to share it with others. If you have your own private repo you can use that
    • mmakhija – docker hub username
    • fluentdcontainer – repo name on docker hub
  • It will take a couple of mins to build the docker. It will show some red color. Don\’t be scared it like I was when I did it for the first time

You can view the image by executing the following command

docker images

Next, you will need to copy fluentd.conf listed below on the docker host. Please remember the path where you are saving it because you will need to specify that when running docker run command.
I have saved the file at which I have specified below \”/fluent/fluentd.conf\”

https://github.com/munishpalmakhija/fluentd/blob/master/fluent.conf

Execute the following command to run the fluentd docker container.

docker run -d -p 24224:24224 -p 24224:24224/udp -p 514:5140 -p 514:5140/udp -p 9880:9880 -v /fluent:/fluentd/etc/  mmakhija/fluentdcontainer:v0

Please ensure to match the name of the image which you had it when you ran docker build command and the path of the fluent.conf file
If everything goes you will see a fluentd container running as below and you can configure ESXi host to forward logs to ip of the docker host where this container is running on port 514 and you should see logs flowing

Please feel free to leave comments or suggestions of if something not working 

0 thoughts on “How to create your own Fluentd container to send logs to VMware Log Intelligence

Leave a Reply

Your email address will not be published.

%d bloggers like this: