Kubernetes Log forwarding to VMware Log Intelligence using Fluentd

munishpalmakhija
In this blog I will walk you through how to use fluentd to forward logs from Kubernetes deployed pods(containers)

Pre-requisites

  • A running Kubernetes Cluster. My setup has with Kubernetes 1.11.1 on CentOS VMs on vSphere. Click Here for detailed instructions 
  • Admin access to the Cluster as we will be deploying fluentd in kube-system name space
  • Application writes to \”stdout\” and \”stderr\” streams 
  • An understanding of VMware Log Intelligence. Click Here to read through if you haven\’t done it yet 

Getting Started

Before to get started, make sure you understand or have a basic idea about the following concepts from Kubernetes (Click Here :

Node            

A node is a worker machine in Kubernetes, previously known as a minion. A node may be a VM or physical machine, depending on the cluster. Each node has the services necessary to run pods and is managed by the master components…

Pod

A pod is a group of one or more containers (such as Docker containers), the shared storage for those containers, and options about how to run the containers. Pods are always co-located and co-scheduled, and run in a shared context…

DaemonSet

A DaemonSet ensures that all (or some) nodes run a copy of a pod. As nodes are added to the cluster, pods are added to them. As nodes are removed from the cluster, those pods are garbage collected. Deleting a DaemonSet will clean up the pods it created. 
We will be using fluentd kubernetes daemon set (Click Here) which will be reading log files for the containers and sending it to VMware Log Intelligence

Options 

There are 2 ways we can send kubernetes logs to VMware Log Intelligence.

Option 1 – Using Syslog fluentd daemon set via Data Collector using syslog protocol

This is the most simplest and the quickest way. You will need to have Data Collector deployed. Click Here to see how to deploy Data Collector






Steps 

Copy the yaml file for syslog (Click Here) and save it locally
Modify the yaml file and to add the Data Collector IP which accepts connections on syslog port (514)
Create fluentd daemon set by executing following command 

kubectl create -f fluentd-syslog.yaml















Once deployed and configured successfully you will see logs from fluentd pod in VMware Log Intelligence

Option 2 – Using Custom daemon set directly to VMware Log Intelligence via HTTPS protocol

You will need to modify the daemon set and install http-out-ext fluentd plugin which can forward logs directly to VMware Log Intelligence using API keys


Steps

First Step is to create your own docker image 

I will be using the debian image as its recommended for Production environments. https://github.com/fluent/fluentd-kubernetes-daemonset/tree/master/docker-image/v1.2/debian-syslog
From the documentation – The following repository expose images based on Alpine Linux and Debian. For production environments, we strongly suggest to use Debian images.

Clone the github repo. Install git (if you don\’t have it already) and execute following command

git clone https://github.com/fluent/fluentd-kubernetes-daemonset.git

Navigate to \”fluentd-kubernetes-daemonset/docker-image/v1.2/debian-syslog/conf\”

Modify the fluent.conf file to following 
Before











After

You will notice that we have added is two environment variables. It is similar to what we have for the standalone fluentd setup. Click Here to view the standalone fluentd installation
Next we will need to modify the docker file to install the http-out plugin \”fluentd-kubernetes-daemonset/docker-image/v1.2/debian-syslog/Dockerfile\”

&& gem install fluent-plugin-out-http-ext -v 0.1.10 \\

Before 



















After

docker build -t docker.io/mydockerhubusername/reponame:v1 ./
It should take a couple of mins 
If you want to push it registry or you can use local image
You can modify the existing yaml file for syslog with following yaml file. Please ensure to generate API Key from VMware Log Intelligence UI
https://github.com/munishpalmakhija/fluentd/blob/master/mm-fluentd-ds.yaml

Execute following command. In my case name of the file is mm-fluentd.yaml

kubectl create -f mm-fluentd.yaml

Once deployed and configured successfully you will see logs from fluentd pod in VMware Log Intelligence

Happy Logging !!!
References
https://www.techrepublic.com/article/how-to-install-a-kubernetes-cluster-on-centos-7/
https://github.com/fluent/fluentd-kubernetes-daemonset
https://docs.fluentd.org/v0.12/articles/kubernetes-fluentd

0 thoughts on “Kubernetes Log forwarding to VMware Log Intelligence using Fluentd”

  13. This is a very interesting article to read. Thanks for sharing the information. Great post.Download the application:Make Money Online…2019 100% WorkingClick Here Now Free bigo liveVideo Streaming Online…2019 100% WorkingClick Here Now Free bigo live apkBest Messaging App…2019 100% WorkingClick Here Now Free video callingOnline Video Chat…2019 100% WorkingClick Here Now FreeLive ChatWatch Series Stream…2019 100% WorkingClick Here Now Free bigo live app

    Reply

  14. This is a very interesting article to read. Thanks for sharing the information. Great post.bigo live is GooD App. connect friends and relatives it helps users to relax, connect friends. BIGO LIVE CONNECTOR FOR PC | BIGO LIVE CONNECTOR Bigo Live For Windows Phone Free Download BIGO LIVE FOR MAC PC FREE DOWNLOAD BIGO LIVE FOR IOS FREE DOWNLOAD | BIGO LIVE FOR IOS BIGO LIVE FOR BLACKBERRY FREE DOWNLOAD BIGO LIVE SIGN UP | HOW TO SIGN UP FOR BIGO LIVE BIGO LIVE PC FOR WINDOWS XP/7/8/8.1/10

    Reply

  27. I am really happy with your blog because your article is very unique and powerful for new reader.Prefer to study this kind of material. Nicely written information in this post,the quality of content is fine and the conclusion is lovely. Things are very open and intensely clear explanation of issues. PHP Training in Chennai | Certification | Online Training Course | Machine Learning Training in Chennai | Certification | Online Training Course | iOT Training in Chennai | Certification | Online Training Course | Blockchain Training in Chennai | Certification | Online Training Course | Open Stack Training in Chennai | Certification | Online Training Course

    Reply

Leave a Reply

Your email address will not be published. Required fields are marked *