concept Kubernetes in category fluentd

This is an excerpt from Manning's book Unified Logging with Fluentd MEAP V02. Login to get full access to this book.

Industry thinking around how log management and the application of logging has been evolving a fair bit in the last four or five years. This at least is in part driven by the rapid progression of Docker and Kubernetes and the effective growth in individual small servers (microservices / macroservices / miniservices) to achieve hyper scaling that these technologies allow and the fact that servers are now far more ephemeral in nature. Other factors such as wider adoption to varying degrees of DevOps have also evolved. The net result is a couple of concepts worth being aware of have developed.

2.5           Fluentd deployment with Kubernetes and Containers

As we have looked at Fluentd and Fluent bit deployments as you might approach the requirement for a native or virtualized environment and referenced some of the mechanisms that would allow us to be automated or containerize these components. Yet, as discussed in Chapter 1, Fluentd has a strong association with containerization and the use of Kubernetes.

Establishing a deployment of a Kubernetes environment and containerization warrants a book in its own right (and we would recommend Kubernetes In Action 2nd Edition for this very purpose - www.manning.com/books/kubernetes-in-action-second-edition). It is, however, worth taking a look at how things work in principle, so as we work through the configuration of Fluentd in the next chapters you will be able to appreciate how the configuration will relate to a Kubernetes deployment. Not only that it may prompt ideas on how and what you may wish monitor with Fluentd when it comes to the microservices themselves.

Fluentd is one of the options for incorporating log management into a Kubernetes environment. This is achieved by defining daemonsets – which tell Kubernetes what pods (collections of containers that work in concert) and individual containers are required, and how they should be run when operating one or more worker nodes (servers providing compute power to a Kubernetes cluster). For example, it is possible to define things such that a Fluentd container will be executed on each worker node.

Listing 2.3 Out of the box Fluentd Daemonset designed for forwarding

apiVersion: apps/v1                         #A
kind: DaemonSet
metadata:
  name: fluentd 
  namespace: kube-system
  labels:
    k8s-app: fluentd-logging
    version: v1
spec: 
  selector:
    matchLabels:
      k8s-app: fluentd-logging
      version: v1
  template:
    metadata:
      labels:
        k8s-app: fluentd-logging
        version: v1
    spec:
      tolerations:
      - key: node-role.kubernetes.io/master #B
        effect: NoSchedule                  #C
      containers:                           #E
      - name: fluentd
        image: fluent/fluentd-kubernetes-daemonset:v1-debian-forward #D
        env:                                #F
          - name:  FLUENT_FOWARD_HOST
            value: "REMOTE_ENDPOINT"
          - name:  FLUENT_FOWARD_PORT
            value: "18080"
        resources:                                #G
          limits:
            memory: 200Mi
          requests:
            cpu: 100m
            memory: 200Mi
        volumeMounts:                             #H
        - name: varlog
          mountPath: /var/log
        - name: varlibdockercontainers
          mountPath: /var/lib/docker/containers
          readOnly: true
      terminationGracePeriodSeconds: 30
      volumes:                                    #I
      - name: varlog
        hostPath:
          path: /var/log
      - name: varlibdockercontainers
        hostPath:
          path: /var/lib/docker/containers

copy

It should be noted that it is possible within an infrastructure being used to establish Kubernetes to run processes such as Fluentd directly on the underlying platform. Whilst this eliminates the abstraction layer of Kubernetes (and the associated overhead) it also removes the opportunity to use Kubernetes to manage and monitor that Fluentd is running.

Figure 2.11 Architecture of Fluentd within Kubernetes as a Daemonset