7 Performance and scaling
This chapter covers
- Tuning Fluentd configuration to maximize resources using workers.
- Distributing workloads with fan-in and fan-out deployment patterns.
- Scaling with fan-out and fan-in deployment patterns
- Implementing high availability and controlling workload distribution with roundrobin or node weighting.
- Deploying Fluentd for microservice deployment patterns.
In the previous chapters we worked with just a single Fluentd instance, but we live in a world of distribution, virtualization, and containerization. A single server solution in the enterprise space is a rarity. The chances are that to provide a level of robustness and flexibility to demand the solution will be configured to scale. With either scaling up (adding more CPUs, memory to a server) and/or scale-out (deploy additional server instances and have workload distributed via load balancing). To provide good availability an organization will have an active server running a solution and a standby configured and being kept in sync. With the synchronization, it becomes possible to startup the standby server on short notice if the first instance fails (active-passive). We also see multiple active servers spread across multiple data centers and this is becoming very conventional as a deployment pattern.