12 Using logs and traces to understand behavior

 

This chapter covers

  • Storing logs in a consistent and structured way in a machine-readable format
  • Setting up a logging infrastructure
  • Using traces and correlation IDs to understand system behavior

In the previous chapter, we focused on emitting metrics from your services and using those metrics to create dashboards and alerts. Metrics and alerts are only one part of what you need to achieve observability in your microservice architecture. In this chapter, we’ll focus on collecting logs and making sure you’re able to trace the interactions between services. This will allow you to not only have an overview of how the system behaves but also go back in time and retrospectively follow each request. Doing so is important to debug errors and to identify bottlenecks. Logs give you a sort of paper trail that documents the history of each request entering your system, whereas traces provide you a way to establish a timeline for each request, to understand how much time it spent in different services.

By the end of this chapter, you’ll have created a basic logging infrastructure and set up the tracing capability. You’ll be able to both monitor the operation of your application and have the tools to audit and investigate in case you need to do so for particular requests. In addition, you’ll be able to identify performance issues by looking into tracing data.

12.1 Understanding behavior across services

12.2 Generating consistent, structured, human-readable logs

12.2.1 Useful information to include in log entries

12.2.2 Structure and readability

12.3 Setting up a logging infrastructure for SimpleBank

12.3.1 ELK- and Fluentd-based solution

12.3.2 Setting up your logging solution

12.3.3 Configure what logs to collect

12.3.4 Finding a needle in the haystack

12.3.5 Logging the right information

12.4 Tracing interactions between services

12.4.1 Correlating requests: traces and spans

12.4.2 Setting up tracing in your services

12.5 Visualizing traces

Summary