A Kubernetes sidecar is a companion container in the same pod as an application container. It moves supporting work, such as logging and monitoring, out of the inference path. section 6.2.5
Use a sidecar when supporting work should not slow successful requests:
- Process and clean logs in parallel, then forward them to a backend or write them to a log file. section 6.2.5
- Avoid placing data-quality checks on the critical path when malformed input could instead receive an error response. section 6.2.5
- Use tools such as whylogs or fluentd as the sidecar logging agent. section 6.2.5
LLM service monitoring
A whylogs or fluentd sidecar captures selected requests or stdout, processes the data, and forwards it to WhyLabs or Prometheus while inference continues. section 6.2.5
Figure 6.10 An example Kubernetes sidecar container, which takes logging out of the critical path. The logging agent would be a tool like a whylogs container or fluentd that captures specific requests or all stdout print statements, processes them, and forwards them to a logging backend like WhyLabs or Prometheus.
Trace the sidecar flow from application output through a logging agent to a monitoring backend. section 6.2.5
