What you will learn in Logs and Telemetry
Using Fluent Bit, Kubernetes, streaming and more
Chapter by chapter
Chapter 8: Stream processors for time series calculations and filtering
What you learn in chapter 8 →Concepts explained in this book
- FluentBitFluent Bit is a lightweight, high-performance telemetry agent for collecting, processing, and forwarding logs and other observability data.
- metricA metric is a usually numeric measurement with a descriptive label that represents the state or activity of hardware or software.
- OpenTelemetryOpenTelemetry is an open standard and observability framework for generating, collecting, and exporting telemetry such as traces, metrics, and logs.
- stream processorA stream processor is a Fluent Bit component that continuously queries, aggregates, filters, and generates events from buffered event streams.
- OTLPThe OpenTelemetry Protocol (OTLP) is a vendor-neutral protocol for transmitting telemetry data between instrumentation, collectors, and backends.
- pluginA plugin is an independently implemented software component that extends an application's capabilities through a defined integration interface.
- output pluginAn output plugin is a Fluent Bit extension that retrieves processed events and stores or forwards them to a destination.
- protocol bufferProtocol Buffers is a language-neutral, platform-neutral mechanism for serializing structured data using generated code and compact binary messages.
- matchAn event match is a Fluent Bit output-selection pattern that determines which tagged events an output receives.
- sidecar processThe sidecar pattern deploys a supporting process alongside a primary application component so they share a lifecycle and execution context.
- time windowAn event window is a defined scope of multiple stream events over which a query computes filtering or aggregate results.
- configurationA configuration is a set of declarative settings that controls Fluent Bit's behavior and processing pipeline.
- signalA signal is an observability event category encompassing logs, metrics, and traces.
- bufferingBuffering is the technique of temporarily storing data between processing stages to absorb differences in their processing rates.
- child spanA span is a trace record representing the start and end of a significant operation, optionally nested within another span.
- stdout output pluginThe stdout output plugin is a Fluent Bit output plugin that writes processed events to standard output.
- MessagePackMessagePack is a compact binary serialization format with libraries for encoding and decoding structured data.
- prometheus_remote_write pluginThe prometheus_remote_write plugin is a Fluent Bit output plugin that sends metrics to systems implementing the Prometheus remote-write specification.
- timestampA timestamp is a time value associated with an event to indicate when it occurred or was received.
- fluentdFluentd is an open-source data collector that unifies the collection and routing of logs and other machine data through pluggable integrations.
- KubernetesA Kubernetes cluster is a group of computing nodes managed together by Kubernetes to run containerized workloads.
- LuaLua is a lightweight embeddable scripting language commonly used to add programmable behavior to host applications.
- stream taskA stream task is a configured Fluent Bit operation that applies a query or processing definition to an event stream or tag.
- regular expressionA regular expression is a pattern language for matching and extracting text according to specified character sequences.
- Kubernetes filter pluginThe Kubernetes filter plugin is a Fluent Bit filter that enriches log events with metadata about Kubernetes Pods, namespaces, and containers.
- SQLiteSQLite is an embedded relational database engine that stores a complete database in a local file without requiring a separate server process.
- WireMockWireMock is a tool for mocking HTTP-based APIs and inspecting or controlling their requests and responses.
- standard outputStandard output is the conventional output stream through which a process writes textual data for a receiving terminal, parent process, or redirected destination.
- Prometheus exporter pluginprometheus_exporter is a Fluent Bit output plugin that exposes processed records in Prometheus metric exposition format over HTTP.
- random input pluginA random input plugin is a Fluent Bit input component that generates records containing predefined fields with randomly generated values.
- multiline.parserA multiline parser is a parser that identifies event boundaries and combines related lines into a single record.
- multiline parsingMultiline parsing is the technique of determining event boundaries across multiple input lines and combining those lines into one event.
- DaemonSetA DaemonSet is a Kubernetes workload controller that ensures a Pod runs on each eligible node or on a selected set of nodes.
- filterA filter is a Fluent Bit processing component that applies custom logic to records as they move through a telemetry pipeline.
- PostgreSQL pluginThe PostgreSQL plugin is a Fluent Bit output plugin that writes event records to PostgreSQL-compatible databases.
- labelA Kubernetes label is a key-value attribute attached to a resource for identification, organization, and selection.
- instrumentationInstrumentation is the insertion or provision of code that records execution behavior and emits telemetry.
- JSONLJSON Lines is a serialization format that represents each JSON value as a separate line.
- forward input pluginA forward input plugin is a Fluent Bit input extension that receives events encoded with the forward protocol.
- GoGo is a statically typed compiled programming language designed for efficient systems and network software development.
- log rotationLog rotation is the practice of renaming or replacing a growing log file and continuing writes in a new file.
- native executableA native executable is a compiled program produced for a specific operating-system and hardware platform and run directly by that platform.
- YAMLYAML is a human-readable data serialization standard that represents structured data through indentation, mappings, and sequences.
- Modify filterA Modify filter is a Fluent Bit filter that conditionally adds, sets, renames, copies, removes, or reorders event record elements.
- WebAssemblyWebAssembly is a portable binary instruction format and execution target for code running in constrained or embedded environments.
- hopping windowA hopping window is a stream-processing time window whose successive intervals advance by a hop duration that can be shorter than the window duration, allowing overlap.
- PostgreSQLPostgreSQL is an open-source object-relational database management system.
- tail input pluginA tail input plugin is a Fluent Bit input component that reads appended records from files, typically following appended content.
- tumbling windowA tumbling window is a non-overlapping time window in stream processing whose successive intervals begin immediately after the preceding intervals end.
- connection poolingConnection pooling is the practice of maintaining reusable database connections so operations can execute without repeatedly establishing new connections.
- pollingPolling is a technique in which a client periodically requests data or checks state from a service.
- tag expansionTag expansion is the inclusion of an input file's full path and related identifying details in an event tag.
- Lua filterA Lua filter is a Fluent Bit filter plugin that executes Lua code to transform or otherwise process event records.
- input callbackAn input callback is a framework-invoked function that retrieves source data and returns it as events for pipeline processing.
- TLSTransport Layer Security is a cryptographic protocol that authenticates endpoints and protects network communications against interception and tampering.
- cardinalityCardinality is the number of distinct label-value combinations or dimensions represented in a telemetry data set.
- file output pluginA file output plugin is a Fluent Bit output extension that writes processed events to files.
- select statementA SELECT statement is a query construct that projects fields or computed values from a stream or tagged event source.
- unix timestamp functionThe unix_timestamp function returns a timestamp as a numeric Unix time value suitable for arithmetic and comparison.
- where clauseA WHERE clause is a query construct that retains records satisfying a predicate over event values.
- dummy input pluginA dummy input plugin is a Fluent Bit input plugin that generates synthetic events internally instead of reading them from an external source.
- telemetry pipelineA monitoring pipeline is an ordered flow that receives, processes, and delivers operational telemetry.
- forward protocolThe Fluent Forward protocol is a data-transport specification for forwarding event records between Fluent Bit, Fluentd, and compatible systems.
- multiline filterA multiline filter is a Fluent Bit filter that combines related records from multiple events into a single multiline event.
- classic configuration formatThe classic configuration format is Fluent Bit's section-based configuration notation using directives such as [SERVICE], [INPUT], [FILTER], and [OUTPUT].
- CPU input pluginThe CPU input plugin is a Fluent Bit input extension that periodically collects processor utilization measurements.
- eventAn event is a discrete record representing an occurrence or data item processed by a telemetry system.
- http_serverhttp_server is a Fluent Bit configuration attribute that enables its embedded web server.
- fluentbit_metrics input pluginThe fluentbit_metrics input plugin is a Fluent Bit input plugin that exposes Fluent Bit metrics for collection by Prometheus.
- Prometheus remote write APIA remote write endpoint is a network address that accepts metrics transmitted according to the Prometheus remote write specification.
- push modelA push model is a data-delivery pattern in which the source sends data to a receiving service when the data becomes available.
- rewrite_tag filterThe rewrite_tag filter is a Fluent Bit filter that conditionally changes event tags to alter routing, optionally emitting a retagged clone or replacing the original tag.
- parsingParsing is the transformation of textual or encoded content into a structured representation according to its syntax.
- cleanup callbackA cleanup callback is a framework-invoked function that releases resources associated with a completed processing cycle.
- callbackA callback is a function invoked by a framework or runtime in response to a lifecycle event or processing opportunity.
- FLBPluginConfigKeyFLBPluginConfigKey is a Fluent Bit Go-plugin interface function that retrieves a named configuration value for a plugin.
- input plugin lifecycleAn input plugin lifecycle is the ordered sequence of registration, initialization, event handling, cleanup, and exit stages for an input plugin.
- retryA retry is a repeated attempt to perform an operation after an earlier attempt fails or its destination is unavailable.
- service blockA service block is the Fluent Bit configuration section containing settings for global service behavior and its embedded monitoring server.
- observabilityObservability is the ability to understand a system's internal state from externally emitted telemetry such as logs, metrics, and traces.
- null output pluginA null output plugin is a Fluent Bit output component that accepts processed events and discards them without forwarding, storing, or sending them to an external destination.
- disk input pluginThe disk input plugin is a Fluent Bit input extension that periodically collects disk activity measurements.
- memory input pluginThe memory input plugin is a Fluent Bit input extension that periodically collects memory and swap measurements.
- network interface input pluginThe network interface input plugin is a Fluent Bit input extension that periodically collects network traffic and error measurements.
- input blockAn input block is a configuration section that declares an event source and its input-plugin settings.
- cgocgo is a Go toolchain facility that enables Go programs to interoperate with C code and libraries.
- action attributeAn action attribute is a metrics-selector configuration field that determines whether matching metrics are included or excluded.
- ltsvLTSV is a log format that represents records as label-value fields separated by tabs.
- backpressureBackpressure is a condition in which a downstream component cannot consume events as quickly as an upstream component produces them.
- http pluginAn HTTP output plugin is a Fluent Bit output component that sends processed event data to a destination over HTTP.
- twelve-factor applicationThe twelve-factor app is a methodology for building software-as-a-service applications according to twelve principles covering configuration, deployment, processes, and operational behavior.
- throttle filterA throttle plugin is a Fluent Bit plugin that limits the rate at which events are released by dropping events that exceed the configured allowance.
- log driverA logging driver is a container-runtime component that routes container log output to a selected logging destination or backend.
- WASIWASI is a standardized system interface that provides WebAssembly programs with controlled access to operating-system-like capabilities.
- content_modifiercontent_modifier is a Fluent Bit processor that modifies fields or attributes in telemetry records.
- lua tableA Lua table is Lua's associative data structure for representing indexed sequences and name-value mappings, including nested values.
- environment variableAn environment variable is a named value supplied by the operating system or process environment and made available to applications at runtime.
- mutual TLSMutual TLS is a TLS mode in which both communicating endpoints authenticate each other with certificates.
- CSVCSV is a delimited text format that represents records as rows of values separated by a configurable delimiter.
- template formatThe template format writes output according to user-defined content containing references to values in an event payload.
- certification authorityA certificate authority is a trusted entity that issues and validates digital certificates and maintains their revocation records.
- certificateA digital certificate is a signed credential that binds an identity to a public key for authenticated communication.
- health check endpointA health endpoint is an HTTP endpoint that reports whether an application or service is operating normally.
- gRPCgRPC is a high-performance remote procedure call framework that commonly uses HTTP/2 and Protocol Buffers for communication.
- OpenTelemetry CollectorThe OpenTelemetry Collector is a vendor-neutral service component that receives, processes, and exports telemetry data.
- telemetryTelemetry data is machine-generated information about software or infrastructure behavior, including logs, metrics, and traces.
- custom parserA custom parser is a parser defined with user-specified expressions to extract structured fields from input content.
- record accessorA record accessor is Fluent Bit expression syntax for selecting values from event records and metadata when constructing or evaluating configuration values.
- stdin input pluginA stdin input plugin is a Fluent Bit input component that reads event data from a process's standard input stream.
- node exporterPrometheus Node Exporter is a Prometheus exporter that exposes operating-system and hardware metrics in Prometheus exposition format.
- node_exporter_metricsnode_exporter_metrics is a Fluent Bit input plugin that collects host operating-system and hardware metrics for Prometheus-compatible export.
- Fluent OperatorFluent Operator is a Kubernetes operator for deploying and managing Fluent Bit configurations and resources.
- nest filterA nest filter is a Fluent Bit filter that groups matching record elements into a nested object or promotes nested elements to their parent.
- Base64Base64 encoding represents binary or textual data as ASCII characters using a 64-character alphabet.
- basic authenticationBasic authentication is an HTTP authentication scheme that transmits a username and password in an encoded authorization header.
- compressionCompression is the encoding of data into a representation that requires less storage or transmission capacity.
- gzipgzip is a lossless compression format and utility commonly used to reduce the size of transmitted or stored data.
- key_exists conditionkey_exists is an expect-filter check that requires a named key to be present in the event record.
- custom resource definitionA custom resource definition is a Kubernetes API extension that defines a new custom resource type.
- ClusterFluentBitConfigClusterFluentBitConfig is a Fluent Operator custom resource for selecting cluster-wide Fluent Bit input, output, and filter configurations.
- FluentBitConfigFluentBitConfig is a Fluent Operator custom resource for selecting namespace-level Fluent Bit input, output, and filter configurations.
These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.