liveBook · Manning

Label-based indexing

As explained in Cloud Observability in Action

Label-based indexing is the creation of an index over selected labels or tags that describe log context rather than the full log text.

Label-based indexing indexes selected labels or tags that describe log context instead of indexing the entire log text. A label might identify service foo or an error condition. section 2.3.3

This approach reduces the scalability and cost challenges associated with full-text indexing, while preserving a way to find relevant log lines through their context. Grafana Loki is a prominent example, inspired by Prometheus. section 2.3.3

Service and error context

A log line from service foo can carry labels identifying the service and an error condition. The index searches those labels, while the remaining log text stays outside the indexed set. section 2.3.3

Table 2.1 Overview of the costs of logs (view table figure)

 

Overhead

Considerations

Instrumentation

Small and simple to add a log line

Widely adopted and supported by programming languages

Agent

Some runtime, lots of options, and few standards for formats (syslog, OTLP)

A runtime footprint, ingestion performance, reliability, and security

Destination

Long-term storage can be expensive.

Data temperature (access recency or frequency) can guide retention

Use the cost overview to compare indexing choices with instrumentation, agent, destination, storage, and retention considerations. section 2.3.3

How Cloud Observability in Action covers it

How deep it goes
Defines the indexing strategy and contrasts its scalability and cost with full-text indexing, but gives no implementation procedure.
Examples
Uses service foo and an error condition as contextual labels, and names Grafana Loki as a practical implementation example.
What it assumes
Assumes familiarity with log lines and contextual labels, while distinguishing indexed context from full log text.
What you can do after
Evaluate indexing strategies by search coverage, scalability, and cost.
How it connects
Connects label-based indexing with full-text indexing, logs, Grafana Loki, Prometheus, and observability.
Cover of Cloud Observability in Action
liveBookmanning.com
Read Cloud Observability in Action on liveBook Get the book at manning.com

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Cloud Observability in Action. The section links point to the parts of the book it draws on.