A full-text search index indexes the textual content of documents or log records, allowing searches across the complete text rather than selected fields. In observability, it makes log lines broadly discoverable, provided they contain actionable formatting and context. section 2.3.3
Use full-text indexing with these trade-offs in mind:
- Searches can find everything in a log line, which is useful when the relevant field or wording is not known in advance. section 2.3.3
- Indexing all text creates scalability and cost challenges, especially as long-term storage becomes expensive. section 2.3.3
- Label-based indexing offers a contrasting approach by indexing contextual tags instead of the complete log line. section 2.3.3
Log-search systems
Elasticsearch and OpenSearch use the traditional full-text approach. Grafana Loki instead popularizes label-based indexing, illustrating the choice between unrestricted text search and lower indexing overhead. section 2.3.3
Table 2.1 Overview of the costs of logs (view table figure)
Compare instrumentation, agent, and destination overhead when operating indexed logs, including ingestion, reliability, security, and retention considerations. section 2.3.3
