liveBook · Manning

Reflections

As explained in Architecting an Apache Iceberg Lakehouse

Reflections are Dremio-managed precomputed representations of source data or query results that can transparently substitute into query plans.

Reflections are Dremio-managed, precomputed representations of tables, views, or query results. They reduce latency and compute costs in federated environments by avoiding repeated remote scans. Dremio’s cost-based planner transparently substitutes a suitable reflection into a query plan, fully or partially, without requiring users to rewrite queries. section 8.3.4 section 8.3.1

Common reflection designs include:

  • Raw reflections accelerate table scans. section 8.3.4
  • Aggregation reflections precompute grouped analytical results, making repeated dashboard queries and large joins faster. section 8.3.4
  • Dremio evaluates reflection statistics, cardinality, and physical layout before selecting an acceleration path. section 8.3.4
  • Administrators can configure reflections on physical or virtual datasets, while Iceberg deployments can use autonomous creation and removal. section 8.3.4

Transparent aggregate acceleration

A dashboard submits an aggregate query to Dremio. The planner detects an aggregate reflection containing the required dimensions and measures, then redirects execution to that precomputed structure without a query rewrite. section 8.3.4

Figure 8.8 When an aggregate query is sent to Dremio, it checks whether any existing reflections would optimize the query and transparently substitutes them. In this example, Dremio detects an aggregate reflection that contains the exact measurements and dimensions the query needs and diverts the query to the optimized data structure.

Shows the planner checking an aggregate query against existing reflections and substituting a matching optimized structure. section 8.3.4

How Architecting an Apache Iceberg Lakehouse covers it

How deep it goes
Explains raw and aggregation structures, planner substitution, metadata selection, configuration, and autonomous management.
Examples
Uses raw scans, repeated BI dashboards, large joins, Iceberg tables, and an aggregate query with matching dimensions and measures.
What you can do after
Apply it via Dremio to accelerate repeated analytical queries without rewriting SQL.
How it connects
Connects reflections to federated querying, Apache Iceberg, materialized views, query planners, cost-based optimization, and caching.

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Architecting an Apache Iceberg Lakehouse. The section links point to the parts of the book it draws on.