liveBook · Manning

Multi-engine architecture

As explained in Architecting an Apache Iceberg Lakehouse

A multi-engine architecture runs workloads on separate configurable query-execution clusters.

Multi-engine architecture isolates query workloads across separate, configurable execution clusters. Independent sizing, scaling, pausing, and resource limits improve reliability, responsiveness, and infrastructure-cost control while reducing noisy-neighbor effects. section 8.3.4

Configure engines around workload requirements:

  • Tune executor count, memory footprint, and concurrency limits for each assigned workload. section 8.3.4
  • Use low-latency engines for interactive analysis, high-throughput engines for scheduled refreshes, and concurrency-focused engines for BI dashboards. section 8.3.4
  • Resize, scale out, or pause engines as demand changes, particularly to reduce cloud compute costs during inactivity. section 8.3.4

How Architecting an Apache Iceberg Lakehouse covers it

How deep it goes
Details cluster sizing, memory, concurrency, scaling, pausing, and workload examples, but does not provide configuration commands.
Examples
Uses interactive analysis, scheduled data refreshes, and BI dashboards as distinct workload assignments.
What you can do after
Operate/configure separate engines with workload-specific resources, scaling policies, and isolation boundaries.
How it connects
Connects workload isolation with query routing, concurrency management, cloud cost optimization, and federated query performance.
Cover of Architecting an Apache Iceberg Lakehouse
liveBookmanning.com
Read Architecting an Apache Iceberg Lakehouse on liveBook Get the book at manning.com

This explainer was generated by Manning's knowledge graph from the text of Architecting an Apache Iceberg Lakehouse. The section links point to the parts of the book it draws on.