liveBook · Manning

Workload separation

As explained in Architecting an Apache Iceberg Lakehouse

Workload isolation is the separation of computational workloads so that resource use or failures in one workload do not adversely affect another.

Workload separation isolates computational workloads so resource-intensive queries or failures in one workload do not degrade another. In a federation layer, dedicated engines or clusters can be tuned for different latency, throughput, concurrency, and reliability requirements. section 8.3.4 section 8.3.7

Apply separation through explicit capacity and routing controls:

  • Assign interactive analysis, scheduled refreshes, and BI dashboards to engines with different executor counts, memory footprints, and concurrency limits. section 8.3.4
  • Resize, horizontally scale, or pause engines as demand changes, balancing responsiveness and cloud compute cost. section 8.3.4
  • Route queries by user group, data source, or query type; Trino Gateway can distribute them across dedicated clusters through one connection URL. section 8.3.4 section 8.3.7
Figure 8.10 Trino clusters are made up of a coordinator node that distributes work among worker nodes. The worker nodes can be horizontally scaled by adding more workers to handle more queries.

The coordinator–worker diagram clarifies how Trino clusters distribute work and scale workers to accommodate additional queries. section 8.3.7

How Architecting an Apache Iceberg Lakehouse covers it

How deep it goes
Explains engine and cluster isolation, sizing, scaling, pausing, and routing, but stops short of scheduler internals.
Examples
Uses dedicated Dremio engines for ad hoc analysis, refresh jobs, and BI dashboards, alongside Trino Gateway cluster routing.
What you can do after
Operate/configure dedicated engines or clusters with workload-specific capacity, scaling, pausing, and routing policies.
How it connects
Connects workload separation with federated queries, multi-engine architecture, query routing, resource management, and semantic layers.

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Architecting an Apache Iceberg Lakehouse. The section links point to the parts of the book it draws on.