liveBook · Manning

Bytes spilled to local storage

As explained in Snowflake Data Engineering

Bytes spilled to local storage is a query resource metric measuring data written to local storage during execution when it cannot remain in memory.

Bytes spilled to local storage is a query-profile statistic that records data written to a warehouse’s local disk when the query cannot keep it in memory. Spilling is a performance signal: reading spilled data back from disk is slower than continuing from memory. For large, complex queries, local disk can also run out, causing writes to remote cloud storage and further slowdown. section 9.2.3

Check the metric in Snowsight’s query profile. Open the Query Details pane after execution, follow the Query ID to the query profile, and inspect the Statistics pane alongside Total Execution Time. This pairs a resource-pressure measurement with the elapsed-time result that matters operationally. section 9.2

Use spill measurements to compare warehouse-sizing choices:

  • Run the same query on each warehouse size and record Total Execution Time, Bytes spilled to local storage, and credits per hour. section 9.2.2
  • Disable cached-result reuse only while collecting repeated execution statistics; otherwise, leave cached-result reuse enabled as the default. section 9.2.2
  • Compare spill values with runtime and estimated credits consumed rather than treating a larger warehouse as automatically cheaper. section 9.2.2
  • Investigate grouping and sorting over large tables when spills appear, because these operations often drive excessive data processing. section 9.2.3
Table 9.2 Comparison of query profile statistics after executing the query from listing 9.1 using differently sized virtual warehouses
Warehouse size
Credits per hour
Total execution time in seconds
Bytes spilled to local storage in MB
X-small
1
22
36.86
Small
2
10
38.06
Medium
4
6
0
Large
8
4
0

This comparison makes the sizing trade-off visible: extra-small and small executions spill roughly 37–38 MB, while medium and large executions show zero local spilling; runtime and hourly credit rates differ at the same time. section 9.2.2

Reduce spilling by filtering earlier

A sales query groups product quantities by store and product, keeps stores selling more than 100 products, and sorts by distance. On an extra-small warehouse, adding a distance_km < 1000 filter reduces the data processed. The observed execution time drops from 22 to 16 seconds; the pattern illustrates rewriting a query to limit records before expensive grouping and sorting. section 9.2.3

Warehouse resizing is one response to spilling, but query reduction is another. First limit the rows a query processes when the business requirement permits it; then rerun the query and examine the same profile statistics. A no-spill result on medium and large warehouses in the comparison demonstrates that memory capacity can remove local spilling, while cost still requires separate evaluation. section 9.2.2 section 9.2.3

How Snowflake Data Engineering covers it

How deep it goes
Explains the memory-to-local-disk spill mechanism, its performance effect, warehouse comparisons, and query filtering; it only notes remote-storage spilling.
Examples
Uses a retailer-sales aggregation across four warehouse sizes and a distance_km < 1000 filter on an extra-small warehouse.
What you can do after
Diagnose/debug with it by reading query-profile spill values, comparing warehouse runs, and reducing processed rows with query filters.
How it connects
Connects spilling with query profiles, Total Execution Time, virtual warehouses, cached results, grouping, sorting, filtering, and credits consumed.

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Snowflake Data Engineering. The section links point to the parts of the book it draws on.