What you will learn in Snowflake Data Engineering
Chapter by chapter
Concepts explained in this book
- COPY commandThe COPY command is a Snowflake SQL command that loads data from staged files into a table.
- Snowflake taskA Snowflake task is a scheduled Snowflake object that executes SQL statements, stored procedures, or Snowflake Scripting blocks.
- SnowparkSnowpark Python is the Python interface to Snowpark for constructing and executing data operations in Snowflake.
- stageAn external stage is a Snowflake object that references data files in an external cloud storage location and stores access and file-format parameters.
- dynamic tableA dynamic table is a Snowflake table whose contents are automatically derived and refreshed from a defining query.
- storage integrationA storage integration is a Snowflake object that defines the credentials and access configuration for connecting to external cloud storage.
- data ingestionData ingestion is the process of importing data from source locations into a system for storage or processing.
- SnowpipeSnowpipe is Snowflake's service for continuously ingesting newly available files from cloud storage.
- data frameA data frame is a tabular data structure that represents a relational dataset and supports operations such as selection, filtering, ordering, grouping, and combining.
- performanceQuery performance is the efficiency with which a database query executes, including its latency and resource consumption.
- incremental data ingestionIncremental data ingestion loads only newly created or changed records from a source system.
- stored procedureA stored procedure is a named, executable program stored in a database and invoked to perform data or control operations.
- role-based access controlRole-based access control is an access-control model that assigns permissions through roles associated with users or processes.
- Snowflake streamA Snowflake stream is a change data capture object that records data manipulation changes to a source object between consumption points.
- data testingData testing is the practice of systematically checking data against expected correctness, completeness, consistency, validity, and other quality conditions.
- schemaA data schema is the structural specification of the fields, types, and organization of a dataset.
- date dimensionA date dimension is an analytical table containing one row per calendar date and associated descriptive or calendar attributes.
- data transformationData transformation is the conversion of data from a raw source representation into information that satisfies defined requirements.
- resource sizeWarehouse size is the configuration that determines the compute resources allocated to a Snowflake virtual warehouse.
- EXECUTE TASK privilegeThe EXECUTE TASK command starts execution of a Snowflake task immediately.
- clustering keyA clustering key is one or more table columns or expressions that determine how Snowflake organizes data in micro-partitions.
- language modelA large language model is a model trained to process, generate, and transform human language.
- extraction layerAn extraction layer is the initial data-pipeline layer that hosts staged source files, external tables, or raw structured and semistructured data.
- external access integrationAn external access integration is a Snowflake object that governs outbound access from Snowflake code to external network locations using specified network rules and secrets.
- functional roleA functional role is an access-control role representing the privileges required for a particular job function or business responsibility.
- validity periodA validity interval is the time period during which a stored record version represents the applicable state of an entity.
- variant data typeThe VARIANT data type is a Snowflake type for storing semi-structured values.
- managed access schemaManaged access is a Snowflake access-control mode in which privilege grants on objects are centrally controlled rather than administered independently by object owners.
- notification integrationA notification integration is a Snowflake object that connects Snowflake to a third-party cloud message queuing service for event notifications.
- data metric functionA user-defined data metric function is a Snowflake function defined by a user to evaluate specified table and column data according to custom testing requirements.
- query profileA query profile is a Snowflake tool that presents details about a query's execution plan, operators, timing, and resource consumption.
- FLATTENFLATTEN is a Snowflake table function that expands elements of a semi-structured array or object into separate rows.
- SnowflakeSnowflake is a cloud data platform that provides data storage, data processing, application development, and analytics capabilities.
- target lagTarget lag is a dynamic-table setting that specifies the maximum permitted delay between the dynamic table and its source tables.
- row-level security policyA row access policy is a Snowflake policy object that determines whether rows are returned based on the querying role and conditions evaluated against row values.
- SYSTEM$STREAM_HAS_DATASYSTEM$STREAM_HAS_DATA is a Snowflake system function that returns whether a stream contains change data available for processing.
- endpointAn API endpoint is a network-accessible interface location representing a specific operation or resource of an application programming interface.
- TASK_HISTORY()TASK_HISTORY() is a Snowflake table function that returns details about current, scheduled, and completed task executions.
- directed acyclic graphA directed acyclic graph is a directed graph whose paths contain no circular dependencies.
- micro-partitionA micro-partition is a Snowflake-managed, contiguous unit of table storage containing a set of rows and associated metadata.
- data warehouse layerA data warehouse layer is a data pipeline layer that stores integrated data collected from multiple source or staging layers and modeled for business use.
- bytes spilled to local storageBytes spilled to local storage is a query resource metric measuring data written to local storage during execution when it cannot remain in memory.
- staging layerA staging layer is a data-pipeline layer that organizes and consolidates ingested data while applying limited transformations before warehouse loading.
- git repository stageA Git repository stage is a Snowflake stage that exposes files fetched from a Git repository for execution or other database operations.
- append-only systemAn append-only strategy is a data-loading technique that adds new records without updating or deleting existing records.
- file formatThe FILE_FORMAT option is a COPY command configuration that specifies file parsing format settings.
- execute immediate fromEXECUTE IMMEDIATE FROM is a Snowflake SQL command that executes SQL statements from a referenced script file.
- Snowflake CLISnowflake CLI is a command-line tool for interacting with Snowflake and performing tasks such as file uploads and connection tests.
- API keyAn API key is a credential string supplied to an application programming interface to authenticate or authorize a request.
- spillingData spilling, or query spilling, is the process of writing intermediate query data from memory to storage when available warehouse memory is insufficient.
- METADATA$ACTIONMETADATA$ACTION is a Snowflake stream metadata column that identifies the DML operation represented by a change record, such as INSERT or DELETE.
- METADATA$ISUPDATEMETADATA$ISUPDATE is a Snowflake stream metadata column containing a Boolean flag indicating whether a change record belongs to an UPDATE operation.
- Snowflake MarketplaceSnowflake Marketplace is a catalog and distribution service for discovering and accessing data products and related offerings in Snowflake.
- holidaysholidays is a Python package that provides holiday calendars and date-based holiday determination.
- database change managementDatabase change management is the practice of controlling, applying, and recovering changes to database objects and their definitions.
- target tableA target table is a table designed to receive loaded or transformed data as part of a data pipeline.
- filteringFiltering is the selection of rows that satisfy specified conditions during query processing.
- role hierarchyA role hierarchy is an access-control structure in which roles are granted to other roles so higher-level roles inherit lower-level privileges.
- SQLSQL is a declarative language for defining, querying, and manipulating data in relational database systems.
- network ruleA network rule is a Snowflake object that specifies permitted network locations and the direction and type of network traffic.
- persisted query resultA persisted query result is a previously computed Snowflake query result retained for reuse when a matching query is executed again.
- masking policyA masking policy is a data-access rule that transforms or obscures sensitive column values for unauthorized users.
- queuingQuery queuing is the waiting of queries for warehouse resources when the warehouse cannot execute them simultaneously.
- future grantA future grant is a Snowflake privilege grant that is automatically applied to objects of a specified type created later in a schema or database.
- semi-structured dataSemi-structured data is data represented with flexible or hierarchical structures rather than a fixed relational table schema.
- search optimizationSearch optimization is a Snowflake table optimization service that maintains search access paths for faster selective queries.
- presentation layerA presentation layer is a data pipeline layer that provides prepared or aggregated data for reporting and analytical consumption.
- StructTypeStructType is a Snowpark type that represents the complete schema of a structured record or table.
- eltELT is a data integration technique that extracts data, loads it into a target system, and then transforms it there.
- ETLETL is a data integration technique that extracts data, transforms it, and then loads it into a target system.
- data qualityData quality is the degree to which data is accurate, complete, consistent, valid, and fit for its intended purpose.
- internal stageAn internal stage is a Snowflake staging object whose data files are stored in cloud storage managed for the Snowflake account.
- notifications serviceAn event notification service is a cloud service that delivers messages or notifications when specified events occur in cloud resources or storage systems.
- declarative data transformation pipelineA declarative data transformation pipeline specifies desired transformations and dependencies while the execution and maintenance are managed by the data platform.
- metadataMetadata is information that describes the content, structure, origin, or characteristics of data, including file-loading details in this context.
- event tableAn event table is a Snowflake table with predefined columns for storing event and log records generated by account features.
- materialized viewA materialized view is a database object that stores the results of a query and maintains them for subsequent access.
- CREATE OR ALTERCREATE OR ALTER is a Snowflake SQL command that creates a database object if it does not exist or changes an existing object to match its supplied definition.
- etltETLT is a data pipeline pattern that performs simple transformations before loading and complex business transformations after loading.
- MERGE statementA MERGE command is a SQL command that conditionally inserts, updates, or deletes rows by matching source and target records.
- full ingestionFull ingestion is a data-loading technique that replaces all data in a target repository with the current data from source systems during each execution.
- metadata columnmetadata$filename is a Snowflake staged-file metadata column containing the name of the file from which a record was ingested.
- multicluster warehouseA multicluster warehouse is a Snowflake virtual warehouse configured to add or deactivate compute clusters according to specified parameters and demand.
- data sourceA source system is an operational or external system from which data is extracted for downstream storage and processing.
- create_dataframecreate_dataframe is a Snowpark session method that constructs a data frame from local data and an optional schema definition.
- cachingData caching is the retention of previously accessed data in faster storage or memory so subsequent operations can retrieve it more efficiently.
- sentiment analysisSentiment classification assigns text to categorical sentiment labels such as negative, neutral, or positive.
- Snowflake Cortex LLM functionA Snowflake Cortex LLM function is a Snowflake-hosted function that invokes a large language model capability such as completion, embedding, extraction, sentiment analysis, summarization, or translation.
- structured dataStructured data is data organized according to a fixed schema or predictable tabular structure.
- cron expressionA CRON expression is a scheduling syntax that specifies recurring execution times.
- credit consumptionCredit consumption is the quantity of Snowflake credits charged for warehouse activity during a period of execution.
- Snowflake CortexSnowflake Cortex is a Snowflake service that provides selected machine-learning and large-language-model capabilities through Snowflake interfaces.
- ACCOUNTADMINACCOUNTADMIN is Snowflake's top-level built-in role for account configuration, monitoring, and maintenance.
- execution environmentA development environment is an environment in which software is developed and modified before formal testing.
- private-key authenticationKey-pair authentication is an authentication method that uses a private cryptographic key for proving identity and a corresponding public key registered with the service.
- root taskA root task is a scheduled task that begins an orchestrated task graph and can have dependent child tasks.
- SYSTEM$LOG_DEBUGSYSTEM$LOG_DEBUG is a Snowflake system function that emits a message at the DEBUG logging level.
- search access pathA search access path is a background-maintained structure that records which column values may occur in each table micro-partition.
- containerA storage container is a logical cloud-storage grouping that holds data objects and controls the locations accessible through a storage configuration.
- storage accountAn Azure storage account is a top-level resource that provides access to Azure Storage services and their data containers.
- external tableAn external table is a Snowflake table whose metadata references data stored outside Snowflake and makes that data queryable.
- cast operatorThe cast operator explicitly converts an expression to a specified data type.
- lateral flattenLATERAL FLATTEN is a SQL construct that combines rows produced from a semi-structured value with columns from the surrounding row.
- STRIP_OUTER_ARRAYSTRIP_OUTER_ARRAY is a Snowflake COPY option that removes an outer array and loads its objects as separate rows.
- TRY_TO_DATETRY_TO_DATE is a Snowflake conversion function that returns a date when conversion succeeds and NULL when it fails.
- Snowpark SessionA Snowpark session is an object representing an authenticated connection through which Snowpark operations are constructed and submitted to Snowflake.
- external network accessExternal network access is a Snowflake capability that permits controlled outbound connections from Snowflake to locations outside the platform.
- cloud storage providerA cloud storage provider is a service that stores files remotely and exposes them for network access.
- external network locationAn external network location is a host or URL outside Snowflake to which network traffic may be directed.
- SYSADMINSYSADMIN is Snowflake's built-in administrative role for creating and managing many account objects and accessing data through granted roles.
- declarative configurationDeclarative database change management specifies the desired state of database objects rather than describing only the sequence of operations needed to reach it.
- LATERAL modifierThe LATERAL modifier allows a table expression to reference columns from preceding relations in the same query.
- SYSTEM$CLUSTERING_INFORMATIONSYSTEM$CLUSTERING_INFORMATION is a Snowflake system function that returns statistics describing how well a table is clustered for specified columns.
- lazy evaluationLazy execution is an evaluation strategy that postpones running data operations until a triggering operation requires their results.
- save_as_tableThe Snowpark save_as_table method executes a DataFrame computation and saves its result as a Snowflake table.
- SECURITYADMINSECURITYADMIN is Snowflake's built-in administrative role for managing grants and administering users and roles.
- USERADMINUSERADMIN is Snowflake's administrative role for managing users and creating roles when granted the relevant privileges.
- notificationAn event notification is a message indicating that a specified event, such as the arrival of a new file, has occurred.
- Snowflake Scripting blockSnowflake Scripting is Snowflake's procedural SQL language for writing blocks of statements with control flow and variables.
- productionA production environment is the environment in which an approved system runs for its intended users and operational workloads.
- test environmentA test environment is an environment in which software is executed to verify its behavior and quality.
- WAREHOUSE_LOAD_HISTORYWAREHOUSE_LOAD_HISTORY is a Snowflake usage-history table containing historical virtual-warehouse workload measurements, including query queuing activity, at five-minute intervals.
- CREATE ROLE privilegeThe CREATE ROLE privilege permits a role to create custom access-control roles.
- requestsRequests is a Python library for sending HTTP requests and receiving HTTP responses.
- dynamic_table_refresh_historydynamic_table_refresh_history is a Snowflake built-in function that returns execution and refresh records for dynamic tables.
- Kafka connectorThe Kafka connector is a framework that reads records from Kafka topics and writes them to Snowflake tables.
- Snowpipe Streaming APISnowpipe Streaming API is a Snowflake programmatic interface for continuously ingesting row-oriented streaming data as it becomes available.
- decentralized governance approachA decentralized governance approach assigns policy definition and application to individual teams.
- JSONJSON is a text-based data interchange format built from objects, arrays, and scalar values.
- data transformation layerA data transformation layer is an organized stage of a data pipeline that stores or presents intermediate outputs produced during extraction, staging, modeling, or presentation.
- selectA SQL command that retrieves rows or computed expressions from tables, stages, or other queryable sources.
- email notificationAn email notification is an electronically delivered message containing information or an alert for one or more recipients.
- SQLROWCOUNTSQLROWCOUNT is a Snowflake Scripting global variable containing the number of rows affected by the most recently executed query.
- copy_historyCOPY_HISTORY is a Snowflake table function that returns loading history for table data loaded by COPY operations.
- REFRESH commandThe REFRESH command instructs Snowpipe to detect eligible files that were already present in cloud storage.
- data governanceData governance is the framework of policies, roles, standards, and controls for managing data availability, quality, security, and compliance.
- promptA prompt is a set of questions or instructions and supporting context supplied to a generative model to guide its output.
- information schemaThe information schema is a database schema containing views and table functions that expose metadata, historical information, and usage data for database objects.
- query statisticsQuery statistics are measurements describing the execution behavior and resource consumption of a database query.
- metadata$file_row_numbermetadata$file_row_number is a Snowflake staged-file metadata column containing the source row number within its file.
- Snowflake built-in loggingSnowflake built-in logging is a Snowflake capability that captures log messages emitted while stored procedures execute.
- data pipeline layerA data pipeline layer is a functional stage through which data passes from extraction to presentation or consumption.
- directory tableA directory table is a Snowflake table-like metadata structure that lists files and their properties for a stage.
- query results cacheA query results cache is temporary storage of completed query results that can be returned for a matching query without re-executing it.
- Python worksheetA Python worksheet is a Snowflake development interface for writing and executing Python code in a worksheet context.
- Snowpark SQL APIThe Snowpark SQL API is an interface for working with Snowpark SQL operations from a local development environment.
- exception blockAn exception block is a Snowflake Scripting block that specifies statements to execute when an exception occurs.
These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.