Schema validation checks incoming records against a predefined, strongly typed schema when they are written. It prevents data pollution and turns the schema into a safety guard between changing data collection and downstream consumers. section 2.3.1 section 2.1.2
In a dataset-management service, enforcement keeps raw ingested data and produced training data consistent with specifications. This protects model-training code from upstream changes and preserves backward compatibility for upstream and downstream clients. section 2.1.2
Use schema validation as part of a controlled dataset workflow:
- Write batch or streaming data into Delta Lake tables in object storage, where table metadata and schema enforcement are managed alongside the data. section 2.3.1
- Keep raw source inputs in bronze tables, clean and transform them into silver tables, then filter and purify results into gold tables. section 2.3.1
- Version schemas when needed, but prefer one schema per dataset when possible; make compatible updates, and create a new dataset type for breaking changes. section 2.1.2
Delta Lake table writes
A Delta Lake table supports querying, inserting, updating, and merging like a SQL table. Schema validation during writes rejects incompatible data before it can pollute the table; table history and time travel then support rollback to an earlier state. section 2.3.1
Figure 2.16 Delta Lake data ingestion and processing workflow. Both stream data and batch data can be saved as Delta Lake tables, and the Delta Lake tables are stored in the cloud object storage, such as Amazon S3.
The workflow locates Delta Lake tables between batch or streaming ingestion and cloud object storage, where schema enforcement accompanies managed table metadata. section 2.3.1
