A production environment is where approved data pipelines execute on live data and provide the required data structures for downstream consumers. It is configured for performance, reliability, and security, making it the operational destination after development and testing. section 15.1
Place production within the delivery pipeline:
- Developers build prototypes, compare approaches, and optimize pipeline code in development. section 15.1
- Test and quality assurance teams verify pipeline functionality, performance, and reliability before release. section 15.1
- Business users may validate functionality, data correctness, completeness, and governance in a user acceptance test environment. section 15.1
- Continuous delivery automates deployment between these environments, while continuous integration runs automated actions after code changes merge into the repository. section 15.0
Pipeline release
A bakery data pipeline ingests operational-system data and customer order JSON files, transforms the data, and moves through development and testing before executing in production for downstream consumers. section 15.0 section 15.1
Table 15.1 Types of data testing and the data they require
|
Test type
|
Data requirements
|
|---|---|
|
Data pipeline functionality
|
Small data sample
|
|
Data pipeline performance
|
Data volume similar to the production data volume
|
|
Data integrity
|
Data that closely resembles the data from the production environment
|
|
Data governance features
|
When production data contains sensitive information that should not be shared in the development and test environments, generated data or masked data can be used for testing.
|
Use testing data that matches the production workload: small samples for functionality, production-scale volumes for performance, representative data for integrity, and masked or generated data for sensitive governance tests. section 15.1
