A test environment is a controlled environment where data pipelines are executed to verify functionality, performance, reliability, data quality, and governance before production deployment. Development teams build and unit-test pipeline code in development, then deploy it to testing for broader validation. section 15.0 section 15.1
Match test data to the validation goal:
- Use a small data sample for pipeline functionality tests. section 15.1
- Use production-scale volume for performance tests. section 15.1
- Use production-like data for integrity tests. section 15.1
- Use generated or masked data when governance tests involve sensitive production information. section 15.1
The test environment sits between development and production. Quality assurance specialists and developers test the pipeline before release, while business users may perform separate acceptance testing in a user acceptance test environment. Smaller organizations may combine development and testing before production. section 15.1
Table 15.1 Types of data testing and the data they require
|
Test type
|
Data requirements
|
|---|---|
|
Data pipeline functionality
|
Small data sample
|
|
Data pipeline performance
|
Data volume similar to the production data volume
|
|
Data integrity
|
Data that closely resembles the data from the production environment
|
|
Data governance features
|
When production data contains sensitive information that should not be shared in the development and test environments, generated data or masked data can be used for testing.
|
Use the testing-data matrix to select representative, appropriately protected data for each validation objective. section 15.1
