Metadata management organizes schema definitions, ownership information, access controls, and related descriptions so data platforms remain governable and trustworthy. Fragmented metadata creates inconsistent governance, manual schema changes, audit friction, and conflicting views of data. section 4.2.3
A practical metadata-management assessment examines:
- Where schemas, ownership metadata, and access policies are stored. section 4.1.3
- How ingestion, processing, and catalog choices affect governance and schema evolution. section 4.1.2
- Whether the existing catalog supports required Iceberg capabilities, including REST-based catalogs for multitable transactions. section 4.1.3
Table 4.3 Findings from interviews and audit
| Domain |
Key findings |
Problems to address |
|---|---|---|
| Storage |
Fragmented infrastructure across HDFS, NAS, and cloud warehouses |
Data duplication, inconsistent governance, and an unclear long-term storage strategy |
| Processing |
Mix of Spark, Kafka, NiFi, and Informatica across batch and streaming jobs |
Tool incompatibility, redundant pipelines, and manual schema coordination |
| Governance |
Metadata scattered across Hive, relational DBs, and manual processes |
Manual schema evolution, audit complexity, and inconsistent access controls |
| Data consumption |
Use of Dremio, Presto, Snowflake, Tableau, and Power BI |
Semantic drift across tools, multiple data copies, and inconsistent user experiences |
Use the audit findings to connect metadata fragmentation with duplicated data, manual coordination, audit complexity, and inconsistent access controls. section 4.2.3
