A technology audit examines how an organization’s data environment works, where systems interact, and where architecture conflicts with organizational needs. It provides an architectural reality check before designing an Iceberg lakehouse. section 4.1.3
Assess the environment across these domains:
- Document storage systems, data locations, regulatory boundaries, workloads, and their performance, cost, and compliance implications. section 4.1.3
- Inventory file formats and identify conversion or staging needs, especially where CSV and JSON dominate instead of Parquet. section 4.1.3
- Review batch and streaming frameworks, metadata and governance systems, query engines, and BI tools for silos, duplication, lineage gaps, and inconsistent semantics. section 4.2.2
Hamerliva Bank
The audit found fragmented HDFS, NAS, S3, and warehouse storage; CSV and JSON conversion needs; Spark, Kafka, NiFi, and Informatica silos; manual schema changes; and duplicated semantics across Dremio, Presto, Tableau, Power BI, and Snowflake. section 4.2.2
Table 4.2 Technical audit findings
| Domain |
Current state |
Pain points identified |
|---|---|---|
| Storage |
HDFS, NAS, S3, Snowflake, BigQuery |
Fragmentation, duplicated data, and inconsistent governance |
| File formats |
CSV, JSON dominant; Parquet used in newer systems |
Performance bottlenecks, widespread need for format standardization |
| ETL/ELT frameworks |
Spark, Kafka, NiFi, Informatica |
Legacy tool reliance, batch/stream silos, inconsistent transformation logic |
| Metadata and governance |
Hive Metastore, relational metadata stores |
Manual schema changes, limited lineage tracking, and governance inconsistencies |
| Query and BI tools |
Dremio, Presto, Tableau, Power BI, Snowflake |
Data duplication across tools, inconsistent semantics, and governance gaps |
Compare audit domains with their current states and identified pain points. section 4.2.2
