Apache Hive provides a SQL layer and table abstraction over distributed files, helping teams query Hadoop data lakes without treating raw files as relational tables. section 1.4 section 2.2
Hive translated SQL into multiple MapReduce stages, often writing intermediate results to disk. This made large batch workloads possible but imposed high latency, resource consumption, and tuning demands. section 1.4
Hive’s table abstraction improves file-based access but leaves important lakehouse limitations:
- The Hive Metastore tracks schemas, partitions, and locations, but metadata operations become fragile at very large partition counts. section 1.5
- Schemas remain tightly coupled to physical files, making column renames, removals, and safe evolution difficult. section 1.5
- Directory listings and file-footer reads support pruning, but late pruning becomes costly on cloud object stores. section 1.5
- Limited transactional guarantees allow conflicting writes, partial updates, duplicate records, and inconsistent query results. section 2.2
