ACID stands for atomicity, consistency, isolation, and durability. Together, these properties make database transactions reliable even when errors or system failures occur. ACID guarantees matter for operational records whose integrity must be preserved, but enforcing them adds latency and can limit horizontal scalability. section 4.3.1
Relational databases are a common fit for transactional data in a digital twin: asset records, work orders, maintenance tasks, and configuration information can be stored in related tables with schema enforcement. A work-order model stores an asset once and links its tasks and work orders to it. Those relationships also support ad hoc questions, such as which tasks a particular worker completed on time for a particular asset. section 4.3.1 section 4.6
ACID is not a requirement for every data store. High-volume sensor streams are often sent to storage that favors ingestion throughput over these guarantees. Critical operational data may need ACID transactions, while analytical data can often tolerate eventual consistency. Choose by weighing consistency against latency, scale, cost, and integration complexity. section 4.3.1 section 4.6
ACID-style transactional storage is not limited to relational databases. A lakehouse can add a transaction log and metadata layer over Parquet files. Delta tables use that layer to support record-level inserts, updates, and deletes without manually rewriting entire files. This differs from a plain Parquet data lake, where changing a record requires reading and writing a replacement file. section 4.3.5
Listing 4.5 An example of using Delta tables to provide ACID transactions
import pandas as pd
from deltalake import DeltaTable, write_deltalake
def main():
table_path = "./delta_demo/sensor_data"
df1 = pd.DataFrame( #1
{
"sensor_id": ["temp_001", "temp_002", "humidity_001"],
"location": ["factory", "warehouse", "factory"],
"value": [23.5, 18.2, 65.4],
"timestamp": pd.to_datetime(["2024-01-15 10:30:00"] * 3),
}
)
write_deltalake(table_path, df1) #2
print("Created Delta table with columns:", list(df1.columns))
dt = DeltaTable(table_path)
df_update = pd.DataFrame( #3
{
"sensor_id": ["temp_001"],
"location": ["factory"],
"value": [24.1], # Updated temperature
"timestamp": pd.to_datetime(["2024-01-15 12:00:00"]),
"unit": ["celsius"],
}
)
dt.merge( #4
df_update,
predicate="target.sensor_id = source.sensor_id",
source_alias="source",
target_alias="target",
).when_matched_update_all().when_not_matched_insert_all().execute()
The local Delta-table example creates sensor records and begins a merge that updates a reading while adding a unit field. section 4.3.5
