# Overview A file-based, open-source storage framework (or [[open table format]]) built by [[Databricks]] that enables building [[Data Lakehouse]] architectures by helping enforce consistencies in [[Data Lakes]]. ## Architecture Diagram ![[Delta Lake 2024-10-17 09.38.24.excalidraw.svg]] ## Summary Characteristics | Type | License | Data Models | ACID | Consistency | Index Structure | Price | Website | | ---------------- | -------------- | ----------- | -------- | ----------- | --------------- | ----- | ------- | | [[File Storage]] | [[Open Source]] | | [[ACID]] | | | | | ## Pro and Cons Summary ### Pros ### Cons ### Limitations # Use Cases # Key Features # Key Characteristics ## Data Storage and Indexing Mechanisms The data is stored as files using the [[Apache Parquet]] storage format. ### [[Locality]] ## High Availability and Fault Tolerance Mechanisms ### Replication and Sharding Options ### Concurrency and Consistency Models A transaction log, referred to as Delta Log, is used as the single source of truth for data within the Delta Lake. It contains ordered records of every transaction performed on the table. It stores JSON files for each committed transaction, which contains operation performed, predicates used, and data files affected (added/removed). # Performance ## Tuning and Optimization Tips Since the Delta Log is a running list of all transactions on the database, it can get very large. All reads are done using the Delta Log, so a smaller log can improve query performance. To do so, [[Compaction]] can be applied to the log with the `OPTIMIZE` command: ```SQL OPTIMIZE my_table ZORDER BY column_name ``` During this process, you must also specify the `ZORDER` parameter to be one or more columns. The concept of [[z-ordering]] is used in Delta Lake to improve the locality of the data needed for your queries. The amount of data can be further reduced through [[Garbage Collection]] using the `VACUUM` command to clean up unused data files, such as uncommitted files and tables that no longer contain the latest table state. ### [[Liquid Clustering]] ![[Liquid Clustering]] # Ecosystem and Integration ## Compatibility with other Tools - Compatible with [[Apache Spark]] ## Available Drivers and SDKs # Security ## Authentication Methods ## Encryption # Notable Users # Documentation Links # Alternatives | Product | Why Consider | | ------- | ------------ | | | | ## Reference #### Working Notes #### Sources