# Overview An [[Open Source]] "[[open table format]]" that allows interoperability between various storage and compute layers. ## Architecture Diagram ![[Apache Iceberg 2024-12-31 13.03.34.excalidraw.svg]] %%[[Apache Iceberg 2024-12-31 13.03.34.excalidraw|🖋 Edit in Excalidraw]]%% ## Summary Characteristics | Type | License | Data Models | ACID | Consistency | Index Structure | Price | Website | | ---------------- | --------------- | ----------- | -------- | ----------- | --------------- | ----- | ------- | | [[open table format]] | [[Open Source]] | | [[ACID]] | | | | | ## Pro and Cons Summary ### Pros of Iceberg #flashcard - Support for [[ACID]] transactions - Efficient metadata management, such as schema evolution <!--ID: 1751507776477--> ### Cons ### Limitations # Use Cases # Key Features ## Copy on Write #flashcard The Apache Iceberg table format is responsible for holding table metadata that is stored in the underlying file system. One of the key design decisions is [[copy on write semantics]]. This means that for each write, a whole new copy of the data is created with the changes. The benefits are: - Writes don't block reads (the reads can continue to use the old copy of the data) - The writes are atomic, because you only point to the new state if the writes are successful - Allows for time travel <!--ID: 1751507776479--> ## Metadata Hierarchy #flashcard But, how is this metadata stored? It needs to be stored in a way that allows for efficient reads (i.e., not reading excess files beyond what is needed). To do so, Iceberg uses a Metadata Hierarchy composed of the following tiers: for each table in a data catalog -> Metadata file (schema, current snap shot ID, list of all snapshots ) -> manifest list (list of partitions and associated metadata files) -> manifest (list of files, file type and stats) -> file <!--ID: 1751507776482--> ![[Apache Iceberg 2024-12-31 13.35.15.excalidraw.svg]] %%[[Apache Iceberg 2024-12-31 13.35.15.excalidraw|🖋 Edit in Excalidraw]]%% **Hidden Partitioning** feature, which allows for advanced partitioning without additional columns in the schema. # Key Characteristics ## Data Storage and Indexing Mechanisms ## High Availability and Fault Tolerance Mechanisms ### Replication and Sharding Options ### Concurrency and Consistency Models # Performance ## Tuning and Optimization Tips # Ecosystem and Integration ## Compatibility with other Tools ## Available Drivers and SDKs # Security ## Authentication Methods ## Encryption # Notable Users # Documentation Links # Alternatives | Product | Why Consider | | ------- | ------------ | | | | ## Reference #### Working Notes #### Sources - [Apache Iceberg - More Than A Table Format | Distributed Systems Deep Dives With Ex-Google SWE - YouTube](https://www.youtube.com/watch?v=cB-vGKZu-eE&t=15s) - [Apache Iceberg - YouTube](https://www.youtube.com/playlist?list=PL-gIUf9e9CCtGr_zYdWieJhiqBG_5qSPa)