# Overview Broadly, an open table format is a way of capturing the organized nature of files/objects within [[object storage]]. In more concrete terms, it describes the underlying files and data using metadata to describe which files belong to the same "table", which files are in the same "partition" of a table, and many other facets of the data (different formats capture different information that enable advanced features like "time travel" and schema management). Regardless of the specifics, the end result is a "thin" interface that sits on top of unstructured storage that adds the facade of structure to applications using the open table format to interact with the data. This illusion of structure and absence of awareness of the differing characteristics on top of [[heterogeneous data]] types is often called [[data virtualization]]. The open table format allows for hiding the underlying format and source of the data to the end user. ![[open table format 2025-01-10 09.15.35.excalidraw.svg]] %%[[open table format 2025-01-10 09.15.35.excalidraw|🖋 Edit in Excalidraw]]%% # Key Considerations ## Types of Open Table Formats As of writing this (January 2025), these formats are still evolving. I think a pro and con list could be outdated within 3 months as these technologies mature, so I'll just list the names: - [[Apache Iceberg]] - [[Apache Hudi]] - [[Delta Lake]] - [[Apache Hive]] - [[Apache Paimon]] - [[DuckLake]] # Implementations Details ## Performance Considerations Details are captured here: [[Understanding Performance in a Data Lakehouse Architecture]] # References - [Medium](https://medium.com/walmartglobaltech/lakehouse-at-fortune-1-scale-480bcb10391b)