awesome-ducklake
Awesome DuckLake
A curated list of awesome DuckLake tools and resources.
Contents
Interfaces
- DBeaver - Universal database tool and SQL client
- DuckDB CLI - DuckDB command-line interface
- DuckDB UI Extension - A user interface for your local DuckDB instance
- Harlequin - The SQL IDE for your terminal
- marimo - Reactive Python notebook
- Sling - Efficient data transfer between various sources and destinations.
Libraries
- Ibis - Portable Python dataframe library
- Narwhals - Lightweight and extensible compatibility layer between dataframe libraries
- Pointblank - Powerful, yet elegant data validation framework for Python
- PySpark - Python API for Apache Spark
Transformation
- dbt Core - Open-source tool that enables transformation using analytics engineering best practices
- lea - Minimalist SQL orchestrator
- SQLMesh - Scalable and efficient data transformation framework
Implementations
- DuckDB - DuckDB core extension for attaching to databases stored in the DuckLake format
- MotherDuck - Cloud data warehouse with support for managed DuckLake
Resources
Documentation
- DuckLake DuckDB extension -
Documentation for the
ducklakeDuckDB extension - Manifesto - The DuckLake manifesto from Mark Raasveldt and Hannes Mühleisen
- Specification - The specification for the DuckLake format
Examples
- DuckDB Python API - Python notebook demonstrating usage of the DuckDB Python API
- Ibis - Python notebook demonstrating usage of the Ibis dataframe library
- Pointblank - Python notebook demonstrating usage of the Pointblank data validation framework
- PySpark - Python notebook demonstrating usage of PySpark
Videos
- DuckLake & The Future of Open Table Formats - An in-depth discussion about DuckLake with Jordan Tigani and Hannes Mühleisen
- DuckLake - The SQL-Powered Lakehouse Format for the Rest of Us - A talk on the design rationale and principles of DuckLake by Hannes Mühleisen
- Introducing DuckLake - A podcast on the DuckLake project by Hannes Mühleisen and Mark Raasveldt