awesome-ducklake
Awesome DuckLake
A curated list of awesome DuckLake tools and resources.
Contents
Interfaces
- DBeaver - Universal database tool and SQL client
- DuckDB CLI - DuckDB command-line interface
- DuckDB UI Extension - A user interface for your local DuckDB instance
- Harlequin - The SQL IDE for your terminal
- marimo - Reactive Python notebook
- Positron - A next-generation data science IDE
- Sling - Efficient data transfer between various sources and destinations
- Shaper - Data Visualisation and Dashboards powered by DuckDB and SQL
Libraries
- dlt - Python library that loads data from various data sources into well-structured, live datasets
- Ibis - Portable Python dataframe library
- Narwhals - Lightweight and extensible compatibility layer between dataframe libraries
- Pointblank - Powerful, yet elegant data validation framework for Python
- PySpark - Python API for Apache Spark
Transformation
- dbt Core - Open-source tool that enables transformation using analytics engineering best practices
- lea - Minimalist SQL orchestrator
- SQLMesh - Scalable and efficient data transformation framework
Implementations
- DuckDB - DuckDB core extension for attaching to databases stored in the DuckLake format
- MotherDuck - Cloud data warehouse with support for managed DuckLake
- DataFusion-DuckLake - DuckLake query engine for Rust, built with DataFusion.
- SwanLake - Arrow Flight SQL server backed by DuckDB, with DuckLake support
- pg_ducklake - Postgres extension for managing DuckLake database
Resources
Documentation
- DuckLake DuckDB extension - Documentation for the
ducklakeDuckDB extension - Manifesto - The DuckLake manifesto from Mark Raasveldt and Hannes Mühleisen
- Specification - The specification for the DuckLake format
Examples
- DuckDB Python API - Python notebook demonstrating usage of the DuckDB Python API
- Ibis - Python notebook demonstrating usage of the Ibis dataframe library
- Pointblank - Python notebook demonstrating usage of the Pointblank data validation framework
- PySpark - Python notebook demonstrating usage of PySpark
- DuckLake + SQLMesh Tutorial: Build a Modern Data Lakehouse On Your Laptop - Tutorial for setting up a local DuckLake with SQLMesh
Videos
- DuckLake & The Future of Open Table Formats - An in-depth discussion about DuckLake with Jordan Tigani and Hannes Mühleisen
- DuckLake - The SQL-Powered Lakehouse Format for the Rest of Us - A talk on the design rationale and principles of DuckLake by Hannes Mühleisen
- DuckLake: Learning from Cloud Data Warehouses to Build a Robust “Lakehouse” - A seminar by Jordan Tigani
- Introducing DuckLake - A podcast on the DuckLake project by Hannes Mühleisen and Mark Raasveldt
Presentations
- DuckLake: The SQL-Powered Lakehouse Format - Presentation by Hannes Mühleisen