About

This publication answers one question at a time: how do you get data from a specific database or stream into a specific warehouse or lakehouse, and what goes wrong along the way?

Each post follows a pipeline that was actually run with OLake, an open source (Apache 2.0) tool that replicates databases, Kafka topics and S3 files into Apache Iceberg tables. Query engines such as Snowflake, BigQuery, Databricks, Redshift, ClickHouse and DuckDB then read those tables in place.

What you will find here

• Sources: Postgres, MySQL, MariaDB, MongoDB, SQL Server, Oracle, DB2, Kafka and S3.
• Destinations: Snowflake, BigQuery, Databricks, Redshift, ClickHouse, DuckDB, and Apache Iceberg read by Spark, DuckDB and Trino.
• In each post: the setup in order, the exact errors hit during the run and how they were fixed, data type mappings, and what the pipeline costs to run.

Where to go next

• The maintained setup guide for every pair is in the OLake integration docs.
• Questions and bug reports go to OLake on GitHub or the community Slack.