These notes follow the setup in the order it has to happen. Wherever it can stop, you get the exact message that comes back and the change that clears it.

The pipeline is change data capture (CDC) from the Postgres write-ahead log (WAL). OLake Go (open source, Apache 2.0) reads the log and commits Apache Iceberg tables into BigLake, which is Google Cloud's Iceberg representational state transfer (REST) catalog. The files sit in a Cloud Storage bucket, and BigQuery reads the catalog directly, so the path has no Datastream, no Kafka and no load job.

I ran this on Postgres 16 with OLake Go v1.1.0 and BigQuery in us-central1. The data was 8 customers and 40 orders, followed by 12 changes.

1. Postgres: settings, role, publication, slot

OLake Go uses logical replication with the built-in pgoutput plugin, available in Postgres 10 and later. Four things have to exist before the first sync:

-- 1. Logical replication. wal_level needs a restart.
ALTER SYSTEM SET wal_level = 'logical';
ALTER SYSTEM SET max_replication_slots = 10;
ALTER SYSTEM SET max_wal_senders = 10;
ALTER SYSTEM SET wal_sender_timeout = 0;   -- so a long first snapshot is not cut off

-- 2. A role for OLake Go
CREATE ROLE olake WITH LOGIN REPLICATION PASSWORD '<password>';
GRANT USAGE ON SCHEMA public TO olake;
GRANT SELECT ON ALL TABLES IN SCHEMA public TO olake;
ALTER DEFAULT PRIVILEGES IN SCHEMA public GRANT SELECT ON TABLES TO olake;

-- 3. A publication and a pgoutput slot
CREATE PUBLICATION olake_pub FOR TABLE customers, orders;
SELECT pg_create_logical_replication_slot('olake_slot', 'pgoutput');

Confirm it took:

SHOW wal_level;                                        -- logical
SELECT slot_name, plugin, active FROM pg_replication_slots;
SELECT pubname, puballtables FROM pg_publication;

Managed Postgres sets wal_level through a parameter group or a server setting, not through ALTER SYSTEM. The rest is identical.

Two rules save trouble later. Give every job its own slot and publication, because two jobs on one slot consume each other's changes and both lose data. And give any table without a primary key ALTER TABLE <table> REPLICA IDENTITY FULL;, so its updates and deletes carry the old row.

2. Google Cloud: API, bucket, service account, key

In the console, inside the project that will hold the tables:

1. Enable the BigLake API under APIs and services.
2. Create a Cloud Storage bucket in the region where BigQuery will run queries. Its name becomes the catalog name in SQL.
3. Under IAM and Admin (Identity and Access Management), Service Accounts, create a service account. Grant it BigLake Editor on the project and Storage Object User on the bucket.
4. On the account's Keys tab, use Add key, Create new key, JSON. The file downloads once.

You do not create the catalog. BigLake creates a single-bucket catalog, named after the bucket, the first time a client calls the endpoint with gs://<bucket> as the catalog path. It appears under BigLake, Catalogs after the first connection test.

The new catalog is in end-user credential mode, where every client brings its own storage credentials. Keep it there. The first error below is what happens otherwise.

3. OLake UI: source and destination

OLake UI comes from the Docker Compose install and listens on http://localhost:8000.

Source. Create a Postgres source, keep Enable CDC on, and enter the Replication Slot and Publication from step 1. My Postgres ran in another container, so the host was host.docker.internal.

Set up source runs the connection test and the CDC checks before anything is saved. A wrong wal_level or a missing slot is reported here, with the setting named and its recommended value.

Destination. Pick Apache Iceberg and the BigLake catalog type. The REST Catalog URI is https://biglake.googleapis.com/iceberg/v1/restcatalog for every project and region, the BigLake Catalog Path is gs://<bucket> with no folder after it, and the Authentication Type is GCP. Paste the whole key file into GCP Service Account JSON and set the GCP Project ID.

Create makes OLake Go call the endpoint, then write and drop a test table in the bucket.

Stop: a 403 on the destination test

There is no long message to quote for this one. The useful part is the host the 403 comes from.

From biglake.googleapis.com, the service account lacks BigLake Editor, the role that lets it create namespaces and commit tables. From storage.googleapis.com, it lacks Storage Object User on the bucket, which is what lets it write the Parquet and metadata files.

Error: the catalog is in vended-credentials mode

failed to test destination: failed to create or get table: rpc error: code = Internal desc = Failed to process request: Malformed request: X-Iceberg-Access-Delegation header must be present and contain `vended-credentials` when credential mode is `CREDENTIAL_MODE_VENDED_CREDENTIALS`.

The catalog was created with, or later switched to, --credential-mode=vended-credentials. OLake Go 1.1.0 does not send the header BigLake requires in that mode. It is tracked as issue 1369 in the OLake GitHub repository.

Switch the catalog back:

gcloud beta biglake iceberg catalogs update <bucket> --credential-mode=end-user

If you run OLake Go from the command line instead of the UI, there is a second way out. Setting "rest_access_delegation": "vended-credentials" in destination.json makes the same test pass. The UI has no field for it yet.

4. The job and the first sync

On the job, under Advanced Settings, I set Target Query Engines to BigQuery, Spark and DuckDB. That list decides the Iceberg delete format: OLake Go only offers formats every listed engine reads. For these three that left equality and positional deletes, and the run used Positional Deletes.

Each table went in as Full Refresh + CDC with Upsert as the ingestion mode.

The historical load took 35 seconds for 48 records (2.23 KB), and that figure includes starting the container. The namespace is built from the job name, the database and the schema, so job postgres_to_bigquery on database shop produced postgres_to_bigquery_shop_public.

5. The first query, and two ways it fails

A BigLake catalog is a BigQuery data source in the same project, so nothing is registered. The bucket is the catalog, the namespace is the dataset, and the name has four parts:

SELECT id, name, city, tier
FROM `<project-id>.<bucket>.postgres_to_bigquery_shop_public.customers`
ORDER BY id;

Error: BigQuery cannot find the dataset

Not found: Dataset <project>:<bucket>.postgres_to_bigquery_shop_public was not found in location US

Nothing is missing. The query ran in a location other than the bucket's region, and the default US multi-region does not see a catalog whose bucket is regional.

In the SQL editor, open More, then Query settings, and choose the bucket's region. With the bq tool, pass --location=<region>. Leaving the location unset also works, because BigQuery then picks it from the table.

Error: INFORMATION_SCHEMA rejects the name

Invalid project ID '<project>.<bucket>'. Project IDs must contain 6-63 lowercase letters, digits, or dashes.

INFORMATION_SCHEMA views expect a dataset-qualified name and do not resolve catalog tables, so the parser reads the first two parts as a project ID. To list columns and types, use this instead:

bq show --schema <project>:<bucket>.<namespace>.<table>

6. The change sync, and the one that hangs

Between the syncs I made 12 changes in Postgres: 7 inserts and 5 updates across the two tables. The follow-up sync took 12 records and 594 bytes off the slot in 58 seconds, and the larger share of that time went to the WAL reader confirming the slot was idle.

The commit report in the sync log shows what was written for customers: addedDataFiles=1, addedRecords=4, addedPositionalDeleteFiles=1, addedPositionalDeletes=2. That is 2 inserts and 2 new row versions in one file, and one delete file retiring the 2 old versions.

BigQuery returned 10 customers on the next query, one per key, without a refresh or an ALTER TABLE.

Stop: the next sync waits on the slot

This one has no error text either, only a sync that hangs after the snapshot. Either another process holds the replication slot, or wal_sender_timeout cut the connection during a quiet period.

Look in pg_stat_replication for a second consumer of olake_slot. Then make sure wal_sender_timeout = 0 is set, as in step 1.

7. A second reader, and its 401

DuckDB 1.5.5 attached the same catalog with a Google access token and read the files with a hash-based message authentication code (HMAC) key for the service account. It returned the same 10 customers. Spark 4.0.1 with Iceberg 1.10.0 did the same from a laptop.

The access token lives an hour. When a DuckDB query starts failing with 401, run CREATE OR REPLACE SECRET with a fresh token.

Not errors, but they will catch you

• numeric arrives as FLOAT64. OLake Go stores decimals as doubles. The order sums in this run still matched Postgres to the cent, but round at query time for exact decimal arithmetic.
• json and jsonb arrive as STRING. PARSE_JSON or JSON_VALUE turns them back.
• The slot pins WAL. A job paused for a weekend on a busy server can hold tens of gigabytes. Drop the slot when a job is retired.
• metadata.json has a ceiling. BigLake rejects one above 1 MB, and every snapshot OLake Go commits is kept in it. Expire snapshots and compact on a schedule.
• Deletion vectors are not an option yet. BigQuery reads them only in preview, so the job stays on positional deletes while BigQuery is a reader.
• This is not native BigQuery speed. Each query reads Parquet from the bucket. For a hot dashboard, load that slice into a native table on a schedule.

The complete guide, with the Spark and DuckDB connection steps and the data type table, is here: Postgres to BigQuery on olake.io.

The pipeline is change data capture (CDC) from the MySQL binary log, read by OLake Go (open source, Apache 2.0) and committed as Apache Iceberg tables into Snowflake's Horizon catalog. Snowflake manages the tables itself, so there is no Kafka, no Debezium and no COPY INTO.

I ran this against MySQL 8.4 with OLake Go v1.1.0 and a Snowflake account on AWS ap-south-1. The data was 8 customers and 40 orders, followed by 12 changes.

One error below stopped that run on its first attempt, and it is marked. The others are what this same setup returns when a single setting is off.

1. MySQL: the binlog, and a user that can read it

OLake Go reads the same stream MySQL replicas use. These go in my.cnf, or in the parameter group on managed MySQL, followed by a restart:

server-id = 1
log_bin = mysql-bin
binlog_format = ROW
binlog_row_metadata = FULL
binlog_expire_logs_seconds = 604800

Then the user:

CREATE USER IF NOT EXISTS 'olake'@'%' IDENTIFIED BY '<password>';
GRANT SELECT, REPLICATION SLAVE, REPLICATION CLIENT ON *.* TO 'olake'@'%';
FLUSH PRIVILEGES;

Verify what the server actually has:

SHOW VARIABLES WHERE Variable_name IN ('log_bin', 'binlog_format', 'binlog_row_metadata', 'server_id');
SHOW GRANTS FOR 'olake'@'%';

ROW format matters because statement-based logging does not carry row images. FULL row metadata puts column names and types into every event, which is how a column added later is recognised.

Error: the source test rejects the binlog

binlog_format: current STATEMENT, recommended ROW

This appears in the OLake UI source test, which names the failing setting along with its current and recommended value. binlog_format and binlog_row_metadata can be changed at runtime with SET GLOBAL. Write them to my.cnf or the parameter group as well, or they revert on restart, and note that log_bin itself needs a restart.

No error, but data goes missing

• A table without a primary key replicates inserts only. Its updates and deletes cannot be matched to a row and are skipped. Add a primary key or a unique index first.
• Aurora MySQL ships with the binlog off. Enable it on the cluster parameter group.

2. Snowflake: database, role, service user, policy, token

Run once as ACCOUNTADMIN in a Snowsight worksheet:

USE ROLE ACCOUNTADMIN;

CREATE WAREHOUSE IF NOT EXISTS OLAKE_WH WAREHOUSE_SIZE = XSMALL AUTO_SUSPEND = 60 AUTO_RESUME = TRUE;
CREATE DATABASE IF NOT EXISTS OLAKE_DEMO;
ALTER DATABASE OLAKE_DEMO SET CATALOG = 'SNOWFLAKE';
ALTER DATABASE OLAKE_DEMO SET EXTERNAL_VOLUME = 'SNOWFLAKE_MANAGED';

CREATE ROLE IF NOT EXISTS OLAKE_ROLE;
GRANT USAGE, CREATE SCHEMA ON DATABASE OLAKE_DEMO TO ROLE OLAKE_ROLE;
GRANT USAGE ON WAREHOUSE OLAKE_WH TO ROLE OLAKE_ROLE;
GRANT ROLE OLAKE_ROLE TO ROLE SYSADMIN;

CREATE USER IF NOT EXISTS OLAKE_SVC TYPE = SERVICE DEFAULT_ROLE = OLAKE_ROLE;
GRANT ROLE OLAKE_ROLE TO USER OLAKE_SVC;

USE SCHEMA OLAKE_DEMO.PUBLIC;
CREATE NETWORK RULE IF NOT EXISTS olake_ip
  MODE = INGRESS TYPE = IPV4 VALUE_LIST = ('<OLake Go egress IP>/32');
CREATE NETWORK POLICY IF NOT EXISTS olake_network_policy
  ALLOWED_NETWORK_RULE_LIST = ('OLAKE_DEMO.PUBLIC.olake_ip');
ALTER USER OLAKE_SVC SET NETWORK_POLICY = olake_network_policy;

SELECT CURRENT_ORGANIZATION_NAME() || '-' || CURRENT_ACCOUNT_NAME() AS account_identifier,
       CURRENT_REGION() AS region;

ALTER USER OLAKE_SVC ADD PROGRAMMATIC ACCESS TOKEN OLAKE_PAT
  ROLE_RESTRICTION = 'OLAKE_ROLE' DAYS_TO_EXPIRY = 30;

Setting CATALOG and EXTERNAL_VOLUME at database level is what turns each new table into a Snowflake-managed Iceberg table. Copy the token secret when the final statement prints it, because it is displayed a single time.

Error: the token statement is refused

Network Policy is required when creating a programmatic access token for user OLAKE_SVC

Snowflake issues a programmatic access token (PAT) to a service user only under a network policy. Create the rule and the policy, attach the policy to the user, then add the token. On a laptop the IP to allow is your public IP, which curl -s https://checkip.amazonaws.com returns.

Error: the destination test fails at the catalog handshake

There is no single string for this one, because Snowflake's reply depends on the cause. OLake Go exchanges the token at /v1/oauth/tokens before it does anything else, and three causes cover nearly every failure:

1. The service user has no network policy, or the sync comes from an IP outside it.
2. The scope does not match the token's ROLE_RESTRICTION. It must be session:role:OLAKE_ROLE.
3. The token has expired. The setup above gives it 30 days, the default is 15 and the maximum is 365.

Reproduce it outside the UI with the call OLake Go makes:

curl -s -X POST "https://<account_identifier>.snowflakecomputing.com/polaris/api/catalog/v1/oauth/tokens" \
  -H 'Content-Type: application/x-www-form-urlencoded' \
  --data-urlencode grant_type=client_credentials \
  --data-urlencode scope=session:role:OLAKE_ROLE \
  --data-urlencode client_secret=<your PAT>

A healthy token comes back as JSON containing an access_token. A broken one returns the same error Snowflake hands to OLake Go.

3. OLake UI: source, destination, job

OLake UI comes from the Docker Compose install and serves on http://localhost:8000.

Source. Choose MySQL and leave Enable CDC on. In this run the host was host.docker.internal because MySQL sat in another container, the database was shop, the user was olake, and Initial Wait Time was 10.

Destination. Choose Apache Iceberg, catalog type Horizon and authentication Programmatic Access Token (PAT). The catalog URI is https://<account_identifier>.snowflakecomputing.com/polaris/api/catalog, the database is OLAKE_DEMO, the scope is session:role:OLAKE_ROLE, and the storage region was AWS_AP_SOUTH_1.

Job. Under Advanced Settings, list every engine that will read the tables as a target query engine. I listed Snowflake, DuckDB and Apache Spark, and OLake Go resolved the upsert type to Positional Deletes on both streams, with sync mode Full Refresh + CDC.

4. The first sync

Error: the snapshot stops before copying a row (my run hit this)

failed to get or split chunks: stats not populated for table[shop.orders]. Please run ANALYZE TABLE to update table statistics

OLake Go plans the parallel snapshot from information_schema row estimates. A table that was just created or just loaded has no estimates yet.

The fix is one statement per table, then Sync Now again:

ANALYZE TABLE customers, orders;

After that the first sync copied 48 records, 8 customers plus 40 orders, in 34 seconds with container start included.

5. The first query

The destination schema takes its name from the job and the source database. Job mysql_to_snowflake on database shop produced mysql_to_snowflake_shop.

Error: Snowflake cannot find the table

SQL compilation error:
Object 'OLAKE_DEMO.MYSQL_TO_SNOWFLAKE_SHOP.CUSTOMERS' does not exist or not authorized.

The table exists. Snowflake upper-cased an unquoted identifier, while the schema and tables were created with their lowercase MySQL names. Quote them:

SHOW ICEBERG TABLES IN SCHEMA OLAKE_DEMO."mysql_to_snowflake_shop";

SELECT "id", "name", "city", "tier"
FROM OLAKE_DEMO."mysql_to_snowflake_shop"."customers"
ORDER BY "id";

6. The change sync

I made 12 changes in MySQL: 2 new customers, 5 new orders, 2 customer updates and 3 order status updates. The next sync read 12 records (563 bytes) from the binlog and took 23 seconds.

The commit report in the sync log gives the file-level picture for customers: addedDataFiles=1, addedRecords=4, addedPositionalDeleteFiles=1, addedPositionalDeletes=2. That is 2 inserts plus 2 new row versions in one data file, and a delete file that retires the 2 old versions. orders looked the same with 8 rows and 3 deletes.

Snowsight showed the result with no refresh. The _op_type column reads u on the two updated customers, c on the two new ones and r on the untouched snapshot rows.

Error: a later sync cannot find its binlog position

idle timeout reached, exiting binlog syncer

There are two causes. If binlog_expire_logs_seconds has already removed the saved binlog file, the position is gone, so run a fresh full snapshot and lengthen retention.

If the file is still there but the server is slow to seek to the position, raise Initial Wait Time on the source. 10 seconds was enough for this two-table database.

Things that raise no error and still surprise people

• decimal becomes FLOAT. An order roll-up still matched MySQL to the cent in Snowflake, DuckDB 1.5.5 and Spark 3.5.6, but round at query time for exact decimal arithmetic.
• datetime and timestamp become TIMESTAMP_LTZ(6) in UTC. json, enum, time and bit(1) arrive as VARCHAR.
• Freshness is a schedule, down to one minute, not sub-second streaming.
• The network policy follows the token everywhere. DuckDB or Spark reading through Horizon must also connect from an allowed IP.
• DuckDB is not an option on an Azure-hosted Snowflake account. It attaches REST catalogs backed by S3, S3 Tables or Google Cloud Storage only.
• Binlog retention costs disk. Seven days on a busy server can be tens of gigabytes.

The AWS Glue variant, the DuckDB and Spark connection steps and the full data type table are in the MySQL to Snowflake guide on olake.io.

The pipeline: Postgres logical replication, read by OLake (open source, Apache 2.0), written as Apache Iceberg tables into Snowflake's own catalog. Snowflake sees them as Snowflake-managed tables, so there is no stage, no COPY INTO and no Kafka.

The run behind this post: Postgres 16, OLake UI with the Postgres source on v1.1.0, an X-Small Snowflake warehouse. It used 8 customers and 40 orders, then 12 changes.

1. Postgres: four settings, one restart

ALTER SYSTEM SET wal_level = 'logical';      -- needs a restart
ALTER SYSTEM SET max_replication_slots = 10;
ALTER SYSTEM SET max_wal_senders = 10;
ALTER SYSTEM SET wal_sender_timeout = 0;     -- so a long first snapshot is not cut off

CREATE ROLE olake WITH LOGIN REPLICATION PASSWORD '<password>';
GRANT USAGE ON SCHEMA public TO olake;
GRANT SELECT ON ALL TABLES IN SCHEMA public TO olake;
ALTER DEFAULT PRIVILEGES IN SCHEMA public GRANT SELECT ON TABLES TO olake;

CREATE PUBLICATION olake_pub FOR TABLE customers, orders;
SELECT pg_create_logical_replication_slot('olake_slot', 'pgoutput');

Check before moving on:

SHOW wal_level;                                         -- logical
SELECT slot_name, plugin, active FROM pg_replication_slots;
SELECT pubname, puballtables FROM pg_publication;

On RDS, Aurora, Azure or Cloud SQL, wal_level is a parameter-group or server setting, not ALTER SYSTEM.

Error: the replication connection is refused

FATAL: must be superuser or replication role to start walsender

The user has SELECT but not the REPLICATION attribute. Run ALTER ROLE olake WITH REPLICATION;. On RDS and Aurora, REPLICATION is reserved, so grant the rds_replication role instead.

Not an error yet, but it will be

• Tables without a primary key replicate inserts only. Give them ALTER TABLE <table> REPLICA IDENTITY FULL; so updates and deletes carry the old row.
• One slot and one publication per job. Two jobs sharing a slot read each other's changes, and both lose data.
• The slot holds WAL until it is read. Pause a job for a week and Postgres keeps a week of WAL. Set max_slot_wal_keep_size and alert on pg_replication_slots.wal_status.

2. Snowflake: database, role, service user, network policy, token

Run once as ACCOUNTADMIN:

CREATE WAREHOUSE IF NOT EXISTS OLAKE_WH WAREHOUSE_SIZE = XSMALL AUTO_SUSPEND = 60 AUTO_RESUME = TRUE;
CREATE DATABASE IF NOT EXISTS OLAKE_DEMO;
ALTER DATABASE OLAKE_DEMO SET CATALOG = 'SNOWFLAKE';
ALTER DATABASE OLAKE_DEMO SET EXTERNAL_VOLUME = 'SNOWFLAKE_MANAGED';

CREATE ROLE IF NOT EXISTS OLAKE_ROLE;
GRANT USAGE, CREATE SCHEMA ON DATABASE OLAKE_DEMO TO ROLE OLAKE_ROLE;
GRANT USAGE ON WAREHOUSE OLAKE_WH TO ROLE OLAKE_ROLE;
GRANT ROLE OLAKE_ROLE TO ROLE SYSADMIN;

CREATE USER IF NOT EXISTS OLAKE_SVC TYPE = SERVICE DEFAULT_ROLE = OLAKE_ROLE;
GRANT ROLE OLAKE_ROLE TO USER OLAKE_SVC;

USE SCHEMA OLAKE_DEMO.PUBLIC;
CREATE NETWORK RULE IF NOT EXISTS olake_ip
  MODE = INGRESS TYPE = IPV4 VALUE_LIST = ('<OLake egress IP>/32');
CREATE NETWORK POLICY IF NOT EXISTS olake_network_policy
  ALLOWED_NETWORK_RULE_LIST = ('OLAKE_DEMO.PUBLIC.olake_ip');
ALTER USER OLAKE_SVC SET NETWORK_POLICY = olake_network_policy;

ALTER USER OLAKE_SVC ADD PROGRAMMATIC ACCESS TOKEN OLAKE_PAT
  ROLE_RESTRICTION = 'OLAKE_ROLE' DAYS_TO_EXPIRY = 30;

The two database settings are what make every table Snowflake-managed, on Snowflake's own storage. You don't need an S3 bucket, an IAM role or an external volume of your own.

Error: the token is refused before it exists

Network Policy is required when creating a programmatic access token for user OLAKE_SVC

Snowflake issues a programmatic access token to a service user only under a network policy. Create the rule and the policy first, then the token. On a laptop the egress IP is your public IP (curl -s https://checkip.amazonaws.com). On a server, it is the NAT gateway or public IP.

Error: the destination test fails at the token exchange

OLake exchanges the token at /v1/oauth/tokens before anything else, so a bad token fails there. On a later rerun of this setup, the cause was a home IP that had changed since the rule was written, and the exchange came back unauthorized_client. Adding the new IP to olake_ip fixed it. The other two causes, in order of likelihood:

1. Wrong scope or role. The scope must be session:role:OLAKE_ROLE, matching the token's ROLE_RESTRICTION.
2. Expired token. The default is 15 days. Re-issue it with ALTER USER ... ADD PROGRAMMATIC ACCESS TOKEN.

Test outside the UI with the same call OLake makes:

curl -s -X POST "https://<account_identifier>.snowflakecomputing.com/polaris/api/catalog/v1/oauth/tokens" \
  -H 'Content-Type: application/x-www-form-urlencoded' \
  --data-urlencode grant_type=client_credentials \
  --data-urlencode scope=session:role:OLAKE_ROLE \
  --data-urlencode client_secret=<your PAT>

A good token returns JSON with an access_token.

3. OLake: source, destination, job

OLake UI runs from the Docker Compose install and listens on http://localhost:8000.

Source. Pick Postgres, turn on Enable CDC, and enter the slot and publication from step 1. If Postgres runs in another container, the host is host.docker.internal.

Set up source runs the connection test and the CDC checks before it saves. All four required checks have to pass. This is where a missing wal_level or a missing slot shows up, before any data moves.

Destination. Pick Apache Iceberg, catalog type Horizon, and authentication Programmatic Access Token (PAT). The catalog URL is https://<account_identifier>.snowflakecomputing.com/polaris/api/catalog, the warehouse is OLAKE_DEMO, and the scope is session:role:OLAKE_ROLE.

Job. Set the target query engines to Snowflake, plus DuckDB or Spark if they will read the tables. OLake picks the update and delete format every listed engine can read and sets it on each stream. In this run that was positional deletes.

Error: Snowflake shows stale rows

A job created without a target query engine defaults to equality deletes, and Snowflake does not read those. The updates land, but Snowflake keeps showing the old rows.

Fix: edit the job's streams and set the target query engine to Snowflake. OLake switches the upsert type, and on the next sync it rewrites the existing delete files into the new format.

4. Query it, and the last error

The first sync copied 8 customers and 40 orders in 41 seconds, container start included. The schema is named after the job, the database and the source schema, so pg_to_snowflake on database shop became pg_to_snowflake_shop_public.

Error: the table "does not exist"

SQL compilation error:
Object 'OLAKE_DEMO.PG_TO_SNOWFLAKE_SHOP_PUBLIC.CUSTOMERS' does not exist or not authorized.

Snowflake upper-cases unquoted names, and OLake keeps the lowercase Postgres names. Quote them:

SELECT "id", "name", "city", "tier", "_op_type", "updated_at"
FROM OLAKE_DEMO."pg_to_snowflake_shop_public"."customers"
ORDER BY "id";

The rule runs both ways: if you create the database as "olake_demo" in quotes, OLake must be given olake_demo, not OLAKE_DEMO.

5. The change sync

12 changes went in: 7 inserts and 5 updates. The second sync read only those 12 records (589 bytes) from the slot and finished in 23 seconds.

For customers that meant one new data file with 4 rows, plus one small delete file that retires the 2 old versions. The sync log's commit report shows it: addedDataFiles=1, addedRecords=4, addedPositionalDeleteFiles=1, addedPositionalDeletes=2.

Each row carries _op_type: r for snapshot rows, c for inserts and u for updates. A deleted row stays as a marker with _op_type = 'd' and null fields, so filter it out in queries that should not see deletes.

Things that are not errors but will surprise you

• numeric arrives as FLOAT. The order totals still matched Postgres to the cent, but round at query time when you need exact decimal arithmetic.
• Timestamps become TIMESTAMP_LTZ(6) in UTC. json, jsonb, uuid and arrays arrive as VARCHAR.
• Freshness is a schedule, down to one minute, not sub-second streaming.
• Small files add up. A table that gets a new file every minute wants compaction. Snowflake compacts manifests for free, and data-file compaction is up to the writer side.

The full guide has the data type table, the AWS Glue variant and the DuckDB and Spark connection steps: Postgres to Snowflake on olake.io.