A Smooth Sea Never Made a Skillful Sailor

This article serves as a follow-up to previous discussions on data ingestion into Snowflake, specifically focusing on the complexities and efforts involved in integrating Change Data Capture (CDC) data with Snowflake in real time. Previous Articles on Snowflake Tour of Snowflake ingestion using CockroachDB and Redpanda Connect Integrating Snowflake with Trino Previous Articles on CockroachDB CDC Emitting Protocol Buffers with CockroachDB CDC Queries Using CockroachDB CDC with Apache Pulsar Using CockroachDB CDC with Azure Event Hubs SaaS Galore: Integrating CockroachDB with Confluent Kafka, FiveTran and Snowflake Using CockroachDB CDC with Confluent Cloud Kafka and Schema Registry CockroachDB CDC using Minio as cloud storage sink CockroachDB CDC using Hadoop Ozone S3 Gateway as cloud storage sink Motivation This article builds upon the previous discussion in Tour of Snowflake ingestion using CockroachDB and Redpanda Connect , where we investigated the process of str...

This is an exploratory article delving into various ways to ingest real-time data into Snowflake. The premise of the article is to examine the level of difficulty and effort required to ingest change data capture (CDC) data into Snowflake. Previous Articles on Snowflake Integrating Snowflake with Trino Previous Articles on CockroachDB CDC Using CockroachDB CDC with Apache Pulsar Using CockroachDB CDC with Azure Event Hubs SaaS Galore: Integrating CockroachDB with Confluent Kafka, FiveTran and Snowflake Using CockroachDB CDC with Confluent Cloud Kafka and Schema Registry CockroachDB CDC using Minio as cloud storage sink CockroachDB CDC using Hadoop Ozone S3 Gateway as cloud storage sink Motivation I work with financial services clients, and it's common to encounter a need for streaming changes in the operational datastore into a data warehouse or a data lake. A former colleague recently reached out for advice on the fastest and most efficient way to load trade data...