A Smooth Sea Never Made a Skillful Sailor

This article serves as a follow-up to previous discussions on data ingestion into Snowflake, specifically focusing on the complexities and efforts involved in integrating Change Data Capture (CDC) data with Snowflake in real time. Previous Articles on Snowflake Tour of Snowflake ingestion using CockroachDB and Redpanda Connect Integrating Snowflake with Trino Previous Articles on CockroachDB CDC Emitting Protocol Buffers with CockroachDB CDC Queries Using CockroachDB CDC with Apache Pulsar Using CockroachDB CDC with Azure Event Hubs SaaS Galore: Integrating CockroachDB with Confluent Kafka, FiveTran and Snowflake Using CockroachDB CDC with Confluent Cloud Kafka and Schema Registry CockroachDB CDC using Minio as cloud storage sink CockroachDB CDC using Hadoop Ozone S3 Gateway as cloud storage sink Motivation This article builds upon the previous discussion in Tour of Snowflake ingestion using CockroachDB and Redpanda Connect , where we investigated the process of str...

Today, we're going to evaluate Hadoop Ozone object store for CockroachDB object store sink viability. A bit of caution, this article only explores the art of possible, please use the ideas in this article at your own risk! Firstly, Hadoop Ozone is a new object store Hadoop Community is working on. It exposes an S3 API backed by HDFS and can scale to billions of files on prem! This article only scratches the surface, for everything there is to learn about Hadoop and Ozone , navigate to their respective websites.

Today, we’re going to explore CDC capability in CockroachDB Enterprise Edition using Minio object store as sink. To achieve this, we’re going to reuse the compose file from the first two tutorials and finally bring this to a close. Without further ado