CDC looks like the answer to everything: real-time data, clean integrations, simple architecture. In practice, it’s not that simple.

In this story, we’ll go through how log-based CDC actually works, when it makes sense, and what can go wrong — including real-world lessons from running it in production, with examples based on SQL Server (and patterns applicable to other databases).

For years, the industry relied on message brokers like Kafka simply moving data from Point A to Point B, forcing downstream microservices and databases to handle all the complex state management. It resulted in message queue overload, high compute costs, and unpredictable latency spikes.

This talk will introduce the paradigm shift: decoupling storage from compute and bringing the computation directly to the data stream.

Can you build a modern data platform without relying on managed services? We did — and in this talk, we’ll show you how to assemble a flexible and fully open-source stack run dbt OSS with Airflow in a real production environment.

We’ll cover orchestration strategies, deployment workflows, and how we manage dependencies and reliability. We’ll also briefly explore how MCP fits into the system as an optional extension, including some experimental LLM-related workflows.