Self-Serve, Automated and Robust CDC pipeline using AWS DMS, DynamoDB Streams and Databricks Delta
Summary
Swiggy's self-serve CDC system ingests transactional data into its data lake in near real time, using AWS DMS for MySQL sources and AWS Lambda with DynamoDB Streams for DynamoDB sources. Databricks Delta merge reconciles inserts, updates and deletes with deduplication, partitioning and schema evolution, handling 500m CDC logs a day across 600+ tables.
Summary from the captured archive. Follow the original source for full context.
More from Databricks
NewsPreparing a CFO performance review with Genie One on Databricks | Genie Demo
The video demonstrates how to use Databricks and Genie One to prepare for a CFO performance review by scheduling automated data analysis, checking contract compliance, and generating reports from dashboards.
NewsAutomatically Route Every Task to the Right Model with Unity Gateway
Unity Gateway manages multiple AI models through a single governed platform and uses smart routing to automatically direct tasks to the most cost-effective and capable model. The demonstration shows how to configure model services, traffic splits, and fallbacks while tracking usage across different development tools.
NewsCreating a Digital Twin with Spark Real-Time Mode and Lakebase
This video demonstrates how to create a real-time digital twin for a manufacturing cell using Databricks tools such as Spark Real-Time Mode, Lakebase, Databricks Apps, and Unity Catalog. It explains how processing sensor readings instantly allows operators to detect and act on equipment faults, such as a hot spindle, before they negatively impact production quality and efficiency.
NewsRespond to Airport Disruptions Using Lakehouse//RT
Lakehouse//RT and Zerobus enable low latency query processing to help airline operations quickly manage flight disruptions and rebook passengers. This technology allows multiple teams to access real-time data simultaneously without creating a duplicate system, significantly improving decision-making speed during emergencies.
TutorialsLakebase Search: Build AI Retrieval for Agents with Lakebase Postgres
Lakebase Postgres introduces lakebase vector and lakebase text extensions to support hybrid search for AI agents. It leverages the Databricks Lakehouse architecture to synchronize and index data for efficient vector and full text search without requiring external databases.
TutorialsGoverning AI Agents with Unity Catalog and Unity Gateway on Databricks | Unity Catalog Demo
The video demonstrates how to govern AI agents using Unity Catalog and Unity Gateway on Databricks. It shows setting up access policies, user defined functions, and role filters to control data and model interactions.