Databricks News: watermark-based incremental ingestion, MCP in AI gateway, Genie, Vector Search
Summary
Databricks now offers watermark-based incremental ingestion from SQL databases without change data feed, allowing for efficient data updates and soft deletion handling. The AI Gateway supports custom MCP servers, enabling integration with external APIs like GitHub for enhanced AI application development.
Summary from the captured archive. Follow the original source for full context.
More from databricks MVP Hubert Dudek
NewsDatabricks Breaking News: Lakeflow Jobs, DABs, Apps, AI, Private Network Gateway
The video covers multiple new updates in Databricks, including Lakeflow connectors, task value loops, state repartitioning, DAB enhancements, SQL functions, serverless version 6, Unity Catalog skills, app spaces, and network gateways. It demonstrates how to configure Lakeflow job tasks, implement task value loops, and manage serverless compute permissions and environments.
Newsdatabricks TaskValue and Lakeflow - orchestrate using Python lists and arrays
NewsDatabricks News: ZeroOps, DABs, Indexes, Genie, sandboxes, migration from PowerBI, secrets
Zero Ops automatically detects errors in jobs and data quality with lineage analysis and proposes code fixes, while DABs now default to direct mode instead of Terraform with automatic state migration. Full-text search indexes deliver 400x faster queries on billion-row tables, Genie automatically converts PowerBI dashboards to Databricks metric views, and Unity Catalog secrets support granular read and reference-only permissions.