What is data pipeline architecture?
Summary
Data pipeline architecture separates ingestion, transformation, storage, and serving into distinct layers, with ELT largely replacing ETL as the dominant approach. Databricks unifies batch and streaming pipelines on a single platform (Lakeflow + Delta Lake + Unity Catalog), eliminating duplicate infrastructure and governance gaps.
Summary from the captured archive. Follow the original source for full context.
More from Databricks Blog
Announcing General Availability of Real-Time Mode for Spark Declarative Pipelines on Databricks
Real-Time Mode is now generally available on Spark Declarative Pipelines, delivering end-to-end processing latencies in the millisecond range for operational streaming workloads. It replaces the need for a separate engine like Apache Flink by unifying batch, ETL, and low-latency streaming in one framework, and ForEach sinks write instantly to operational stores for use cases like fraud detection and live personalization.
Smarter cost controls for serverless
Serverless cost controls in Databricks now extend beyond infrastructure settings to cover who can run workloads, how costs are attributed, and how much teams can spend, and you can already apply them today with usage-policy tags, billing system tables, access controls, and budget alerts. Rate limits are now available to help prevent runaway workloads, with shared budgets coming soon to keep team spending within bounds.
How Procore is powering construction analytics with OpenSharing
Procore moved its construction analytics product from an API-driven, once-a-day refresh to OpenSharing on Databricks, so customers can now pull data into their downstream systems and business processes hourly. OpenSharing distributes hundreds of curated tables with partitioned per-customer access across hundreds of thousands of data shares, doubling data availability for end customers and internal product teams.
Voice Agents are here. Context turns them into real customer value.
Voice agents only create real customer value when they have context on who is calling, so they can adapt to that person, stay within set rules, and improve with every call. On Databricks, the agent's reasoning sits next to your governed data, with context and ontology grounding each answer and a learning loop fed by every call, and the DIVA blueprint offers a starting point after you measure one high-volume routine call.
Vital Skills: Upskilling Genie Code with Healthcare and Life Sciences skills
Vital Skills is a set of open-source Agent Skills that gives coding agents healthcare and life sciences domain expertise, and Genie Code uses it to apply HLS-specific tools and best practices on Databricks. It addresses general agents silently picking clinical thresholds and defaulting to generic statistical methods. On BiomniBench, Genie Code with Vital Skills outscored Genie Code alone on supported in-scope tasks, with better consistency, accuracy and rigor, fewer missed steps and retries, and closer adherence to industry standards.
From spot fixes to measured gains: improving document extraction accuracy with ai_extract and MLflow Prompt Registry
Improve ai_extract accuracy by matching the fix to the problem: schema descriptions for field-level rules, prompts for document-level logic, precision mode for structural complexity, and citations to diagnose errors. Version prompts and schemas in MLflow Prompt Registry, deploy or roll back via aliases, and compare candidates against human-reviewed ground truth before promotion, with Genie Code automating much of this eval-and-fix loop.