Brickster Newsletter

Lakebase + Native Lakehouse Sync: the OLTP/OLAP split is finally fading

Almost every data team runs two parallel systems. A transactional database (Postgres, Aurora, Cloud SQL) backs the application: customer logins, agent state, session data, billing events. A separate analytical platform (Databricks, Snowflake) holds the warehouse where the dashboards and ML models live. Between them sits an ever-growing forest of ETL jobs, sync delays, schema drift, and "the prod DB and the warehouse don't match again" Slack threads.

Lakebase has been Databricks' answer to that split. It's a managed Postgres service that lives inside the lakehouse, under the same Unity Catalog governance and the same compute fabric. What was missing until earlier this month was the automatic link between Lakebase and the rest of the lakehouse.

That gap closed on May 12 with the Public Preview of Native Lakehouse Sync Public Preview. It replicates Lakebase Postgres data into Unity Catalog managed tables automatically. No separate pipeline, no external compute, no reverse-ETL job. The app writes to Postgres; the analytical view in Delta updates within a minute, with the same governance applied across both.

🧠 What this unlocks

The combination matters most for workloads that previously needed two systems and an ETL crew to keep them honest:

• AI agents with real memory. Agents need fast reads (vector lookup, document retrieval) and fast writes (conversation state, working memory, entity tracking). Lakebase plus Delta plus Vector Search now sit on one governance plane. Agents read and write production data without crossing system boundaries.
• SaaS apps backed by lakehouse data. Your customer-facing API queries Postgres for low-latency reads. Native Lakehouse Sync replicates that data into Delta automatically, so the analytics team works from the same source. No reverse-ETL needed.
• Operational dashboards on live data. The "data is 6 hours stale" complaint shrinks to under a minute when the dashboard reads the Delta replica of the app's Postgres writes.
• Governed write paths from notebooks. Data scientists can write back to production-grade Postgres tables under the same Unity Catalog access controls that govern the rest of the lakehouse.

💡 Why this matters

1. It closes Databricks' biggest gap. Until Lakebase, the honest answer to "what about the transactional database?" was "use Aurora next to us." Native Lakehouse Sync replaces that hand-off with a single managed surface.

2. It enables AI-native applications. Autonomous research agents, customer-support agents, code-review agents. They all need both transactional state and analytical history. Building on top of two separate systems is painful. One system with one governance model removes the pain.

3. It changes the buying decision. "OLTP plus OLAP under one governed platform, with automatic sync between them" is a very different pitch from "OLAP only, bring your own database."

The takeaway: the OLTP/OLAP split has been a fact of life for 30 years. Lakebase plus Native Lakehouse Sync is the most concrete major-vendor attempt to fold it under a single governed platform. The implications reach from how you build agent memory to how you negotiate your next data-warehouse contract.

OpenTelemetry comes to Unity Catalog. Governed traces, no SaaS log tax.

If you've tried to monitor a production AI agent or LLM application, you've hit the same bottleneck. Observability is expensive and a security risk at the same time.

Every time an agent executes a multi-step reasoning loop, calls a database tool, or queries an LLM, it generates a firehose of trace logs. Ingesting those logs into external tools like Dynatrace or Datadog racks up a steep bill. Worse, those traces contain sensitive data (proprietary code, schema layouts, customer PII) that compliance teams hate seeing sent outside the company firewall.

The pieces started landing on April 9, 2026, when Databricks introduced serverless OpenTelemetry (OTel) tracing directly into Unity Catalog via its new Zerobus Ingest engine, now in Beta. A follow-up post on May 22 tied the pieces together: MLflow Tracing, Databricks Apps, and Model Serving all routing OTel data into governed Delta tables.

By implementing standard OTLP gRPC collector services natively on serverless compute, Databricks lets any OTel-compliant SDK push traces, metrics, and logs straight to the lakehouse in real time. No intermediate message queues, no broker fleet to maintain.

🔌 Where it integrates

Telemetry lands directly into pre-created Delta tables and views under a configurable prefix in your chosen catalog (spans, logs, metrics, plus annotations and unified trace views). It's integrated across the main Databricks surfaces:

• Databricks Apps 📱 Beta
  Apps built on Flask, Streamlit, or Gradio run with native OTel sidecars that capture system logs and web traffic automatically. (Learn more)
• MLflow Tracing 🧪 Beta
  Records multi-turn agent conversations, token usage, and intermediate tool executions natively in OTel format inside UC. (Docs)
• Model Serving 🧠 Available
  Captures inference payloads, confidence scores, and request latency into Unity Catalog Delta tables automatically. (Docs)
• Custom SDKs and Collectors ⚙️ Beta
  Configure any external OTLP/gRPC client to authenticate via OAuth and stream traces directly to the workspace OTel endpoint. (Learn more)
• Lakebase Postgres 🐘
  Lakebase metrics and logs can also be exported via OpenTelemetry Collector to common observability backends (Grafana Cloud, New Relic, Datadog), or routed into Unity Catalog through the same path described above. (Docs)

💡 What it changes for teams

For organizations with established observability frameworks (e.g. streaming application metrics to Dynatrace), this opens up a useful hybrid path:

1. Real governance, query-time PII masking. Because trace tables live in Unity Catalog, standard security policies apply. If an agent outputs sensitive code or customer logs, security teams can use SQL-based column masking and row filters to automatically redact PII at the query layer.

2. Bypassing SaaS log taxes. Storing high-volume trace histories in SaaS APMs gets expensive fast. The Zerobus architecture lets you route high-volume raw developer traces directly to cheap object storage in the lakehouse, reserving APMs for aggregated system alerts.

🔄 The flywheel: Ingesting traces as Delta tables lets you build custom SQL dashboards to monitor agent costs, or run MLflow evaluations on real-world request history to check for hallucination trends. Without exporting any data.

The takeaway: observability isn't a separate infrastructure silo anymore. By storing traces directly in Unity Catalog, Databricks makes telemetry queryable, secure, and cost-effective.

🤖 Stay ahead of the Databricks curve

Ask the Brickster Assistant about anything in the archive. Cited answers, grounded in the archive.