Unity Catalog

dbt users on Databricks: do your snapshots handle files that arrive out of order?

If you build SCD2 history with dbt snapshot on top of batch files (daily exports, CDC drops, API extracts), there's a failure mode worth knowing about. Snapshots apply data in the order it arrives. So if Tuesday's file lands after Wednesday's: - with strategy ='check', the late (older) value becomes the "current" row - with strategy ='timestamp', the change is skipped and that period just disappears from history Every run stays green and every test passes. Nothing tells you. We found out when someone asked why an employee's promotion showed up as a demotion. We ended up building dbt-iterative-snapshot, a custom materialization for Databricks. You add sequence_by='batch_date', and on each run it: 1. checks which batches it has already processed (it keeps an audit table) 2. if a new batch is older than one already processed, rolls back the snapshot from that point 3. replays the batches in order, one MERGE per batch It also gives you changed_attributes (which columns changed), per-batch row counts, and an error log for duplicate keys instead of failing the run. dbt Hub: https://hub.getdbt.com/mercedes-benz/dbt_iterative_snapshot/latest/ GitHub: https://github.com/mercedes-benz/dbt-iterative-snapshot It's dbt-databricks only for now (Delta MERGE, Unity Catalog). PRs for other adapters are welcome. I'm curious how others handle this. Do you rebuild SCD2 from raw with window functions, loop dbt snapshot over dates, or just hope files arrive in order? Happy to answer questions, and honest feedback on edge cases is very welcome. Read more: https://reliabledataengineering.com/posts/article_dbt_iterative_snapshot/ submitted by /u/Alone-Security7044 [link] [comments]

Genie Code tasks in jobs are GA

A job task runs your prompt autonomously, reads upstream outputs, calls tools, and returns a link to the resulting Genie Code chat. Pair with performance insights on serverless jobs to auto-open Genie Code for query tuning. Scheduling agents on Databricks has never been easier. If you have Genie Code enabled for skills, MCP servers, and of course a well constructed Unity Catalog this is a game changer to use in your every day worklfows. Oh and yeah you can create in a Asset Bundle https://preview.redd.it/zbrtlo0w0fyh1.png?width=2044&format=png&auto=webp&s=a4b5d43de61e2794238815ad6c16ab9bc228ebcf https://preview.redd.it/m18u7o0w0fyh1.png?width=1894&format=png&auto=webp&s=76508191b06e7f3eddee29731365bf56038a36a4 submitted by /u/Wise_Ear_4064 [link] [comments]

Unity Catalog external location access error after metastore attachment

My client wants to maintain Unity Catalog object definitions in a database-project-style folder structure and deploy any new or modified objects through Databricks Asset Bundles (DAB). Is this approach supported using DAB deployments through GitHub Actions?

submitted by /u/Ok-Tomorrow1482 [link] [comments]

What broke when we move to Unity Catalog

We recently moved our workspace to Unity Catalog. It went okay, but a few things broke along the way: -Old table names stopped working. Our code used two-part names like schema.table. We had to update everything to the three-part format catalog.schema.table. -Some jobs failed on shared clusters. A few of our older Python jobs didn’t work in shared access mode. We switched those jobs to single-user clusters. -People lost access to data. Permissions didn’t carry over the way we expected. We had to set up groups and grant access again. Lesson learned: test a few jobs first before moving everything. Has anyone else gone through this? What broke for you, and how did you fix it? submitted by /u/No_Ambition8323 [link] [comments]

Anyone playing around with Databricks Apps yet? Thoughts & initial impressions

Hey everyone, I’ve been testing out the new Databricks Apps feature recently, and I wanted to see how others in the community are using it. Historically, if we wanted to spin up a Streamlit app or a custom dashboard connected to Databricks, we had to deal with separate hosting (App Service, EC2, etc.), network security groups, and handling OAuth tokens separately. With Databricks Apps, running a Streamlit/Dash app directly inside the workspace using Unity Catalog for permissions has been a game-changer for internal tooling. That said, I'm curious: Are you using it for production workloads or just quick internal prototypes? How are you handling CI/CD deployments for these apps? Have you run into any limitations with serverless compute or scaling yet? submitted by /u/SalamanderOpen859 [link] [comments]

Beyond the Dashboard: Why AI Data Assistants Fail Without Unity Catalog Governance

For years, the promise of self-service BI has been a moving target. We gave business users drag-and-drop tools, semantic layers, and natural language query bars. Yet, time and again, we hit the same wall: trust. When a business user asks an AI tool, "What was our fleet downtime last quarter?" , the AI is only as good as the underlying data fabric. If your data is trapped in fragmented silos, uncatalogued, and un-governed, conversational AI quickly becomes an expensive generator of confident misinformation. This is where the paradigm is shifting. With the introduction of tools like Databricks Genie , we are moving past rigid dashboards into the era of conversational analytics. But as Solution Architects, our job isn't just to look at the UI layer—we have to look at the spine of the architecture. Here are three architectural realities I’ve learned while implementing modern Lakehouse patterns: 1. AI Without Governance is Just Automated Chaos You cannot point an LLM at a raw storage bucket and expect accurate insights. Genie relies entirely on the metadata, table descriptions, and foreign key relationships defined in Unity Catalog . If your governance layer is messy, your AI's answers will be messy. Unity Catalog isn’t just a security tool anymore; it is the semantic foundation for enterprise AI. 2. The Shift from "Building Pipelines" to "Curating Context" Tools like Lakeflow Connect and automated ingestion frameworks are taking over the heavy lifting of ETL. As data engineers and architects, our value is no longer measured by how many custom Spark scripts we write to move data. Our value is measured by how well we curate the Gold layer —structuring tables, defining clear column descriptions, and setting up quality expectations so that tools like Genie can actually understand the domain context. 3. Real-Time Trust Beats Real-Time Speed In heavy industries—like managing IoT telemetry for gas turbines—getting data in 5 seconds doesn't matter if the data quality is flawed. Integrating Delta Live Tables (DLT) Expectations as a quality firewall before data hits the consumption layer ensures that both human analysts and AI assistants are querying verified, pristine data. The Takeaway Conversational AI and natural language agents like Genie are the future of data consumption. But they are built on a simple architectural truth: Garbage in, hallucination out. If we want business users to trust AI-driven insights, we have to start by modernizing our foundations with robust governance, automated pipelines, and strict data quality gates. Curious to hear from other architects out there: Have you started deploying conversational AI tools to business users yet? What are the biggest governance or data trust hurdles you've had to overcome? submitted by /u/SalamanderOpen859 [link] [comments]

Databricks Unity Catalog Foreign Catalog – Timezone Mismatch with AWS RDS PostgreSQL Affecting Query

A full month-end finance app built on Databricks Apps

Below are the things covered in the video: 1. Overview: drag the month slider and every figure on the page moves with it, including a cash flow diagram from sales through to profit 2. P&L: the statement drawn as a ladder from revenue down to profit, plus a bridge showing why we beat or missed budget 3. Cafes: ranking, a map of the Front Range, a margin heatmap by month, and how fast each new cafe paid back its fit-out 4. Menu: which items earn their place and which are just popular 5. Cash: a 13-week forecast, overdue invoices by age, and the supplier payment calendar 6. Close: the month-end checklist as a board, with how long each close took historically 7. What-if: move price, volume, wage and ingredient levers and watch profit change 8. A one-page monthly report that prints to A4 Under the hood: seven Delta tables in Unity Catalog, read through a SQL warehouse, with the app running as a Databricks App in Python. All the finance logic (consolidation, budget variances, the close tracker, the cash forecast) sits in one file that a finance analyst could read. submitted by /u/oneaffidavit1 [link] [comments]

Genie App Builder

I'm building an app using the Databricks Genie app builder (micro app), and I'm trying to figure out whether it's possible to grant users access to Unity Catalog (UC) tables directly from within the app. But looking at the API scopes, I guess its not possible yet. https://preview.redd.it/82hz0mr358xh1.png?width=797&format=png&auto=webp&s=f5c476db84cbfdb9ac4ea826c91857c62a5830d5 submitted by /u/9gg6 [link] [comments]

Increase Unity Catalog tables-per-schema quota

Had a call with a Databricks PM this week and found out about all the cool things!

Spent 30 minutes on a product discovery call with u/saad-the-engineer from Databricks on Tuesday. Supposed to be about Unity Catalog triggers for jobs. We wandered. At one point I was explaining how I run bronze to silver on most client builds. It’s table-driven, with three Delta tables holding the config: one for table settings, one for column mappings (bronze names to business-friendly names), and one for DQ expectations that get loaded into the declarative pipeline. Genie Code actually picks up the pattern pretty well since it all lives in UC. Saad’s response was basically “cool, have you looked at sdp-meta?” I had not. It’s the Labs metadata framework for Spark Declarative Pipelines. You define bronze and silver in a JSON or YAML onboarding file and one generic pipeline builds the whole graph at runtime. Pretty much what I built by hand. Then he showed me pyDABs (aka Python for DABs). If you’re tired of hand-writing every task in YAML, you can loop through notebooks in a path and generate tasks in code. Mutators let you apply things like notifications or tags across every job in one shot. Every time I brought up something I’d hacked together, he had a “we’ve got that too.” I mentioned DQX and he said it’s one of those Labs projects that works so well they’re in no hurry to touch it. I also got a preview of some cool things that are coming (Stay tuned!) Best conversation of my week, and it was only Tuesday. For those of you running metadata-driven pipelines: did you adopt sdp-meta or roll your own? And what made you pick one over the other? Side note, Scott Haines and I are doing a live session on the Iceberg updates at the end of the month. Let me know if you are interested and I'll share the link! submitted by /u/OkImprovement7010 [link] [comments]

What’s new in Databricks - September 2026?

Databricks shipped the following major GA launches in September 2026: 👉 Spark Declarative Pipelines (SDP): Lakeflow Spark Declarative Pipelines now feature structured maintenance windows for continuous runs, zero-reprocessing migrations (import_checkpoint), multi-source append flows, and explicit cascade deletion (cascade: true) for Unity Catalog tables. 👉Governance and ABAC Security: Metastore-level Attribute-Based Access Control policies and explicit DENY rules are now GA in Unity Catalog, automatically enforcing data protection and access rules across all present and future catalogs. 👉Connectors and Data Applications: Lakeflow Connectors reached GA for Anaplan, Anysphere, and Celigo, while Genie Ones MCP server was officially made generally available for AI assistant workflows. Full article 👉 https://community.databricks.com/t5/product-platform-updates/what-s-new-in-databricks-september-2026/ba-p/173896 submitted by /u/Youssef_Mrini [link] [comments]

Governed Tag Propagation (Beta) in Unity Catalog - How It Works and What to Consider

Cross-subscription/workspace data access pattern for PII data in Azure Databricks?

We are setting up Azure Databricks for a client with two fully isolated workspaces ( Dev and Prod ). They have completely separate: Workspace IDs Azure Resource Groups Azure Subscriptions The Problem: PII data from a source system is currently landing in an Azure storage location that only the Prod workspace has access to. Because of this, we can read the data in Prod , but not in Dev . We now need to grant a specific group of analysts read-only access to this source PII data directly from the Dev workspace. Questions: What is the standard/cleanest pattern to achieve this? 1) Cross-subscription Azure Managed Identity / Storage Credential grants? 2) Unity Catalog cross-workspace binding / Delta Sharing? Any advice, reference architecture, or lessons learned would be greatly appreciated! submitted by /u/Miserable-Neat4246 [link] [comments]

Databricks Unity Catalog now supports automatic Governed Tag Propagation

Hi, A new cool feature called Tag Propagation appeared in Unity Catalog. It automaticaly propagates tags to downstream tables and views based on UC lineage. Tag propagation - Azure Databricks | Microsoft Learn I've put together a short article explaining how to enable the feature and walking through the small demo step by step. Feel free to check it out: Databricks Unity Catalog: Governed Tags Can Now Propagate Automatically! | by szymon_dybczak | Databricks Community Articles | Oct, 2026 | Medium submitted by /u/szymon_dybczak [link] [comments]

Have you ever changed a Delta table schema and only discovered the impact later?

When a column gets renamed, removed or changes its data type, the table itself might be fine, but downstream jobs can start failing or producing unexpected results. How do you guys find all the dependent jobs and tables before making schema changes.. Is Unity Catalog lineage enough in real projects or do you use other checks too? submitted by /u/Bhanuprakash_1947 [link] [comments]

Announcement | Your data, your storage, your rules: A 2026 guide to Unity Catalog managed tables

Databricks Unity Catalog: The Safe Way to Govern AI

Lakebase CDF to Unity Catalog vs firewall on storage

I have a use case where Lakebase CDF (transparent syncing of tables from PostgreSQL to Unity Catalog) would be super convenient. However I found that currently the CDF doesn't support storage accounts behind a firewall, accessible through a private endpoint, etc. They have to be publicly accessible, which is a no-no for most corporate data. Can anyone from DataBricks share whether it's on your roadmap to add support for this and when? submitted by /u/djfeelx [link] [comments]

Using Lakehouse Federation to bypass Bronze ingestion for near real-time pipelines; feasibility and security questions

(Used AI to refine my question better and structure the question) TL;DR: Trying to reduce our pipeline runtime to under 3 minutes by bypassing Bronze ingestion and using Lakehouse Federation. Looking for: ​A setup guide/checklist I can hand to our IT Admins to establish the initial connection. ​Advice on a security flaw: How do I prevent users from bypassing our custom Unity Catalog RLS/PII controls simply by materializing (CTAS) data out of the federated catalog? Hey everyone, I’m working through a performance optimization challenge in Azure Databricks and wanted to bounce my intended approach and a few assumptions off the community. The Context -> The Workflow: 10–15 users trigger a Databricks job multiple times a day, sometimes concurrently. -> The Process: We pull data from a source database, apply transformations, and write out about 8 tables into Unity Catalog using a Medallion architecture. The largest table is 20M rows, which we ingest incrementally with a 2-day lookback. -> The Bottleneck: The pipeline currently takes 4–5 minutes (optimized down from 9). Our target is under 3 minutes. Since the source-to-Bronze ingestion alone takes 1.5 to 2 minutes, I am looking into Lakehouse Federation. If we bypass the initial ingestion and query the frequently refreshed datasets directly from a federated catalog, we should comfortably hit that sub-3-minute target. Before I commit to this architecture, I’m hoping to validate a couple of things since I don't have an admin playground environment to test this out myself: Division of Setup / Admin Privileges Because I don't have Account Admin access, I want to understand exactly what I need to ask IT to configure versus what I can do myself. Is there a clear guide or documentation that outlines the standard division of setup? For example, do admins just need to create the CONNECTION object and grant me CREATE FOREIGN CATALOG on the metastore, leaving me to map the schemas? I just want to know the exact steps so I can submit a precise IT ticket. RLS & PII via Control Tables on Federated Data I am independently working on refactoring our data security posture. Currently, we manage PII and Row-Level Security (RLS) using custom "control" tables in Unity Catalog. This works perfectly for our native Databricks tables. My assumption with Federation is that while I can apply these same functions to the foreign tables, anyone with read access to the federated catalog could simply run a CREATE TABLE AS SELECT... (materialize the data), bypassing the dynamic filters entirely and exposing the raw PII. -> Is my understanding of this design flaw accurate? -> Has anyone successfully implemented control-table-based RLS on federated catalogs, or is there a better, more standard way to handle this in a federated setup? Appreciate any insights or reality checks from anyone who has implemented this in production! submitted by /u/aRTy_k1d [link] [comments]

Federation vs. ETL: Why I’m starting to hate moving data just to look at it.

I’ve been working heavily with Unity Catalog’s Query Federation lately, and it’s forcing me to rethink my "ELT everything" default. For a long time, the standard architecture was: Copy everything from operational DBs (SQL Server, Oracle, etc.) into the Lakehouse so we can join it. But the "ETL Tax"—the storage costs, the stale data, and the pipeline maintenance—is starting to feel like overkill for ad-hoc reporting. Why Query Federation is winning me over: Pushdown is clean: With Query Federation, Databricks pushes the SQL via JDBC to the source. The remote DB does the compute, and I just get the result back. It saves me from creating yet another massive ingestion job that I have to monitor. The "Governance" Win: This is the biggest selling point for me. I can manage security in Unity Catalog (UC) and it "wraps" the remote data. My users get the same catalog.schema.table experience, and I don’t have to manage 50 different connection strings for them. Automatic Lineage: Finally, I can see where data in a report actually came from, even if it originated in a legacy SQL Server. My Rule of Thumb (The "Architecture Bridge"): Use Federation for: Ad-hoc reporting, dashboards needing "live" data, and rapid data exploration. Stick to ETL for: Multi-terabyte aggregations, complex transformations (SCD Type 2, deduplication), and historical reporting. I’m not going to stress out a production operational DB with a massive GROUP BY at 10 AM. submitted by /u/SalamanderOpen859 [link] [comments]

Stop migrating data to Databricks: Are you using Lakehouse Federation yet?

I’ve been diving deep into Unity Catalog lately, and the "Lakehouse Federation" feature is genuinely a game-changer for reducing ETL bloat. For a long time, the default answer to "how do I analyze this external data?" was "copy it into the lake." We’re finally moving away from that. Here is the breakdown for anyone trying to cut down on unnecessary data movement: The two ways to federate: Catalog Federation (The "Data-Centric" approach): How it works: Connects Databricks to external metadata (AWS Glue, Hive, Snowflake Iceberg). The Magic: Databricks compute does the heavy lifting (reading raw files in S3/ADLS). It’s fast, low-cost, and creates a "live" bridge to your storage. Best for: Incremental migrations and long-term hybrid storage. Query Federation (The "Database-Centric" approach): How it works: Pushes SQL via JDBC to remote databases (Postgres, MySQL, SQL Server, Synapse, etc.). The Magic: The remote DB does the compute. You get the result set back without touching the raw underlying tables. Best for: Ad-hoc reporting against operational/production databases where you don't want to break their performance with heavy ETL loads. Why it’s actually useful: Centralized Security: Unity Catalog acts as the single security guard. You don't need to manage separate credentials for your users on the remote DB. Automatic Lineage: This is the biggest win for me—Databricks automatically maps the lineage from the remote system to your report. No more "mystery data." Unified Syntax: You query everything using catalog.schema.table regardless of whether it’s sitting in Databricks, Snowflake, or an on-prem SQL Server. submitted by /u/SalamanderOpen859 [link] [comments]

Building an Expense Management App with the Genie App Builder Leveraging Lakebase + AI for Image Scanning

I previously did not consider Databricks a great place to build apps on with AI. With the release of Genie App Builder (still in beta), that has changed, and now I see it as a serious player in the AI app building space. Before I start, quick disclaimer: I am part of the Databricks MVPs, work for a consulting company that is a Databricks partner, etc. That said, this review is fully based on my experience, my own words, my own testing, etc. Always try things out yourself! Things I appreciated: +By far, this feels like the most polished Databricks beta that has been released. If the goal is to make this easy for business users to develop applications easily and securely, this feels like it is already there. Granted, I have not tested with multi-users yet, but it feels very polished already. +Native integration with Databricks features like Lakebase. +Even though it has native integrations to Databricks features, it didn't force these features in building what I did when it was otherwise unnecessary. For example, the dashboard it built as part of the application was not an AI/BI Dashboard. I would have expected it do that, but it didn't, and rightfully so, as it would not have been truly the best thing for this use case, so truly a customer first thing here. +Respects governances/access controls, meaning I can develop an application freely in the sandbox environment, but if I want to deploy it, organizational safeguards, Unity Catalog, and other controls built on Databricks apply. Things that could be better: -With the goal of empowering citizen devs (or i should just say, business users that are technical enough), there needs to be an easier way to send request for access/capability etc to admins. Right now, the builder tells you through the chat, but that is not as intuitive as having a clear UI-embedded to-do things before deploying etc. One step further would be that the developer can notify the administrator of this need. -I found that when I developed an application that had to use a sandbox only capability when access was otherwise unavailable (see above), just hung when I tried to reopen the dev experience for it. I had to troubleshoot through the chat interface as to why that was happening to figure this out. Additional Note : Currently free at the moment, but pricing is not yet available since we are in beta. Will be interesting to see how the overall value of this capabilities plays out factoring the ease & governance vs compute & token costs. Closing Thoughts I think this is on track to be one of Databricks' best released feature, and a serious entry in the space that is otherwise feel with undifferentiated app building solutions. Is it creating a new category? No. But it feels like an Apple-level product, where being the best at executing is sometimes better than being the first to do something. And I believe it will pay off as businesses look more carefully as to where they spend their hard-earned money on. From a "data + AI platform vendor" perspective, this is currently the best offering of its kind. From an "app building vendor" perspective, there are still solid players out there, but for enterprise focused apps, in the top 3 of options I would consider, and for sure the first one I would try if my stack was heavily Databricks focused. submitted by /u/JosueBogran [link] [comments]

Monitoring a custom Agent (Responses) serving endpoint with an experiment on the Unity Catalog

CUSTOMER STORY | CARIAD unifies global automotive data with OpenSharing and Unity Catalog

Is Delta Sharing the end of external ETL? My notes on Open vs D2D.

Just spent some time digging into the Unity Catalog and Delta Sharing documentation. Here’s the TL;DR on the two patterns for anyone looking to implement this: Open Sharing (The "Universal" Path): Audience: Recipients without Databricks (using Power BI, Pandas, Tableau). Mechanism: Presigned URLs + Bearer Tokens. Constraint: Tabular data only (tables/views). Security: Can be restricted via IP filtering. D2D (The "Native" Path): Audience: Both parties are on Unity Catalog-enabled Databricks. Assets: Tables, Views, Volumes, Notebooks, and Models. Pros: No token management; Zero egress costs if in the same region. Performance: DBR 16.2+ uses scoped credentials, making it basically as fast as a local table read. Observations: The "Zero-Copy" aspect is the real winner here. The audit trail in system tables alone covers about 95% of standard compliance needs (GDPR/SOC). The Coastal Community Bank case study (2 months to 2 days for onboarding) seems to be the gold standard for ROI here. Discussion: Is anyone actually using the Open Sharing side with external vendors in production? How are you handling the token rotation/security for non-Databricks clients? Or are you finding that most of your partners are already on the Databricks ecosystem? submitted by /u/SalamanderOpen859 [link] [comments]

Databricks ABAC Now Follows Your Data: External Engines and Views

Databricks ABAC row filters and column masks now reach external Apache Spark (GA) and Unity Catalog views (Beta) submitted by /u/Lenkz [link] [comments]

What's new in Databricks AI/BI (Oct 2026): multi-fact modeling, Row Zero spreadsheets, Excel Add-in GA

If you work in BI on Databricks, this monthly roundup might be useful. This month's edition covers: Relationships now support multi-table schemas and measures across fact tables with different granularities (via shared dimensions) "Ignore filters" on dashboard widgets, handy for calendar hierarchies Row Zero acquisition: spreadsheets on billions of rows with a live connection to governed data Excel Add-in is GA: browse Unity Catalog, pull metric views in as native pivot tables, and ask Genie One from inside Excel Full post: https://aibilakehouse.substack.com/p/beyond-the-star-schema-spreadsheets Disclosure: all three authors work at Databricks. Happy to answer questions or hear what you'd like covered next. submitted by /u/Good_Robin [link] [comments]