Delta Sharing

How do you manage, in a Databricks Asset Bundles-based CI/CD, the permissions and access resources that a bundle cannot declare?

We have a DABs-based CI/CD in a monorepo and want to split it: deploy the DABs from the data-code repo and keep the Terraform infrastructure in a separate repo. The original idea was that Terraform would change at a different pace and its deployments would be manual, with no automated dependency on DABs deploys. However, at work they’re now saying this infrastructure part should be tied to the DABs deployment and not be manual, and I’m unsure where it should live and how to automate it. We mainly run into four cases: Cross-boundary grants: our pipeline needs to read a table owned by another team, so we need to grant SELECT on that table/schema to our service principal. Postgres grants: a job needs to connect to a Lakebase database and requires a GRANT on a Postgres schema, table, or role. Share contents: we need to add or remove tables and views from a Delta Sharing share consumed by an external consumer. Notification destinations: a job or pipeline needs to send alerts to a notification destination (email, Slack, webhook) that the bundle cannot declare. Today, all of these live in Terraform outside the bundle, so every time the data code needs one of these changes, we have to coordinate a separate MR and deployment in addition to the bundle deploy. Where do you put these in your pipeline: inside the bundle, in a separate Terraform stack applied after the bundle deploy, in the platform repo, or another way? What issues have you run into with each approach? Tx so much submitted by /u/Appropriate_Tip_9580 [link] [comments]

Cross-subscription/workspace data access pattern for PII data in Azure Databricks?

We are setting up Azure Databricks for a client with two fully isolated workspaces ( Dev and Prod ). They have completely separate: Workspace IDs Azure Resource Groups Azure Subscriptions The Problem: PII data from a source system is currently landing in an Azure storage location that only the Prod workspace has access to. Because of this, we can read the data in Prod , but not in Dev . We now need to grant a specific group of analysts read-only access to this source PII data directly from the Dev workspace. Questions: What is the standard/cleanest pattern to achieve this? 1) Cross-subscription Azure Managed Identity / Storage Credential grants? 2) Unity Catalog cross-workspace binding / Delta Sharing? Any advice, reference architecture, or lessons learned would be greatly appreciated! submitted by /u/Miserable-Neat4246 [link] [comments]

CUSTOMER STORY | CARIAD unifies global automotive data with OpenSharing and Unity Catalog

Is Delta Sharing the end of external ETL? My notes on Open vs D2D.

Just spent some time digging into the Unity Catalog and Delta Sharing documentation. Here’s the TL;DR on the two patterns for anyone looking to implement this: Open Sharing (The "Universal" Path): Audience: Recipients without Databricks (using Power BI, Pandas, Tableau). Mechanism: Presigned URLs + Bearer Tokens. Constraint: Tabular data only (tables/views). Security: Can be restricted via IP filtering. D2D (The "Native" Path): Audience: Both parties are on Unity Catalog-enabled Databricks. Assets: Tables, Views, Volumes, Notebooks, and Models. Pros: No token management; Zero egress costs if in the same region. Performance: DBR 16.2+ uses scoped credentials, making it basically as fast as a local table read. Observations: The "Zero-Copy" aspect is the real winner here. The audit trail in system tables alone covers about 95% of standard compliance needs (GDPR/SOC). The Coastal Community Bank case study (2 months to 2 days for onboarding) seems to be the gold standard for ROI here. Discussion: Is anyone actually using the Open Sharing side with external vendors in production? How are you handling the token rotation/security for non-Databricks clients? Or are you finding that most of your partners are already on the Databricks ecosystem? submitted by /u/SalamanderOpen859 [link] [comments]

Time to Swap the Cookies for Jetfuel - New Dataset & New Databricks Genie Tutorial

Most of you know samples.bakehouse . Great for a first query. Perfect for a quick demo. But after years of cookie sales, it's a little overbaked. Time to swap the cookies for jet fuel. ✈️ Together with the OpenSky Network , I brought a full day of global air traffic to Databricks Marketplace: 696 million real ADS-B position reports, messy just like real life. Myself, I used Genie for the whole journey: EDA, data exploration, a Apache Spark Declarative Pipeline, and a Lakeflow Job. Then I went a step further and read the same Marketplace data with open-source tools only, using OpenSharing and pandas. The result is this hands-on tutorial: Marketplace + Unity Catalog: get the data as a governed table Genie Agents: find anomalies in plain English Genie Agents: explore and visualize with maps and charts Genie Code: a Spark Declarative Pipeline, bronze to gold, with data quality rules Genie Code: a Lakeflow Job with schedule, retries, and email alerts Databricks Apps: your coding agent, governed by Unity Catalog OpenSharing: the open-source client in VS Code with pandas Everything runs on Databricks Free Edition (free, no credit card). 📖 Tutorial: Databricks Genie for Data Engineers and Data Scientists 💻 GitHub: databricks/tmm/DSDE-Genie-Tutorial 🛫 Dataset: OpenSky Network full-day dataset on Marketplace What's the first thing you'd query in a day of global air traffic? P.S. For the record, we still love bakehouse! 🍪❤️ [Disclaimer: I'm one of the two people who baked it.] submitted by /u/CompetitiveBet8978 [link] [comments]

Multi-region model serving on Databricks with OpenSharing

Scaling Delta Sharing across Non-Databricks Recipients: Handling Credential Rotation and Lifecycle

Using Delta Sharing to hold ISV entitlement data outside the customer's metastore — reasonable fit?

Automatic change data feed is now generally available!

With automatic CDF, Databricks computes row-level changes at read time using row tracking, rather than materializing those changes during every write. Use change data feed on Databricks | Databricks on AWS Why does that mattre? - Better write performance for MERGE INTO and UPDATE workloads - No need to enable CDF individually on every eligible table - Lower storage overhead compared with legacy CDF - The same familiar APIs still work: table_changes() and readChangeFeed - Works with batch processing, Structured Streaming, and Databricks-to-Databricks Delta Sharing For Delta Lake, the main requirements include: • Databricks Runtime 19 LTS+ • A managed table or external table in Delta Lake format with row tracking enabled And if you’re already using legacy CDF, migration is really simple.Once the table meets the requirements, disable legacy CDF: https://preview.redd.it/0dddizd442oh1.png?width=710&format=png&auto=webp&s=fddc536d6aeca423b2e418f4856ae06843393749 submitted by /u/szymon_dybczak [link] [comments]

How to do cross-cloud sharing with OpenSharing with added security (demo)

Hey folks! In this demo, Akram from Databricks' product team shares how you can leverage SecureConnect to better your security posture when doing cross-cloud sharing on OpenSharing! If you have no idea what OpenSharing is, how it applies to you, or how we got from Delta Sharing to OpenSharing, also encourage you to watch this video: https://youtu.be/0mfuNybtmdE Hope you find this helpful! submitted by /u/JosueBogran [link] [comments]

Databricks and OpenSharing questions

This my first time looking into use OpenSharing to share data externally and would to get some idea on if I am on the right path. My current idea is as follow: ETL my datasets using CDF from source to a catalog_curated, multiple tables In a catalog_shared, create views for each that apply filters (rows or joins) and these are what will be published and shared with external. I understand when sharing a view via OpenSharing, pushdown will not work and the whole views will get materialized temporarily when query, which I know may be an issue when external uses one of our watermark column. However, our dataset is not that big and access to the data is not going to be frequent, sharing a persisted version of the views will solve that, but is the added complexity worth it? I also could add the filters as part of the ETL config, but I am thinking it might be too complex for my current need. And when we need to expand the filters, all we have to do is change the views. I think the goal for me right now is simplicity and maintainability over complex ETL work, unless there is a reason to do so. As stated, this is my first time working with OpenSharing/DeltaSharing. Any comments, suggestions, or best practices are greatly appreciated. Thanks submitted by /u/Hungry-Tax6101 [link] [comments]

Who pays cross-cloud egress for Delta Sharing from GCP to Azure?

Databricks OpenSharing: How Global Distribution Reduces Egress Costs & Improves Performance (Demo with Databricks Product Lead)

If you are a data provider on Databricks, OpenSharing (formerly Delta Sharing) is a fantastic tool for distributing data to your customers. During my time at a former job, there was nothing better than to work with customers that were able to consume data from Delta Sharing. On at least two occasions though, we received unpleasant surprises: egress fees from our cloud providers. Those egress fees were most often times avoidable. For example, we would make sure that our customers and our company agreed on the cloud/region combination we would serve from/to, allowing us to build one-off pipelines, etc. But, that required manual work, created staleness issues, etc. While in our situation, we could sometimes overcome the same cloud vendor but different region situation, distributing to a region we just didn't have anything setup for OR a completely different cloud was a no-go. All that to say, Global Distribution aims to solve this and more. No need to have an account with another cloud vendor to serve cross-cloud. No need to have have data distributed to multiple regions within your AWS/Azure/GCP. No need to build/maintain pipelines. Just use Global Distribution. Check out the video and see Akram Chetibi who leads OpenSharing explain/demo it, and love to hear your thoughts. submitted by /u/JosueBogran [link] [comments]

Got error using Opensharing Get Table API to foreign iceberg table

Databricks OpenSharing Explained: Delta Sharing for the AI Era (w/ Databricks Product Leader)

Hey everyone! In this video, a Databricks product leader joins me to explain what OpenSharing, and what happened to Delta Sharing! Delta Sharing is one of the features I used often at a previous role to provide data to multiple customers, so it will be interesting to see how the journey with the expanded focus (data + AI sharing) goes. submitted by /u/JosueBogran [link] [comments]

Announcement | OpenSharing, Marketplace Apps, and SecureConnect for the AI Era

Databricks Hands Delta Sharing to the Linux Foundation and Levels It Up

Got error when access delta sharing table with iceberg endpoint