Row Filters

Fail-closed ABAC Policies and early Error detection

Hello everybody, since we are gonna use ABAC Policies extensively in the near Future I was wondering If there is a good/Standard method of checking all tables for conflictinf Policies? Let's slay WE have a catalog-wide RowFilter Policy in place and implement a Schema-level row Filter Policy inside this catalog. Is there an easy way to find all tables this causes conflicting row-filters without going through alles tables as each relevant Principal? submitted by /u/Coco_Games [link] [comments]

Using Lakehouse Federation to bypass Bronze ingestion for near real-time pipelines; feasibility and security questions

(Used AI to refine my question better and structure the question) TL;DR: Trying to reduce our pipeline runtime to under 3 minutes by bypassing Bronze ingestion and using Lakehouse Federation. Looking for: ​A setup guide/checklist I can hand to our IT Admins to establish the initial connection. ​Advice on a security flaw: How do I prevent users from bypassing our custom Unity Catalog RLS/PII controls simply by materializing (CTAS) data out of the federated catalog? Hey everyone, I’m working through a performance optimization challenge in Azure Databricks and wanted to bounce my intended approach and a few assumptions off the community. The Context -> The Workflow: 10–15 users trigger a Databricks job multiple times a day, sometimes concurrently. -> The Process: We pull data from a source database, apply transformations, and write out about 8 tables into Unity Catalog using a Medallion architecture. The largest table is 20M rows, which we ingest incrementally with a 2-day lookback. -> The Bottleneck: The pipeline currently takes 4–5 minutes (optimized down from 9). Our target is under 3 minutes. Since the source-to-Bronze ingestion alone takes 1.5 to 2 minutes, I am looking into Lakehouse Federation. If we bypass the initial ingestion and query the frequently refreshed datasets directly from a federated catalog, we should comfortably hit that sub-3-minute target. Before I commit to this architecture, I’m hoping to validate a couple of things since I don't have an admin playground environment to test this out myself: Division of Setup / Admin Privileges Because I don't have Account Admin access, I want to understand exactly what I need to ask IT to configure versus what I can do myself. Is there a clear guide or documentation that outlines the standard division of setup? For example, do admins just need to create the CONNECTION object and grant me CREATE FOREIGN CATALOG on the metastore, leaving me to map the schemas? I just want to know the exact steps so I can submit a precise IT ticket. RLS & PII via Control Tables on Federated Data I am independently working on refactoring our data security posture. Currently, we manage PII and Row-Level Security (RLS) using custom "control" tables in Unity Catalog. This works perfectly for our native Databricks tables. My assumption with Federation is that while I can apply these same functions to the foreign tables, anyone with read access to the federated catalog could simply run a CREATE TABLE AS SELECT... (materialize the data), bypassing the dynamic filters entirely and exposing the raw PII. -> Is my understanding of this design flaw accurate? -> Has anyone successfully implemented control-table-based RLS on federated catalogs, or is there a better, more standard way to handle this in a federated setup? Appreciate any insights or reality checks from anyone who has implemented this in production! submitted by /u/aRTy_k1d [link] [comments]

Databricks ABAC Now Follows Your Data: External Engines and Views

Databricks ABAC row filters and column masks now reach external Apache Spark (GA) and Unity Catalog views (Beta) submitted by /u/Lenkz [link] [comments]

Build a Document Knowledge Base on Databricks

Locking the folder does not lock the answer. On Databricks you can grant the volume, the table, and the search index. None of those grants decides which page a person may read. The index does not apply the row filter. Someone who can query it gets whatever the question asks for.The label on each chunk has to be set by your own service. The person asking does not get to choose it. submitted by /u/Worth_Phase1358 [link] [comments]

ABAC on Views is now in Beta in Azure Databricks!

Until now, ABAC row filters and column masks were primarily applied to tables. With this update, the same policy model can now be enforced when sensitive data is exposed through views as well. September 2026 - Azure Databricks | Microsoft Learn For example, a view such as sales.customer_view can expose customer data while a policy masks sensitive columns or filters rows depending on the querying principal. Here's a simple example of usage: CREATE POLICY mask_pii APPLY TO all_account_users EXCEPT admins ON tables.tagged sensitivity=high USING COLUMN MASK mask_email(email) Notice one thing - because Unity Catalog considers views as tables, use ON TABLE view_name to target a specific view. There is no separate FOR or ON VIEW syntax. I’m really glad to finally see this feature. I’ve personally been waitng for it for quite some time, since our team relies heavily on views as the consumption layer ;) submitted by /u/szymon_dybczak [link] [comments]

Time-Travel on Tables with ABAC RLS/CM - Beta

Hey all! I’m a Product Manager at Databricks who just launched Beta support for time travel on tables with ABAC policies. You can now query historical data using VERSION AS OF or TIMESTAMP AS OF while keeping ABAC row-filters and column-masks enforced. The main design question we faced was how policies should interact with historical schemas. We chose a security-first model: Load the requested historical snapshot. Match it against the current schema. Validate the match. Renamed or dropped historical columns block the query to prevent data leak, while columns added later are excluded from the result. Apply the active ABAC policies to the historical data and return query results. Here's an example: How time-travel works on ABAC-protected tables This fail-closed behavior is especially important when a policy references a column that did not exist historically. Treating the missing column as null or assigning a default could change the row filter’s result and expose restricted rows. Databricks blocks the query instead. Why fail-closed behavior is critical I'm curious if the community has any feedback on the approach. Any input will be valuable as we step towards the GA milestone. For more information, please see our documentation . submitted by /u/alicks22 [link] [comments]

Row-level security for a RAG agent: what Unity Catalog enforces, and what you have to build

Ontology ranked the snippet. I wrote the gate I wanted before trusting a Genie space.

A commercial user asked what yield was last season. Two definitions live in the same lakehouse. One is plot-level, moisture-adjusted, borders out. The other is as-harvested, and a dashboard still uses it. Genie returned the second number. Not a hallucination. OntoRank did what it does: creator, usage, link to a certified asset, freshness. The loud dashboard won. Nobody had certified the measure the board pack is allowed to quote. That is the failure I care about. A green demo is not a release gate. What I actually require before that space faces a user who will paste the sentence into Slack: Golden questions written before the space exists. If you write them after you have tuned instructions, you are scoring an overfit prompt. Benchmarks are not instructions. Genie does not learn from the benchmark SQL. The number lives in a Metric View. A column comment and a knowledge-store formula are hints. Ontology will rank all three. Only the Metric View is a contract every surface compiles. Same KPI from MEASURE(), from the certified dashboard, and from Genie. If they disagree, the space stays off Slack. Permissions checked as the asker. Inside a Genie Agent, compute is the author’s warehouse and data access is the end user. That split does not automatically hold in Teams, Copilot on maker credentials, or a Databricks App on its default service principal. The row filter still runs. It runs for whoever showed up on the SQL. A written kill line. I use Databricks’ “above 80% before UAT” as the hold line, not the ship line. Ship is 90% and zero fails on a regulated definition. Any miss on that definition takes the space down. A person does that. Not a service principal. Genie Code can draft the Metric View and the MERGE. ZeroOps can propose a fix and wait for Approve. Neither owns the definition. If the job stays green for eleven days because a key was slightly wrong, that is still your incident. I wrote the four pieces up so I would stop re-explaining this in reviews. Argue the 80 vs 90 line if you want. I will not move a regulated KPI to “looks fine in the demo.” submitted by /u/Worth_Phase1358 [link] [comments]

Row-level security when the filter column is not on your table

Managing Row-Level Security (RLS) vs. Views: Performance impacts at scale

Best practice to enforcing row-level security across multiple catalogs sharing same schema structure

Read this if you use Streaming Tables in Lakeflow Spark Declarative Pipelines

🚀 We’re excited to announce that Lakeflow Spark Declarative Pipelines (SDP) now supports creating “vanilla” (i.e., non STREAMING) MANAGED TABLES and writing to them via one or more append flows , using the new CREATE TABLE ... FLOW ( SQL ) and create_table() (Python) APIs . What is this Beta? This Beta allows creating a managed table that is populated by append flows: CREATE TABLE ... FLOW (SQL) / create_table() + @append_flow (Python) create a managed table written by one or more flows. Fan multiple sources into one table — declare several flows targeting the same managed table. Full table surface works: partitioning, liquid clustering, expectations, row filters, table properties, and private (pipeline-local) tables. import_checkpoint on append_flow , which migrates an existing Structured Streaming workload into a pipeline without reprocessing the source — the flow imports the query's existing checkpoint and resumes from the last committed offset with state intact. Example (Python): from pyspark import pipelines as dp dp.create_table("combined") dp.append_flow(target="combined") def from_a(): return spark.readStream.table("source_a") u/dp.append_flow(target="combined") def from_b(): return spark.readStream.table("source_b") Example (SQL): CREATE TABLE events PARTITIONED BY (bucket) FLOW INSERT BY NAME SELECT id, bucket FROM STREAM read_files('abfss://my_path', format => 'json'); Where do we need help? We are in Beta, so there might be some rough edges. Please take this for a spin and share your feedback here . What’s next? Managed Tables support for other flow types (AutoCDC, Replace Using, and Replace Where) is coming soon! Learn more CREATE TABLE ... FLOW (SQL reference) — https://docs.databricks.com/aws/en/ldp/developer/ldp-sql-ref-create-table-flow create_table (Python reference) — https://docs.databricks.com/aws/en/ldp/developer/ldp-python-ref-create-table import_checkpoint on append_flow — https://docs.databricks.com/aws/en/ldp/developer/ldp-python-ref-append-flow Questions, feedback, or help: comment below or share feedback in the form: https://forms.gle/7bGP5FYN7P1Z4WP27 submitted by /u/SlightImagination250 [link] [comments]

Ingest image in databricks for powerbi ? a poc and any idea welcome

Spent some time this week on a POC that started from a business constraint: about 3,000 photos to ingest every week, sensitive enough that we can't just drop a shareable link in a dashboard, and they need to end up in Power BI where people already work, with row-level security. That combination rules out the easy answer. No public links, no loose files in blob storage floating around outside governance. The images had to live inside Delta on Databricks so Unity Catalog could handle access control, and Power BI had to be able to render them directly from the table. What the simple poc below does: Reads the images with Spark's binaryFile source (recursive lookup, glob filter on *.jpg) to pull path, modification time, size and raw bytes into one DataFrame. Encodes the binary content as a base64 data URL, so the image itself lives inside the row instead of behind a link. Then the actual blocker: Power BI caps text fields at roughly 32,766 characters, and a real photo's base64 string blows straight past that. So each string gets split into ~32,000-character segments and exploded into multiple rows, each tagged with its index and total length. On the Power BI side, a single DAX measure puts it back together in the right order before rendering: ​ Image_concat = IF( HASONEVALUE(images_in_delta[image_name]), CONCATENATEX( images_in_delta, images_in_delta[segment], , images_in_delta[split_index] ) ) Not elegant, but it's what gets a full-resolution image through a hard platform limit without touching the sensitivity requirement. The PySpark side, stripped to what matters — reading the images and doing the chunking: from pyspark.sql import functions as F from pyspark.sql import DataFrame #READ ALL THE IMAGES images_df = spark.read.format("binaryFile") \ .option("recursiveFileLookup", "true") \ .option("pathGlobFilter", "*.jpg") \ .load("/Volumes/main/image_ingest/image_sample") def add_base64url_from_image_binary(df: DataFrame, max_len: int = 32000) -> DataFrame: df_with_b64 = df.select( "*", F.concat(F.lit("data:image/jpg;base64,"), F.base64(F.col("content"))).alias("base64url") ) df_with_split_info = df_with_b64.select( "*", F.ceil(F.length(F.col("base64url")) / F.lit(max_len)).cast("int").alias("num_segments"), F.length(F.col("base64url")).alias("total_length") ) df_split = ( df_with_split_info .withColumn( "split_index", F.explode(F.sequence(F.lit(0), F.col("num_segments") - 1)) ) .select( "*", F.substring( F.col("base64url"), F.col("split_index") * max_len + 1, F.least(F.lit(max_len), F.col("total_length") - F.col("split_index") * max_len) ).alias("segment") ) .drop("base64url", "content") ) return df_split def add_image_name(df : DataFrame) -> DataFrame : return df.withColumn("image_name", F.regexp_replace(F.col("path"),".*/([^/]+)$", "$1")) df_images = add_base64url_from_image_binary(images_df) df_images = add_image_name(df_images) df_images.write.mode("overwrite").format("delta") \ .option("mergeSchema", "true") \ .saveAsTable("main.image_ingest.images_in_delta") Nothing here is exotic engineering — the interesting part was realizing early that the constraint wasn't really "how do we store images in Delta," it was "how do we get a sensitive image through Power BI's text field limit without ever exposing it outside the governed table." Once that was clear, the chunking workaround fell out naturally. At 3,000 images a week this holds up. If volume goes up meaningfully, I'd want to revisit whether inlining every image is still the right call versus resolving binary content on demand. Curious if others have hit the same Power BI ceiling with sensitive image data and landed on something cleaner than manual chunking. Have you any other idea than this ? submitted by /u/Data-space_men [link] [comments]

Your Row Filter Works. The Agent Just Isn't Who It's Filtering

What is the recommended approach to enforce row-level security in Unity Catalog for external BI tool