Skip to content
All topics

Databricks SQL

Recent items mentioning Databricks SQL across the Databricks ecosystem — releases, news, videos, and community Q&A. Updated hourly.

60 recent items12 releases5 news4 videos39 community threads

What is Databricks SQL?

Databricks SQL is the data warehouse side of the platform. It runs directly on your data lake and supports ANSI SQL with Delta Lake extensions, so you get warehouse-style SQL over the tables you already have instead of loading copies into a separate system. Queries run on SQL warehouses, compute dedicated to SQL workloads.

It carries the tooling analysts expect: a SQL editor with AI assistance and version history, AI/BI dashboards, alerts that monitor query results and deliver notifications, and a metrics layer for defining business metrics with consistent calculations. It also covers a fair amount of ETL, since streaming tables and materialized views can be defined and refreshed directly in Databricks SQL and queries can be scheduled as jobs.

SQL warehouses come in serverless, pro, and classic types, and Databricks recommends serverless when available: capacity management, patching, upgrades, and performance optimization are handled by Databricks. Query history, execution plan inspection, and automatic recommendations for inefficient queries are built in.

What's the difference between serverless, pro, and classic SQL warehouses?

All three run Databricks SQL queries. Databricks recommends serverless SQL warehouses when available, citing instant and elastic compute, minimal management overhead, and lower total cost of ownership, since capacity management, patching, upgrades, and performance optimization are handled by Databricks. Pro and classic remain as alternatives where serverless isn't offered.

Can I use Databricks SQL for ETL, not just BI queries?

Yes. You can define and refresh streaming tables and materialized views directly in Databricks SQL, and schedule SQL queries as jobs for automated data processing and reporting workflows.

Do I need to move my data into Databricks SQL first?

No. Databricks SQL is built on lakehouse architecture and runs directly on your data lake with ANSI SQL and Delta Lake extensions, so the warehouse sits on data where it already lives rather than on copies loaded into a separate system.

Does Databricks SQL include dashboards and alerts?

Yes. AI/BI dashboards with AI-assisted authoring are built in, and alerts can monitor query results, evaluate conditions, and deliver notifications automatically. There's also a metrics layer for defining business metrics with consistent calculations.

Sources: Databricks SQL overview (Databricks docs) · SQL warehouses (Databricks docs)

What's happening in Databricks SQLAI synthesis · updated 1d ago

The agentic code converter went GA for SQL on Oct 29 1, while the SDK releases between Oct 15 and Oct 21 expanded the SQL warehouse API: Realtime SQL warehouse support landed first 810, then new customerVersionLabel, maxDbu, and reydenClusterSizeMultiplier fields on warehouse create, edit, and get calls 567. Watch for breaking changes: replicationConfig is removed from CreateShare and ShareInfo in the Java, Go, and Python SDKs 567, and effectiveServerlessComputeId is gone from job tasks 810. Practitioners are also asking how to debug remote_query() performance 2.

Generated daily from the 10 most recent items mentioning Databricks SQL. Click any [N] to jump to the source.

Reddit

Agentic code converter is GA for SQL

Migration · GA Agentic code converter is GA for SQL Genie Code converts T-SQL, Snowflake, Redshift, Oracle, BigQuery, and Teradata SQL to Databricks SQL. It validates each file and fixes errors before writing to a workspace folder. Work in a migration project, or run /migrate in a Genie Code chat for one file or folder. Afterward, keep chatting to clear action items. The Notebooks target is not part of this GA. https://preview.redd.it/u5m0quj71fyh1.png?width=1808&format=png&auto=webp&s=1a5f16a1d66fa88e9bc171e10efb16c33dadfa2b Looking to migrate workloads but don't have the labor to do so. This is going to continue to expand its SQL dialect but taking an agentic approach to migrations really speeds up migrations and cuts down on time/money from labor submitted by /u/Wise_Ear_4064 [link] [comments]

00Wise_Ear_4064yesterday
Reddit

How to debug remote_query() performance?

I have a job which queries some Azure SQL DBs connected through UC managed connections (no lakehouse federation) via remote_query() function, and facing pretty long execution times (several minutes) when reading a few thousand rows only. When running the same query via SSMS on the target host, it returns same result within milliseconds. Is there any way how I can debug the pure execution time from Databricks side? I'd like to understand if the queries run slow, or if there's any Databricks or network overhead accounting responsible. Note that remote_query() runs T-SQL in this scenario, not Databricks SQL. So there's no SQL conversion involved at any point in time. submitted by /u/Sea_Basil_6501 [link] [comments]

00Sea_Basil_65012d ago
HackerNews

Show HN: 10x faster, 3x cheaper ETL

For the past few months our team has been working on an agentic framework that takes large, expensive SQL workloads and compiles them to Rust, backed by a formal proof of correctness. This is the sort of work I used to do by hand as a systems performance engineer, but now it happens in hours instead of months. We first redid our internal pipelines with this technique and they're now 10x faster and 3x cheaper. Next, we went live with our Snowflake and Databricks SQL customers. They're seeing similar results with critical production workloads that originally cost 6-7 figures per year. We'll be publishing benchmarks and code samples over the coming weeks; for now, you can read a bit more (and watch our anime launch trailer) on the site. Happy to answer questions about how it all works. --- top comments --- [gtirloni] Why was SQL slow? Where did processing shift to?

11karamazov3d ago
Reddit

A full month-end finance app built on Databricks Apps

Below are the things covered in the video: 1. Overview: drag the month slider and every figure on the page moves with it, including a cash flow diagram from sales through to profit 2. P&L: the statement drawn as a ladder from revenue down to profit, plus a bridge showing why we beat or missed budget 3. Cafes: ranking, a map of the Front Range, a margin heatmap by month, and how fast each new cafe paid back its fit-out 4. Menu: which items earn their place and which are just popular 5. Cash: a 13-week forecast, overdue invoices by age, and the supplier payment calendar 6. Close: the month-end checklist as a board, with how long each close took historically 7. What-if: move price, volume, wage and ingredient levers and watch profit change 8. A one-page monthly report that prints to A4 Under the hood: seven Delta tables in Unity Catalog, read through a SQL warehouse, with the app running as a Databricks App in Python. All the finance logic (consolidation, budget variances, the close tracker, the cash forecast) sits in one file that a finance analyst could read. submitted by /u/oneaffidavit1 [link] [comments]

00oneaffidavit16d ago
Reddit

Databricks Genie One — How to Restrict Agent Creation and Scheduled Tasks?

We use Databricks Genie One, and some users are creating unnecessary Genie Agents and scheduled tasks. Removing Consumer access or Databricks SQL access affect users’ ability to view dashboards, run SQL queries, or use other Databricks features Questions: Can workspace admin prevent regular users from creating Genie Agents? Can scheduled-task creation be disabled while users can still chat with Genie? submitted by /u/--playground-- [link] [comments]

00--playground--2w ago
Reddit

What happens when too many ppl run queries on the same Databricks SQL Warehouse?

Do the queries just wait in a queue, or does the warehouse automatically handle the extra load.. Also, how do you figure out when it's time to increase the warehouse size instead of letting queries wait... Curious how this works in real projects. submitted by /u/Bhanuprakash_1947 [link] [comments]

00Bhanuprakash_19472w ago
Databricks CommunityCommunity Articles

How to set budgets and alerts on SQL warehouses so self-service never surprises Finance

003w ago
Reddit

Auto stop option on databricks apps

Want to request a feature for databricks app, it should have an auto stop option. I know Databricks markets it as a web app that should be readily available, but I think customers deserve the option to turn auto-stop on or off, like we do for, say, SQL Warehouses. This has been a major blocker for us suggesting apps to leadership, the engineering team has build some really useful apps for usecases like Data modelling and stuff which is used when needed but they running 24*7 all time is wasting compute and getting flagged in finops P.S i know we can have a databricks job which can shut down and start the app during working hours but we have a global team working round the clock so it's not feasible for us. If we have someone from databricks apps team this is a highly requested thing from alot of people around the community. submitted by /u/sumit671 [link] [comments]

00sumit6713w ago
Databricks CommunityTechnical Blog

Rebuilding an 800-Line PySpark Pipeline in 200 Lines of Databricks SQL

003w ago
Reddit

system.query.history is now Generally Available in Databricks

Until now, answering "which query is eating our warehouse?" or "who changed that table?" meant clicking through the Query History UI one workspace at a time. Now it is a SELECT. The table logs every statement run on SQL warehouses, serverless compute, and Lakeflow pipelines, across all workspaces in the same region. Read more on LinkedIn: https://www.linkedin.com/posts/cenh_databricks-dataengineering-systemtables-ugcPost-7511094425977016320-L7uj/?utm_source=share&utm_medium=member_desktop&rcm=ACoAABmJHrsBNAC3x3H1M58JRKoHv_l4D61n0-8 submitted by /u/Lenkz [link] [comments]

00Lenkz1mo ago
Reddit

Laya off the benchmark: can a zero-shot decision model route real SQL traffic?

Weekend-ish experiment on Databricks. One table, TPC-H orders, 15M rows, living in two places at once: Lakebase (Databricks' managed Postgres), a continuously synced copy with a btree index on the key. Sub-100ms point lookups, useless for a GROUP BY over 15M rows. Delta behind a serverless SQL Warehouse. Great at scans and aggregations, slow at fetching one row. The synced table is the nice part: native continuous Delta to Lakebase Postgres (needs a PK + CDF), so it's one dataset under one Unity Catalog, replication handled by the platform instead of a homemade pipeline. Then I put laya (convaiinnovations/laya, a non-autoregressive zero-shot decision model, vanilla, no fine-tuning) behind a FastAPI Databricks App. It reads each query and picks the engine. MLflow traces input to decision to execution. I submitted by /u/Limp-Park7849 [link] [comments]

00Limp-Park78491mo ago
Databricks CommunityData Engineering

New Databricks SQL window fuction: counter_diff

001mo ago
Reddit

What if Databricks showed the cost directly next to every job and run?

Databricks give us a lot of cost data, but usually not where we actually need it. When I open a job run, I can see duration, compute, tasks and logs, but not what that run actually costs. When I create a cluster or SQL warehouse, I choose the size first and DBU, and discover the real cost later. That’s the problem we’re trying to solve, bringing cost directly into the Databricks UI. We created a free browser extension, with no backend or telemetry https://finopsway.com/ submitted by /u/Significant-Guest-14 [link] [comments]

00Significant-Guest-141mo ago
Reddit

Before you move a job to serverless: the limitation list is the real decision, not the startup time

Most classic-vs-serverless comparisons stop at "serverless starts faster and Databricks manages it". True, but that is not what decides it in practice. Where they differ structurally: with classic compute the control plane asks your cloud provider for VMs, and the cluster runs in your own account and network, so you pick instance types and own the networking, and you pay the cloud provider for machines plus Databricks for DBUs. With serverless, the docs say resources run in "the serverless compute plane, which is managed by Databricks" and you provision nothing in your own cloud account. Your tables stay in your own storage in both cases. What actually decides it is the limitation list: - R is not supported - compute-scoped libraries and compute-scoped init scripts are not supported - only Spark Connect APIs are available, so the Spark RDD APIs are out - streaming: Trigger.AvailableNow() is supported, Trigger.Continuous(interval) and Trigger.ProcessingTime(interval) are not - maximum runtime is 7 days; a run that exceeds it is terminated and not retried So: serverless for SQL warehouses, ad hoc exploration and spiky jobs where waiting for a cluster is the real cost. Classic when you need a specific instance type or GPUs, cluster-scoped libraries, R, RDD code, a continuous stream, or your own network setup. I drew both paths side by side as a short animated diagram (2:36): https://youtu.be/s8452E2Y6WM Series in order: https://www.youtube.com/watch?v=B5iHmoYgnqY&list=PLDB5WDkDOYF4 For those who have moved production jobs across: what caught you out that is not on the documented list? submitted by /u/Broad-Cherry-5999 [link] [comments]

00Broad-Cherry-59991mo ago
Reddit

Databricks pipeline works in one workspace but fails in another — where would you start?

Imagine the same pipeline and code are deployed to two Databricks workspaces. Workspace A: catalog.schema.table → Query works → Pipeline works Workspace B: catalog.schema.table → Catalog not accessible → Pipeline fails The catalog exists, the table exists, and the SQL itself is valid. Would you first investigate: Workspace-to-catalog assignment Unity Catalog permissions USE CATALOG USE SCHEMA SELECT privileges Service principal permissions External locations Storage credentials Cluster access mode SQL Warehouse identity I'm curious about the actual troubleshooting order people follow for these kinds of Unity Catalog issues. What would you check first? submitted by /u/hanshu6576 [link] [comments]

00hanshu65761mo ago
Reddit

Manager wants us to "use AI." Thinking about an AI-driven data testing framework for DevOps promotions. Sanity check?

Although we are using genie code alot but manager wants some functionality based on AI. ( maybe that’s hood goal). Our devs hate manually writing tests, so I'm drafting an automated testing gate for DevOps promotions (Local ➔ Dev ➔ QA). Wanted review with all of you. The Proposed Architecture: 1. Extract Metadata: Pull column tags, schemas, and lineage from Databricks Unity Catalog. 2. AI-Generated Tests (Llama via ⁠ai_query⁠ ): LLM reads metadata to draft SQL data checks (nulls, types, basic business logic). 3. Persist & Cache: Save SQL rules to a table. Re-generate only when schema hashes change so bug-fix retests stay 100% deterministic. 4. Execution: Run the generated SQL on a SQL Warehouse (fast, cheap, no LLM cost per data row). 5. Alerting: Feed error logs to LLM for a 2-sentence summary and send directly to Teams via Webhook (avoiding ignored email reports). How does it sound like? Is it really worth it? Anybody using this or any other AI based functionality to make devs life easy. submitted by /u/Terrible_Mud5318 [link] [comments]

00Terrible_Mud53181mo ago
Reddit

How do you parse an xml that's in string format?

I have xml data that's for God knows why it's in string format. And the fking thing is so messed up or at least I think it's messed up because it's not consistent. Sometimes there is something else in there. Xml is something like this: .... ... ... Regex is not an option. Substring is not an option. Because it's so messed up it's not consistent. I just need to somehow parse and get that Name, Age, Gender values using Sql/Databricks sql Please, help you mate. submitted by /u/pineapple_brownies [link] [comments]

00pineapple_brownies1mo ago
Reddit

No more UNION ALL-ing all of your SDP pipeline event log tables for monitoring

https://preview.redd.it/3w0j4ss3inoh1.jpg?width=2048&format=pjpg&auto=webp&s=50bc79280a546544e8f795947e5af6b211e9596a Hi folks, Databricks PM here - I wanted to share an exciting update that you no longer have to manually publish and combine your pipeline event logs for monitoring across pipelines and workspaces. We just launched the beta for the pipeline events system table (system.lakeflow_pipeline_events_preview.pipeline_events). Key features: All pipeline events (regardless of cluster start) are automatically captured without any manual enablement or maintenance. Events are aggregated in a central system table without requiring any custom aggregation logic. This data is available at close to real time latency (based on our internal testing we achieve a P99 latency of less than 1 minute). An admin can grant a single user access and they can query events for every pipeline in the table. Fine-grained access controls support which scopes the visibility to only the pipelines the user has access to is coming soon. The data remains in the system table even after pipeline deletion and is retained for 13 months. If you want longer retention this is also possible with the configurable retention feature for System Tables. Query ergonomics are better now with the use of VARIANT. Here are some sample queries in case you want to try them out: -- The latest error for each pipeline that has errored in the last 7 days, with the outermost exception. -- The exception chain is ordered with the root cause last, so read element -1 for the root cause. -- On many errors only the first element carries error_class and sql_state. SELECT workspace_id, pipeline_id, event_time, event_type, message, error.exceptions[0].error_class AS exception_error_class, error.exceptions[0].sql_state AS exception_sql_state FROM system.lakeflow_pipeline_events_preview.pipeline_events WHERE level = 'ERROR' AND event_time >= current_timestamp() - INTERVAL 7 DAYS QUALIFY ROW_NUMBER() OVER (PARTITION BY workspace_id, pipeline_id ORDER BY event_time DESC) = 1 ORDER BY event_time DESC -- Flow throughput for a specific pipeline SELECT origin.flow_name, date_trunc('HOUR', event_time) AS hour, SUM(variant_get(details, '$.flow_progress.metrics.num_output_rows', 'BIGINT')) AS rows_written FROM system.lakeflow_pipeline_events_preview.pipeline_events WHERE pipeline_id = ' ' AND event_type = 'flow_progress' AND event_time >= current_timestamp() - INTERVAL 7 DAYS GROUP BY origin.flow_name, date_trunc('HOUR', event_time) ORDER BY hour DESC, rows_written DESC -- Data quality: failed expectations by dataset, per update, in the last 1 day SELECT pipeline_id, update_id, origin.dataset_name, expectation.name AS expectation_name, SUM(expectation.failed_records) AS failed_records FROM system.lakeflow_pipeline_events_preview.pipeline_events LATERAL VIEW explode(variant_get(details, '$.flow_progress.data_quality.expectations', 'ARRAY >')) AS expectation WHERE event_type = 'flow_progress' AND event_time >= current_timestamp() - INTERVAL 1 DAY GROUP BY pipeline_id, update_id, origin.dataset_name, expectation.name HAVING SUM(expectation.failed_records) > 0 ORDER BY failed_records DESC; Beyond single queries you can build alerting (using Databricks SQL alerts) and dashboards. We will share a new dashboard template soon - I will update this post once its available. Call outs: This is in beta right now, if you are not opted in we will not capture your event log data. Enablement: Toggle on the “ Lakeflow Pipeline Events System Table ” from the account level preview. Docs are linked here , would love to hear your thoughts on how you will use it or what else you want to see to improve observability! submitted by /u/brickster_123 [link] [comments]

00brickster_1231mo ago
Reddit

Databricks is too expensive for small teams" is usually a workload problem, not a platform problem

Every few weeks someone posts a version of “our bill is going from $1k to $5k a month, is Databricks even worth it at our size?” The answer isn’t really about company size. It comes down to how you’re using the platform. All-purpose compute being used for scheduled jobs. Interactive clusters are convenient, but they can increase costs quickly. If a notebook runs on a schedule, moving it to jobs compute can make more sense. SQL warehouses sized for peak usage and left running. Using auto-stop and choosing a warehouse that can scale when needed can help avoid paying for idle capacity. Continuous triggers on jobs that don’t need them. This one gets misdiagnosed a lot. The fix usually isn’t “rewrite it as batch,” which costs you checkpointing and exactly-once. It’s Trigger.AvailableNow , which processes what’s available and shuts the cluster down. Databricks recommends it for incremental batch processing. If the table needs a 15-minute refresh, that’s a scheduled job with an AvailableNow trigger, not a cluster running at 3 AM. Once these three areas are addressed, the bill for a small team can often come down to a much more reasonable baseline. Then the more interesting question is: are you actually getting value from Unity Catalog, Delta, and the broader BI, ETL, and ML capabilities, or are you mainly paying for Spark compute that you don’t really need? If your data fits comfortably in Postgres or ADX, you have one main consumer, and you don’t need much governance or lineage, Databricks may not be necessary. No amount of cost tuning changes that. For teams running Databricks on relatively small workloads, what actually made it worthwhile for you? Was it a specific technical requirement, governance, or simply the convenience of having everything in one platform? submitted by /u/AbilyticsEng [link] [comments]

00AbilyticsEng1mo ago
Reddit

Do small companies actually use Databricks?

Sometimes I feel like Databricks is way too expensive. It feels like using a huge truck to move a single grain of sand. My company needs real-time data, but our data volume simply does not justify the use of Spark Structured Streaming. Despite this, they are insisting we move to Databricks. I'm worried our data infrastructure costs will jump from $1,000/month to $5,000/month or more due to the running costs of Databricks SQL Warehouses. Currently, I use Azure Container Apps with KEDA and Python, which helps me manage scaling and keep costs low. We ingest into Event Hubs, use ADX (Azure Data Explorer) as our OLAP warehouse, and archive cold data in a data lake. With this setup, I manage to process all our data with very low latency. When I tested this on Databricks Structured Streaming, I actually got higher latency and much higher costs. Would love to know what you guys think. submitted by /u/Puzzled-Mail-9092 [link] [comments]

00Puzzled-Mail-90921mo ago
Databricks CommunityData Engineering

What is a Warehouse in Databricks SQL?

001mo ago
Reddit

Query works in Databricks SQL but fails through JDBC

I’ve noticed cases where a query runs successfully in the Databricks SQL editor but fails when the exact same query is executed through a JDBC-based application or data quality tool. Has anyone run into this? Was the issue related to query wrapping, session settings, SQL dialect differences, or JDBC driver behavior? How do you usually troubleshoot these cases? submitted by /u/No_Ambition8323 [link] [comments]

00No_Ambition83231mo ago
Reddit

What type of compute do you guys use in Databricks?

Hey everyone, I've been using Databricks for a while, but I'm still curious about how other teams decide which compute option yo use. There are so many choices now - serverless,job clusters, all-purpose clusters, SQL warehouses, etc. - and sometimes it's not really obvious which one makes the most sense. And when do you go with serverless vs a normal cluster? I'm also wondering if people are mainly choosing based on cost, performance, startup time, or just what their team is already comfortable with. Would be interested to hear what you're using in your projects and what made you choose it. Especially if you've switched from one type of compute to another and noticed a real difference. submitted by /u/Delulu62134 [link] [comments]

00Delulu621341mo ago
Reddit

Does Photon support CSV?

Ok so I know that the docs state that CSV is supported. But when trying to read a very standard CSV file I get poorer than expected performance and in the Spark UI I see that it's not using a Photon scan operator, just regular Scan CSV followed by a Row to Columnar conversion operator. https://preview.redd.it/1amu93oqqslh1.png?width=260&format=png&auto=webp&s=4ca1c1ffec6cb6f6cd484b4645b0b5d7c5f6be86 I'm not running anything complex: df = ( spark.read .format("csv") .option("header", "true") .schema(schema) .load(csv_path) ) df.write.mode("overwrite").saveAsTable("...") I also checked reading the same CSV and a difference CSV via DB SQL (using `COPY INTO`) and see low task time spent in Photon + a row to columnar operator. https://preview.redd.it/g1tlotk8rslh1.png?width=940&format=png&auto=webp&s=f3d2b22ebab4f716c5b6831d96305c082e1126a8 DBR 19 + Serverless SQL Warehouse (current) Can anyone explain whether or not CSV is supported and in what conditions? This was quite a surprising find as I assumed that Photon supported pretty much everything. submitted by /u/Common_Jaguar474 [link] [comments]

00Common_Jaguar4742mo ago
Reddit

Moving ~150 Tableau dashboards to Databricks AI/BI - anyone done this at scale?

We are considering moving from Tableau to Databricks AI/BI Dashboards and wanted to see if anyone here has made a similar move. We currently have around 150 Tableau dashboards used by 1,000+ users, with a mix of technical and non-technical users. Most of the underlying data already lives in Delta tables, so naturally we're looking at whether it makes sense to consolidate and use Databricks for the dashboarding layer as well. For those who have done this, how has the experience been? Anything you wish you knew before migrating? Any major limitations compared to Tableau, especially around visualization, performance, or the experience for non-technical users? I'm also curious about cost at scale. With 1,000+ users potentially hitting dashboards, did SQL warehouse usage become a concern? Would you feel comfortable replacing Tableau completely with AI/BI Dashboards today, or do you still find yourself needing Tableau for certain use cases? Would really appreciate hearing about real-world experiences. submitted by /u/Immediate_Bus5250 [link] [comments]

00Immediate_Bus52502mo ago
Databricks CommunityData Engineeringanswered

databricks SQL UDF in select statement

002mo ago
Stack Overflow

"List columns" statements when querying Databricks via ODBC

I've got an old .NET app running on Windows Server. The app submits queries to a serverless SQL Warehouse. I've noticed that each query ( SELECT statement) submitted by the app results in three statements in the SQL Warehouse. One statement containing the query submitted by the app, and two statements that look like Listing columns 'catalog : foo, schemaPattern : foo, tablePattern : resultssettable, columnName : null' . The query might take 200ms to execute. The "Listing columns" statements take about 90ms each to execute. In aggregate, these "Listing columns" statements double the duration of queries sent to this warehouse. How do I get rid of these freak'n "Listing columns" statements? Here is what I've done/tried: Tables in the query are full qualified with catalog and schema. Query specifies the names of the columns to retrieve (i.e., not a SELECT * FROM ... ). I have Use Native Query enabled in the DSN. I have Fast SQLPrepare enabled in the DSN. Enabling Ignore Tables Metadata from All Schemas in the DSN makes no difference. I configured the Databricks ODBC driver to pool connections. I have the 64-bit Databricks ODBC driver installed, version 2.11.00.1005. I don't have access to the .NET apps source. I have been using the following PowerShell script to recreate the issue and test configuration changes: $dsn = "DSN=Test Connection" $sql = @" SELECT col1 , col2 , ... FROM dev.foo.bar WHERE whatever = 1; "@ $conn = New-Object System.Data.Odbc.OdbcConnection $dsn try { $conn.Open() $cmd = $null $cmd = $conn.CreateCommand() $cmd.CommandText = $Sql $cmd.CommandTimeout = 30 $adapter = New-Object System.Data.Odbc.OdbcDataAdapter $cmd $adapter.MissingSchemaAction = [System.Data.MissingSchemaAction]::AddWithKey $table = New-Object System.Data.DataTable "foobar" [void] $adapter.Fill($table) return $table.Rows.Count } finally { if ($reader) { $reader.Close() } if ($cmd) { $cmd.Dispose() } } I feel like I missing […truncated]

.netodbcdatabricksazure-databricks
10Adam2mo ago
Reddit

Databricks SQL Alert - Scheduling Questions

Databricks SQL Alert — Question Set Help needed, I created a SQL alert in databricks to notify me when a new entry is added to a dataset/table within the last 24 hours. SQL SELECT COUNT(*) AS new_ FROM catalog.schema.table WHERE xyz IN ('a', 'b', 'c', 'd') AND datetime >= CURRENT_TIMESTAMP() - INTERVAL 24 HOURS; Within the Alerts panel, I set the condition as follows: First row Column name: new_ Operator: > Static value: 0 I then added my email address in the Notifications section and set "When alerting, notify" to Always . I then selected View Alert , where I created a schedule — let's say every day at 9:00 AM . Questions a) How exactly does the schedule work here? I am asking because I selected "Always" under the notification settings, but I have also created a schedule to run every day at 9:00 AM. What is the relationship between these two settings? b) Let's suppose I build this alert and create the schedule as described above. Person B (sales dept rep) also subscribes to this alert. I do not want Person B to receive an alert unless the new record belongs to the Sales department. For example, if a new record is added that belongs to a department other than Sales, and the scheduled alert runs at 9:00 AM, will Person B still receive an email notification? Thanks in advance for any clarification! submitted by /u/SquareLong2523 [link] [comments]

00SquareLong25232mo ago
Stack Overflow

Databricks Dashboard - Scheduling Questions

( FYI, I did write the initial draft of this message by myself, and then later asked AI to proof read it and later pasted the same here, Apologies in advance!) I recently developed and published a Databricks dashboard for a personal project, and I want to learn how the scheduling feature works. For context, I have read the Databricks documentation, Manage scheduled dashboard updates and subscriptions | Databricks on AWS , but I couldn't find answers to the questions below. Databricks Dashboard — Question Set 1 I published the Databricks dashboard using Individual Data Permissions . I then added another person as a participant, created a schedule, and subscribed to the schedule from both accounts. Let's call them Person A (publisher + subscriber) and Person B (subscriber). Questions a) Assume Person B works in the Sales department and should only be able to see Sales data. Since the schedule was created by Person A, when the scheduled run occurs and Person B receives the email notification they subscribed to, will the PDF attachment contain all departments' data , or will it contain only the data Person B is permitted to see? b) After the scheduled run, if Person B opens the dashboard using the link in the email notification, or accesses the dashboard directly through Databricks, what data will they see? Will they see all department data , or only the data they are permitted to see based on their individual data permissions? My understanding is that when Person B visits the dashboard, it does not automatically refresh just because a scheduled dashboard update has occurred. I may be misunderstanding how this works, so I'd appreciate some clarification. Databricks SQL Alert — Question Set 2 I also created a SQL alert to notify me when a new entry is added to a dataset within the last 24 hours. Please note: The screenshot below is not from my actual example. I am only including it so that new developers like myself can relate to what I am referring to. enter image descr […truncated]

databricks
-60Rahul Sawant2mo ago
Reddit

Tagging individual ai_query() calls in SQL?

Essentially the title, but for background we’re using ai_query() in Databricks SQL Warehouse and our pipeline makes 3 LLM calls in parallel for different steps. We can see token usage/cost in system.ai_gateway.usage, but we can’t tell which call came from which step. From what I can tell, request tags are possible via the Python/rest sdk, but not through ai_query() in SQL. Has anyone found a workaround, or is moving the calls out of SQL currently the only option? Edit for clarity: the main thing we’re trying to do is join each pipeline step back to system.ai_gateway.usage so we can attribute token usage/cost to steps a/b/c etc submitted by /u/hulioshort [link] [comments]

00hulioshort2mo ago
Databricks CommunityTechnical Blog

Load Testing Databricks SQL Warehouses with JMeter — Part 3: Running and Analyzing Results

002mo ago
Databricks CommunityTechnical Blog

Load Testing Databricks SQL Warehouses with JMeter — Part 2: Concurrency and the Test Plan

002mo ago
Databricks CommunityTechnical Blog

Load Testing Databricks SQL Warehouses with JMeter — Part 1: Inputs and Configuration

002mo ago
Databricks CommunityTechnical Blog

Serve Tableau Reports Directly from Databricks SQL

003mo ago
Databricks CommunityMVP Articles

Understanding EXPLAIN FORMATTED in Databricks SQL

003mo ago
Databricks CommunityGenerative AI

DBSQL MCP output limit

004mo ago
Databricks CommunityGenerative AI

Querying Metric Views via Classic/Pro SQL Warehouses

004mo ago
Databricks CommunityTechnical Blog

Incremental REPLACE WHERE Flows Brings Targeted Refreshes to SDP and DBSQL

004mo ago
Databricks CommunityAdministration & Architectureanswered

Restrict certain queries on SQL Warehouse

004mo ago