Lakebase

Serving the Lakebase Data API Through Azure API Management

Migrating Europe from RDS Postgres to Lakebase

submitted by /u/Limp-Park7849 [link] [comments]

How to add a resource to an app space?

Hi, I was testing out the Genie App Builder today and wanted to created an app which connects to a Lakebase postgres table. However, Genie told me the resource postgres is missing in the app (space) and I should add it. In the app settings I can't add the resource because the app is managed/restricted by the app space. But in the app space I cannot find the option to add the postgres resource either (according to the documentation only "User authorization", "Serverless usage policy", and "Permissions" can be configured https://learn.microsoft.com/en-us/azure/databricks/dev-tools/databricks-apps/app-spaces#configure ). Is it possible to add resources to app spaces? Or am I limited with them? submitted by /u/IIDraxII [link] [comments]

Test on a Branch, Not on Patients: Agentic Drug Substitution with Lakebase

AI Agent Memory

Databricks supports agent memory storage. The amazing part is that Lakebase stores the memory, so we can also inspect the saved record. We can save "User prefers Mexican recipes" and recall it through a client with the question "What should I cook for dinner?" more news https://medium.com/databrickscommunity/databricks-news-genie-cli-app-builder-agent-memory-dabs-41820427fcc2 submitted by /u/hubert-dudek [link] [comments]

Quest 4 Completed – Building and Deploying My First Databricks App with Lakebase

Databricks Apps/Jobs Deployment Issue on New Workspace: Due to Quota?

My company sells a data analytics application to professional sports teams. The application is built as a Databricks App, deployed into a customer's workspace. The underlying data model is seeded by ETL that lives in my own company's workspace, with the gold tables shared as pinned version views into the customer's workspace, where a daily job syncs the views into Lakebase on the customer side. I built this on Databricks partially because the customers typically don't have robust IT teams in-house. Rather than requiring them to self-manage a bunch of infrastructure or host my own costly SaaS backend, I figured I could save a lot of complexity by setting up a Databricks workspace for them and deploying the application code into it via DAB from a central repo. I'm having an issue with a new customer that I onboarded yesterday. They created a new Azure subscription for the purpose of deploying the Databricks workspace + our application. Everything deployed successfully, but the Databricks App stops itself after approximately two hours. At first I thought it was a transient issue and restarted the app, but it keeps recurring. Troubleshooting has been challenging, because App logs are not visible when an App is stopped. I was able to get the following from the CLI ( databricks apps get -o json ): "app_status": { "message": "App status is unavailable.", "state": "UNAVAILABLE" }, "compute_size": "MEDIUM", "compute_status": { "message": "App compute was stopped due to workspace or account status.", "state": "STOPPED" }, Both the Azure subscription (Azure Plan) and the Databricks workspace (Premium + RBAC) do not appear to be in a trial state. Both were set up in the past 48 hours. I suspect this is some kind of new workspace restriction on the Databricks side, but I'm not sure what can be done about it. The App is deployed into a few different workspaces and runs without issue, but all of the workspaces have existed for a while. This is the first Workspace that was deployed specifically for the App. I found this thread on Databricks Community that describes a similar issue, but it appears to have died without a resolution. Has anyone run into a similar issue, and is there anything that can be done about this? It's likely that many of our customers will need a new Databricks workspace to deploy our application, so if this is a recurring issue, we may need to rethink our entire architecture. submitted by /u/Minute_Visual_3423 [link] [comments]

LTAP without lakebase?

Is there any world where an LTAP architecture can be created that does NOT include lakebase, or databricks for that matter? It seems to me that it isn't much more than a marketing term for selling lakebase. Every public doc about LTAP has references to lakebase sprinkled all over it. Lakebase is mentioned from the beginning to the very end. As I understand LTAP is essentially defined as a shared storage layer for OLTP and analytics workloads. And that sort of storage is an implementation detail that is ONLY found in the deepest and most proprietary parts of lakebase. (IMO, Microsoft has had their own flavor of LTAP for many years, in the form of Fabric DW. That engine looks and acts similar to a TSQL database, yet emits lakehouse files in storage. Yet I suppose it lacks some very particular features, or performs badly on certain disqualifying OLTP workloads. The disqualifications will always allow a databricks salesman to refuse to call it LTAP. If these disqualifications aren't purely subjective, then maybe we also need to define LTAP in terms of a certain TPS threshold and a certain set of minimal DBMS capabilities. I haven't seen anything like that.) submitted by /u/SmallAd3697 [link] [comments]

Performance Test - Batch-Loaded Unique Constraint Validation in Lakebase

The vendors that create big-data engines keep saying it is so challenging and costly to enforce uniqueness constraints in their managed catalogs (eg. in technologies like UC managed tables, that live on top of lakehouse blobs). They say it "doesn't scale". And that it gets more challenging at large volumes. But this is like anything else - customers should still be empowered to use features that may not always scale well. Would this problem get easier if they offered it only for certain data types like 32 bit unsigned integers? Lets consider Lakebase. Lets say I'm a user of Lakebase, and I'm running sequential batch transactions to insert a consecutive series of 1000 integers at a time that are validated by the database to be unique. How long would it take until those transactions to become slower than 100ms? Should I expect to to insert all 4 billion integers and never see any transaction take longer than 100ms? Now lets do a similar thing in a UC managed table , where we insert the same sequentially increasing 1000 integers at a time, and I check for the uniqueness of these myself in my serializable insert transactions. (Because the UC managed table won't help me with that.) Obviously my uniqueness validations would start to slow down over time. As I approach the limit of 4 billion integers I would expect the insert operations to take longer than lakebase , but possibly never more than one second per transaction. Does that sound about right? (If necessary I might use cheats, like break apart the integer into four columns of one byte each, then use partitioning). TLDR, validating uniqueness with 32 bit integers may be harder in a UC managed-table than for lakebase. But IMO that does NOT mean customers should be doing the heavy lifting ourselves. These lakehouse-based technologies should give us a mechanism for it, even if there are restrictions (ie. certain data types, tables of no more than 4 billion rows, etc). submitted by /u/SmallAd3697 [link] [comments]

Lakebase: Postgres That Actually Likes Your Lakehouse

How to Sync Lakebase Tables to Delta with Lakehouse Sync

Participate in a Lakebase user study!

👋 Hi r/databricks , I'm Connie from the Databricks UX team! The Databricks Platform team is conducting a usability evaluation to understand how Lakebase users set up, configure, and debug LTAP (Lake Transactional/Analytical Processing). We are hoping to connect with individuals using Lakebase on data and app development teams . Study details are as follows: Topic: Lakebase LTAP usability Duration: 60-min remote session over Google Meet Study date(s): October-November, 2026 (rolling) Compensation: $25 USD " thank you " gift card for the study session, if your company policy permits. Please fill out the screener survey if you’re open to participating in a 60-min call in the upcoming weeks, or can connect me with a member of your team who may be the right fit. Completing this screener survey does not guarantee eligibility into the study. The UX team will be reviewing responses for inclusion. Thank you for taking the time to help us build a better product! submitted by /u/Logical-Novel4483 [link] [comments]

How do you manage, in a Databricks Asset Bundles-based CI/CD, the permissions and access resources that a bundle cannot declare?

We have a DABs-based CI/CD in a monorepo and want to split it: deploy the DABs from the data-code repo and keep the Terraform infrastructure in a separate repo. The original idea was that Terraform would change at a different pace and its deployments would be manual, with no automated dependency on DABs deploys. However, at work they’re now saying this infrastructure part should be tied to the DABs deployment and not be manual, and I’m unsure where it should live and how to automate it. We mainly run into four cases: Cross-boundary grants: our pipeline needs to read a table owned by another team, so we need to grant SELECT on that table/schema to our service principal. Postgres grants: a job needs to connect to a Lakebase database and requires a GRANT on a Postgres schema, table, or role. Share contents: we need to add or remove tables and views from a Delta Sharing share consumed by an external consumer. Notification destinations: a job or pipeline needs to send alerts to a notification destination (email, Slack, webhook) that the bundle cannot declare. Today, all of these live in Terraform outside the bundle, so every time the data code needs one of these changes, we have to coordinate a separate MR and deployment in addition to the bundle deploy. Where do you put these in your pipeline: inside the bundle, in a separate Terraform stack applied after the bundle deploy, in the platform repo, or another way? What issues have you run into with each approach? Tx so much submitted by /u/Appropriate_Tip_9580 [link] [comments]

Announcement | Stripe Data Pipeline now syncs real-time Stripe data to Lakebase Postgres

Lakebase Search: Vector, Keyword, and Hybrid Retrieval Inside Databricks Postgres

Lakebase - OLTP Workloads on Databricks!

IoT device + Databricks Free Edition

tried out Free Edition for first time this year, it is surprisingly capable. Apps, Lakebase, Serverless Jobs, ML Flow, Genie worked well without any issues. created a demo with sending real IoT data to workspace. https://youtu.be/_RcoBxriu9Q submitted by /u/ramgoli_io [link] [comments]

Lakebase CDF to Unity Catalog vs firewall on storage

I have a use case where Lakebase CDF (transparent syncing of tables from PostgreSQL to Unity Catalog) would be super convenient. However I found that currently the CDF doesn't support storage accounts behind a firewall, accessible through a private endpoint, etc. They have to be publicly accessible, which is a no-no for most corporate data. Can anyone from DataBricks share whether it's on your roadmap to add support for this and when? submitted by /u/djfeelx [link] [comments]

Lakebase - Part 2

Announcement | Lakebase and Agentic SDLC: Branching Databases for Coding Agents

Announcement | Load terabytes of data in minutes into Lakebase Postgres

Building an Expense Management App with the Genie App Builder Leveraging Lakebase + AI for Image Scanning

I previously did not consider Databricks a great place to build apps on with AI. With the release of Genie App Builder (still in beta), that has changed, and now I see it as a serious player in the AI app building space. Before I start, quick disclaimer: I am part of the Databricks MVPs, work for a consulting company that is a Databricks partner, etc. That said, this review is fully based on my experience, my own words, my own testing, etc. Always try things out yourself! Things I appreciated: +By far, this feels like the most polished Databricks beta that has been released. If the goal is to make this easy for business users to develop applications easily and securely, this feels like it is already there. Granted, I have not tested with multi-users yet, but it feels very polished already. +Native integration with Databricks features like Lakebase. +Even though it has native integrations to Databricks features, it didn't force these features in building what I did when it was otherwise unnecessary. For example, the dashboard it built as part of the application was not an AI/BI Dashboard. I would have expected it do that, but it didn't, and rightfully so, as it would not have been truly the best thing for this use case, so truly a customer first thing here. +Respects governances/access controls, meaning I can develop an application freely in the sandbox environment, but if I want to deploy it, organizational safeguards, Unity Catalog, and other controls built on Databricks apply. Things that could be better: -With the goal of empowering citizen devs (or i should just say, business users that are technical enough), there needs to be an easier way to send request for access/capability etc to admins. Right now, the builder tells you through the chat, but that is not as intuitive as having a clear UI-embedded to-do things before deploying etc. One step further would be that the developer can notify the administrator of this need. -I found that when I developed an application that had to use a sandbox only capability when access was otherwise unavailable (see above), just hung when I tried to reopen the dev experience for it. I had to troubleshoot through the chat interface as to why that was happening to figure this out. Additional Note : Currently free at the moment, but pricing is not yet available since we are in beta. Will be interesting to see how the overall value of this capabilities plays out factoring the ease & governance vs compute & token costs. Closing Thoughts I think this is on track to be one of Databricks' best released feature, and a serious entry in the space that is otherwise feel with undifferentiated app building solutions. Is it creating a new category? No. But it feels like an Apple-level product, where being the best at executing is sometimes better than being the first to do something. And I believe it will pay off as businesses look more carefully as to where they spend their hard-earned money on. From a "data + AI platform vendor" perspective, this is currently the best offering of its kind. From an "app building vendor" perspective, there are still solid players out there, but for enterprise focused apps, in the top 3 of options I would consider, and for sure the first one I would try if my stack was heavily Databricks focused. submitted by /u/JosueBogran [link] [comments]

Things I miss on Databricks

Here are a couple of things I miss on Databricks: Group jobs per category. We all know that jobs can become messy sometimes. Connect jobs in Git more easily Data Quality. Like for real, where users can establish rules, create scans, quarantine, break pipelines, and etc. I don’t want to take my data outside to do it. App Genie Builder that won’t force me going to bundle Scale to zero apps without needing to use App Genie Builder Similar to 1, grouping queries, we all know that it can get messy with time Auto TTL in Lakebase (don’t know if it’s possible, but that’s a very good functionality) What else do you miss in Databricks? submitted by /u/Select_Morning8221 [link] [comments]

Databricks Acquires Responsive

Responsive is joining Databricks to accelerate development of Lakebase. submitted by /u/sai-nageshwaran [link] [comments]

Building on Lakebase | Episode 1: Meet Lakebase

I thought Databricks Lakebase was expensive

I used to think Databricks Lakebase was pretty expensive. A few Databricks architects I've talked to thought the same, especially when you factor in syncing data between Lakebase and Lakehouse. With LTAP coming, that second part should become less of an issue. Recently, we were adding Lakebase pricing to our free FinOpsWay extension so you can see estimated costs directly in the Databricks UI when creating a database. And I was honestly surprised by the numbers. Compute was cheaper than I expected, especially with scale-to-zero. Obviously, it depends on the workload, but I think I dismissed Lakebase too quickly without actually looking at the pricing. Anyone running Lakebase in production? Curious how much you're actually spending and whether it's worth it. submitted by /u/Significant-Guest-14 [link] [comments]

When (if at all) does Databricks plan to have bidirectional sync between Lakebase and Lakehouse?

I’m working on a project that could really benefit from bidirectionally syncing table entries that come in through an application hitting Lakebase as well as those batched into the Lakehouse from an external API. Do we know when bidirectional sync is expected? submitted by /u/ceeej777 [link] [comments]

Announcement | Lakebase Search: State-of-the-art full-text and vector search for Postgres

Group-based auth with Lakebase Data API

Lakebase CDF/WAL2Delta cannot write to UC managed storage with Private Endpoin

Built a live Helsinki tram tracker on Databricks with Lakeflow, Lakebase, Genie and Apps

Databricks Lakebase Search - GA Announcements

submitted by /u/Wise_Ear_4064 [link] [comments]