LLM
Recent items mentioning LLM across the Databricks ecosystem — releases, news, videos, and community Q&A. Updated hourly.
Databricks shipped ai_decide, an AI Function that returns structured decisions from unstructured text faster and cheaper than general LLMs, running in SQL at batch scale or over REST in real time for prompt routing, metadata tagging, and agent quality evaluation 4. The same shift toward decision-focused models that output typed probabilities instead of free text is the theme of the September AI newsround, alongside the general availability of the Genie one Model Context Protocol 3. On the application side, Vital Skills gives Genie Code open-source healthcare and life sciences skills, which stop general agents from silently picking clinical thresholds, and it outscored Genie Code alone on BiomniBench 1. A community MDM app combines rules, ML, and LLMs for entity resolution 2.
Generated daily from the 4 most recent items mentioning LLM. Click any [N] to jump to the source.
Vital Skills: Upskilling Genie Code with Healthcare and Life Sciences skills
Vital Skills is a set of open-source Agent Skills that gives coding agents healthcare and life sciences domain expertise, and Genie Code uses it to apply HLS-specific tools and best practices on Databricks. It addresses general agents silently picking clinical thresholds and defaulting to generic statistical methods. On BiomniBench, Genie Code with Vital Skills outscored Genie Code alone on supported in-scope tasks, with better consistency, accuracy and rigor, fewer missed steps and retries, and closer adherence to industry standards.
Custom MDM App on Databricks: Combining rules, ML, and LLMs for Entity Resolution
Introducing ai_decide: make fast decisions on your governed data
Databricks has introduced ai_decide, a new AI Function that processes unstructured text to return structured decisions faster and at lower cost than traditional LLMs. Optimized for decision-making rather than text generation, it runs directly on governed data at batch scale in SQL or in real time over REST for tasks like prompt routing, metadata tagging, and agent quality evaluation.
Databricks Micro Apps, App Spaces and Genie App Generator
Databricks launched App Spaces and Serverless Micro Apps at the end of last week (in Beta). Over the weekend, I tested migrating two of my production Databricks Apps to App Spaces. App Migration 1: Blocked by Zero Egress My Data Portfolio Project Creator required internet access to pull data stacks from live job postings. Then, the LLM needs internet access to research for open data sources to use. In standard Databricks Apps, this runs cleanly. In App Spaces, there is zero external internet egress including for LLMs. The app can reach internal workspace resources, but it cannot touch the outside web. If your app relies on third-party APIs, external databases, or web scraping, App Spaces is a non-starter until Databricks opens network egress. Workload 2: Success on Internal FinOps My client-facing DBU cost observability app reads workspace usage data and writes directly to Lakebase. Because it requires zero external network calls, the migration worked. Both the micro app compute and Lakebase scale to zero when idle. Cold starts take roughly 30 seconds (though in beta, you occasionally need a quick browser refresh once it spins up). For internal, low-frequency administrative tools, this turns a continuous monthly compute bill into pennies. (+ App Spaces, Genie App Generator, and Micro Apps are free while in beta) The next part is less about App Spaces and more just general best practice for Databricks Apps that I see people miss. Stop Using Delta Lake as an OLTP Database Databricks Apps are software applications, not batch analytics notebooks. If your app writes application state, session data, or row-level CRUD directly into analytical Delta tables, you need to rethink that design. For app transactions, use Lakebase (serverless Postgres which also scales to 0). Both are governed under UC, but Lakebase gives your app the low-latency transactional engine that application engineering actually requires. My Verdict on the Beta App Spaces solves the idle compute problem that has plagued Databricks Apps since launch. But until Databricks allows us to deploy directly from existing Git repos and opens external network egress, it remains limited to internal-only use cases. Curious on other peoples experience... How has it been for others? submitted by /u/OkImprovement7010 [link] [comments]
Can Jev replace your LLM judge? Part 2: Testing harder answers
TypeSafe's Jev model evaluates MLflow QA answers roughly seven times faster and significantly cheaper than larger LLMs, but it misses subtly incorrect technical details such as reversed roles and omitted permissions. Ahead of Jev integration arriving in MLflow 3.17, practitioners can use its confidence probabilities to fast-track clear verdicts while escalating borderline cases to a larger model for review.
The deployment decalogue
I am an ML engineer, but I come from a software engineering background: years of full-stack work, with heavy DevOps and Terraform experience. I come from teams that deploy to production five times a day with real continuous deployment. And honestly? Pressing the button still feels weird sometimes. Every engineer knows that feeling, no matter how good the safety net is. So I wrote down the list that settles it. Ten commandments, one flow, written with data scientists and ML teams in mind, but it works for batch jobs, realtime inference, and LLMs alike. Answer honestly, and if all ten are true, you can ship to production anytime, in any form or way. submitted by /u/SuspiciousPavement [link] [comments]
How we eliminated $1 million a year of wasted AI agent spend in one hour
Broken MCP tool calls and silent retries can quietly waste over $1 million annually in tokens and engineering hours across AI agent fleets. Tracing tool calls with Unity Gateway and analyzing spend with Genie One enables teams to rapidly deploy fixes and design tools that gracefully handle ambiguous LLM inputs.
How are you actually setting up AI/LLM evals in Databricks end-to-end? Looking for a step-by-step production workflow
We have a product where Databricks is our backend , and we’re now trying to properly evaluate and improve the quality of the AI-generated answers in our application. I’m looking for advice from people who have actually implemented LLM/GenAI evaluations in Databricks in production . I’d really appreciate an end-to-end, step-by-step explanation of how you would set this up from scratch. Specifically, how would you approach: Define what a “good answer” means Accuracy / correctness Relevance Completeness Groundedness / faithfulness to our data Hallucination rate Citation/source correctness Following user instructions Response consistency Latency and cost Create a benchmark / golden dataset Should we manually create a set of representative user questions? How many questions are enough to start? Should each question have an expected answer? Should we store expected SQL/results, expected sources, or just an expected natural-language response? Where should this benchmark dataset live in Databricks? How do you keep it updated as the product evolves? Set up automated evaluations What Databricks/MLflow tools should we be using today? MLflow evaluation? LLM-as-a-judge? Custom scorers? Human evaluation? How do you combine these rather than relying on one score? Evaluate RAG / data-grounded answers Our AI answers questions based on enterprise data in Databricks. How do you separately evaluate: Retrieval quality Whether the correct tables/documents were selected Context relevance Groundedness Final answer correctness Whether the model invented something that wasn’t in the retrieved data Evaluate text-to-SQL / analytics questions If a user asks something like: “What was average building occupancy last month?” should we evaluate: Generated SQL Tables selected SQL execution result Final natural-language answer separately? What is the recommended architecture for this? Guardrails Where should guardrails sit in the architecture? For example: Prevent hallucinated numbers Prevent querying unauthorized tables Detect PII Prevent prompt injection Enforce tenant/user permissions Block unsupported questions Force answers to cite their source Return “I don’t know” when confidence is too low Should guardrails be part of evaluation, inference, or both? Production monitoring Once this is live, what should we log for every AI request? For example: User question → retrieved context → generated SQL/tool calls → query result → final response → model → prompt version → latency → tokens → cost → evaluation scores → user feedback Is that roughly the right model? Regression testing When we change: System prompt Model Retrieval strategy SQL generation logic Tools Temperature Data sources how do you automatically run the benchmark again and determine whether the new version is actually better? Do you set minimum score thresholds before allowing something to deploy? Human feedback How are people incorporating thumbs-up/down or analyst review into their evaluation datasets? Do production failures automatically become new benchmark cases? Making answers more precise This is ultimately my main goal. If our AI currently gives an answer that is “mostly correct,” what is the systematic process for figuring out why it isn’t fully correct? Is the best workflow something like: Production traces → identify failure → categorize failure → add to benchmark → improve retrieval/prompt/tool → run eval → compare against baseline → deploy → monitor Or is there a better approach? I’m especially interested in what the ideal Databricks-native architecture looks like: User Question → Agent / LLM → Retrieval / SQL / Tools → Databricks data → Response → MLflow tracing → Automated evaluators → Benchmark dataset → Regression testing → Production monitoring If you’ve implemented something like this, I’d love to know what you would build first, second, third, etc. Even a practical example like: Week 1: create benchmark Week 2: add tracing + scorers Week 3: add regression tests Week 4: add guardr […truncated]
AI-Enabled Advisory Services for Higher Education
Databricks has introduced a GenAI-powered solution that automates and scales the quality review of higher education advisory calls by transcribing conversations, scoring performance against institutional rubrics, and surfacing insights. The blog post demonstrates this architecture on a single, governed platform, providing code notebooks and a Genie space for natural language data exploration.
Solution Accelerator Series | Building Common Sense Product Recommendations With LLMs
Guide to Agentic Systems and AI Agents
Agentic AI systems are autonomous software platforms that perceive, reason, execute multi-step tasks, and learn with minimal human intervention, unlike traditional generative models. These systems use LLMs as reasoning engines with external tools and memory to complete complex workflows, with enterprise adoption spanning customer service to financial risk.
End-to-End RAG Workflow: How Retrieval Augmented Generation Works
Databricks now offers a five-stage RAG workflow for connecting LLMs to external knowledge bases, enabling accurate, domain-specific answers without model retraining. Production RAG requires careful selection of embedding models, vector database indexing, chunking strategies, and hybrid search, with independent evaluation of retrieval precision and generation faithfulness.
Solution Accelerator Series | Building a Chatbot With Large Language Models (LLMs)
What is document AI?
Document AI transforms messy, high-volume documents into structured data for downstream systems, offering value beyond just faster processing. While generative AI makes it more adaptable for summarization and extraction, accuracy still relies on validation and human review, with governance becoming central due to sensitive data.
How Ecolab rebuilt retail intelligence on Databricks and Anthropic Claude
Ecolab rebuilt retail intelligence on Databricks and Anthropic Claude, converting 700-page FDA manuals into real-time answers for frontline staff using Foundation Model APIs and cutting compliance report compilation from two weeks to under two minutes. The solution, a native Databricks App with Lakebase Postgres and Unity Catalog, unifies nine siloed data sources and employs a multi-agent orchestration framework with Judge LLMs and MLflow tracing for personalized, continuously refined intelligence.
AI Serving Platform That Adapts to Your Model
Databricks now offers a fully managed AI serving platform that automatically adapts to your model's resource needs, from scikit-learn to 70B LLMs, without manual configuration. This results in up to 90% lower infrastructure costs and <10ms p99 latency overhead for customers migrating from self-managed stacks.
Announcing the Databricks storage ecosystem: Governing the enterprise data estate, wherever it lives
The Databricks Storage Ecosystem now natively connects hybrid and on-premises storage platforms to Databricks via OpenSharing, enabling centralized data governance and GenAI scaling across your entire hybrid infrastructure. Run Databricks Serverless Compute, Genie, and LLMs directly on your on-premises datasets with a zero-copy architecture, instantly turning isolated data into active, AI-ready assets.
Solution Accelerator Series | Large Language Models (LLMs) for Customer Service Analytics
Accelerating LLM Inference with Prompt Caching for Open‑Source Models on Databricks
Databricks now supports prompt caching for open-source models across all workloads, automatically accelerating LLM inference by reusing repeated prompt prefixes. This feature boosts throughput by 2.5x and reduces P50 latency by 3x for models like GPT-OSS, with no setup required.
LLM Vs AI: A Practical Guide to Differences, Use Cases, and Tools
LLMs are a subset of AI, and this guide clarifies their practical differences, use cases, and tools. Understand how LLMs fit into the broader AI landscape and what that means for your Databricks workflows.
AI Applications: Tools, Use Cases, and Platforms
AI applications span four capability tiers, each with distinct data requirements and evaluation frameworks, and enterprise deployments often stall due to inadequate data infrastructure. Production-grade model development, from prompt engineering to pretraining, is increasingly accessible with open-source LLMs, but requires pre-built governance and monitoring infrastructure for successful deployment at scale.
Are LLM agents good at join order optimization?
LLM agents can improve Databricks join order optimization, achieving 1.3x latency reduction in 80% of cases by reasoning through runtime statistics. This prototype demonstrates LLM agents' potential to act as data-driven DBAs, addressing cardinality misestimation challenges in complex SQL queries.
Introducing MLflow AI Gateway: Governed, Observable Access to LLMs
MLflow AI Gateway provides a single, secure endpoint for all LLM providers, complete with usage tracking and native tracing. This new feature offers governed, observable access to LLMs for Databricks practitioners.
MemAlign: Building Better LLM Judges From Human Feedback With Scalable Memory
MemAlign, a new framework for aligning LLMs with human feedback, is now available, offering competitive or better quality than state-of-the-art prompt optimizers at significantly lower cost and latency. It achieves this through a lightweight dual-memory system, making it a valuable tool for building better LLM judges.
Introducing DeepEval, RAGAS, and Phoenix Judges in MLflow
MLflow has integrated over 50 LLM-as-a-judge metrics from DeepEval, RAGAS, and Arize Phoenix directly into the MLflow Scorer API. Practitioners can now run and compare these third-party evaluations alongside native MLflow judges in a single API call and unified visual UI to debug and evaluate RAG pipelines and agents.
Unity Catalog AI 0.2.0
Unity Catalog AI 0.2.0 adds Gemini and LiteLLM integrations for using UC functions as AI tools and introduces function wrapping APIs with dependency management support. AutoGen integration now requires version 0.4.x and Databricks client now only supports serverless endpoints.