Databricks AI Search

Release v0.191.0 adds SDK methods for AgentKit memory and session store permissions, budget spend status queries, feature engineering materialization sets, and starting or stopping vector search endpoints. It also adds external spend inclusion for budgets, unhealthy-table notifications for data quality, relative error options for approximate count-distinct and percentile functions, and table trigger conditions and timing controls.

Building live intelligence: Databricks AI Search supports Streaming Tables, Materialized Views, and Managed tables using Iceberg

AI Search in Databricks now natively supports Streaming Tables, Materialized Views, and Managed tables using Iceberg v3+, in addition to Delta tables. You can run the same AI Search workflow from the UI or API across these sources without building separate synchronization pipelines, supporting real-time intelligence, personalized experiences, and cross-engine AI applications.

Release v0.170.0 adds startEndpoint() and stopEndpoint() methods for vector search endpoints, along with new STARTING, STOPPING, and STOPPED status values. It also adds OneDrive ingestion support for pipelines, an includeExternalSpend option for budgets, and new relativeError, TableTrigger, and regex fields.

This release adds start and stop methods for workspace-level vector search endpoints, along with new starting, stopping, and stopped status values. It also adds OneDrive as a pipeline ingestion source, an include_external_spend option for budgets, regex filtering for identity visibility, trigger conditions and timing controls for table triggers, and a relative_error parameter for approximate count-distinct and percentile functions.

NEAREST BY Join: Scaling Vector Search in Databricks Runtime

Databricks Runtime now features NEAREST BY, a new SQL join that executes exact or approximate batch vector searches directly against your Lakehouse data. Powered by a fused Photon operator with a custom blocked GEMM kernel, it turns standard liquid-clustered Delta tables into partition-pruned vector indexes with no external vector store to sync or manage.

Real-Time Retail Intelligence: Building E-Commerce Recommendations with Lakebase and AI Search on Databricks

A complete reference architecture is now available for building a real-time, multi-stage e-commerce recommendation engine on Databricks using AI Search for candidate retrieval, Lakebase for online feature serving, and Model Serving for low-latency inference. By unifying batch precomputation and real-time scoring on a single governed platform, this design replaces fragmented ML stacks to eliminate glue code and deliver end-to-end lineage from raw clickstream to production predictions.

Lakebase Search: State-of-the-art full text and vector search for Postgres

Lakebase Postgres now includes a built-in search engine, generally available on AWS and Azure, that enables native semantic, keyword, and hybrid search directly alongside operational data without separate ETL pipelines. Featuring a serverless architecture that scales to zero and decouples storage from compute, it delivers twice the throughput at a quarter of the cost of cloud Postgres with pgvector on 100-million-vector benchmarks.

Direct deployment state version 3 is a breaking change requiring Databricks CLI v1.8.0 or later. New capabilities include Docker credential helpers for Databricks Artifact Registry, AI Gateway service support in bundles, file-change job run triggers, and bundle improvements for resource reference handling, deployment reporting, and state management.

The Terraform deployment engine is now deprecated; migrate to the direct deployment engine for continued support. AI runtime tasks now automatically package local directories into tarballs during deployment, and the direct engine fixes several convergence issues with empty grants, webhook notification ordering, pipeline configuration, and vector search index creation.

SSH connect now supports specifying serverless usage policies via --usage-policy-id, and bundle deploy/destroy are more robust against transient app states and full workspaces. Bundle validation is stricter for grants while empty field values are now dropped, fixing deployment failures and spurious permission drift.

From prototype to production: High QPS for Databricks AI Search

High QPS for Databricks AI Search is now generally available, enabling endpoints to scale to thousands of queries per second with a single configuration parameter. Teams can support peak production traffic for real-time search, personalization, and entity resolution without managing replicas or load balancers, backed by built-in UI observability for QPS, latency, and health.

Scaling document classification to 100k+ labels

Databricks practitioners can scale document classification to over 100,000 labels by pairing SQL-native vector search with the AI Classify function to retrieve and select from a shortlist of candidate labels. Across three benchmarks, this hybrid approach beat the best cost-efficient frontier model by five points of accuracy at roughly a hundredth of the token cost.

Building a soccer coaching app on Databricks

Coach's Corner is a Databricks App that processes 25 fps match tracking data into a sub-second 2D/3D tactical bench with replays, event analytics, a scout chat, and an opponent-dossier agent. The end-to-end solution is powered entirely on the Databricks platform, utilizing Lakeflow pipelines for data refinement, DBSQL and Lakebase for rapid querying, and Unity Catalog-governed AI tools like Genie, Vector Search, and MLflow tracing.

End-to-End RAG Workflow: How Retrieval Augmented Generation Works

Databricks now offers a five-stage RAG workflow for connecting LLMs to external knowledge bases, enabling accurate, domain-specific answers without model retraining. Production RAG requires careful selection of embedding models, vector database indexing, chunking strategies, and hybrid search, with independent evaluation of retrieval precision and generation faithfulness.

What is Vector Search?

Vector search retrieves information based on meaning and context using embeddings, solving keyword-only search limitations for use cases like RAG and recommendations. Databricks AI Search adds reranking, metadata filtering, and automated index updates to improve relevance and simplify operations for production systems.

The CLI improves SSH error reporting with server logs, extends timeouts to 45 minutes for GPU accelerators, fixes environment variable handling to prevent default profile fallback, and corrects Cursor integration hanging. Bundle deployments prevent WAL corruption after failed deploys, fix resource references used multiple times within the same field, add postgres_roles and postgres_databases resources for Lakebase branches, and eliminate spurious recreates when names are backend-normalized.

This release adds new Postgres data API methods to the workspace client, introduces Dynamics 365 as a catalog connection type, and adds capacity reservation and sync configuration fields across compute, database, and vector search services. Additionally, the resource ID field in bundle deployment operations is no longer required, which represents a breaking change.

This release fixes an issue where tag policy and assignment operations failed for hierarchical tag keys containing slashes. It also introduces a breaking change by making the ResourceId field optional in bundledeployments.Operation and adds support for Dynamics365 connection types and EndpointId in vector search indexes.

The direct deployment engine is now generally available and the default for new bundle deployments, while existing deployments continue using their current engine. The CLI adds a quickstart command, version checking, authentication verification fixes for account console profiles, and a new genie_spaces bundle resource for managing Databricks Genie spaces.

This release introduces the AI Search and Bundle Deployments services to the workspace client, while removing the legacy bundle service and package. Additionally, it adds new configuration fields across multiple services, including custom retention hours for catalogs and schemas, auto-clustering options for pipelines, and facet filtering for vector search queries.

The Databricks Python SDK v0.116.0 adds new workspace services for AI Search and Bundle Deployments, along with configuration options for Vector Search facets, Delta Live Tables auto-clustering, and custom catalog retention hours. This release also introduces a breaking change by removing the legacy bundle workspace service and package.

Databricks Asset Bundles now includes a select flag in the direct deployment engine to plan and deploy specific resource subsets, along with support for Terraform references and automatic transient HTTP retries. The experimental open command expands to support additional workspace resources such as volumes and vector search endpoints, while notebook task paths now correctly preserve Lakeflow Designer files.

Bring Databricks into Kiro IDE with the AI Dev Kit Power

The Databricks AI Dev Kit Power now offers a one-click setup to integrate Kiro IDE with the full Databricks platform, providing AI-assisted development grounded in your workspace's Unity Catalog metadata. This new path, alongside a lighter PAT-based option, ensures your AI assistant writes SQL with actual columns and respects all row, column, and tag-based grants.

The provider adds service principal Git credential management via principal_id on databricks_git_credential and enables permission management for Agent Bricks resources. Key fixes include metastore external_access_enabled now properly sent in PATCH requests, vector search index timeout increased to 75 minutes and made configurable, and workspace_id now accepting connection IDs alongside numeric workspace IDs.

The error reported when a direct-only resource (catalogs, external locations, vector search endpoints) is used with the terraform engine now also suggests setting bundle.engine: direct in databricks.yml, in addition to the DATABRICKS_BUNDLE_ENGINE environment variable (#5295).

This release introduces a breaking change that renames the min_qps configuration and CLI flag to target_qps for vector search endpoints, and it stops applying bundle name prefixes to these endpoints. Additionally, it improves the interactive authentication profile pickers, enhances OS keyring token storage behavior, and fixes bundle bugs related to nested notebook directory generation and multi-profile authentication propagation.

This release fixes an issue where the Databricks CLI --profile fallback was broken. It also introduces new API methods for workspaceClient.supervisorAgents() and workspaceClient.vectorSearchEndpoints(), along with several breaking changes related to Example and Tool fields, and minQps in Vector Search endpoints.

This release introduces new methods for managing supervisor agents and vector search endpoints, along with several new fields for connector options, ingestion sources, and customer-managed keys. Breaking changes include the removal of min_qps fields from vector search endpoint configurations and making guidelines and description fields optional in certain services.

The CLI now supports a --limit flag for paginated list commands and caches host metadata lookups for faster repeated invocations. Bundles gain support for Vector Search Endpoints and prompt before destroying Lakebase resources.

Benchmark Your Way to Better RAG and Agents:Tuning Vector Search with MLflow

Tuning vector search for RAG and AI agents requires isolating one configuration knob at a time against a fixed evaluation set rather than relying on default settings. Using MLflow retriever evaluation to track metrics like precision@k and nDCG@k lets teams systematically measure and compare the impact of embedding models, chunk sizes, and rerankers side by side.