Cost Optimization
Recent items mentioning Cost Optimization across the Databricks ecosystem — releases, news, videos, and community Q&A. Updated hourly.
Lakebase Postgres cuts operational spend when teams sync only working subsets of Lakehouse data, align sync modes with freshness needs, and leverage serverless scale-to-zero alongside cache-fitted compute 2. In parallel, community practitioners are targeting idle compute by monitoring all-purpose clusters via the API 1 and evaluating third-party cost optimization tools 3.
Written October 2026 from the 3 most recent items mentioning Cost Optimization at that time. It refreshes when this topic next has enough new material. Click any [N] to jump to the source.
Databricks cost optimization: monitoring all-purpose clusters with the API
A practical guide to cost optimization with Lakebase Postgres
Lakebase Postgres delivers low, predictable costs when you sync only the working subset of Lakehouse data, match sync modes to actual freshness needs, and right-size compute so hot data fits in cache. Combining these optimization practices with serverless scale-to-zero, tuned PITR, and shared storage ensures strong performance and availability without paying for unused resources.
Databricks Cost Optimisation 3rd Party Tools
Does anyone have experience with third party optimisation tools like espresso.ai or Zipher? Espresso has an enticing pricing model but I wonder if dumping a whole heap of logs and system stats into Claude would get me 80% of the way there. submitted by /u/Agitated-Western1788 [link] [comments]
NBCUniversal’s Seamless Migration: Unlocking Scalable Analytics with Databricks
NBCUniversal cut costs by 30% by moving from a slot-based reservation model to Databricks' dedicated job compute, letting parallel pipelines scale independently and consistently hit SLA targets. Partner EXL executed the migration in phases with custom accelerators for code conversion, data migration, and automated validation, giving NBCUniversal a unified platform for ML development, real-time analytics, and collaborative data engineering.
The Open-Weight Revolution: A Game Changer for Our LLM Cost Optimization Odyssey
Snowflake to Databricks Migration in 12 weeks and cut cost per run by ~77%. AMA.
Lovelytics wrapped up a Snowflake-to-Databricks migration; 847 DBT models, 35 Info Mart tables, \~77% lower cost per run on a 2XL warehouse. **TL;DR What helped:** * Treated the migration as engineering, not translation. Each dbt model was tested in isolation, not just row counts vs Snowflake. * Routing macro to resolve cross-layer references at runtime, so the same codebase could read from Snowflake, federated Snowflake, and Unity Catalog without forking logic. * Dual model trees in one repo, which let the migration stay in lockstep with live Snowflake changes. * Script-generated wave selectors enabled parallel builds while preserving dependency order. * Used reference-slice validation subsets vs. waiting on full mart refreshes. **TL;DR Cost reduction:** * Reworked joins to use narrow staging dimensions instead of wide marts where possible. * Added incremental predicates to reduce MERGE target scans. * Split wide models into parallel sub-models where the dependency graph allowed it. * Copied static reference data into Delta instead of repeatedly reading it through federation. * Loaded static copies into Delta rather than reading via federation (predicate pushdown is poor). Happy to go into the gotchas: HASH() not being portable, Snowflake MERGE tolerating duplicate keys that Delta doesn't, NULL ordering, and timestamp handling. AMA [Full Blog Post](https://community.databricks.com/t5/technical-blog/partner-blog-847-models-12-weeks-77-less-inside-r1-s-snowflake/ba-p/157284)
Scaling for MHHS: how Octopus Energy achieved a 50x cost reduction in margin data engineering
Octopus Energy achieved a 50x cost reduction in their margin data engineering pipelines by re-architecting on Databricks for UK MHHS regulation. They leveraged Delta Lake Change Data Feed and Databricks Serverless to process 48x more data at a fraction of the original cost, improving freshness from weekly to daily.
[ebook] The Guide to Databricks Cost Optimization
[](https://www.reddit.com/r/databricks/?f=flair_name%3A%22General%22)Most cost optimization guides tell you to "right-size your clusters." Cool. Which ones? By how much? This guide actually answers that. Five core strategies, written by a Principal Data Engineer who's built production Databricks environments, for the people who own the bill. Download it for free (Unlike Your Databricks Bill.) [https://c.select.dev/guide-databricks-cost-optimization?utm\_source=reddit&utm\_medium=organic&utm\_campaign=databricks\_26](https://c.select.dev/guide-databricks-cost-optimization?utm_source=reddit&utm_medium=organic&utm_campaign=databricks_26)
[ebook] The No-BS Guide to Databricks Cost Optimization
Most cost optimization guides tell you to "right-size your clusters." Cool. Which ones? By how much? This guide actually answers that. Five core strategies, written by a Principal Data Engineer who's built production Databricks environments, for the people who own the bill. Download it for free (Unlike Your Databricks Bill.) [https://c.select.dev/no-bs-guide-to-databricks-cost-optimization?utm\_source=reddit&utm\_medium=organic&utm\_campaign=databricks\_26](https://c.select.dev/no-bs-guide-to-databricks-cost-optimization?utm_source=reddit&utm_medium=organic&utm_campaign=databricks_26) [](https://www.reddit.com/submit/?source_id=t3_1tdyfhd&composer_entry=crosspost_prompt)
How nOps Rebuilt Their Cloud Optimization Platform on Databricks Lakebase, and Why Other ISVs Should Too
nOps migrated its production cloud cost optimization platform to Databricks Lakebase, replacing a separate relational database and custom ETL pipelines with managed PostgreSQL directly integrated into the Lakehouse. The move provides an architectural blueprint for ISVs looking to eliminate synchronization overhead and unify operational and analytical data under a single governance model.
NewsDatabricks Apps vs Model Serving: Authentication, Cost, and Performance Compared
Databricks Apps are now the recommended first choice for deploying agents due to their flexibility in handling full-stack applications with multiple components, offering faster iteration and local testing compared to Model Serving. Model Serving remains suitable for use cases prioritizing high QPS, governance features like AI Gateway, inference tables, and guardrails, or when scaling to zero is acceptable for cost optimization.
NewsGPU Accelerated Spark Connect
This video demonstrates how to accelerate Spark Connect using GPUs for both Spark SQL and ML workloads. It details the architecture, deployment, and benchmark results showing significant speedups and cost savings compared to CPU-only execution.
NewsSaving Millions From Millions: Navigating Towards Cost-Efficiency in Pinterest's Spark Jobs
Pinterest built a three-layer cost optimization system for Spark jobs using observability platforms (collecting real-time metrics and event logs), platform innovations (Apache Celeborn remote shuffle service and Kubernetes burst-aware memory allocation), and fine-grained management tools. Production deployments achieved 30-50% cost reductions per job through resource tuning while improving cluster utilization.
NewsInnovating Retail Data: Unilever’s Transformation with Databricks DLT
Unilever replaced its legacy data pipelines with Databricks Delta Live Tables using a medallion architecture, enabling serverless streaming, automated data quality checks, and unified governance through Unity Catalog. The migration delivered 25% infrastructure cost reduction, 200-500% faster data processing, and real-time analytics at scale.
NewsFrom Code to Insights: Leveraging Advanced Infrastructure and AI Capabilities
Insulet migrated its data infrastructure from Azure to Databricks, implementing a data mesh architecture with medallion layers, Unity Catalog governance, and Delta Sharing for partner data access. The company achieved real-time analytics, GXP-compliant deployments, and FinOps cost optimization, demonstrating how to balance innovation with operational efficiency.
NewsSponsored: Kyvos | Analytics 100x Faster Lowest Cost w/ Kyvos & Databricks, Even on Trillions Rows
Kyvos is a universal semantic layer and analytics acceleration platform that delivers sub-second query performance on massive datasets with high concurrency. The platform utilizes smart aggregation technology to pre-aggregate data efficiently on disk, allowing companies to replace complex legacy cubes and aggregate tables with a single, highly scalable object.
TutorialsDatabricks Cost Management: Tips and Tools to Stay Under Budget
Databricks cost management involves tracking usage with billing system tables and cloud tools while optimizing expenses through proper compute sizing and automated job clusters. The platform also provides methods like cluster policies, custom tagging, and alerts to control budgets and attribute costs across different business units.
TutorialsDBA Perspective—Optimizing Performance Table-by-Table
The video explains how to scale database administration and optimize lakehouse performance by analyzing metadata to identify tables with too many small files or too few files. It then demonstrates how to resolve these issues by using Databricks runtime updates, table optimizations, Z-ordering, and proper retention and cleanup policies.
EventsMedia and Entertainment Experience at Data + AI Summit 2022
The Communications, Media and Entertainment program at Data + AI Summit 2022 is curated in one place for data teams and executives. It includes an industry forum with HBO, LaLiga, Condé Nast and others, a partner lounge, and over 10 technical talks from companies such as AT&T, Netflix and HuuugeGames.
NewsAccelerate Spark-Based Analytics by Minimizing the Barriers of Large, Complex Hadoop Migration
WANdisco and Avanade present a methodology for migrating petabytes of data from on-premises Hadoop to Azure, removing a common blocker to Spark-based analytics. The approach minimizes migration risk and targets petabyte-scale migration with zero downtime, zero data loss and faster time-to-value.
NewsNLP-Focused Applied ML at Scale for Global Fleet Analytics at ExxonMobil
ExxonMobil uses Databricks to ingest structured and unstructured equipment maintenance logs for its global fleet from legacy systems and apply NLP to millions of records. The resulting insights support outlier identification, capacity planning, prioritization of cost reduction opportunities and discovery for cross-functional teams.
NewsFrugalML: Using ML APIs More Accurately and Cheaply
FrugalML learns where paid ML prediction APIs are strong or weak and picks the best sequential strategy for using them within a budget. Experiments with APIs from Google, Microsoft, Amazon and others show up to 90% lower cost at the best single API's accuracy, or up to 5% better accuracy at its cost.