Kubernetes
Recent items mentioning Kubernetes across the Databricks ecosystem — releases, news, videos, and community Q&A. Updated hourly.
Data teams are actively navigating migrations from managed Databricks Jobs to self-hosted Spark jobs on Kubernetes 2. Concurrently, practitioners deploying ~1GB+ embedding models are encountering frequent OOMKilled memory failures under tight 2Gi Kubernetes pod limits 1.
Written October 2026 from the 3 most recent items mentioning Kubernetes at that time. It refreshes when this topic next has enough new material. Click any [N] to jump to the source.
UnityCatalog 0.7.0
Unity Catalog 0.7.0 enables native Apache Iceberg REST catalog support by default, allowing external query engines to read, write, and manage Iceberg tables directly. Dropping managed tables, volumes, or models now automatically and permanently purges underlying cloud storage files after a default seven-day delay.
How to fit ~1GB+ embedding model into a 2Gi Kubernetes pod? Getting OOMKilled
Hi, Deploying a FastAPI to Kubernetes that uses a multilingual sentence-transformers embedding model (ONNX backend, CPU only). My source data lives in a Delta table in Databricks, and the app reads from it and generates embeddings. The app takes user input (text) at embeds it, and compares it against stored embeddings for semantic similarity. So at least the query embedding has to happen live. Pod resources: - CPU: 1 request / 2 limit - Memory: 2Gi request = 2Gi limit (platform policy requires memory request and limit to be 1:1) Current docker setup: - Multi-stage Docker build (python:3.11-slim) - CPU-only PyTorch - The model is downloaded at build time, so it's baked into the image Image breakdown: - HF model cache: ~1.1 GB - torch: ~650 MB - pyarrow, scipy, transformers, pandas: ~100–150 MB each - Plus sklearn, onnxruntime, mlflow, and others The pod gets OOMKilled at startup or shortly after. With a ~1GB model, torch, pandas/pyarrow, and the ONNX runtime session all in one process, I think I'm just over 2Gi. What I'm trying to figure out on where do you store large models in production? Would love to hear what setups have worked for you. Thanks! submitted by /u/runningnozone [link] [comments]
Databricks Jobs to Kubernetes Spark Jobs migration
I'm discussing some architectural changes with my data team, and one of them is about cutting costs. The idea is to migrate Databricks Jobs to Spark jobs on Kubernetes. We've done a PoC and it works well. But despite the clear cost savings, I'm getting pushback from management and they're resisting the change. Can someone explain the reason behind this? It seems clear that you pay double for running jobs in Databricks, and you could save a lot of money. Yes, doing this means we bypass Unity Catalog, but we could convert only the Bronze and Silver layers. submitted by /u/worldpwn [link] [comments]
MLflow 3.16.0
MLflow 3.16.0 makes the redesigned trace explorer the default interface, introducing natural-language custom trace views via the MLflow Assistant, session grouping for multi-turn conversations, and span links. The update also adds Unity Catalog model service support for built-in evaluation judges, per-user AI Gateway budget policies, and fail-closed authorization by default.
Data Engineers, what does your actual day-to-day work look like? And what should I learn next?
I’m currently trying to transition deeper into Data Engineering and would really appreciate some perspective from people who are already working in the field. I have 1.3 yrs experience as a Junior Python Developer. What I want to do is slowly transform into a Data Engineer. How would you suggest my choice? Basically what I do is make web scraping scripts to get the data from web and give the data in excels. Our company is currently not using git or CI/CD or anything like that. The problem I’m running into is that when I look at Data Engineering jobs on Naukri, LinkedIn, etc., the requirements seem endless. One job asks for Python, SQL, Airflow and AWS; another wants Spark, Kafka and Databricks; another wants Snowflake, dbt, Terraform, Kubernetes, CI/CD, etc. It becomes difficult to understand what I should actually prioritize. So I’d like to hear from people who are actually working as Data Engineers . What does your day-to-day work look like? What kind of problems do you solve, what technologies do you use regularly, and which skills have turned out to be genuinely important in your job? More importantly, based on my current experience, what would you suggest I improve or learn next to become a stronger candidate for Data Engineering roles? Are there any gaps that you think I should focus on, or technologies/concepts that are worth learning through projects rather than just studying theoretically? I’m not really looking for a generic “learn SQL → Python → Spark → AWS” roadmap. I’m more interested in understanding the reality of the job and getting advice from people who have actually gone through the transition. If you’re a Data Engineer with 1–5+ years of experience , I’d especially appreciate your perspective. Even a short description of what you work on and what you wish you had learned earlier would be extremely helpful. Thanks in advance! submitted by /u/Stunning-Space8032 [link] [comments]
How Databricks Uses AI to Accelerate Incident Investigation
Databricks' AI SRE now handles over 2,000 daily investigations across 150+ teams, helping engineers troubleshoot 100s of microservices spread across 1,500 Kubernetes clusters in 70+ regions and three clouds. Rather than relying on black-box reasoning, the system uses composable agentic runbooks and ties every diagnostic recommendation to verifiable raw evidence, embedding a context-first approach into its design.
MLflow 3.15.0 introduces an MCP Registry for registering and sharing Model Context Protocol servers, enhances the Assistant with multi-provider LLM support and per-session token usage tracking, and enables proxy-less artifact transfers via presigned URLs to reduce server load and timeouts on large files. Additional improvements include sharable Runs table views, multi-modal image attachments for LLM judges to evaluate vision tasks, and numerous bug fixes across tracing, evaluation, gateway, and UI components.
Show HN: Ingestr CDC – open-source CDC replication in Go
Hi all, this is Burak, one of the founders at Bruin. I created ingestr to make data ingestion easy. ingestr is a CLI tool that can ingest data from 130+ sources. I have shared it on HN after our Go rewrite as well, which made it the fastest ingestion tool in the space. However, ingestr had always been a batch tool. I am personally a big fan of batch workloads due to their simplicity and have built ingestr around that assumption as well; however, over time, the cracks started to show when we started working with larger orgs. Turns out there are some scenarios where CDC proves beneficial: - For legacy systems where it is not possible to introduce cursor columns due to technical, but mostly organizational, concerns, it becomes impractical to deploy batch pipelines. - For systems that do not have a way to reliably know the update timestamp, also due to legacy reasons. Think usecases where the columns are updated without the timestamp being updated. - For hard deletes. Even though I do believe there are ways to solve each of these, it ended up putting us in a disadvantage, and we decided to build the CDC connectors instead. ingestr CDC works in two modes now: batch (bad name, I know) and stream. The batch mode is reading the changelog entries from the last load until the starting timestamp, and the streaming mode keeps reading and landing them to the destination databases. ingestr has a few advantages compared to a more traditional debezium + kafka setup: - it's a standalone Go binary and does not require any additional infra. - it has very low resource consumption, ~100MB baseline. - it can run on your own computer during development, and can be converted into a streaming prod deployment when it is ready. - supports 20+ destinations already, primarily analytical platforms like snowflake, databricks, generic iceberg destinations, etc. i would love to hear any feedback on what we could do to make it easier for cdc workloads! https://github.com/bruin-data/ingestr --- top comments --- [mercutio93] Man I have had a lot of gripes with debezium... the setup alone in a kubernetes cluster with strimzi kafka connect... And dealing with the kafka connect errors and fixing replication slots... ehhh was painful. Does this tool make it any well easier. One problem I expirienced in production systems with staging doubles is that sometimes we do database restores from production onto staging and that makes the sinks and sources go haywire... does ingestr handle this well? Is support for kubernetes/helm anywhere in the pipeline
Can Databricks Jobs Run on Kubernetes Clusters?
MLflow 3.13.0 introduces Role-Based Access Control with Admin UI, automatic trace archival to S3, and one-click observability for Claude Code and other coding agents. Breaking changes include a redesigned permission system (legacy APIs removed), MLServer removal from pyfunc serving, and requirement for MLFLOW_ALLOW_FILE_STORE=true flag for local file-based stores.
MLflow 3.13.0rc0 completely overhauls Role-Based Access Control with unified permission APIs and a new Admin UI, and integrates Claude Code, OpenAI, Ollama, and OpenClaw as native assistant providers in the AI Gateway. The release adds trace archival with seamless retrieval, GenAI agent stress-testing, Kubernetes Helm chart support, and database replica routing for horizontal scaling.
MLflow 3.11.1 introduces AI-powered issue detection in traces, AI Gateway budget alerts and spending controls, trace graph visualization, native Databricks gateway provider, and pickle-free model serialization. TypeScript SDK packages are now @mlflow-scoped and LiteLLM is no longer required for GenAI evaluation.
Enterprise-Scale MLflow Operations and Security Practices at LY Corporation
LY Corporation deployed a managed MLflow architecture on Kubernetes handling over 600,000 daily requests across 40 services while maintaining strict data isolation. By leveraging Athenz sidecar proxies and SPIFFE-based mTLS for automated OAuth 2.0 authentication, the platform secures machine-to-machine communication from training pods without modifying open-source MLflow.
UnityCatalog 0.3.0
UnityCatalog 0.3.0 adds support for Spark 4.0 and Delta 4.0 with new credentials and external locations APIs enabling flexible external storage management. The release includes Kubernetes deployment via Helm charts and fixes for Delta time-travel SQL queries.
NewsSaving Millions From Millions: Navigating Towards Cost-Efficiency in Pinterest's Spark Jobs
Pinterest built a three-layer cost optimization system for Spark jobs using observability platforms (collecting real-time metrics and event logs), platform innovations (Apache Celeborn remote shuffle service and Kubernetes burst-aware memory allocation), and fine-grained management tools. Production deployments achieved 30-50% cost reductions per job through resource tuning while improving cluster utilization.
NewsInternet-Scale Analytics: Migrating a Mission Critical Product to the Cloud
Akamai migrated its high-volume internet security analytics platform from an on-premises Hadoop architecture to a cloud-native setup using Azure blob storage, Kafka, Kubernetes, and Databricks. The resulting architecture decouples raw data storage from streaming metadata via Kafka, utilizing custom shared libraries and sharding strategies to achieve low-latency ingestion, cost-effective scaling, and improved performance.
TutorialsHow To Make Apache Spark on Kubernetes Run Reliably on Spot Instances
This video covers strategies for running Apache Spark on Kubernetes reliably using spot instances, focusing on how to avoid spot interruptions and handle them gracefully when they occur.
NewsManaging Straggler Executors at Apache Spark 3.3
This video explores how Apache Spark 3.3 addresses the challenges of straggler executors and tasks that slow down streaming and batch workloads. It introduces a new feature called executor rolling and explains how built-in rolling policies and custom plugins help decommission poorly performing executors.
ReleasesApache Spark on Kubernetes—Lessons Learned from Launching Millions of Spark Executors
The video explains the design and implementation of Apple's self-service elastic analytics platform, built on Kubernetes to scale Apache Spark. It shares lessons learned and pitfalls encountered while supporting Spark at scale, focusing on orchestration, security, monitoring, and workload optimization.
NewsBuilding Metadata and Lineage Driven Pipelines on Kubernetes
This video discusses using the Kubeflow Pipelines framework on Kubernetes and how its integration with MLMD and Tekton can be leveraged for better metadata and artifact tracking, lineage, and pipeline execution. It also introduces Kubeflow Pipelines v2 features like intermediate representation and smart compilers to reduce dependency on backend architectures.
NewsDistributed Machine Learning at Lyft
Lyft built a distributed machine learning platform using Kubernetes, Spark, and Fugue to handle various data and compute sizes across the machine learning lifecycle. The platform addresses scalability and infrastructure challenges while improving user experience and cost monitoring.
NewsEfficient and Multi-Tenant Scheduling of Big Data and AI Workloads
The video discusses the challenges of big data and AI scheduling and how they can be addressed using a multi-tenant environment on Kubernetes with Apache YuniKorn. It details key features like queue management, workload queuing, and gang scheduling to optimize resource utilization and application performance in cloud-native infrastructures.
NewsHow socat and UNIX Pipes Can Help Data Integration
Containerized CLIs, one reading and one writing data, are reliably orchestrated on Kubernetes for data integration through a TCP-based process abstraction built with socat and UNIX pipes. Architectures for horizontally scaling these workflows are compared, and the approach powers all of Airbyte's Kubernetes deployments, syncing TBs of data daily.
NewsThe Rise of ZStandard: Apache Spark/Parquet/ORC/Avro
Zstandard, a fast compression algorithm, is traced through four Apache Spark use cases: shuffle file compression to cut local disk IO on K8s, event log compression where it replaced LZ4 as default, ORC and Parquet data files, and MapStatus serialization. The talk also looks at Avro support targeted for Spark 3.2.
NewsSuperworkflow of Graph Neural Networks with K8S and Fugue
Superworkflows combine different computing frameworks and environments in one machine learning workflow, using Fugue as an abstraction layer and Kubernetes to host containerized Spark and PyTorch environments. Node2Vec, a graph neural network, runs on graphs with hundreds of millions of vertices and edges as the worked example, with task partitioning and efficiency benchmarks.
NewsScaling your Data Pipelines with Apache Spark on Kubernetes
Rajesh Thallam and Sougata Biswas show how to run Apache Spark applications on Google Kubernetes Engine (GKE) and Google Cloud Dataproc, with data and ML pipelines orchestrated by managed Apache Airflow through Google Cloud Composer. Topics include key traits of Spark on Kubernetes, autoscaling, and a demo of Airflow-orchestrated analytics pipelines.
NewsIgnite A Revolution With Enterprise Apache Spark On Multi-Cloud Kubernetes
Enterprise Apache Spark on multi-cloud Kubernetes is presented as the way to modernize heavy data platforms in the final wave of digital transformation. The talk argues for combining Spark's capabilities with the cloud principles of modern applications, delivering the ideal user experience where enterprise data lives, on-premises.
NewsHybrid Apache Spark Architecture with YARN and Kubernetes
Lyft runs ETL and ML Spark workloads on Kubernetes with the Spark on k8s operator, but growing scale caused reliability issues with IP allocation, container images, IAM roles and the control plane. A hybrid architecture on Kubernetes and YARN supports continued growth, with a dynamic runtime controller for per-environment config overrides and easy switchover.
NewsMonitor Apache Spark 3 on Kubernetes using Metrics and Plugins
Apache Spark metrics and the improved plugin interface in Apache Spark 3 support data-driven performance troubleshooting for Spark on Kubernetes and cloud environments. Plugins extend collection to OS and container metrics such as CPU, I/O, memory and network, plus cloud filesystem access, feeding a performance dashboard that complements the Spark WebUI.
NewsInfrastructure Agnostic Machine Learning Workload Deployment
Running Spark and machine learning workloads as containerized Docker packages on Kubernetes offers an infrastructure-agnostic alternative to managed cloud Spark clusters, improving operational efficiency and reducing compute cost. It walks through porting the same pipelines from a laptop Kubernetes setup to on-prem or cloud environments by changing a few configurations.
NewsFugue: Unifying Spark and Non-Spark Ecosystems for Big Data Analytics
Fugue is an abstraction layer over Spark and non-Spark tools that represents ETL and machine learning pipelines end to end in a SQL-like language extendable with Python. The talk covers a K8S Spark environment for interactive, batch and streaming work, Kafka and Kinesis extensions, and unified interfaces over Spark ML, Scikit Learn, XGboost and Tensorflow.
NewsSimplify and Boost Spark 3 Deployments with Hypervisor-Native Kubernetes
A reference architecture integrates Spark 3, with Kubernetes as its scheduler, and VMware Pacific, an enterprise-grade Kubernetes natively integrated with the vSphere 7 hypervisor. Demos cover bringing up a hypervisor-native Kubernetes cluster, deploying Spark 3 and running machine learning tasks with its ML libraries, alongside vSphere and Kubernetes sizing guidance.
TutorialsDeploying Apache Spark Jobs on Kubernetes with Helm and Spark Operator
Scala Spark jobs deploy onto any Kubernetes environment using Helm and the Spark Operator, shown through a live coding demonstration. The approach makes deployments more scalable with less need for custom configuration, resulting in boilerplate-free, highly flexible deployments.
TutorialsDeploy and Serve Model from Azure Databricks onto Azure Machine Learning
A PySpark multi-class classifier trained on Azure Databricks predicts semantic types of tabular data, using PySpark UDFs for featurization and Random Forests tuned with PySpark MLlib. Azure Machine Learning registers and deploys the model onto Azure Kubernetes (AKS) with a PySpark runtime, serving it as HTTPS endpoints with key-based and token-based authentication.
NewsCloud-Native Apache Spark Scheduling with YuniKorn Scheduler
Lyft and Cloudera run Apache Spark on cloud-native, Kubernetes-based infrastructure for data engineering and machine learning. The open-source YuniKorn resource scheduler manages quotas, resource sharing and auto-scaling to schedule large-scale Spark jobs efficiently on Kubernetes in the cloud, with a look at the overall architecture.
NewsRunning Apache Spark Jobs Using Kubernetes
Running Spark on Kubernetes builds a single workflow for Spark-based data preparation and ML tasks, using Kubernetes for isolated workloads, resource limits and on-demand scaling. Yaron Haviv explains how this unifies analytics and data science on one cloud-native architecture without a separately managed big data cluster.
NewsEnabling Push Button Productization of AI Models
AI platforms for deploying ML models to production at scale use a fully automated continuous delivery process built on streaming microservice architectures with Docker containers and Kubernetes. Data scientists push model code that is built, tested, deployed and activated automatically, with monitoring, model health indicators and retraining to reduce total cost of ownership.
NewsRunning Apache Spark on Kubernetes: Best Practices and Pitfalls
Running Apache Spark on Kubernetes in a stable, performant, cost-efficient and secure way is covered through lessons from building Data Mechanics, a serverless Spark platform powered by Kubernetes. Topics include core setup, configuration for performance and resource sharing, dynamic allocation and cluster autoscaling, data I/O performance, monitoring and security, and current limitations.
NewsImproving Apache Spark Downscaling - Christopher Crosbie (Google) Ben Sidhom (Google)
Apache Spark downscaling becomes critical in serverless-like environments and with preemptible or spot instances, where frequent node removal can trigger repeated expensive recomputation. Google's Christopher Crosbie and Ben Sidhom explore approaches to improve scale-down on open source cluster managers such as YARN and Kubernetes, from job scheduling to block and shuffle placement.
NewsReliable Performance at Scale with Apache Spark on Kubernetes - Will Manning, Matthew Cheah Palantir
Palantir runs Apache Spark on Kubernetes, with its largest deployment averaging about 5 million Spark pods per day, and shares lessons from migrating off YARN, including shuffle performance, CPU limit throttling and missing dynamic allocation. The talk also covers Palantir's open-sourced Kubernetes scheduler extension for Spark workloads and a pluggable external shuffle service API.