Photon
Recent items mentioning Photon across the Databricks ecosystem — releases, news, videos, and community Q&A. Updated hourly.
Photon is now the execution engine behind two new Databricks Runtime features. NEAREST BY uses a fused Photon operator with a custom blocked GEMM kernel to run exact or approximate vector search as a SQL join on liquid-clustered Delta tables, with no external vector store to sync 2. The native IP functions, GA on DBR 18.3+, are also optimized in Photon and complete IP CIDR joins up to 3.1x faster and 6.4x cheaper than an alternative, without UDFs or regex 3.
Generated daily from the 3 most recent items mentioning Photon. Click any [N] to jump to the source.
Optimize serverless job performance with query metrics & performance insights
Hi folks, Databricks PM here - I wanted to share an update about serverless jobs observability since this is a topic I get asked about all the time. One of the hardest adjustments for users adopting serverless is changing your mindset from how you monitored classic jobs and investigating very low level details (think deep diving into the Spark UI) to understanding how you can monitor serverless jobs and have confidence in their performance. Query Profile already simplifies performance investigations for serverless jobs by providing in depth metrics on query execution. The addition of query performance insights (generally available since September) have made this even easier: Databricks flags queries that have optimization potential directly in the UI and you can take action via Genie Code by using the “Optimize” button which will propose optimizations directly on the asset (e.g. in your notebook for code optimization). Here is the list of insights that are currently covered: Query optimization insights: COVERAGE_FILTER_KEYS_CLUSTERING, COVERAGE_FILTER_KEYS_PARTITIONING, COVERAGE_PHOTON, EXPLODING_JOIN, FLOW_FULL_RECOMPUTE, REDUNDANT_AGGREGATION, REDUNDANT_JOIN, SELECTIVE_JOIN, WIDE_PROJECTION Data layout insights: CONCURRENT_WRITE, COVERAGE_STATS_DELTA, COVERAGE_STATS_OPTIMIZER, DATA_FILE_SIZE, DATA_SKEW, MANUAL_DATA_LAYOUT Compute and resource insights: DATA_SPILL, EXCESSIVE_QUEUE_TIME, IO_THROTTLING Applied accelerations: AUTO_LIQUID_CLUSTERING, HISTORY_BASED_JOIN_STRATEGY, SHORT_QUERY_PRIORITIZATION, METRIC_VIEW_MATERIALIZATION_MATCH Having this available only at the query level can make it cumbersome to figure out which queries are worth optimizing for your job, so we have brought these insights to the Jobs UI. So how can you try this out for your jobs? Open any serverless job that has a recent run ( You can either drill down and pick which query you want to optimize or you can use the Optimize button directly from the job run sidebar and let Databricks prioritize insights which are estimated to help achieve the most performance improvements for a given job. Timeline view Side note: we also roll up query metrics on data volume and number of queries on the job and task level now to help you quickly identify if a change in job performance can simply be attributed to a change in data volume. Limitation: the roll up of metrics and insights is currently capped at 100 queries You can check out our documentation to learn more. If you have tried it out already, I would love to hear any feedback! Are there any other insights or query performance metrics that you are missing? What other observability improvements would you like to see for serverless jobs? submitted by /u/brickster_123 [link] [comments]
NEAREST BY Join: Scaling Vector Search in Databricks Runtime
Databricks Runtime now features NEAREST BY, a new SQL join that executes exact or approximate batch vector searches directly against your Lakehouse data. Powered by a fused Photon operator with a custom blocked GEMM kernel, it turns standard liquid-clustered Delta tables into partition-pruned vector indexes with no external vector store to sync or manage.
IP Functions are Generally Available, bringing high-performance network analytics to the Lakehouse
Native IP functions are now generally available on Databricks Runtime 18.3+, providing built-in support to parse, validate, canonicalize, and join IPv4 and IPv6 addresses and CIDR blocks without UDFs or regex. Optimized in Photon across SQL, PySpark, and Scala, these native functions run demanding network workloads in seconds and complete IP CIDR joins up to 3.1x faster and 6.4x cheaper than another leading cloud data warehouse.
Photon enabled but a large share of the plan is falling back, cost up and runtime flat
Photon enabled but a large share of the plan is falling back, cost up and runtime flat
Does enabling Photon improve performance while lowering compute costs?
Does Photon support CSV?
Ok so I know that the docs state that CSV is supported. But when trying to read a very standard CSV file I get poorer than expected performance and in the Spark UI I see that it's not using a Photon scan operator, just regular Scan CSV followed by a Row to Columnar conversion operator. https://preview.redd.it/1amu93oqqslh1.png?width=260&format=png&auto=webp&s=4ca1c1ffec6cb6f6cd484b4645b0b5d7c5f6be86 I'm not running anything complex: df = ( spark.read .format("csv") .option("header", "true") .schema(schema) .load(csv_path) ) df.write.mode("overwrite").saveAsTable("...") I also checked reading the same CSV and a difference CSV via DB SQL (using `COPY INTO`) and see low task time spent in Photon + a row to columnar operator. https://preview.redd.it/g1tlotk8rslh1.png?width=940&format=png&auto=webp&s=f3d2b22ebab4f716c5b6831d96305c082e1126a8 DBR 19 + Serverless SQL Warehouse (current) Can anyone explain whether or not CSV is supported and in what conditions? This was quite a surprising find as I assumed that Photon supported pretty much everything. submitted by /u/Common_Jaguar474 [link] [comments]
Detecting Photon fallback in-cluster + safe right-sizing from system tables
TutorialsMaster Databricks 2nd Ed: Lesson 3 - Understanding Clusters
This video explains Databricks clusters, detailing their components like driver and worker nodes, configuration options such as autoscaling and Photon acceleration, and how to create and manage them within Azure. It also covers common interview questions related to cluster sizing and performance tuning, emphasizing that Databricks clusters are essentially Spark clusters enhanced with the Databricks runtime for cloud environments.
TutorialsTop Performance and Cost Optimizations for DLT
Performance optimization for DLT involves identifying CPU, memory, and IO bottlenecks and adjusting instance types, partition sizes, and autoscaling strategies to balance cost and speed. Databricks offers built-in optimizations like liquid clustering, Photon vectorized execution, microbatch pipelining, and Enzyme to reduce workload latency and cost automatically.
NewsTransforming Data Pipeline Management With a Targeted Proof of Concept
Capital One's credit line increase program required processing trillions of records across 160 customer behavior and financial features using data from multiple sources, but their existing compute infrastructure was timing out. By migrating to Databricks' Photon processing engine, they achieved an 80% reduction in time and cost per job, completing operations 60 times faster.
TutorialsMaster Databricks and Apache Spark Step by Step: Series Update - What's Changed?
The video reviews major updates and new features added to Databricks and Apache Spark since the original training series was published. It highlights key advancements such as Delta Lake lakehouses, Unity catalog governance, Mosaic AI integration, adaptive query execution, Photon engine performance, and enhanced workflows.
NewsUnderstanding Databricks & Apache Spark Performance Tuning: Lesson 02 - Spark Hardware
Proper compute resource selection and cluster configuration are foundational for optimizing Databricks and Apache Spark performance. The video details how hardware choices like worker size, node count, storage speed, and Photon acceleration directly impact workload efficiency and prevent costly shuffles and spills.
NewsDatabricks and Delta Lake: Lessons Learned from Building Akamai's Web Security Analytics Product
Akamai uses Databricks and Delta Lake to power its Web Security Analytics platform, delivering fast sub-seven-second queries and real-time ingestion over petabytes of security data. The engineering team presents best practices for optimizing both performance and cost at this massive scale, including custom load balancing, daily data partitioning, and scheduled scaling of SQL warehouses.
NewsBest Practices for Running Efficient Apache Spark™ Workloads on Databricks
The video demonstrates best practices for optimizing Apache Spark workloads and developing applications on the Databricks platform. It highlights key features such as Unity Catalog for governance, Delta Live Tables, the VS Code extension, Databricks Asset Bundles, workflows, and predictive I/O for faster queries and updates.
NewsBest Exploration of Columnar Shuffle Design
This video details the columnar shuffle design and implementation within the Gluten project for Apache Spark. It explains how native libraries like Velox and Intel processors handle columnar data serialization, spilling, and compression to improve performance.
NewsPhoton for Dummies: How Does this New Execution Engine Actually Work?
Databricks presents Photon, a high-performance native execution engine for Apache Spark that is written in C++ and uses vectorized columnar processing. The session details how Photon integrates with the Catalyst optimizer and Java virtual machine to accelerate data warehousing workloads like joins and aggregations without requiring code changes.
NewsBuilding a Lakehouse on AWS for Less with AWS Graviton and Photon
AWS Graviton2 instances and Photon on Graviton are now available in public preview on Databricks. Graviton2-powered instances provide up to 40% better price-performance compared to comparable x86-based instances.
NewsRadical Speed on the Lakehouse: Photon Under the Hood
Photon is Databricks’ next-generation query engine built to provide faster performance and lower latency for workloads on the Lakehouse platform. It uses native code and vectorized execution to run queries significantly faster than traditional execution engines without requiring code changes.
NewsDelta Lake, the Foundation of Your Lakehouse
Delta Lake is the foundation of the Databricks Lakehouse platform, combining data reliability, performance, and governance across cloud storage. Recent innovations in Delta Lake improve query performance for reads and writes through better file sizes, data skipping, metadata processing, and accelerated merges.
NewsHow AT&T Data Science Team Solved an Insurmountable Big Data Challenge on Databricks
AT&T's data science team uses the RAPIDS Accelerator for Apache Spark and the Photon runtime on Databricks to process extremely large personalization datasets of 2TB to 50TB. Speedups and costs are compared against the regular Databricks runtime Apache Spark environment, with results shown for both options.
NewsRadical Speed for SQL Queries on Databricks: Photon Under the Hood
Photon, a key query execution component of the Databricks Lakehouse platform built on Delta Lake, is written in C++ and uses vectorized query processing to exploit data and instruction level parallelism in modern CPUs. It is fully compatible with the Apache Spark DataFrame and SQL APIs, so workloads run without code changes.
EventsAli Ghodsi and Rohan Kumar (CVP, Microsoft) | Fireside Chat | Keynote Data + AI Summit NA 2021
Rohan Kumar, CVP of Azure Data at Microsoft, tells Ali Ghodsi what customers such as Grab and ABN AMRO achieve with data and AI on Azure Databricks. He also explains how innovations like Photon and Delta Sharing are coming to life on Azure.
EventsDatabricks SQL & Photon Updates with Demo | Reynold Xin | Keynote Data + AI Summit NA 2021
Databricks Chief Architect Reynold Xin presents performance improvements and simplified administration in Photon and Databricks SQL. A demo shows the SQL development experience, backed by an engine with improved concurrent querying capabilities.