Lakeflow Connect added managed marketing connectors and Lakeflow Jobs released task output in for_each loops.
Databricks expanded Lakeflow Connect with native managed connectors for ad and marketing platforms, including Salesforce, HubSpot, Google Ads, and Meta Ads, landing raw ingestion data into Unity Catalog [29]. Accompanying guides demonstrated how to model these feeds for marketing analytics [28]. Within Lakeflow Jobs, support for passing task outputs inside for_each task loops entered public beta [3], while Databricks delivered updates to Lakeflow Designer [14].
Teams adopting Lakeflow Connect shared early production experiences and raised infrastructure questions [19]. Users deploying the SQL Server change data capture gateway asked how to select specific virtual machine types [10], handle elastic compute requirements [1], and avoid connection saturation on source transactional databases [22]. Practitioners also shared architectural patterns for managing upstream schema drift and executing historical backfills alongside ongoing change streams [6].
Lakeflow Jobs sparked discussion around production orchestration maturity. Teams evaluated replacing legacy Airflow schedules with data-aware triggers in Lakeflow [16, 26], though engineers debated whether the platform provided adequate tooling for service level objectives and SLA monitoring across large fleets [15, 25]. Community members also worked through practical job configurations, such as implementing conditional task retries [8] and enabling predictive optimization for streaming tables in declarative pipelines [18].
Everything cited
- [1]Lakeflow connect SQL server ingestion can be made elastic? community · 2026-09-30
- [2]system.query.history is now Generally Available in Databricks community · 2026-09-30
- [3]Lakeflow Jobs update! Task output in foreach now available in Beta! community · 2026-09-30
- [4]Time to Swap the Cookies for Jetfuel - New Dataset & New Databricks Genie Tutorial community · 2026-09-29
- [5]Trigger a job automatically on databricks DABs bundle deploy video · 2026-09-28
- [6]Operationalizing Lakeflow Connect:Handling Upstream Schema Evolution & Historical Backfill community · 2026-09-24
- [7]Running open-Jev in SQL on Databricks news · 2026-09-24
- [8]Conditionally retrying a task on Lakeflow Jobs community · 2026-09-24
- [9]How I built agent-based security reviews on Databricks news · 2026-09-24
- [10]How can I configure Lakeflow Connect SQL Server CDC Gateway to use a desired VM type? community · 2026-09-24
- [11]I didn't want to choose between doing engineering and staying up to date with Databricks, so I built this. Hope it's useful to you too! community · 2026-09-23
- [12]Real-Time SAP Accounts Receivable on Databricks Lakeflow Declarative Pipelines community · 2026-09-23
- [13]Data Engineering Project On Real Company | Real Data | Pyspark | Databricks | Production Ready community · 2026-09-23
- [14]New features in Lakeflow Designer community · 2026-09-22
- [15]Anyone running actual SLOs on Lakeflow Jobs, or is it still “it failed, rerun it”? community · 2026-09-22
- [16]Do you still use Airflow with Databricks? community · 2026-09-21
- [17]Environments on classic compute community · 2026-09-19
- [18]Predictive Optimization for Streaming tables in Lakeflow pipelines community · 2026-09-19
- [19]Lakeflow Connect in production: What has your experience been? community · 2026-09-19
- [20]system tables data retention community · 2026-09-18
- [21]BrickTalk Recording | One Platform, Any Source: Unifying Enterprise Data with Lakeflow Connect community · 2026-09-18
- [22]Lakeflow connect CT pipeline keeps running multiple connections in source community · 2026-09-15
- [23][Discussion] Lakeflow Jobs: If your data catalog could trigger a job on any change what would it be? community · 2026-09-15
- [24]Private Network Gateway community · 2026-09-13
- [25]Best practices for SLA monitoring and automated retries across hundreds of Lakeflow Jobs community · 2026-09-12
- [26]igrating from Cron-based Airflow to Lakeflow's Data-Aware Triggers — Real-world experiences? community · 2026-09-12
- [27]Lakeflow Connect SQL Server Connector community · 2026-09-12
- [28]Tutorial: Transform your Lakeflow Connect ad data into visual and conversational analytics community · 2026-09-11
- [29]Unify your marketing data with Lakeflow Connect news · 2026-09-11
- [30]What is Data Transformation? news · 2026-09-03