Using dbt with Databricks: Architecture decisions that determine success
Summary
Databricks users who skip dbt incur compounding costs. A solution architect explains key architecture decisions and when to act to ensure success.
Summary from the captured archive. Follow the original source for full context.
More from dbt Labs Blog
How to put generative AI directly into a dbt pipeline with plain SQL
Google Cloud shows how to call BigQuery's AI SQL functions from inside dbt models, so generative AI steps like sentiment scoring run as ordinary SQL transformations in the pipeline. It extends the pattern to LLM-generated product briefings, all written in plain SQL without separate orchestration code.
How Sigma runs a 2,883-model semantic layer on dbt and Dagster
Sigma manages its semantic layer as version-controlled code, running 2,883 dbt models orchestrated with Dagster. The post walks through how that code-based approach works at scale, giving Databricks practitioners a concrete reference for treating a semantic layer like any other software project.
ClickHouse + dbt: Building a GitOps workflow at Twilio
Twilio consolidated its Kafka, Spark, and Elasticsearch stack into a single ClickHouse pipeline, with dbt models and a GitOps workflow governing how changes are made. The post walks through how that setup works, which is useful for Databricks practitioners weighing how to manage analytics pipelines through version-controlled, declarative workflows.
How KOHO built a Claude-powered self-serve analytics tool on years of dbt governance
KOHO built a Claude-powered self-serve analytics tool on top of years of dbt governance, and argues that AI analytics only works when a governed dbt foundation and a strong data culture are already in place. The post walks through why that groundwork, rather than the model itself, is what makes self-serve analytics trustworthy.
Why an AI agent's revenue query was $3.5 million wrong, and how AWS fixed it with dbt Semantic Layer
AWS shows how a plausible-looking AI agent revenue query came out $3.5 million wrong, and how grounding the agent in the dbt Semantic Layer prevents that kind of error. The post argues that supplying business context, such as governed metric definitions, is what keeps AI-generated numbers trustworthy.
How DoorDash scaled merchant-facing analytics to thousands of users in weeks, on dbt and ThoughtSpot
DoorDash used a dbt-defined semantic layer to take merchant-facing analytics from pilot to thousands of users in just weeks. Learn how pairing dbt with ThoughtSpot let the team scale access without rebuilding metrics for each new audience.