Snowflake

Show HN: Dbprint – Database Context for Agents

CLI that produces an offline snapshot of a database (DDL, column- and relation-level statistics, values with frequencies), combines it with annotations written by you or an agent, and exposes it as MCP and as a web UI. Works with PG, MySQL, Snowflake, ClickHouse, Databricks, BigQuery. How to try it: uvx dbprint context \ arboretum.seedbank.germination_trial \ --purpose query \ --project https://github.com/jakubro/dbprint/blob/main/docs/format/v1/examples/production/.dbprint.yaml Features: - Philosophy: snapshots are meant to sit next to your code and versioned so that you have a history of changes and agents can hit them when searching codebase. - Offline: no need to expose live credentials to agents. - CI: "dbprint check" can verify schema/data drift. - Annotations: complement statistics by documenting the meaning behind data. An agent can prepare them for you ("dbprint annotate"). - Redaction: values can be masked/excluded/hashed either explicitly with FQNs or automatically based on inferred sensitivity and value shape. - Format: Snapshots are YAML and markdown, so greppable and usable by other tools. Why I built this: SQL queries produced by agents were inaccurate, and the overall time to result was slow and costly because of finding the answers in codebase or in live database.

Agentic code converter is GA for SQL

Migration · GA Agentic code converter is GA for SQL Genie Code converts T-SQL, Snowflake, Redshift, Oracle, BigQuery, and Teradata SQL to Databricks SQL. It validates each file and fixes errors before writing to a workspace folder. Work in a migration project, or run /migrate in a Genie Code chat for one file or folder. Afterward, keep chatting to clear action items. The Notebooks target is not part of this GA. https://preview.redd.it/u5m0quj71fyh1.png?width=1808&format=png&auto=webp&s=1a5f16a1d66fa88e9bc171e10efb16c33dadfa2b Looking to migrate workloads but don't have the labor to do so. This is going to continue to expand its SQL dialect but taking an agentic approach to migrations really speeds up migrations and cuts down on time/money from labor submitted by /u/Wise_Ear_4064 [link] [comments]

Open source benchmark for Snowflake, Databricks, BigQuery, Redshift, Fabric, ClickHouse and MotherDuck: 244 queries, 1 GB to 1 TB, up to 100 concurrent users, every compute size

submitted by /u/dan_the_lion [link] [comments]

Show HN: 10x faster, 3x cheaper ETL

For the past few months our team has been working on an agentic framework that takes large, expensive SQL workloads and compiles them to Rust, backed by a formal proof of correctness. This is the sort of work I used to do by hand as a systems performance engineer, but now it happens in hours instead of months. We first redid our internal pipelines with this technique and they're now 10x faster and 3x cheaper. Next, we went live with our Snowflake and Databricks SQL customers. They're seeing similar results with critical production workloads that originally cost 6-7 figures per year. We'll be publishing benchmarks and code samples over the coming weeks; for now, you can read a bit more (and watch our anime launch trailer) on the site. Happy to answer questions about how it all works. --- top comments --- [gtirloni] Why was SQL slow? Where did processing shift to?

MiniLake Is a Local Databricks API Emulator with SQL, Delta Lake and Spark

--- top comments --- [gavinray] If this actually works, this is such a huge boon. Let me share why: I work on connectors to various datasources, mostly SQL DB's The most useful tests are the integration tests, sending actual queries to the source For most DB's we have a giant Docker Compose which runs a local instance But things like Snowflake, Databricks, Redshift, Athena suck to test because we need an external hosted instance I would kill to be able to run test versions of all of these locally [akbarian] Getting tired of the standard AI design, all websites look the same now [ngruhn] You should call it "pond". [lysecret] Want to hijack this someone made experience recently with https://ducklake.select/ ? We need to setup a warehouse soon and this looks enticing. [asabla] Oh wow! This would've been so useful at my previous work place. I'll save this one for later. Looks pretty decent too for homelabs! I wonder if there is any plans on extending it further? (just a bit)

Title: Data Engineering Roadmap: Azure + Databricks vs AWS + Snowflake + Airflow

Hi everyone! I’ve learned Python, SQL, and PySpark, and I’m looking for guidance on what to learn next to become job-ready for a Data Engineering role. I’m currently considering two learning paths: Option 1: Azure + Databricks Option 2: AWS + Snowflake + Airflow Which path would you recommend for someone starting out in Data Engineering, considering job opportunities, industry demand, and long-term career growth? I’d also appreciate recommendations for: Free learning resources and YouTube tutorials Hands-on, end-to-end Data Engineering projects A practical roadmap to become job-ready If you’re already working as a Data Engineer or have experience with these technologies, I’d love to hear your advice if you know any free learning resources, YouTube tutorials, or end-to-end project tutorials please share the links 🙏 Thanks in advance for your suggestions! submitted by /u/harishvangara [link] [comments]

Databricks vs. Snowflake: Are we comparing features instead of foundations?

submitted by /u/Zealousideal_Ant5362 [link] [comments]

Databricks Article Summary:

Just recently publish my whole thesis on Databricks on this subreddit. Got really great feedback so figured I would write a summary for those too busy for a 20 minute read. The Core Thesis: Databricks is one of the largest and most influential companies mostly lying behind the scenes. Databricks is a platform that allows organizations to store, process and analyze massive amounts of data. Built on their open-source platform Apache Spark, Databricks created software that they released publicly. This software eventually gained extremely widespread adoption and built much of Databricks’ moat. Led by Ali Ghodsi, the 47-year-old Iranian-Swedish-American computer scientist, Databricks has leapfrogged competition in the last decade and is now ahead of its main competitor Snowflake in valuation, revenue, and revenue growth rate. The Numbers to Know: $190 billion valuation. $5.2 billion in 2025 revenue. 71% - 2025 revenue growth. 27x - valuation to revenue multiple using Databricks’ estimated July 2026 ARR. 140% net revenue retention in 2024–2025. 74% - last disclosed gross margin cited, down from 80%. $5 billion raised in the last funding round despite $15–20 billion in investor interest. 80% of Fortune 500 companies using Apache Spark. The Bear Thesis: Databricks trades at 27x annualized revenue which assumes that their fast growth will continue to be sustainable. In addition, while AI is certainly beneficial for Databricks in accelerating revenue, their gross margins are starting to compress, down to a (still healthy) 74% from a previously elite 80%. The company has already acknowledged further compression as compute costs rise. If gross margins compress significantly, even if Databricks’ revenue continues growing at it’s current pace, the company may not be profitable enough to justify its valuation. The Verdict: Databricks was one of the most impressive companies I have researched to date. A few years ago, Databricks was behind its main competitor Snowflake in revenue and valuation. Today, that script has flipped on its head with Databricks being ahead of Snowflake in valuation and revenue. This is largely due to Ali Ghodsi and the incredible team at Databricks and it is partly for this reason that Databricks trades at a slight premium in comparison to Snowflake. Their sticky moat, strong leadership and impressive revenue growth are all strong reasons to invest in Databricks. However, I think due to the premium valuation perhaps the right call with Databricks at this moment is to have patience. I will particularly be watching the revenue growth to see if the company is able to maintain the revenue growth they currently have and watch to see if the gross margins continue to compress. This is certainly a company I want to add to my portfolio, at the right time and right price. submitted by /u/Roadtochessmaster [link] [comments]

Stop migrating data to Databricks: Are you using Lakehouse Federation yet?

I’ve been diving deep into Unity Catalog lately, and the "Lakehouse Federation" feature is genuinely a game-changer for reducing ETL bloat. For a long time, the default answer to "how do I analyze this external data?" was "copy it into the lake." We’re finally moving away from that. Here is the breakdown for anyone trying to cut down on unnecessary data movement: The two ways to federate: Catalog Federation (The "Data-Centric" approach): How it works: Connects Databricks to external metadata (AWS Glue, Hive, Snowflake Iceberg). The Magic: Databricks compute does the heavy lifting (reading raw files in S3/ADLS). It’s fast, low-cost, and creates a "live" bridge to your storage. Best for: Incremental migrations and long-term hybrid storage. Query Federation (The "Database-Centric" approach): How it works: Pushes SQL via JDBC to remote databases (Postgres, MySQL, SQL Server, Synapse, etc.). The Magic: The remote DB does the compute. You get the result set back without touching the raw underlying tables. Best for: Ad-hoc reporting against operational/production databases where you don't want to break their performance with heavy ETL loads. Why it’s actually useful: Centralized Security: Unity Catalog acts as the single security guard. You don't need to manage separate credentials for your users on the remote DB. Automatic Lineage: This is the biggest win for me—Databricks automatically maps the lineage from the remote system to your report. No more "mystery data." Unified Syntax: You query everything using catalog.schema.table regardless of whether it’s sitting in Databricks, Snowflake, or an on-prem SQL Server. submitted by /u/SalamanderOpen859 [link] [comments]

Launch: Assury Cross platform agent lifecycle mangement

Hey HN, I am David Girvi, co-founder of Assury.ai. We are practitioners who became vendors last year. I wanted to share that we have launched agent lifecycle management for cross-platform agents. One centralized spot for all your agents. AWS, Databricks, Snowflake, Salesforce, etc. Lifecycle does auto-discovery in every environment, plus auto-tagging. We bring everything into a tamper-proof, searchable audit trail. Easily see who or what teams own the agents etc. We also have Kill/resume controls based on tagging so you can granularly shut down certain groups during an incident. We built this also to handle swarms, as one of our design partners has 30k agent swarms that spin up and down for research. Our thesis is that we want everyone to use the agents that will help them with their work. We want to be the observability, audit, governance, and incident response for all those different platforms. We started with one of the first MCP gateways that did execution level governance, then the endpoint with hooks, then our agent deception technology, and now lifecycle management. We are a small team of practitioners building product for our customers, not a marketing team. Cheers.

Connecting to Snowflake via Workload Identity Federation

Which Came First For You... Spark or Databricks?

Saw a post this morning claiming you shouldn't touch Databricks until you've already mastered Apache Spark. That advice is backwards. It treats learning like a 90s waterfall project and traps engineers in tutorial hell. The fundamentals are non-negotiable: - You need SQL and Python. - You need to understand distributed shuffles, partitions, and memory spills. - You need to understand Parquet structures, metadata footers, and row groups. - You need to understand Delta Lake transaction logs and ACID guarantees. If you don't know what happens under the hood, you will write brittle pipelines that burn cloud budgets. The false assumption is that you must learn these concepts in an abstract vacuum before touching a modern platform. You don't learn networking by memorizing RFC packet headers on a whiteboard for six months before opening Wireshark. You fire up the tool, capture packets, and inspect the traffic. Databricks, Fabric, or Snowflake is your network analyzer. It is the laboratory. Nobody in enterprise production is setting up bare-metal Spark clusters on a home lab just to learn how shuffles work. And running local[*] on a laptop masks real-world distributed realities: - Local filesystems use POSIX atomic renames. Cloud object stores (S3, ADLS) do not. You cannot truly grasp why Delta's _delta_log exists until you see it write commits against cloud storage. - A local JVM masks network serialization, executor skew, and cross-node shuffle overhead. The caveat: don't treat the platform as a black box. Don't just click "Run All" on serverless compute and assume you know Spark. Use the platform to inspect the engine: - Run .explain(True) and read the physical plan. - Open the Spark UI to watch tasks, stages, and memory spill. - Inspect the JSON commits inside _delta_log after a merge operation. - Compact small files and see how file pruning changes scan times. You don't master the theory first and then graduate to a platform. You master the fundamentals by using the platform as your workbench. Learn the platform. Learn the engine. Learn the storage layer. Do it concurrently, in the environment where production actually lives. submitted by /u/OkImprovement7010 [link] [comments]

Show HN: Like LeetCode, but multi-file and multi-step

On LeetCode you write one function, it passes, and you're done. Most AI labs such as Anthropic and OpenAI, as well as some big tech companies such as Meta, Snowflake and Databricks, don't stop in one function and instead ask multi-step questions that are closer to real-world problems. Say you're building a key-value store. Step 1 is get, set, delete, count. Once your tests pass, Step 2 unlocks: expiring keys. Then nested transactions. Then atomic batches. I built this because I couldn't find a good place to practice for multi-step, multi-file questions. The questions are based on ones people have reported from those loops. Check it out at https://loopprep.dev. There are 24 questions, 3 free, and no signup to try them. Happy to answer questions. --- top comments --- [MiroslavPokorny] Writing code in a browser like this or leet code is wrong on so many levels its not funny.

Need some advice on Snowflake vs Databricks

submitted by /u/Ok_Independent_343 [link] [comments]

Snowflake’s AI generated slop blog

submitted by /u/Useful-Swordfish4946 [link] [comments]

Databricks vs Snowflake comparison

Are there any unbiased comparisons between these two popular platforms? Seen a lot but most of them are biased views, based on experience and commercial motives. submitted by /u/medici2022 [link] [comments]

Databricks vs Snowflake Pyspark Performance

Data Engineers, what does your actual day-to-day work look like? And what should I learn next?

I’m currently trying to transition deeper into Data Engineering and would really appreciate some perspective from people who are already working in the field. I have 1.3 yrs experience as a Junior Python Developer. What I want to do is slowly transform into a Data Engineer. How would you suggest my choice? Basically what I do is make web scraping scripts to get the data from web and give the data in excels. Our company is currently not using git or CI/CD or anything like that. The problem I’m running into is that when I look at Data Engineering jobs on Naukri, LinkedIn, etc., the requirements seem endless. One job asks for Python, SQL, Airflow and AWS; another wants Spark, Kafka and Databricks; another wants Snowflake, dbt, Terraform, Kubernetes, CI/CD, etc. It becomes difficult to understand what I should actually prioritize. So I’d like to hear from people who are actually working as Data Engineers . What does your day-to-day work look like? What kind of problems do you solve, what technologies do you use regularly, and which skills have turned out to be genuinely important in your job? More importantly, based on my current experience, what would you suggest I improve or learn next to become a stronger candidate for Data Engineering roles? Are there any gaps that you think I should focus on, or technologies/concepts that are worth learning through projects rather than just studying theoretically? I’m not really looking for a generic “learn SQL → Python → Spark → AWS” roadmap. I’m more interested in understanding the reality of the job and getting advice from people who have actually gone through the transition. If you’re a Data Engineer with 1–5+ years of experience , I’d especially appreciate your perspective. Even a short description of what you work on and what you wish you had learned earlier would be extremely helpful. Thanks in advance! submitted by /u/Stunning-Space8032 [link] [comments]

Lakehouse Federation (Snowflake) — large query results fail to download from internal stage

Snowflake OR Databricks

Show HN: Ingestr CDC – open-source CDC replication in Go

Hi all, this is Burak, one of the founders at Bruin. I created ingestr to make data ingestion easy. ingestr is a CLI tool that can ingest data from 130+ sources. I have shared it on HN after our Go rewrite as well, which made it the fastest ingestion tool in the space. However, ingestr had always been a batch tool. I am personally a big fan of batch workloads due to their simplicity and have built ingestr around that assumption as well; however, over time, the cracks started to show when we started working with larger orgs. Turns out there are some scenarios where CDC proves beneficial: - For legacy systems where it is not possible to introduce cursor columns due to technical, but mostly organizational, concerns, it becomes impractical to deploy batch pipelines. - For systems that do not have a way to reliably know the update timestamp, also due to legacy reasons. Think usecases where the columns are updated without the timestamp being updated. - For hard deletes. Even though I do believe there are ways to solve each of these, it ended up putting us in a disadvantage, and we decided to build the CDC connectors instead. ingestr CDC works in two modes now: batch (bad name, I know) and stream. The batch mode is reading the changelog entries from the last load until the starting timestamp, and the streaming mode keeps reading and landing them to the destination databases. ingestr has a few advantages compared to a more traditional debezium + kafka setup: - it's a standalone Go binary and does not require any additional infra. - it has very low resource consumption, ~100MB baseline. - it can run on your own computer during development, and can be converted into a streaming prod deployment when it is ready. - supports 20+ destinations already, primarily analytical platforms like snowflake, databricks, generic iceberg destinations, etc. i would love to hear any feedback on what we could do to make it easier for cdc workloads! https://github.com/bruin-data/ingestr --- top comments --- [mercutio93] Man I have had a lot of gripes with debezium... the setup alone in a kubernetes cluster with strimzi kafka connect... And dealing with the kafka connect errors and fixing replication slots... ehhh was painful. Does this tool make it any well easier. One problem I expirienced in production systems with staging doubles is that sometimes we do database restores from production onto staging and that makes the sinks and sources go haywire... does ingestr handle this well? Is support for kubernetes/helm anywhere in the pipeline

Show HN: Connect DuckDB to any database that has an ADBC driver

Hi everyone, I'm a PhD student in databases at CMU. Over the past few months, I've been interning at Columnar and building a community extension for DuckDB that lets you query Snowflake, Databricks, BigQuery, PostgreSQL, MySQL, and any other system with an ADBC (Arrow Database Connectivity) driver. The extension supports querying ADBC databases directly through a read_adbc table function. It also supports using ATTACH to connect to an ADBC database and then running SELECT, INSERT, COPY, and CTAS statements as if the database were local to DuckDB. You can install it from DuckDB by running: INSTALL adbc FROM community; LOAD adbc; The ADBC extension is open source and available under the Apache-2.0 License. We plan to contribute this extension to the Arrow project so it can become an official ADBC library. Give it a try, and leave feedback if you have ideas about how to make it better. Leave an issue on the GitHub repo if you hit any rough edges! --- top comments --- [eveerett] Super cool project! Curious your thoughts on LakeSail. It can run single node on your laptop similar to DuckDB all the way to a cluster like Spark (it adopts the Spark interface, but built entirely in Rust without the JVM). Might be an interesting look for ya. [StarlaAtNight] Haven’t tried it yet, but awesome work! This seems huge. I try to use DuckDB whenever feasible, and I also tend to work with Snowflake, Databricks, and BigQuery (as well as Fivetran which uses DuckDB for local dev/debug mode). Excited to have ADBC-enabled connections w/ DuckDB, and have a feeling this is going to start coming up as a use case a lot more

Show HN: Dex – Cost-aware analytics engineering skills for agents

Hi I’m Marco, co-founder of Exmergo. Me and my team created Dex to help Analytics Engineers do real work with Claude Code (and any other coding agent). We’ve found that Data and Analytics teams are stuck between a rock and a hard place cost-wise: - On one side, they are using some of the most expensive consumption-billed software on the planet (Snowflake, Databricks etc.). - On the other side, Anthropic and Open AI want you to tokenmax (and with data analytics it’s very easy to burn your context window). So we created Dex, our open source skills plugin (Apache-2.0), to solve both of these problems: - Dex forces the agent to use a cost guard when performing exploration queries and transformations. - Dex uses a set of tooling that makes it hard to get burned when transforming data. It achieves these things with tight control scripts that the SKILL.md files are pointed towards when using /dex:explore, /dex:transform and /dex:maintain. Bonus: Dex makes agents really good data exploration, building sql and dbt models and detecting drift. 76% performance on ade-bench with Claude Sonnet 5 (and, per our measures, 2.5x cheaper than Fable 5). Install on any agent with this command in your terminal: npx skills add exmergo/dex Install on Claude Code running these commands (separately): /plugin marketplace add exmergo/exmergo-agent-plugins /plugin install dex@exmergo If you want to see more visual examples you can go through the README or browse here: https://www.exmergo.com/dex Let me know if this helps your analytics workflow and makes your agents more cost-aware.

Ingest data from snowflake to databricks

Snowflake to Databricks Migration in 12 weeks and cut cost per run by ~77%. AMA.

Lovelytics wrapped up a Snowflake-to-Databricks migration; 847 DBT models, 35 Info Mart tables, \~77% lower cost per run on a 2XL warehouse. **TL;DR What helped:** * Treated the migration as engineering, not translation. Each dbt model was tested in isolation, not just row counts vs Snowflake. * Routing macro to resolve cross-layer references at runtime, so the same codebase could read from Snowflake, federated Snowflake, and Unity Catalog without forking logic. * Dual model trees in one repo, which let the migration stay in lockstep with live Snowflake changes. * Script-generated wave selectors enabled parallel builds while preserving dependency order. * Used reference-slice validation subsets vs. waiting on full mart refreshes. **TL;DR Cost reduction:** * Reworked joins to use narrow staging dimensions instead of wide marts where possible. * Added incremental predicates to reduce MERGE target scans. * Split wide models into parallel sub-models where the dependency graph allowed it. * Copied static reference data into Delta instead of repeatedly reading it through federation. * Loaded static copies into Delta rather than reading via federation (predicate pushdown is poor). Happy to go into the gotchas: HASH() not being portable, Snowflake MERGE tolerating duplicate keys that Delta doesn't, NULL ordering, and timestamp handling. AMA [Full Blog Post](https://community.databricks.com/t5/technical-blog/partner-blog-847-models-12-weeks-77-less-inside-r1-s-snowflake/ba-p/157284)

Sqlit – A lazygit-style TUI for SQL databases

sqlit is a lazygit-style TUI for SQL. Connect and query your database from the terminal in seconds, no config files or documentation to read first. It supports all the major databases: SQL Server, PostgreSQL, MySQL, SQLite, MariaDB, FirebirdSQL, Oracle, DuckDB, CockroachDB, ClickHouse, Snowflake, Databricks, Supabase, Cloudflare D1, Turso, Athena, BigQuery, Spanner, Redshift, IBM Db2, SAP HANA, Teradata, Trino, Presto, Apache Flight SQL, Apache Impala, SurrealDB, and osquery. A few things that come built in: - Keyboard focus: Context aware keybindings always visible - Docker integration: auto-detects and connect to running database containers - Vim-style query editor with customizable keybindings. - Fuzzy filter in results window. - SSH tunnels, OS-keyring credential storage, password manager integration. - Autocomplete for tables, columns, and procedures. - Cloud CLI integration (browse external DBs via Azure / AWS / GCP CLIs). - Themes (Rose Pine, Tokyo Night, Nord, Gruvbox). Install: `pipx install sqlit-tui` (also works with `uv tool install` and `pip`). Built with Python and Textual. First shared here in December (https://news.ycombinator.com/item?id=46276002) - a lot has shipped since. Repo: https://github.com/Maxteabag/sqlit Feedback welcome, especially on what's still missing for daily-driver use. My goal is to make an aesthetic tool that makes it easy and enjoyable to connect and query data, and do that one thing only, really well. --- top comments --- [cesarmanzocode] Very nice UX direction. One thing I’ve consistently missed in terminal DB tools is schema-aware query history/search across multiple databases. Feels like that would fit this aesthetic really well. [qsera] Looks amazing. But before I can commit to trying it, I would like to know how much LLM involvement was there. [teejmya] Why is this in Ask HN? Shouldn't it be in Show HN?

Why should I use Databricks Over Snowflake ?

Where do you Qlik/Talend with the likes of Databricks?

Hi Folks, Long time Qlik Analytics user and since Talend merger have been involved in data engineering side of things as well. While I already know the major differences and benefits Databricks/Snowflake and new age BI solutions like Sigma and Omni provides over other vendors. Wondering where does the market see vendors like Qlik ?

[PARTNER BLOG] 847 Models, 12 Weeks, 77% Less: Inside R1's Snowflake-to-Databricks Migration

Seeking Volunteers with Lakehouse, Fabric, Databricks, or Snowflake Experience

[BLOG + video] Snowflake and Databricks benchmarks

We put Snowflake and Databricks head-to-head across 5 scenarios. 𝗦𝗻𝗼𝘄𝗳𝗹𝗮𝗸𝗲 𝘄𝗼𝗻 𝟰 𝗼𝘂𝘁 𝗼𝗳 𝟱 𝘀𝗰𝗲𝗻𝗮𝗿𝗶𝗼𝘀: \- Sequential queries: 34% faster, 17% cheaper (at $2/credit) \- Concurrent queries: 38% faster, 39% cheaper \- Cold start: 54% faster (Databricks startup time: \~7 sec. Snowflake: sub-second. Every. Single. Time.) \- DML (delete + insert): 59% faster, 32% cheaper thanks to elite query pruning that treated 6B rows like 6M 𝗗𝗮𝘁𝗮𝗯𝗿𝗶𝗰𝗸𝘀 𝗰𝗹𝗮𝗶𝗺𝗲𝗱 𝘁𝗵𝗲 𝗼𝗻𝗲 𝘁𝗵𝗮𝘁 𝗺𝗮𝘁𝘁𝗲𝗿𝘀 𝗳𝗼𝗿 𝗱𝗮𝘁𝗮 𝗲𝗻𝗴𝗶𝗻𝗲𝗲𝗿𝘀: CTAS (Create Table As Select): 58% faster, 71% cheaper when writing billions of rows across multiple table shapes If your workload is heavy on dbt materializations, large table builds, or data pipeline writes, Databricks has a real edge here. If your workload is analysts running queries, dashboards, and incremental refreshes, Snowflake Standard looks compelling, even vs. Databricks Enterprise pricing. \- Read the full methodology and results: [https://select.dev/posts/snowflake-vs-databricks-showdown](https://select.dev/posts/snowflake-vs-databricks-showdown) \- Take a look at the repo: [https://github.com/get-select/snowflake-databricks-benchmark](https://github.com/get-select/snowflake-databricks-benchmark)

Show HN: Matterbeam, a company-wide write-ahead log for your data

Hey HN. I'm Michael, founder of Matterbeam. Been chewing on the core ideas of it for over ten years, building toward it for three. demo: https://www.youtube.com/watch?v=YuhujARUmhA whitepaper: https://matterbeam.com/whitepaper Short version: companies build their data infra on point-to-point pipelines and one place to put all the data. Source A goes to warehouse B. Team C wants the same data shaped differently? Build another. Eventually a mess of brittle ETL nobody wants to touch. Matterbeam puts existing ideas together in a different way. Source data collected as immutable, time-ordered facts into a log. Destinations replay and transform those facts, from any point in time, into the target they need. One source, many uses. My last startup was acquired by Pluralsight in 2014. I ended up leading product architecture and data there for about five years. Working with really brilliant, product and data people that I would have said were doing everything _right_. Yet no one in the company was happy with data. It made me question if something more fundamental wasn't broken. A key inspiration came from Martin Kleppmann's 2015 talk "Turning the Database Inside Out." (https://www.youtube.com/watch?v=fU9hR3kiOK0) Most databases internally do something interesting: a write-ahead log (durable, append-only, time-ordered) as a source of truth, and derived structures are created (B-trees, indexes, materialized views) optimized to serve different read patterns. What if you took that pattern and blew it up to org scale? Your uses become materializations. Warehouse, RAG vector db, graph db, any new use created when needed with a late transform and a new emitter. A few comparisons: We aren't Kafka. Kafka is lower-level. My first attempt at this was at Pluralsight using Kafka as the log. It was crazy expensive and complicated to operate. For Matterbeam we built cloud-native: object storage gives durability, ephemeral compute avoids coordination, we don't need 100ms latency for most jobs. Allowed us to avoid a lot of Kafka's complexity. We aren't Fivetran. Fivetran is a managed pipeline. We're a utility. One customer replaced Fivetran when they brought us in. Saved them money, but that wasn't the goal, suddenly projects they estimated at five months started taking two days. A two-year migration compressed into months. Their PMs started asking to use Matterbeam for everything. We aren't a warehouse or lake. Snowflake and Databricks are great at what they're great at. The push to centralize all data in these systems was a mistake. We aim to be the layer underneath. Basically fulfill the original promise of the data lake: collect without a use case, materialize when you figure out what you need, in the shape and system you need. What's broken: This doesn't fit cleanly into "what does this replace" buckets. Most people agree data is broken but then lament "data is hard" or some form of "my team isn't doing it right." Nobody's actively looking to solve the deeper problem. Hard to find new customers even with glowing testimonials. Connector coverage. Fivetran has hundreds. We have way fewer in production. We're working on it, we're using AI, you can write your own pretty quickly. Still, if your stack needs fifty SaaS integrations on day one, we struggle. We're early. Handful of paid customers. Not large-enterprise-ready no SOC2, HIPAA etc yet. Also, conscious decision not to be open source. Long list of reasons, separate post. I'd love feedback on: How would you position or market this? It feels like category creation, which I know is hard. Does the mental model land, or is there a piece where you go WAT? If you've built CDC-into-warehouse, Kafka-plus-schema-registry, or rolled a data backbone, what's the part you'd have wanted an easier solution for? Blog, testimonials, marketing video on the site. I'll be watching the thread. Be brutal, I can take it (I think).