Information chess recreation: Databricks vs. Snowflake, half 1

June was fairly a month by post-lockdown requirements. Not solely did stay occasions return with a vengeance after a pair years of countless Zoom marathons, however the begin of summer season noticed a confluence of occasions from arguably the info world’s hottest trio: in sequential order, MongoDB, Snowflake and Databricks.

There could also be stark and refined variations in every of their trajectories, however the frequent thread is that every is aspiring to develop into the next-generation default enterprise cloud knowledge platform (CDP). And that units up the subsequent act for all three: Every of them must attain exterior their core constituencies to broaden their enterprise enchantment.

As a result of we’ve acquired so much to say from our June journey report with the trio of knowledge hotshots, we’re going to separate our evaluation into two elements. Right now, we’ll give attention to the chess recreation between Databricks and Snowflake. Tomorrow, partially 2, we’ll make the case for why all three corporations should step exterior their consolation zones if they’re to develop into the next-generation go-to knowledge platforms for the enterprise.

The info lakehouse units the agenda

We famous that with analytics and transaction processing, respectively, MongoDB and Snowflake might ultimately be on a collision course. However for now, it’s all in regards to the forthcoming battle for hearts and minds in analytics between Databricks and Snowflake, and that’s the place we’ll confine our dialogue right here.

The grand context is the convergence of knowledge warehouse and knowledge lake. About 5 years in the past, Databricks coined the time period “knowledge lakehouse,” which subsequently touched a nerve. Nearly everybody within the knowledge world, from Oracle, Teradata, Cloudera, Talend, Google, HPE, Fivetran, AWS, Dremio and even Snowflake have needed to chime in with their responses. Databricks and Snowflake got here from the info lake and knowledge warehousing worlds, respectively, and each are actually working into one another with the lakehouse. They’re not the one ones, however each arguably have the quickest rising bases.

The lakehouse is solely the means to the top for each Databricks and Snowflake as they search to develop into the info and analytics vacation spot for the enterprise.

To oversimplify, Snowflake invitations the Databricks crowd with Snowpark, so long as they’re prepared to have their Java, Python or Scala routines execute as SQL features. The important thing to Snowpark is that knowledge scientists and engineers don’t have to alter their code.

In the meantime, Databricks is inviting the Snowflake crowd with a brand new SQL question engine that’s much more purposeful and performant than the unique Spark SQL. Sarcastically, in these scuffles, Spark is at the moment on the sidelines: Snowpark doesn’t (but) assist Spark execution, whereas the brand new Databricks SQL, constructed on the Photon question engine, doesn’t use Spark.

The trick query for each corporations is how to attract the Python programmer. For Snowflake, the query is whether or not user-defined features (UDFs) are probably the most performant path, and right here, the corporate is investing in Anaconda, which is optimizing its libraries to run in Snowpark. Databricks faces the identical query, provided that Spark was written in Scala, which has historically had the efficiency edge. However with Python, the variations could also be narrowing. We consider that Snowflake will ultimately add functionality for native execution in-database of Python and maybe Spark workloads, however that may require vital engineering and received’t occur in a single day. 

In the meantime, Databricks is rounding out the info lakehouse, broadening the capabilities of its new question engine whereas including a Unity Catalog as the muse for governance, with fine-grained entry controls, knowledge lineage and auditing, and leveraging companion integrations for superior governance and coverage administration. Andrew Brust offered the deep dive on the brand new capabilities for Delta Lake and associated initiatives similar to Undertaking Lightspeed in his protection of the Databricks occasion final month.

Who’s extra open, and does it matter?

Databricks and Snowflake additionally differ on open supply. This could be a subjective idea,  and we’re not about to revisit the controversy once more. 

Suffice it to say that Databricks claims that it’s much more open than Snowflake, given its roots with the Apache Spark undertaking. It factors to enterprises that run Presto, Trino, DIY Apache Spark or industrial knowledge warehouses straight on Delta with out paying Databricks. And it extends the identical argument to knowledge sharing, as we’ll word under. To settle the argument on openness, Databricks introduced that remaining options of Delta Lake are actually open supply. 

In the meantime, Snowflake makes no apologies for adhering to the normal proprietary mode, because it maintains that’s the simplest option to make its cloud platform performant. However Snowpark’s APIs are open to all comers, and for those who don’t wish to retailer knowledge in Snowflake tables, it’s simply opened assist for Parquet recordsdata managed by open-source Apache Iceberg as the info lake desk format. After all, that results in extra debates as to which open-source knowledge lake desk storage is probably the most open: Delta Lake or Iceberg (OK, don’t neglect Apache Hudi). Right here’s an outdoor opinion, even when it isn’t really unbiased.

Databricks makes open supply a key a part of its differentiation. However excluding corporations like Percona (which makes its enterprise delivering assist for open supply), it’s uncommon for any platform to be 100% open supply. And for Databricks, options similar to its notebooks and the Photon engine powering Databricks SQL are strictly proprietary. As if there’s something improper with that.

Now the hand-to-hand fight

Information warehouses have been recognized for delivering predictable efficiency, whereas knowledge lakes are recognized for his or her functionality to scale and assist polyglot knowledge and the flexibility to run deep, exploratory analytics and complicated modeling. The info lakehouse, an idea launched by Databricks almost 5 years in the past, is meant to ship one of the best of each worlds, and to its credit score, the time period has been adopted by a lot of the remainder of the {industry}. The operable query is, can knowledge lakehouses ship the constant SLAs produced by knowledge warehouses? That’s the context behind Databricks’ promotion of Delta Lake, which provides a desk construction to knowledge saved in open-source Parquet recordsdata.

That set the stage for Databricks’ TPC-DS benchmarks final fall, which Andrew Brust put in perspective, and naturally, Snowflake responded. On the convention, Databricks CEO Ali Ghodsi up to date the outcomes. Watching him extoll the aggressive benchmarks vs. Snowflake rekindled cozy recollections of Larry Ellison unloading on Amazon Redshift with Autonomous Database. We usually take benchmarks with grains of salt, so we received’t dwell on precise numbers right here. Suffice it to say that Databricks claims superior worth efficiency over Snowflake by orders of magnitude when accessing Parquet recordsdata. After all, whether or not this displays configurations consultant for BI workloads is a matter for the specialists to debate.

What’s attention-grabbing is that Databricks confirmed that it wasn’t religiously tied to Spark. Truly, right here’s a enjoyable truth: We discovered that roughly 30% of workloads run on Databricks will not be Spark.

As an illustration, the newly launched Photon question engine is an entire rewrite, slightly than an enhancement of Spark SQL. Right here, Databricks changed the Java code, JVM constructs and the Spark execution engine with the confirmed C++ utilized by all of the family names. C++ is way extra stripped down than Java and the JVM and is way extra environment friendly with managing reminiscence. The outdated is new once more.

That is an space the place Snowflake units the agenda. It launched the trendy idea of knowledge sharing within the cloud roughly 5 years in the past with the knowledge sharehouse, which was premised on inside line organizations sharing entry and analytics on the identical physique of knowledge with out having to maneuver it. 

The thought was a win-win for Snowflake as a result of it offered a option to develop its footprint inside its buyer base, and because the bulk of Snowflake’s income comes from compute, not storage, extra sharing of knowledge means extra utilization and extra compute. Subsequently, the hyperscalers hopped on the bandwagon, including datasets to their marketplaces.

Quick ahead to the current and knowledge sharing is behind Snowflake’s pivot from cloud knowledge warehouse to knowledge cloud. Particularly, Snowflake cloud must be your group’s vacation spot for analytics. A key draw of Snowflake knowledge sharing is that, if the info is inside the identical area of the identical cloud, it doesn’t have to maneuver or be replicated. As a substitute, knowledge sharing is in regards to the granting of permissions. The flip aspect is that Snowflake’s inside and exterior knowledge sharing can lengthen throughout cloud areas and totally different clouds, because it does assist the mandatory replication.

The newest replace to Snowflake Information Market, which is now renamed Snowflake Market, is that knowledge suppliers can monetize their knowledge and, in a brand new addition, their UDFs through a Native Software Framework, which certifies that these routines will run inside Snowpark. They will promote entry to the info and native apps sitting in Snowflake with out having to pay any fee to Snowflake. The secret’s that this should occur inside the Snowflake walled backyard as {the marketplace} solely covers knowledge and apps residing in Snowflake.

Final month, Databricks got here out with its reply, asserting the opening of inside and exterior knowledge marketplaces. Not like Snowflake, it runs inside a single area and cloud, because the Databricks service doesn’t at the moment have cross-region or cross-cloud replication options. {The marketplace} goes past datasets to incorporate fashions, notebooks and different artifacts. One of many options of Databricks market is knowledge cleanrooms, during which suppliers keep full management over which events can carry out what evaluation on their knowledge with out exposing any delicate knowledge similar to personally identifiable info (PII), a functionality that Snowflake already had.

There are a number of primary variations between the Snowflake and Databricks marketplaces, reflecting coverage and stage of improvement. The coverage distinction is about monetization, a functionality that Snowflake simply added whereas Databricks purposely refrained. Databricks’ view is that knowledge suppliers is not going to probably share knowledge through disintermediated bank card transactions, however will as a substitute depend on direct agreements between suppliers and customers. 

The hands-off coverage by Databricks to knowledge and artifacts in its market extends to the admission charge, or extra particularly, the dearth of 1. Databricks says that suppliers and customers in its market don’t should be Databricks subscribers.

Till not too long ago, Databricks and Snowflake didn’t actually run into one another as they focused totally different audiences: Databricks specializing in knowledge engineers and knowledge scientists growing fashions and knowledge transformations, working via notebooks, whereas Snowflake appealed to enterprise and knowledge analysts via ETL and BI instruments for question, visualization and reporting. That is one other case of the sheer scale of compute and storage within the cloud eroding know-how limitations between knowledge lakes and knowledge warehousing, and with it, the limitations between totally different constituencies.

Tomorrow, we’ll take a look at the opposite aspect of the equation. Databricks and Snowflake are fashioning themselves into knowledge locations, as is MongoDB. They’re every hot-growth database corporations, and they’re going to every should enterprise exterior their consolation zones to get there.

Keep tuned.

That is the primary of a two-part collection. Tomorrow’s publish will define the subsequent strikes that Databricks, MongoDB and Snowflake ought to take to enchantment to the broader enterprise.