Databricks showcased a new no-code data management tool, powered by a generative AI assistant, at its ongoing Data + AI summit, which is designed to help enterprises eliminate data engineering bottlenecks slowing down AI projects.
Called Lakeflow Designer, the tool is designed to empower data analysts build no-code ETL (extract, transform, load) pipelines — a task, typically, left to data engineers who are always busy, thus creating a barrier to accelerate AI projects or uses cases, said Bilal Aslam, senior director of product management at Databricks. Lakeflow Designer is currently in preview.
While enterprises employ low-code or no-code tools for data analysts to curtail the load on data engineers, these tools lack governance and scalability, Aslam said, adding that Lakeflow Designer is designed to alleviate these challenges.
“Lakeflow Designer addresses a key data management problem: data engineering bottlenecks are killing AI momentum with disconnected tools across data ingestion, preparation, cleansing, and orchestration,” said Michael Ni, principal analyst at Constellation Research. “Lakeflow Designer blows the doors open by putting the power of a no-code tool in analysts’ hands, while keeping it enterprise safe.”
Ni called Lakeflow Designer the “Canva of ETL” — instant, visual, AI-assisted development of data pipelines — yet under the hood, it’s Spark SQL at machine scale, secured by Unity Catalog.
Advisory firm ISG’s director of software research, Matt Aslett, said the new tool is expected to reduce the burden on data engineering teams but pointed out that data analysts are highly likely to still be working with data engineering teams for use cases that have more complex integration and transformation requirements that require additional expertise.
Lakeflow Designer makes collaboration between data analysts and engineers in an enterprise easier as it allows sharing of metadata and CI/CD pipelines, meaning these can be inspected, edited by engineers if required, Ni said.
The tool also supports Git and DevOps flows, providing lineage, access control, and auditability, Ni added.
Like Aslett, the analyst pointed out that the new tool is likely to aid enterprises in less complex use cases, such as regional margin tracking, compliance, metric aggregation, retention window monitoring, and cohorting, although it supports custom development.
Lakeflow Designer is part of Lakeflow, which will now be generally available. Lakeflow has three modules — Lakeflow Connect, Lakeflow Declarative Pipelines, and Lakeflow Jobs. Designer is integrated within Declarative Pipelines.
United in purpose, divided in approach
Lakeflow Designer, despite being very similar to rival Snowflake’s Openflow, differs in its approach, analysts say.
“Lakeflow and OpenFlow reflect two philosophies: Databricks integrates data engineering into a Spark-native, open orchestration fabric, while Snowflake’s OpenFlow offers declarative workflow control with deep Snowflake-native semantics. One favors flexibility and openness; the other favors consolidation and simplicity,” Ni said.
Both the offerings also differ in maturity, ISG’s Aslett said. While Snowflake’s OpenFlow is relatively new, Lakeflow has matured in functionality over the years, with Designer being its latest tool. “The Connect capabilities were acquired along with Arcion in 2023. Declarative pipelines functionality is the evolution of DLT (Delta Live Tables), and Jobs is the evolution of Databricks Workflows,” Aslett added.
Separately, Databricks also released another pro-code integrated development environment (IDE) for data engineers, which unifies the data engineer’s full pipeline lifecycle — code, DAGs, sample data, and debugging — all in one integrated workspace.
Releasing Lakeflow Designer and the new IDE together is a strategic play, according to Ni, as the lakehouse provider is targeting both ends of the pipeline maturity curve — low-code to move fast and the full IDE to scale and maintain pipelines.