Sponsored by: Prophecy | Automate Data Prep with Specialized Claude Code

Summary

Prophecy uses a specialized Claude Code agent to let business users describe data goals in natural language and automatically generate production-quality Spark pipelines on the Databricks Data and AI platform, without writing code.
The agent relies on domain-specific building blocks called Gems as ETL primitives to produce governed, visual data preparation workflows that bridge the gap between business analysts and data engineers.
This video also demonstrates migrating legacy Alteryx workflows directly into Prophecy pipelines that execute on Databricks SQL warehouses, eliminating data copies and closing enterprise governance gaps.

Chapters

FAQs

What is Prophecy's specialized Claude Code agent for data prep?

Prophecy's specialized Claude Code agent is a domain-specific AI that understands ETL concepts and Databricks-native building blocks called Gems. It lets business users describe a data preparation goal in natural language, then automatically builds a visual pipeline and generates production-quality Spark code—without requiring any coding from the business user.

How does Prophecy solve the problem of business users needing ETL without writing code?

Instead of requiring repeated back-and-forth between business analysts and data engineers—a cycle that can stretch data products to weeks—Prophecy's agent lets business users define the goal themselves. The agent builds and validates the pipeline, reducing delivery time from weeks to minutes or hours.

What are Gems in Prophecy's data pipeline approach?

Gems are domain-specific, reusable ETL primitives that serve as the building blocks of Prophecy's visual pipelines on the Databricks Data and AI platform. They provide the specialization that allows the agent to generate correct, governed Spark code rather than generic code that might not meet enterprise production standards.

How does Prophecy support migrating Alteryx workflows to Databricks?

Prophecy includes an Alteryx import feature that converts existing Alteryx workflows into Prophecy pipelines. Those pipelines then execute on Databricks SQL warehouses, giving teams cloud-scale and governance without needing to redesign their legacy ETL logic from scratch.

Full transcript

[00:07] Thank you for joining me on one of the last sessions here at the data bricks conference. Um I'm sure you want to hear more about our products and things, but I'm really excited to share with you what we have today. Uh my name is Nathan. Uh I'm one of the lead for deployed engineers at Prophecy and I'm
[00:19] excited to take you through a brief demo about what we actually do and how we bring value to you and your users communities. So for today's agenda, it's really going to be just three-fold. There will be no death by PowerPoint. You've seen enough
[00:31] of that today. In the beginning, I'm just going to start talking off with a little bit of the context, why we exist, what problem we're trying to solve. And then I'm going to bring for you two demos. The first demo just showing that through prophecy, how you can agentically build a data pipeline in a
[00:44] jify. And then secondly, if you have legacy ETL workflows, maybe something living on say Alterix for example, how you can easily import that into a prophecy pipeline that executes on data bricks SQL warehouses.
[01:00] So the first problem and this is actually more of a question that I have for you. How does your business team currently prep data for data bricks? Um typically what we see in the field is that it's always a conversation between data engineers and data analysts or
[01:12] business analysts. Business users will be sending requirements for data engineers to basically build for them. But oftent times it's not really exactly what you expect to see. So it's usually an iterative iterative cycle where requirements get sent in. We give you
[01:25] something back for the business users. it's not what they want. Because of that, business uh data products can actually take weeks on end in order to produce rather than just probably the few minutes if not hours that's needed to actually accomplish it. So that's one
[01:39] of the first that's one of the the first disparities that we see today. On the left, you have that dichconomy between business users and engineers. Well, what if we give them a tool? Um and that's that leads to our second problem, which
[01:51] is enabled but with a little bit of anarchy. There exists a lot of desktop applications today that do data preps. But the problem with that is that data bricks is based in cloud. So if you want to work with anything on your desktop, it remind it requires copies of data
[02:05] ending up in people's local desktops. That's not really an acceptable alternative for a lot of enterprise data governance uh requirements. We need a solution that enables the business users to prepare data as if they had the if
[02:17] they h as if they had the capabilities of that of a data engineer but within the framework of a govern solution centered around data bricks. This is where prophecy comes in. Prophecy is the agentic data prep
[02:31] solution that's designed for the simplicity and ease of use for a business user but built on top of a sophisticated data engineering platform. So what I mean by that is that for business users when it comes to building
[02:43] a data pipeline, they don't have to know code. Our users today are actuarials, accountants, investment managers. These are folks that didn't go to school to learn how to code, but they still have requirements for building data pipelines
[02:55] as far as prepping data for downstream use cases. And they accomplish that within prophecy by either dragging and dropping which is kind of the more canonical uh approach or using our prophecy agent which obviously is built
[03:08] on top of claude understands your intent and will build the data pipeline for you and allow you to iterate and test on it as well. And from the data engineering side we want to make sure that our tool fits very nicely into your enterprise
[03:20] tech stack. Everything we generate is open source code. I'm going to show it to you that underneath the covers everything is simply just open source SQL. We use DBT core as the compiler so that we build the SQL tool the SQL from
[03:32] that. That way we check directly into your git. You want to schedule it on your enterprise platform for enterprise scheduling or maybe check it into your CI/CD processes. This is all going to be available out of the box.
[03:46] So without further ado, let's get into our first demo. Uh the first demo I want to show you is how quickly you can build a data pipeline just with a few simple prompts. And for my example, I'm actually going to grab it straight out of the website of AltterX. So maybe
[04:00] you've heard of Altterx before, maybe you have Alterix seats. I'm not sure what your affinity towards it is, but from my perspective, from our perspective, we noticed that a lot of enterprises are looking to move away from desktop solutions. What you see on
[04:12] the screen right now is an example of one of the use cases, one of the challenges that people put together. This use case is one that's you know medium complexity. Basically you're an analyst. You need to figure out you need
[04:24] to do a little bit of analysis into why banks have failed over time. Altrix will give you the two sample data sets. Go ahead and accomplish these two tasks over here. Figure out the number number of days a bank has exist existed for and
[04:37] then also determine which bank failed in the shortest amount of time. Now with an alterix this would be like a drag and drop solution. I would imagine that it would take around 45 minutes to a couple hours in order to accomplish this and
[04:50] get it running in a in just a prototype stage within Prophecy. I want to show you that I can take I can do all this, get my answers, and do more in just a few minutes. So, what I'll do first is
[05:02] I'm actually just going to copy this prompt, go straight into Prophecy, and just pop it directly into my agent here. Now the title of this uh presentation is data prep with claude uh
[05:14] with claude right and some of you might be thinking well I could just use claude to do this right well one thing about claude is that it's actually quite non-deterministic you can have the same prompt 10 times and it will it it can potentially come up with 10 different
[05:27] approaches to try to figure out the answer and you might get the same answer eight out of 10 or nine out of 10 times but as an analys as an analyst that's actually having to make business decisions that's not really acceptable
[05:39] With Prophecy, what we've done is we've trained our agent, our Prophecy agent, directly on top of Claude. We've given it its specialized list of skills to make sure that everything it produces follows our definition of standardized
[05:52] highquality code. So, that's what our agent is doing right now. In fact, our agent is doing a number of things. Notice here that I gave it a prompt, but I didn't really tell it or give it any kind of clues as to what data sources to use. That's because our agent is
[06:05] intelligent enough to go through Unity catalog as well as any other data sources that I permitted to see. So, if I take a look over here, um, I can actually go ahead and see all the different data sources that I'm
[06:17] currently connected to. Oh, uh, I just lost internet. Oh, internet's back, I hope. There we go. Internet's back.
[06:31] So what what the prophecy agent is doing is it's actually simply just going through all of the different data sets that I have connected to. So this is all governed and defined by Unity catalog. That's great. My agent can only read and
[06:44] write to tables that could actually that actually exist and I have access to. But if I wanted to, I could connect it to more data. So it's not just limited to reading and writing data inside Unity Catalog. I can connect it to external uh
[06:57] on-prem or even cloud-based databases or data warehouses. But more but more commonly we can also connect directly into file storage systems. This is Excel Excelss or CSV files that's living inside your SharePoint share drive. Uh
[07:10] even your even your shared network drive. We don't really care at the end of the day. We can connect to whichever data source you want us to do. Our agent is simply just reading the metadata to understand what's inside each of these
[07:23] files. No need to read any data at all. Now, what's going on on the left hand side though is really interesting because what we've done over here is we've integrated Claude into our solution. But we're not just producing code. We want to give you the answer,
[07:36] but we want to give you an answer that you can verify visually. And that's how and that's what's being produced over here. You'll notice that within the screen cla agent has generated a couple of these different components. The
[07:49] prophecy lingo for these is they're called gems. Within the world of Altterx, you may know them as tools. Each one of these gems you see here represents a step within the data transformation process. Right? So, you're reading it from left to right.
[08:02] Now, when I get my answers at the end of the day, I can also verify step by step what prophecy did in order to accomplish this task. It knew that in order to get me the answer for which bank failed in the shortest amount of time, I had to
[08:15] read data from these two different data sets. I have to then join the data together and then accomplish all of these following steps that you see over here. And for you for those of you who don't want to actually go into each of these gems, you can still drag and drop
[08:28] and do all of this all of this within here. But what I like to do is I like to just have prophecy explain the pipeline for me. So it's given me the answer. I wanted to actually explain the pipeline just from this perspective as well. You
[08:41] can see that what prophecy does. We already have the metadata. We already see the code. It's not very difficult for us to generate a highle summary as well as a gem by gem or a step-by-step description of what this pipeline
[08:53] actually does. You can see on the left hand side I have my answer over here. Um the original alter question was find find me the bank that failed in the shortest amount of time. Waterford Village Bank is the name of the bank. It only lasted for about
[09:06] 879 days. If I wanted to now examine this pipeline and see what the data actually looks like, I certainly can. All of these components are easily quer or easily executable and all it's doing
[09:18] is it's sending SQL code directly to your SQL warehouse. I can open up the preview of this do a little bit of analysis into my sample data that I see here. I can take a look at some expanded profiling to see any kind of data
[09:30] quality issues like missings or nulls and I can deal with it just using my AI platform right here. So, so far so good. Um, I was able to accomplish this task in just a single
[09:42] prompt. I did get a little internet hiccup, but that's okay. Um, a minute and a half is all I need. Let's just say that now that I'm taking a look at my taking a look at this graph or this sample output. I can see that for all
[09:54] the banks that have failed, I can also see the states that they belonged in as well as well as their acquiring institutions. So, why don't we actually try this? I have some time. Yeah, I got 10 minutes. I got time. Let's go ahead and add an additional prompt. And I'm
[10:07] just going to take this prompt I have here. I want it now to show me the top 10 acquiring institutions and give me a bar chart and also show me the failed banks and group it into four
[10:19] major geographic regions. And I'm going to just let Claude define that geographic region for me. I'm pretty sure it's smart enough to do that. And exactly what you have going on going on here, what the agent is doing is it's comprehending the intent, but it's doing
[10:32] something more than just building a pipeline. This part is a little bit underrated. It's also doing unit tests. So, some of you data engineers are familiar with doing unit tests and data tests. We want to make sure that what we produce actually turns out a result. So,
[10:45] that's exactly what the agent is trained to do. We've included our skills that teach it how to do unit testing on the fly. It knows how to generate these unit tests to make sure every single one of these gem components you see here actually produces a result. And as you
[10:59] can see here, it compiled successfully. You see these additional results. Once it finishes testing, it's also going to uh display the answers over here as well. While this is thinking, one thing I want to show you, I want to go
[11:12] directly into the code. I did mention earlier that Prophecy is an open-source solution. This means that all the code that we've been generating underneath the hood is open- source code. We're just using DBT core as the build. We
[11:25] don't use DBT cloud because that costs money and you would have to pay that. We use We use DBT core because it's free and it's portable. I know that some of you might be thinking about using LakeFlow Designer using Spark declarative pipelines. Fantastic tool
[11:38] I'm pretty sure. But when when data bricks announced LakeFlow Designer, it simply just affirmed what we've been doing for 10 to 15 years already. And we've been working directly with the end users, the business users, they like to see this kind of user interface. They
[11:51] want to have it portable as well, so they don't get locked into one particular cloud data warehouse. DBT is DBT. I can run this on a variety of data warehouses such as BigQuery or Red Shift. if I if I wanted to as well.
[12:07] So going back into the visual, I was able to produce the additional pipelines. It was able to answer for me the questions. I did want I did say I wanted to see it in a bar chart and a pie chart as well. So if I switch over to this tab over here, what it's going to show me is just a live dashboard
[12:20] that's running directly against data living in my warehouse or my external sources as well. Okay, so this is really cool. It's an Agentic data prep solution. I just typed in two prompts and was able to and was
[12:33] able to build all of this code, these nice visuals. If I wanted to do additional train changes, it's always just a matter of drag and drop. Right now, this is really cool, but I have a lot of tech debt from my existing ETO uh
[12:45] from my existing ETO provider. Right, which goes into my next demo where I want to show you that we also have the ability to import workflows directly from any kind of thirdparty legacy ETL
[12:58] provider. It's not just Alterix. We work with uh we work with Abeneio, Informatica and data stage as well. We can take those mappings and convert it directly into prophecy pipelines as well. I can see the screen went dark. Uh
[13:10] there we go. Um so let me jump back into the prophecy workspace and show you how easy that import's going to be. I'm going to go directly into here. I'm going to import a workflow. These are
[13:23] the different providers that we can we can significantly help you out with. Usually our transpolation our import uh results are about 90% success rate. The remaining 10% is just going to be a matter of some manual configurations you
[13:35] would do on our workspace which is also low code and very easy. But let's take an AltterX one for example. So I'll go to AltterX. I get to choose what language I want to convert the pipeline into. Do I want to
[13:47] write it into SQL or do I want to actually just generate Spark code? That that option is entirely up to you. I'll do SQL.
[13:59] Pick the team. Pick the project. The project just simply means what repository I I want to actually save it into. This is a git repo. You guys own the code. Select the warehouse. And the next step here, which is probably the important part, is simply just picking the file
[14:13] that I want to use. So, I'm going to grab this example here. This is a zip file that contains an Alterx workflow. It's a it's a it's a mapping file, if you will. First step I'm going to do is just to validate to make sure that all of the
[14:26] components are actually convertible in the first place. This process that I'm showing you here right now does not involve AI at all. All our trans all our import capabilities are just purely deterministic. They're just compilers
[14:38] that we built that we've been building for the past 10 years. So now that I hit the transpile button, this basically just converts that code from Alterix into prophecy pipelines that are going to run very nicely on data bricks.
[14:52] Hit the import button. This is this is basically the report card that shows you everything that I converted over from Altterix. And let's actually take a look and see what Prophecy actually produces. And you can
[15:05] see over here that this is a pretty pretty big pipeline. If I were to fit this on the screen over here, this is what it would look like if I took an AltterX workflow, converted it into a prophecy pipeline. All the sources and
[15:17] all the targets are being carried over. If it was reading a file from an SFTP site, it certainly can do that. And now that it's inside prophecy, all the context awareness is there as well. I can start asking it questions such as
[15:30] very simply, explain this pipeline to me. What are the sources and targets? So I can now start inquiring about what
[15:43] this pipeline actually does. And this agent is not just there to answer questions. It's there. instead of do things, right? I showed you earlier, type in a few prompts, it's going to produce the whole pipeline. If I wanted to type in more prompts to have it generate parameters or or new
[15:57] visualizations and have it have it email to you. All of these are things that we can do right out right out of the box, too.
[16:09] Highle overview is going over here. Now, it's just going to be nicely cleaned up. I can see the sources and the targets. And once again, just to reiterate the point, this code that we've generated will not lock you into any product because this code at the end of the day
[16:22] is based on open- source DBT uh DBT code. This is all just SQL that you you can actually go in manually and edit if you'd like. So we support both a visual to a code as well as a code to a visual
[16:34] kind of uh birectional birectionality. It means that now your analysts and your engineers can live in the same space. their productivity is going to be significantly boosted because they're no longer just exchanging requirements left and right. They're able to work within
[16:47] the same workspace as well. So to sum it up uh just in terms of some of the things that we covered
[17:00] just want to mention that from a prophecies perspective we want to be your solution for introducing data prep in a govern in a governed manner to your greater analyst community. Uh we have integrated cloud code but a specialized version of cloud code to make sure that
[17:12] the experience is a lot more deterministic. You have the visual workflows that you can still drag and drop directly into in order to make edits and fine-tune customizations. Everything is going to be integrated
[17:24] into your existing enterprise tech stack, meaning that it checks directly into your git. It's open source. You're not locked in with any particular vendor. We have the complete data life cycle. And if you have to import any existing workflows to get out of like
[17:37] existing contracts for legacy ETL tools, we're here to help as well. I see that we ended this session a little bit earlier. I wanted to thank you for your time. Um, if you have any questions, I'm going to stick around and help answer anything. We also have a
[17:50] booth over there in the back around zone 600. Enjoy the rest of your time at Summit. Thank you again for your time.

Learn more about the Databricks Data and AI platform.

The information provided herein is for general informational purposes only and may not reflect the most current product capabilities or configurations.