Leveraging GenAI for Synthetic Data Generation to Improve Spark Testing and Performance in Big Data
Summary
- Testing challenges: Spark jobs often work in development but fail in production due to insufficient datasets and inadequate documentation, creating gaps between local and production environments on the Databricks Data and AI platform.
- Synthetic data approach: Generative AI can create realistic synthetic datasets for local testing, enabling developers to identify performance bottlenecks, optimize job performance, and recognize scalability issues early while keeping costs low.
- Deployment confidence: This video demonstrates how synthetic data testing improves deployment practices and enhances the reliability of Spark jobs across development, staging, and production environments.
Leveraging GenAI for Synthetic Data Generation to Improve Spark Testing and Performance in Big Data

Talk by Satej Kumar Sahu, Principal Data Engineer, Zalando SE
Testing Spark jobs in local environments is often difficult due to the lack of suitable datasets, especially under tight timelines. This creates challenges when jobs work in development clusters but fail in production, or when they run locally but encounter issues in staging clusters due to inadequate documentation or checks. In this video, we’ll discuss how these challenges can be overcome by leveraging Generative AI to create custom synthetic datasets for local testing. By incorporating variations and sampling, a testing framework can be introduced to solve some of these challenges, allowing for the generation of realistic data to aid in performance and load testing. We’ll show how this approach helps identify performance bottlenecks early, optimize job performance and recognize scalability issues while keeping costs low. This methodology fosters better deployment practices and enhances the reliability of Spark jobs across environments.
Here’s more to explore:
Production ready data pipelines for analytics and AI: https://www.databricks.com/solutions/data-engineering
The Big Book of Data Engineering: https://www.databricks.com/resources/ebook/big-book-data-engineering-2nd-edition
See all the product announcements from Data + AI Summit: https://www.databricks.com/events/dataaisummit-2025-announcements
FAQs
Why is testing Spark jobs in local environments difficult?
Local environments typically lack suitable datasets, making it hard to test jobs realistically. Jobs may work locally but fail in production clusters due to data variations, undocumented edge cases, or performance issues at scale.
How can generative AI help with Spark testing?
GenAI can generate realistic synthetic datasets tailored to your testing needs. This video shows how to create varied and representative data that captures edge cases without requiring production data access.
What benefits does this approach provide?
Synthetic data testing enables early identification of performance bottlenecks, optimization before deployment, recognition of scalability issues, and cost reduction compared to testing on large production clusters.
How does this improve deployment practices?
By testing thoroughly in local environments with synthetic data, teams can deploy with greater confidence. This approach reduces failures in staging and production, leading to more reliable Spark jobs across all environments.
Full transcript
Hey hello everyone hope you are able to hear me. Uh so I work at Zando and today's topic is about leveraging genai for synthetic data generation and at the same same time about improving spark testing and the performance and big data. Just a very short disclaimer that all the thoughts expressed here are totally my own and personal and does not reflect my employers or my previous employers you know any work uh environment.
Uh so the primary goal of today's session is we have talked about you know how databicks is striving towards democratization of data about democratization of AI how to put AI and data into the hands of software engineers. Now what is the third aspect we are trying to target here? This is about democratization of operational optimization. So this is the one expect which we'll try to uh cover here. This is a very optimal you know kind of environment which you can think about when you see in the horizontal expect you see the environments where developers can work in the local they can push their changes to the staging and then the production environment and at the same time in the vertical expect you can see there are data sets you have compute you have the test cases now there are some components like when you're developing in a local environment you have variable based data sets or maybe a very small uh you know park file or delta file.
The second could be when you're testing in the local environment you just have a local data compute based on your driver and the third is for testing your using unit tests when you push to the staging environment. uh most of the time what happens is you do not have a production data you know staging data set you rely on your production data set and more often than not it is more about how your company cultures because maybe you're relying on some data sets from some different teams which do not have the staging data set.
So you're you know you have to work on the production data set and when you're testing on the staging environment you're more often than not using an interactive cluster and there you're doing some manual test or running a job uh just to you know make sure that your job is running out of out of the box. The real uh challenge happens when you move to a production. This is where you're using a job cluster but you're not sure about when your test cases will fail or how do you you know debug it.
So I think to optimize this out you'll see more often than not developers spending a lot of time actually finding out the problem rather than solving that. So sometimes you might hear your you know data engineers working on a particular problem uh which might be over a week or maybe to speak a minimum of 3 to 5 hours and they might be saying okay I need to fix a particular bug because maybe it's over the it's it's out of memory error or some other aspect of you know big data engineering.
So this is one aspect which we'll be trying to target in today's session. We all know about databicks AI assistant which is a very handy tool if you see it in the right. So whenever you write something and you make some error it can give you some handy tools and you can very quickly question it and if you see the context switching is very less here because you're typing on the left you can switch to the right and very quickly search the answer and it gives you the option to copy the answer back to your notebook.
Now what happens when you run something in the production? So when you run something in the job cluster, the AI assistant doesn't have any context of what your error is, what your data cluster is. So it cannot give you the direct fix for it. So you can ask some questions of it like I'm facing this particular error. What should I do? But you have to do some context switching here. And it doesn't the main I think the drawback is that it doesn't have the context around what your data engineering job is doing.
If you go around some documentation around spark UI and data bricks spark UI is the go-to document where you can find what is the issue which is happening. And if you see here this is a sparki and some of the uh pages here like what is the timeline of an event which is happening and uh some of the executors which are added removed. So if you go to the data bricks documentation they have a very clear guideline as to what do you need to do where you need to start with you start with your event timeline.
See you know what are the executors which are added removed which are some of the jobs which are taking a long time and if that is not the case you move on to the you know the task how many tasks have completed and what is the spill memory dis spill and eventually to some of the summary metrics. And once you figure out the problem, then comes the solution part which it could be an optimization solution. You need to write some optimized code or you need to make some configuration changes very specific to your maybe SQL or you know the environment which you're working upon or the format change.
Maybe it requires you're working on park you need to convert it to delta and the solution guideline is also in the data bricks document. Now if you think about this one of the very core challenges is the documentation is somewhere in the browser and you are on another tab where you're facing the whole job cluster error. Now you have to switch a lot right and you could be an expert here with 10 to 15 years of experience in data engineering and this could be a very you know cool expect for you to debug it find a problem but think about someone like 3 to 5 years experience they come into your company uh they want to work on this and they face this problem so one of the very first things which they will say is how to optimize this they have two options one is to go through this documentation read it one by and figure out some of the concepts because I think the documentation is clear but if you go into it very deep into some of the structural elements which you need to understand.
The second part is once you understand it you have to follow the sequence like checking from one page to another to see where the actual issue lies and then you know make some changes to it and eventually see if it fixes your problem. So the actual problem in this if you recognize is finding out the problem not solving it. So one expect is all of the spark UI which we saw right now is all generated from a single log file.
In some cases it could be more than one log file but it's a single log file in a simple case where the spark UI parses it and then shows you that information in a very structured you know page format. This is a very uh local file which I have generated. If you see the format here, it's very similar to a JSON L format where you see that the spark listener error started and some you know very simple uh statements here which says about tasks and other statements.
These are some of the expects like the spark listener application has started the application end event. Uh the listener log start event and executor added. So all these are different events which are added to that log file. Now this is where the event log is. So it tells you about the completed jobs and the stages that comprised of and the number of tasks within each particular stage and it's a very hierarchical uh element of uh size.
Now one expect if you think about is that it's very manual enough it's very sequential if you think about if you think about the expect you have to go through some documentation if you have your context you can find it and query about it and correlate in this expect if you think about the large language model if you feed this information about the problem statement and the sequence of events it needs to follow to find your problem it can do a better job at it because these are kind of very sequential steps following one at a time.
And if I think about these five lines which are from the documentation of data bricks, it tells us what to do in a stepwise manner to find out the problem. If I just feed these statements to the large language model, it can also go through my logs instead of me going through the EUI. So it can find that solution for me. And this is what uh a very few short example I can give it. I can give it some examples. Okay, this is a log.
This is what I what the solution looks like. And if you think like there's a user query to it and the event context and there's a prompt and also the solution dock because not only it has to find a problem but also give a solution to me and this is more of a better way to put it. Now if you think about it, it's a very simple large language model with the context feeding and it's more about prompt engineering. It's simple.
It's a good starting point. But some of the cons are you don't find the exact solution because your context could be very long and you always have a you know very small prompt window. You cannot feed everything into it. And when you think about the solution, the problems and all of it, it's a lot of documentation around Now this is what a classic rag looks like for everyone to and I hope everyone knows about rag model where you're feeding some document and you're trying to search based on similarity.
The questions to ask you is could rag help us here with some prompt and some trade-offs. So here one configuration is we can give an AWS S3 to be mounted for your datab bricks. So that whatever your events are generated are stored in S3 and when your S3 is stored you can have a lambda job which can you know fetch the data and put it into kind of open search which is an embedded data store. Here you can use any data store for your example and you can query it based on a rag query.
So this is a very simple rag architecture. So some of the pros of it is that you have a improved context window. You could explore different models which give you better groundedness and you know help. When we using rag you have better groundedness because it's more documentoriented. One of the side effects is that your context could be missing because whenever you are chunking your data some of the relation might be missing and you still have a large context window.
This is where graph database can help you because if you see here you have a job and you have stages below it and you have tasks related to it. So it's a very related schema if you think about it and at a very particular time as for the documentation you need to start with the job layer then you proceed to the stage and if you think about it you don't need all of the information from the contextual log event you start with the small and then proceed to the next.
So this is where if you format it and store it in the graph layer with relationship to each other you can get a very small context and then figure out if that solution helps you. So if you see here whenever the log is generated in AWS S3 here in particular then you have the AWS lambda you're also passing the relationships and storing it in the graph DB. You can replace the graph DB with any graph database of your choice here and you're using the AWS bedrock or any large language model to query your graph database.
So when you query your graph database, you're fetching the relationship first. Let's say the particular job ID or the jobs related. Once you have the jobs, then you can query your vector database to get more information relevant to it. Now this was about the problem statement but what about finding the solution for it. So this is where if you see on the very left part now we have another part here where we have the solution documents and this is what we are passing through AWS lambda.
So now we have two vector stores which you can query to find the problem and after you find the problem you go to the solution part. Now the second part is all this is is very good enough when you have a very static document but let me bring you to one use case here in particular where if you think about the documentation suggests that once you go from one step to another all of the configuration and the changes might not help you at one particular time you need to go into the data cluster and see what is your data size you have to calculate that you have to apply some formula.
Let's say your suff is around 580 GB here. These are some of the steps which you need to follow to calculate the approximate shuffle partition size. So this is where a large language model cannot help you. It can only suggest or maybe hallucinate some numbers for you. But doing this mathematically is not possible for it. So this is where if you think we can bring an agentic model here and if you see the Python tool on the top right this is where you can use the Python tooling interactive to get you that calculation.
So it knows that particular step that it needs to get you that calculated number of partition size it can calculate it and give it back the result to you. So this is another incremental improvement on top of it. Now we got the solution, we got the problem because I think finding the problem was the very crucial part. How about validating it? Because again I have to run it somewhere. It costs me some you know dollars to run it.
This is where we can use a datab bricks tool because we can have a databicks interface through its APIs and command line interface where once we get the solution from the overall ecosystem of large language model we can apply that in a database tool. One challenge with this aspect is you cannot run everything on your big data because it will cost and it may not be the case that the very first solution you're applying is the right solution.
So you have to try the you know the different approaches of solutions it is trying to give you. This is where I think synthetic data sets can help you because you don't work on the whole production data set and eventually you have your regulation of data sets. This is where if you think about the law of scalability, you have a very large compute which works upon terabytes of your data. But now when you have a bring it to a very small compute size, it can eventually give you similar results when you work on a very small data set comparably.
That is how the law of scalability works here. So now if I bring the S3 data from production from left using a large language model or you know a faker model, you can simply replicate your data. Large language models have the capability to hallucinate. So it's always good to have some guard rails to make sure that you're not creating some very random data and once you have the you know the small data set you can work on a very small compute.
Now this is where at the top you see it's generating the small data set and you can now run multiple small datab bricks environments which are small enough and which are not costly for you to try it so that it can give you hints that this was the most optimal solution for you. Now very similarly the other aspect of it that feedback is very important. We saw a very simple uh one loop mechanism here. But if you think about it's a very iterative solution.
You can go to the end apply it and get the feedback and again think about a new solution to it. And if you think about as data engineers we are doing that we go about the solution find the problem apply the solution apply it and see what are the result and we go back and feedback you know. So I think agentic AI really helps you here. So this is one example where you know the this is the output which the model gave. So the shuffle read and shuffle right it got this output from the logs and it also has the cluster size information and here it is with the Python tool.
It is able to calculate the optimal shuffle size which is you know the number of partition should be 80 at the bottom. And the final solution which it gives me is that this configuration should be what which I should apply it to. So some of the uh results of the findings are here but I I would like to point the last point here where the time spent by data engineer to find a problem from 5 to 8 hours to just 5 to 15 minutes is very productable because I'm not talking about the time to find the solution but to find the problem we don't want to spend time you know finding the problem we want to solve the problem.
Right? So just a few points here I just wanted to share. User experience is very important when you're thinking about creating this interface because where do you show this because it has to be very contextual when the data engineer is working. Should I pack it all together? It may not be because we very experienced engineers who may not need this. You have to think about how do I start incrementally by asking the user okay what do they need in first the third is call to action.
So whenever you're having a user interface it's helpful let's say not to expose the whole datab bricks environment to them but ask them okay I have this datab bricks tool who do you like me to uh apply the solution here. So that gives you uh you know kind of the feedback too which you can receive and we didn't talk about memory here but whenever we are conversing with a large language model or NLP interface it's important to remember the memory or the past conversation so that it can give you very quick results and the last two points are I think observability is very important because you cannot just expose this whole system to all data engineers you may have some guard drills around you know the cost structure around the responsibility of the data engineers the importance of the projects so that because sometimes it can get out of your hands the cost you know environment altogether uh this is a very small uh you know introduction about myself uh there are some of the books I have written for all attendees of the conference there's a datab bricks 20 code which you can apply there's another book coming for generative AIdriven application development with Java uh in the summer of 2025.
I would love to hear you know any feedback and you know how this idea resonated with all of you. Uh thank you so much.
Learn more about the Databricks Data and AI platform.
The information provided herein is for general informational purposes only and may not reflect the most current product capabilities or configurations.