Improve AI Training With the First Synthetic Personas Dataset Aligned to Real-World Distributions
Summary
- Real-world grounded personas: NVIDIA and Databricks present a novel compound AI approach combining LLMs and probabilistic graphical models to generate synthetic personas aligned with real demographic statistics.
- Addresses critical training challenges: The open-source Nemotron-Personas dataset solves bias, licensing, and persona-skew limitations of existing synthetic data generation methods on the Databricks Data and AI platform.
- Diversity and quality at scale: Leverage synthetic personas with real-world distributions to improve model training while addressing model collapse and data provenance concerns head-on.
Improve AI Training With the First Synthetic Personas Dataset Aligned to Real-World Distributions

Talk by Dane Corneil, Senior Staff Applied Scientist, NVIDIA ; Yev Meyer, Principal Research Scientist, NVIDIA
A big challenge in LLM development and synthetic data generation is ensuring data quality and diversity. While data incorporating varied perspectives and reasoning traces consistently improves model performance, procuring such data remains impossible for most enterprises. Human-annotated data struggles to scale, while purely LLM-based generation often suffers from distribution clipping and low entropy. In a novel compound AI approach, we combine LLMs with probabilistic graphical models and other tools to generate synthetic personas grounded in real demographic statistics. The approach allows us to address major limitations in bias, licensing, and persona skew of existing methods. We release the first open-source dataset aligned with real-world distributions and show how enterprises can leverage it with Gretel Data Designer (now part of NVIDIA) to bring diversity and quality to model training on the Databricks platform, all while addressing model collapse and data provenance concerns head-on.
Databricks Named a Leader in the 2025 Gartner® Magic Quadrant™ for Data Science and Machine Learning Platforms: https://www.databricks.com/blog/databricks-named-leader-2025-gartner-magic-quadrant-data-science-and-machine-learning
Build and deploy quality AI agent systems: https://www.databricks.com/product/artificial-intelligence
See all the product announcements from Data + AI Summit: https://www.databricks.com/events/dataaisummit-2025-announcements
Chapters
0:00Introduction to personas1:05Persona Hub and application4:44Benefits of targeted synthesis8:01Need for real-world grounding9:51Probabilistic graphical models14:41PGM versus LLM approaches16:55Analyzing persona correlations19:26Personality trait integration21:13Putting it all together22:29Nemotron-Personas dataset26:16Persona record walkthrough28:31Persona diversity examples31:33Professional persona clustering32:52Nvidia NeMo framework33:32Compound AI system benefits35:57Data designer implementation37:09Conclusion and questions
FAQs
Why use synthetic personas over real data?
Synthetic personas address limitations of human-annotated data (scale, licensing) and purely LLM-generated data (distribution clipping, low entropy) while avoiding model collapse.
What are personas in the context of synthetic data?
Personas are compressed representations of knowledge carriers—unique profiles like 'golfer,' 'pediatric nurse,' or 'data scientist' used to diversify and improve synthetic data generation.
How are these personas grounded in real-world data?
The Nemotron-Personas dataset combines LLMs with probabilistic graphical models and real demographic statistics to ensure synthetic personas reflect actual population distributions.
Can enterprises use this dataset on Databricks?
Yes, the open-source Nemotron-Personas dataset is available for use with Gretel Data Designer on the Databricks Data and AI platform to enhance your model training workflows.
Full transcript
0:00 Introduction to personas
All right, I think we're going to get started. Hopefully you all can hear us. Thank you very much for coming so early to the session. Uh D and I are very excited to um share this work with you, discuss um a new synthetic personas data set that we have developed and how you may want to use it in improving your AI models. And so here's a quick look at the agenda. We'll briefly touch on what personas are. Uh we will discuss why one would want to align personas to real world distributions. uh how we went about grounding this dist uh these personas in these distributions.
We'll present the data set uh and then we'll touch on uh some of the capabilities uh that allowed us to generate this data set pretty quickly and how you can use them as well. Okay, so what are personas? Um you know the word itself is not uh new. So in a traditional context, we're all familiar with persona being a description of an individual unique characteristics, their knowledge, their life experiences, their profession perhaps.
1:05 Persona Hub and application
However, what's interesting is that last year um 10 cent uh published this paper on scaling synthetic data generation with 1 billion personas where they put a completely new angle uh on personas. And specifically what the group suggested is to view personas as compressed representations of knowledge carriers. What does that mean? Well, if we think about the world body of knowledge, think internet uh data, data that is readily available for us.
If you look at a specific article, uh you may say, well, you know what, this specific piece of data is very much applicable to a specific persona. It could be that persona uh wrote that article, created that data. It might be that they read it. It might be that they're really interested. Maybe it's their hobby. And so the idea was to take that body of world knowledge and compress it down to personas. And so some examples of personas include a golfer, a pediatric nurse, perhaps a data scientist attending data and AI summit.
Um and what Tensson has done is published a persona hub with many such personas generated and um the idea then is to use these personas to really inject them and decompress that knowledge at generation time. So if you're inferencing from an NLM, if you're generating synthetic data, you may want to use these personas to diversify the kind of data that you get as a result. Um and what they have demonstrated is that personas really enable us to enforce this diversity and complexity at scale.
Right? So think about generating not just a few records but thousands and millions of records. How do you ensure that the data stays diverse? And so one of the examples on the bottom is um very simple. You would want to let's say you want to create a logical reasoning meth uh a logical reasoning problem involving a particular persona. You have a choice of which persona to use right? could be persona 1, two, three, four, however many personas you generate.
Um, and to make this even more explicit, you can literally create any kind of data with a particular persona in mind. And so you can create a math data set, you can create a coding data set. Uh, here are a few examples from the persona hub. Uh, so the initial uh, release included 200,000 rows. And just looking at some of these persona descriptions, you see that they mostly capture a profession of an individual. So you have a political analyst specialized in El Salvador's political landscape.
Uh perhaps you have a determined basketball player who aspires to be the star athlete of the school. Um or you have an engineer with a shared sense of humor who has known the comedian since grade school. And so out of the gate, you see that a the focus is primarily on what these people are doing, but b sometimes there just isn't enough information to really get at the persona, right? So in the case of this engineer, uh with a shared sense of humor, shared with whom, right?
And when we say the comedian, which comedian? Um and this gets at some of the issues with using uh internet data. Um this data set uh was also published with a non-commercial license and that's an important consideration because it becomes challenging to use something like this if you wanted to build an application especially enterprise uh grade application. Here are a few examples of how personas can be used. So again remember we can take a persona and we can very much target what kind of data we want to generate.
4:44 Benefits of targeted synthesis
So the focus is on targeted synthesis. Here we're showing an example of generating math problems. And so you use an input persona. In this example, a librarian who is curious about technology and occasionally asks for text support from their family member. And you get a synthesized problem. Librarian is trying to organize uh the library's digital database. The library has a total of 10,000 books and you basically end up setting up a math problem.
And so imagine doing this at scale. You can be leveraging thousands to millions of these personas. Um and that's what Tensson demonstrated, right? So taking just an off-the-shelf model and using as little as just um a million samples, uh you can um really achieve uh pretty great performance. What you will note here is going from Quen uh QN27B um to something u fine-tuned on 1 million examples gave a 15 point boost um on this specific uh benchmark.
And what's interesting is that uh this performance also increases with the number of samples that you're generating, right? Really getting at that diversity piece. The more diverse personas you bring in, the better you do. Um so, okay, this is great, right? And obviously Tencent is using it. Um not that many foundation model providers are disclosing exactly what they're doing. Uh Len Institute for AI is one of those great places that publishes um and opens up quite a bit.
And they also in their recent technical report on Tulu 3 emphasized how important it is to synthesize data for target skills. Um and there's you know the section 312 where they say that uh correctly point out that generating diverse and high quality data at scale is quite non-trivial. Um SLMs are susceptible to falling into repetitive modes of patterns. Right? And we all have seen this if you just interact with an LM.
Um you see how it often uh produces data that is very similar. And so they conditioned on the Persona's data set from Persona Hub to generate prompts targeting specific skills such as instruction following math and code. And so in the paper they linked to these data sets. These are also published um on hugging face. And you can see they're very specifically targeted towards math towards code. Really the world is your oster um as to what you uh can do with this.
Uh real quick just to emphasize how big um of a portion of the overall data set this is. What we're showing here is the number of samples as a function of number of tokens in sample on the lock scale. And so you see this nice distribution. Keep in mind it's lock scale on the bottom. On the bottom we're showing um uh number of tokens per sample, right? And again think for example long context, right? You would want to generate you would want to target uh synthetic data generation to produce something in long context.
You also want it to be diverse. And so what I'm highlighting here is uh in blue this is the persona math portion. This is the coding portion. Uh this is another um math focused uh data set. And so it's really a really sizable part of the overall mix. And so all of this is great. Uh we're using this. Why in the world would we want to align personas to real world distributions? And really the answer is that uh internet data is not really the data that you want for your transformer.
8:01 Need for real-world grounding
So this is a message that uh and others have been repeating um um often recently. Uh why is that? Well, because internet data is not really well uh set up and it's often low quality. But in this particular instance, the statement applies to personas themselves, right? We're trying to use personas to diversify the data, but personas themselves are generated using the very LLM we're trying to improve, right? because we're taking publicly available data and we're trying to extract personas from that.
And so, how do you make sure that there's actual diversity in there? Uh, furthermore, when working with WebEx data, the challenge is that the articles, the data you're interacting with, uh, ends up being pretty biased, right? It's pretty biased towards the types of people represented in that data. And so the data that's um usually available on the internet tends to focus on working professionals, people who are most active on the internet and covering a pretty narrow age range.
Uh in another data set published by HuggingFace find personas, the author specifically called this out um that there is a very strong bias towards personas in the education and scientific domain. Right? And so what happens if your application needs to cover other domains? And so here are just a few examples. Um, if you wanted to enhance texttospech models and you wanted to touch on a specific demographic of the population or you wanted to improve customer support chatbots or you wanted to customize your enterprise application towards the specific domain and vertical you're operating in, right?
Well, then you're kind of out of luck because there is no way for you to really get um at these personas. And so, how can we capture a more realistic diverse persona distribution than what is available from web text data? And so with that, I'll hand it over to who will explain how we've done it. Thanks. Um, so yeah, that leads us into the some of the methods that we actually used to generate this persona data set. And the first was uh census data and probabistical models.
9:51 Probabilistic graphical models
So the core idea here is that we can turn the rich statistics available from census data and in our case we specifically focused uh to start on the US census into a generative model of a subset of American demographic details and then we're going to use that to seed personas. So that's going to allow us to uh generate personas that better represent the diversity of um the target population in this case the United States.
So very quick refresher um what are probabilistic graphical models? Hopefully that's uh maybe ringing a bell. Um these describe a network of variables with correlated states that are defined by joint probability distributions. And when that network happens to form a directed cyclic graph or a DEG uh we can use a method called ancestral sampling to sample realistic combinations of variables. So just to ground this in our particular case we always start with a zip code.
So for each individual persona we start by sampling a zip code proportional to the number of people who live in each zip code. So right off the bat that allows us to try sort of enforce the geographical statistics of uh the US. We then start to sample additional variables. So one we sample is um sex conditioned on zip code and we do that based on the census statistics. So with the zip code level statistics that give us the differential um distributions of uh sex within each zip code and so on.
So then for instance we can sample age condition on the first two variables. So both sex and zip code. So just to give an example if we imagine that zip code happened to correspond to a retirement community in Florida and maybe sex happened to be female both of those would uh push the age distribution uh to have higher probability mass on older ages. So this is how we get realistic combinations of variables um within our persona data set.
So yeah, we can continue to sample more demographic details condition on some or all the variables already sampled. So here we have the PGM that we actually used where as I mentioned we always start in the top left corner with the zip code and the majority of the statistics that we are using here were from the census bureau. So that's everything in uh blue boxes. The other uh main source that we used was a paper by Rosminadal published in 2023 um looking at race and ethnicity data for first middle and last names and that was first of all because it's a a massive source of names within the US um from both uh voter registration files cross reference with census data.
So about 150 to 160,000 names each for first and middle and over 300,000 names for last. But it also has statistics on the overall prevalence of these names um and prevalence within um different ethnic communities. So that means that we can carry through that locality specificity and for instance get uh people with more Spanish names centered in Puerto Rico and also correlations between the names themselves. So for instance, if someone has a first name of South Asian descent, it's somewhat more likely that they'll have a last name also of South Asian descent.
So these are some of our actual demographic details uh that um a subset of which we use to eventually see the persona. As I mentioned, we always start with those locality uh details. So those zip codes with their corresponding city and state. Um here we have uh the names which are generated both from uh that sex variable as well as the ethnic background statistics at the zip code level. And you can see that people don't always have a middle name sort of reflecting um the real world.
And then we get into some of these deeper demographic statistics which are also going to allow us to capture the richness of this uh distribution. Um so we have education uh sampled based on zip code level statistics um that's overall education level and bachelor's field if it applies and then we also have occupation which uh reflects the influence of um education and age. So uh solutions do exist for sampling these types of personal demographic details um on demand.
14:41 PGM versus LLM approaches
A very common one is Faker. Uh but these uh existing tools typically generate fields independently without um correlations between them and without reference to real world statistics. So these are really focused on just having data to fill up generally for instance test databases rather than reflecting um real world uh statistics. Um, and by capturing those correlations, the PGM allows us to significantly improve within sample fidelity.
So, uh, do the fields make sense within a single sample? So, for instance, I think an example I already mentioned is that persons in Puerto Rico in the data set are much more likely to have Spanish names, although not certain, um, than in the mainland US. Uh so another solution that you could use for these types of personal details is an LLM um directly and we do use LMS later on um to generate the persona fields but we did not find in our testing that this was a very good solution for demographic details themselves um for seating the data set in particular we actually developed an internal um LM based solution for this problem um actually an SLM fine-tuned on demographic details combined with GPT4 and compared to that solution we had developed um we found that the PGM had significantly improved across sample fidelity.
So one way we see this is in this um age distribution. The PGM essentially by design in blue reflects the actual age distribution you would see within the US population whereas the LLM was very heavily focused on people in their 30s. um which in part reflects the biases of the original chaining distribution, but also just the difficulty that you can run into uh generating independent samples from an LLM for the same prompt and hoping for uh you know some sort of diversity in the result.
Um and generally that's not the case out of the box. Another place that we see um the uh LLM run into some failures is in uh geographic diversity. So this was with our existing LLM solution. Um and it tended to heavily over represent people on the west coast and in states like uh Colorado and New York. This is relative to population. So we're already taking into account that people um tend to live more in those areas.
16:55 Analyzing persona correlations
Um and still again relative to population under reppresent people in areas like the Midwest. So digging more into the data set itself. Um some of the correlations that we see for instance are between uh the age variable and the uh marital status variable which is uh also used to see these personas. So uh for younger ages as you would expect we see most people fall in the never married category. Then we see this trend uh towards married presently and then later the emergence of um labels like divorced or widowed which again just reflect the census statistics.
We see things like geographic diversity and educational attainment. Um so for instance uh there's a higher proportion of people with a bachelor's degree 25 years of age or older in Colorado relative um to the rest of the country. again directly pulling uh from those census statistics. Um another interesting one is labor force participation. So we include a notin workforce label within our occupation category. And you see that um is much the not in workforce label is much more prevalent towards the younger end of this pyramid and towards the older end as people retire out of the uh labor force.
So in addition to these correlations um we can also get uh a huge amount of specificity within the data set um using these labels that we get from the census itself. So for instance uh the US census tracks uh over 560 unique uh occupation categories and we are able to correlate that um with uh education and age. Um, so that allows us to capture the diversity within our downstream personas of the occupations that actually exist in the US while still maintaining some realism.
For instance, that you know there are more cashiers in the population than firefighters. Okay, great. So the PGM allows us to generate these realistic details for a person um that are representative of the US population and that's going to act as prompt seed data for generating a rich and complex persona. Um so we're also going to layer on attributes like personality, skills, and interests while still staying grounded in that target population.
19:26 Personality trait integration
So one of those um personality is what I'll talk about next. Um so in order to generate even more diversity in these personas uh we turned to uh personality science and behavioral science in particular the ocean or big five framework. So this measures personality across five different dimensions um openness to experience conscientiousness extraversion agreeableness and eroticism. And essentially what we do is roll the dice for each one of the personas within this data set on where they're going to sit um along this axis.
So for instance, for one persona we might get someone who is curious and appreciative of art and open to unc unconventional thinking because they score towards the higher end of openness to experience. On the other hand, that same persona might be more uh disorganized and have trouble with deadlines because they score lower on conscientiousness uh and so on. So maybe this is someone who sits sort of in between on the introversion extroversion scale.
They're very agreeable, compassionate, cooperative um and they tend to be uh fairly emotionally stable. So this allows us to get diverse personas even within someone um or even within two different samples with very similar demographic statistics. Okay. So yeah, that brings us to putting this all together. Um how do we do that? So as I mentioned um we uh seed from these two different sources um the PGM uh which gives us these synthetic personal details and ocean the personality traits um which give us these uh personality details.
21:13 Putting it all together
We feed those into the LLM and then we use those to generate uh these four different passages sort of expanding on and adding a narrative to those underlying details. So the four different passages we generate uh as an in between deliverable is cultural background, skills and expertise, goals and ambitions and hobbies and interests. And then we have a second stage um with a second LM. In our case, it happens to be the same as the first uh that takes the output of those in between passages and the original um details uh from the PGM and from ocean within the prompt and generates all of those personas that are really um the meat of the data set.
Um so with that I think I'll turn back to talk about the actual personas. Okay. So um now that we have this machinery and you will you can tell that it is getting pretty complex right but the whole idea is to bring in these real world distribution to ground it in the actual science and the actual statistics both with respect to the epigraphics but also uh personality science. Um so what we're going to do first is show the data set just so you can get a feel for it and then we'll discuss how one is able to actually do that uh much faster and target um other personas.
22:29 Nemotron-Personas dataset
So uh we have just released this at the data and AI summit um uh this u neatron personas data set reinforcing the Nvidia neatron brand and um really we think of this as an open source release to further advance AI research. So, synthetic personas are being used for um lots of different use cases. A big one is obviously generating synthetic data for uh next state-of-the-art um models. And so, we want to make sure that the open source community can benefit from this, build on top of this uh and really advance the state-of-the-art.
Uh very importantly, this data set is a truly open-source data set. It's released with a uh creative common license. uh it includes 100,000 records with 22 fields, six persona fields, and 16 contextual fields. These contextual fields are actually really important because if you want to be able to zone in on a specific subset uh on a on very specific personas, you want to be able to slice and dice that data. Um as I mentioned, comes with a true uh open source license.
Uh there are no llama or other proprietary fibers in there. uh it's using a fully Apache 2 uh solution to generate this. Um the data set has 54 million tokens. Roughly half of that is just persona tokens. The rest of it is um spent on the contextual fields. Uh it's a compre includes a very comprehensive coverage across demographic, geographic and personality traits distributions that we discussed just now. over 560 distinct occupations uh presented in this data all of them grounded in real world distributions and as mentioned down to the zip code right because it's incredible the kind of data publicly available data we actually have but most LLMs today actually not leveraging that data right so when you think about macro diversity um and that sample fidelity it becomes really challenging to generate uh data that is diverse and complex we're also covering uh 50 states as well as Puerto Rico and Virgin Islands and really importantly uh there's u a big coverage of typically underrepresented groups uh along multiple axis.
So think age typically the elder population is not represented in the data that you can find online. Uh same with uh rural personas or anything regarding education and occupation. And so uh real quick uh this is the schema and so you can see we have a 100,000 records. Uh some of the fields are missing very deliberately. For example, bachelor's field is not available for everybody. also not everybody goes to college, right?
And so this is really speaks to representing what's happening um in the US. Uh over 9,000 different cities are represented um over 18,000 different uh localities and um to really reinforce the overall approach um all of these fields have really complex interdependencies, right? They're really rich patterns represented. These are not just independently sampled fields. Um these are conditional distributions between these fields.
There there's a lot of very rich patterning that you would expect to see uh in the real world. Um and a lot of it thanks to uh the probabistic graphical model that we uh show here. And so here is an example. We'll walk through this record. This is just one record uh of a person in their 50s. And so let's u zoom in on this. So here the overall persona is Natividad. A 50-year-old pharmacist with free spirit balances her relaxed approach to obligation with a genuine desire to improve her community's health one personalized prescription at a time.
26:16 Persona record walkthrough
And there's also professional persona description. Right? So the uh persona description uh we showed first is quite generic. Uh we can zone in on a specific facet of an individual. Right? to the professional persona. In this case, Niad, a pharmacist with a holistic approach, combines her scientific expertise with a curiosity for alternative medicines, offering personalized care and education to her diverse Tampa community, often going the extra mile to reach undeserved patients.
Um, and what we're highlighting here are some of the demographic details and other fields that are that you see shine through this data. So first of all we're able to generate first name and last name here. We know that this is uh a female individual individual of age 50. Uh this is coming from the ocean framework uh that you see represented in the personas right that curiosity appreciative um appreciation of art, new ideas, varied experiences open to unconventional thinking.
You can see that infused into the persona. Um you can see the occupation that was generated like specifically for that local for that zip code because we're talking about a pharmacist who went to uh school obviously to become a pharmacist right so there is a lot underpinning that persona and then in terms of skills and expertise uh we have pharmacist who has a strong scientific background with expertise in pharmarmacology pharmaceutical care and patient counseling her openness to new ideas right so you see the ocean framework come through as well um led her to explore complimentary therapies and alternative medicines, making her knowledgeable in areas often overlooked by traditional pharmacists.
She's also skilled in Spanish, a result of her upbringing and her commitment to serving Tampa's diverse community. And so you can see sort of this tapestry within the data, how intricately intertwined it is uh continuing to other slices of an individual, right? Because it's not just profession that is important to us in our lives. Uh we can generate a sports persona and arts persona. for the sports persona case. Um, again, Tampa native cheers for the lightning uh during hockey season.
28:31 Persona diversity examples
If you look it up, it is uh geographically accurate, but her true athletic passion lies in early morning yoga, finding balance and resilience on her mat, just as she does in her life. So again, you see the ocean framework coming through. With respect to the arts, an artist at heart finds inspiration in the works of Peruvian painter de Chislo and American photographer blending their influences into her own expressive art and photography.
Where is this coming from? Well, um, Tampa native, right? So, obviously local sports teams, um, but also cultural background, right? This was one of the things we generated before generating personas. Born and raised in Peru. Um, her cultural background is deeply rooted in the vibrant heritage. She is proud of her indigenous ancestry and incorporates traditional Peruvian beliefs and customs into her life. Her open-mindedness allows her to appreciate and blend these influences with her American experiences, creating a unique cultural identity.
She loves sharing her culture through food, stories, and art. And here's the one that touches on the arts um inclinations. Her curiosity and appreciation for art led her to explore various artistic pursuits. She enjoys painting in her free time, often incorporating Peruvian motifs, right? So you see that the Peruvian background is injected into this. She's also interested in photography and that ends up in um informing the arts persona um capturing the vibrant colors and textures of her surroundings.
So, um, again, this is just a randomly selected record where you can see all of these intricate details, uh, ultimately informing the persona. Um, to wrap this up, you know, you you can basically do pretty much anything you want with this, right? These are just examples of the kinds of personas we're able to generate. You can zone in on a travel persona, a culinary persona. And what's remarkable is just degree of specificity you get this uh you get with this once you're able to ground an LLM uh in this high quality data.
And again just to contrast this with the other approaches. This is most likely data that this LLM has never seen and it's also presented in a format that is amenable to this. Right? So you can't just take census data, shove it into an LM and expect to generate um data like this. Um, and again emphasizing what Karpathy has been saying for a while that the internet data is really not the data uh you need for your transformer.
And so you can see um I'm going to skip this because we're running out of time and I want to make sure people can ask questions. But you can again see the ocean framework coming in with respect to the travel persona, culinary persona, the kind of things they may be open to or averse to. Um same thing with conscientiousness um and uh cultural background. Um what we have also done is looked at this data set um and just in an effort to um really emphasize how diverse truly diverse and complex of a data set one gets.
31:33 Professional persona clustering
Uh we looked in this example just at the uh professional persona slice of an individual. Again profession doesn't determine everything we do but if you look at the existing data sets most of the focus is on what is it they do so that we can target towards specific skills. And so what we're doing here is just doing um we're using uniform manifold approximation with clustering just to highlight the sheer number of clusters.
There are more than 560 that are even in the occupation uh in the data that we used to ground the LLM with. Um and uh it's it's really incredible to see this level of diversity just in one specific persona. And so you see a retired STEM educator with people of similar background close to her a community center volunteer, a seasoned nurse, a dedicated physician, um a customer service representative, a mechanical engineer, a retail employee.
And so again, what's really interesting about this is that it is extremely um complex and it also reflects the diversity that we expect to see in the population. Okay, the overall approach that we discussed was uh incredibly complex, incredibly compound, but generating this data doesn't need to be that complicated and this is really where um the Nvidia Nemo framework uh comes in. And so what is Nvidia Nemo? Uh it's an enterprise platform for end to end generative AI development.
32:52 Nvidia NeMo framework
Uh it's an enterprisegrade platform that comes with many modular microservices. It's deployable with standard APIs, comes with security, stability, and scalability. And most importantly, it uh lets you create this data flywheels including the one uh we're discussing just now. So you can literally go from data preparation to training customization to deploying a generative AI solutions. Uh here's an example of a data flywheel.
If you're in the enterprise, you can get going with enterprise data. Maybe just maybe you would want to generate synthetic data and leverage other services to develop your custom model. And so with respect to the synthetic data generation, we have several new cap capabilities coming soon. Uh uh data designer and um emo safe synthetics. Um and I just really wanted to quickly touch on Nemo. And so uh sorry on the data designer.
33:32 Compound AI system benefits
This is Gretle data designer now part of Nvidia and it's really a compound AI system approach to synthetic data generation. Right? So again going back to what we presented uh we weren't working just with one model and we weren't working just with an LLM. We're working with PGMs and there's a very intricate interplay and so uh we set out to build this compound AI system. For those not familiar with what a compound AAI system is u it's something that was introduced by the um artificial intelligence um research.
Um and the definition is that it's a system that tackles AI tasks using multiple components including multiple calls to models, retrievers or external tools. Um and what we're showing uh on the right is a very quick graphical representation. The idea is that you're not using just a single LLM, right, to generate synthetic data. Um you use a compound AI system. And the reason to do that is because if anyone tried to generate synthetic data, they know that doing so involves so much more than just hitting an LLM, right?
There is often quite a bit of pre-processing, post-processing, data seating, right? You have to handle diversity that we discussed in this talk. um you may want to validate data in introduce execution of data if you're generating code for example and so what a compound AI system lets you do is really leverage the best tool for the job um and make sure that uh all all the models play well together to give you the highest uh quality possible and with that comes a differentiated and future proof approach because essentially it doesn't matter what the latest state-of-the-art model is because you're always able to generate higher quality and so this is exactly what we use to generate the this data set.
Um, so you can think of this diagram we showed before as being completely abstracted away behind data designer. And so this is the really cool part that data designer really enables us to uh very quickly iterate on the data, prototype it, see it, um, and make progress very quickly. Um, so it's a multi-stage compound approach that we presented. But in data designer, this would look very simple. Um we would simply say hey I would like to uh sample uh um persons in the range from 18 to 114.
35:57 Data designer implementation
114 is the oldest age uh in the US. And you can very quickly add you could say I would like to add a first name. I would like to add age perhaps. Uh you can define what kind of personas you would like in this case you know a traditional persona and a professional persona. And you can uh make the PGM that is running under the hood doing all of this for you that's supporting the sampling with um uh persona sampling. You can then inject uh all of these demographic details using Ginga syntax in this case um in the uh personas um column on the bottom where we say you know based on the following profile of an individual where you provide name sex age marital status generate self-contained persona descriptions that capture how persona attributes and profile can uh combine to create a unique individual's perspective and approach to various facets of life.
And so this incredible amount of complexity that we discussed can now be done in less than 50 lines of code. And the best part is that now that we have released this data set, uh you can actually do it even faster because these personas are now available to you via data designer. And so you could literally say, hey, I would like to get a professional persona. Um and so uh really this becomes a very different way of interacting and building with data and really designing data.
37:09 Conclusion and questions
So it's really in the name uh data designer. Um and so with that we'll conclude uh the talk and if we have uh a few minutes um u go to questions and so the conclusion is that um persona data sets are already contributing to improved model performance but at the moment they very much lack diversity and real world grounding and what neutron neatron persona's data set does uh is that a it allows uh people to really explore freely because it's a truly open source release um but it's also grounded real world demographic statistics sourced from the US census and so it brings that realw world diversity and complexity and patterns that we expect to see.
Um and it's built directly into the Nemo ecosystem. Uh all right, I'm going to skip lessons learned. Um yeah, thank you. With that, we'll take a few questions.
Learn more about the Databricks Data and AI platform.
The information provided herein is for general informational purposes only and may not reflect the most current product capabilities or configurations.