Skip to main content

Trillions of Data Records, Zero Bottlenecks for Investor Decision-Making

Summary

  • Real-time market data: J. Goldman transformed trillions of live market records into scalable, instant insights for investment teams without developer bottlenecks.
  • Self-service access: Databricks computational power and Unity Catalog governance enabled non-technical teams to explore live data in a web-native interface.
  • Competitive edge: Fastest time-to-market for investor decisions by eliminating infrastructure constraints and enabling true data democratization.

Trillions of Data Records, Zero Bottlenecks for Investor Decision-Making

Watch: Trillions of Data Records, Zero Bottlenecks for Investor Decision-Making
Talk by Carlos Capellan, CTO, J. Goldman & Co., L.P. ; Ryan Sullivan, Head of Data, J. Goldman & Co., L.P
In finance, every second counts. That’s why the Data team at J. Goldman & Co. needed to transform trillions of real-time market data records into a single, actionable insight — instantly, and without waiting on development resources. By modernizing their internal data platform with a scalable architecture, they built a streamlined, web-native alternative data interface that puts live market data directly in the hands of investment teams. With Databricks’ computational power and Unity Catalog’s secure governance, they eliminated bottlenecks and achieved the fastest time-to-market for critical investor decisions possible. Learn how J. Goldman & Co. Innovates with Databricks and Sigma to: Ensure live, scalable data access across trillions of records in a flexible UI Empower non-technical teams with true self-service data exploration
Intelligent analytics for everyone: https://www.databricks.com/product/business-intelligence
BI meets AI: https://www.databricks.com/resources/ebook/business-intelligence-meets-ai
Explore the only unified platform for agent systems: https://www.databricks.com/product/artificial-intelligence
See all the product announcements from Data + AI Summit: https://www.databricks.com/events/dataaisummit-2025-announcements

FAQs

How does J. Goldman handle trillions of real-time market data records?

Using Databricks scalable infrastructure and Unity Catalog security governance, J. Goldman built a streamlined web-native interface that delivers live market data directly to investment teams instantly, eliminating traditional bottlenecks.

What is the business impact of faster data access for investors?

Accelerated time-to-market for investment decisions gives J. Goldman competitive advantage in fast-moving markets. Teams make trades based on the latest insights without waiting for development resources.

How did Sigma integrate with Databricks for this solution?

Sigma provides a flexible, web-native BI interface on top of Databricks, enabling business users to explore trillions of records without SQL expertise or IT intervention.

What are the key technologies in this architecture?

Databricks handles distributed computation and stores trillions of market data records. Unity Catalog provides fine-grained access control, while Sigma delivers the user-facing analytics interface for self-service exploration.

Full transcript

Hi there. Guess we can go ahead and get started. I think actually there's a few folks still coming in here. All right, we'll wait like 10 seconds. Um, I'm the CTO of Jigleman LP. Uh, we're a hedge fund in New York City. Next year will be marking our 40th year in business, which uh hedge fund years are kind of like dog years. So 40 years is a pretty impressive run. And I want to tell you a little story about all data.
So today you'll be hearing from myself and from my colleague, our head of data. We're going to tell you a little bit about the story of how we got here in the before times and then how joined our team with a mandate to provide alt data to our to our investment staff and how he built that with data bricks and sigma. So let's talk about the first part life before alt data. So a little bit about us. We strive to embrace nimleness.
When you've been in business for 40 years, you've been through a lot of good times and bad times in our industry. who had to deal with the dotcom crash, the '08 financial crisis, COVID, even April this year was was pretty wild for the markets. So, nimleness for us has always been essential about helping us be successful at what we do. The same thing applies for our technology. Of course, it's a constant arms race. Of course, it's true of every single industry, right?
There's always disruption. There's always things getting more complex. You can hear that in every other room, in every other presentation up and down this hallway. But on the tech side, what we think helps us be successful is maintaining that constant imperative to keep modernizing, to make sure we're always figuring out how to build up, embrace new technologies, and be nimble so when the business makes that ask, we're ready.
We're ready to execute. Let's talk a little bit about fund needs and wants so you can get what you want, right? It's very common with hedge funds to have a very strong reflection in the culture of the mindset of the founder, right? Every every startup has that fact, right? Your culture is always driven by the founder. And so you have very strong entrepreneurial cultures and a very much of a get it done now mindset, right?
It's very easy to go out and make a trade in the market. Why can't we roll out that thing in technology with the same urgency and and efficiency? We all know tech, of course, is never an overnight thing. not even a same day thing. But at the same same time, we have lots of regulators. Uh in our case, we have the Securities and Exchange Commission. The SEC always has these requirements for us to make sure we're following.
So, we have to make sure we're doing our compliance archive. We have to make sure we're doing disaster recovery. We have to make sure we're doing cyber security. It's very important when the regulator shows up at your doorstep to be able to show them that you've been doing these things, following these procedures, and produce the information they want to see. So you have these two competing imperatives uh in our field and of course the way it also boils down to too is that we're a small operation.
We're 100 people. I'm sure there are lots of folks who are on teams that size right among you know companies that have hundreds of thousands of employees. So we have to basically make sure that we're doing we're always trying to punch above our weight and making sure that when we have that staffing each hire is really providing a lot of value and that when we're outsourcing we're making sure that we can actually provide good oversight and direction because otherwise of course outsourcing is very problematic.
So let's talk a little bit about the classic hedge fund line of business stack and what that looks like. So first of all you always have your order match system right? This is your kind of your origination of all your data. All your trades start here, right? You have to be able to capture all that information accurately in near real time. All your trades, all your average prices, all your tickers, which sub account it might go into and so on, right?
And so it's very much a real-time system. Uh you have to also do lots of compliance checks. You have to make sure that you're not trading names on the list of restrictions. You have to make sure you're not over going over some threshold that might trigger a filing, a regulatory filing. You have to make sure you don't have a trade that might have some tax implications you don't want to be have. So all those things are very important that these things happen in real time.
And then of course the end of the day is a whole data exchange piece happening with data flying around to different systems and different counterparties. The next piece is your accounting system. What makes this different from your run-of-the-mill system is you have to also be pushing in everyday prices, right? So every single thing, every asset you own has to be priced every day. And you have to be able to generate your profit and loss with prices, right?
No prices, bad data. You have to also handle the exceptions that happen every day. Every day you have exception handling with corporate actions. Corporate actions are at the bane of every back office's existence at every fund, right? Stock splits, corporate actions, mergers, spinouts, reorgs, just always a pain in the neck. Has to be handled every day. All that hopefully flows cleanly into your data warehouse. We acquired one from a vendor about eight years ago and the standard things you do they do warehouse right you want to track your KPIs you want to generate your risk management statistics and you want to be able to do that ad hoc analysis for the business.
So what kind of information do you get asked by the business? The most important piece of course is attribution. So you have some profit and loss, right? You have some some you may you made or lost x amount of of money for a specific time period, maybe the month, the quarter. Where did it come from? How do you break it out? So you have to be able to slice and dice the data and be able to understand did it come from specific tickers, which tickers made us money, which PMS made us money, which portfolio managers made us money, which asset classes, which countries and so on, which sectors.
So all that has to be able to be rolled up and then sliced out. Very common, right? BI type scenario. Risk management is a little more niche to our field. So understanding the overall risk of a portfolio. We see risk risk is generally volatility. Volatility is standard deviation. That's a whole rabbit hole. We will not go down that rabbit hole now. But basically understanding that piece, understanding the correlation of the market, right?
If we're just matching the overall market, that's not what our investors are paying us for. They want to know that we're outperforming the market. They want to know that we're returning a positive return in both good times and bad times. Understanding overall which stocks in our portfolio have more or less risk, right? As you add more names to a portfolio, your overall risk goes down. A rabbit hole will also not go down.
And finally, liquidity. So if I had to go to cash, how long can I do that? Right? Equities are very easy. You can you can turn us into cash in a few days. If you had lots of bonds or other credit instruments that can take months or even years sometimes for the really illquid stuff to turn. So it's a very important metric for the business to know and of course on their desk 8 a. m. correct and not late. So our BI journey well with our with our data warehouse vendor we quickly realized it was very unwieldy to try and do direct queries of the data. it was just not working from the standpoint of trying to have that flexibility and ad hoc analysis.
So we realized it was important to follow the data mart pattern and essentially we provided that readonly schema and in data bricks world I guess we call it that like silver medallion architecture right and the idea was that here is this data being provided centrally and those teams can iterate from that and create whatever data they need based off of that central core schema. So it was a very clean breakdown of responsibilities which was very nice when a report didn't run and you had to figure out whose fault it was.
In the second piece, data viz, we jumped right into Tableau. And again, this is about 10 years ago. So, or eight to 10 years ago, we were doing um it was all server-based and you know, Tableau cloud was still, you know, rather new at the time. And so, we quickly realized that running our Tableau views live was very impractical, right? You you have to be doing extracts in Tableau if you have want any kind of performance out of out of your reports.
Unfortunately, what also happens is that your end users tend to do one of those select star from table, join every table, and you have a gigantic monolithic table with, you know, a jillion rows and then you end up doing that full refresh every day and it's just forklifting all the data from your SQL server to Tableau server and it's just the worst of all worlds. We discovered the hyper API. We said okay we can use this make our offline extracts we can we can actually do some updates right until very recently you could not do incremental updates you could do some appending but that use case didn't work for us and so we we we built that and it worked but a lot of care and feeding to get there so how do we do well good news and bad news right we have the daily risk data for the business the business does get what it wants we can make reports we can generate you know we can do that ad hoc analysis.
We can answer the questions from the business when they ask those questions, but those extracts really were just so much care and feeding and really just too fragile for our needs. Also worse, it was too many hops for the endto-end data flow because essentially your information was starting in that order management system and then flowing into your accounting system and then flowing into your data warehouse and then flowing into that data mart and finally into that Tableau extract.
So way too many hops to get from point A to point you know Z and be useful to the business. And of course, one of the cardinal sins, right, which is too much data movement. In Spark World, we talk about bringing the compute to the data, right? We want to make sure that we're not we're avoiding data movement. And here with XRX, we're doing precisely that, right? We're doing a ton of data movement to just kind of forklift just about everything from the server side, the SQL server side to the uh Tableau server side.
So all in all, workable, fragile, not ideal. So then we got a second bite of the apple. We had a blank sheet of paper. We had a second chance to try and do it right and try and use some newer stuff. So we were talking just now about how entrepreneurial entrepreneurial cultures exist at hedge funds, how you have that get it done now mindset. And so the word came down, we just hired a data team. Start next week. What do you got?
Right? They have to be working right away. Be productive right away. Oh, and by the way, yes, we know you're an Azure shop, but all these alt data vendors, they provide all this data in AWS and S3 buckets. So figure that out because I don't think you want to be copying terabytes of data between clouds to have everything run in Azure. So better have an answer for that piece, too. And by the way, don't even think about trying to ETL this data to your SQL server box.
It's going to fall over because yes, we could go out and get SQL enterprise, have a cluster, do always on availability, right? All that jazz. We're not going to do that at our size. So, you better have an answer. Well, of course, the answer was you need a cloud data warehouse and you need it yesterday. So, the word came down, well, hey, I used a certain, you know, winter themed cloud data warehouse at my last job. Let's try it here.
Um and about after about and that worked well for us, right? It it certainly had very fast time to value. We were able to spin out very quickly. We were able to have a lot of success with Snowflake's marketplace. A lot of the vendors we needed and needed to access quickly were available and you had that wonderful model of just having the data peer in your cloud data warehouse, right? It's it's ideal, right? Who wants to deal with ETL or ELT when you don't have to?
But at about after 18 months or so, we realized we were starting to kind of outgrow some of the things that we could do with Snowflake at the time. And of course, I understand both platforms are kind of in this race to the top and converging on the same point, but we really felt like we were missing out some of the capabilities that we enjoy today in data bricks, right? We we we were able to get a lot of value out of the whole workflow uh and jobs infrastructure.
We're able to get a lot of value out of all the work you can do with Pispark, right, over pure SQL. And of course, at the same time, of course, SQL, I I think it's safe to say now is is is at this point certainly a first class citizen uh in data bricks today. So, we got to this point now where our all data team was able to execute extremely quickly. In about a year's time, they were able to walk in the door, build out all their processes, map all their data, build their underlying data sets, get to their gold gold medallion architecture, and build out a whole ton of views that the business needed in Sigma.
And that was enabled through these key pieces, right? Which you've all seen before. Unity catalog is fantastic, right? Permission at once. Data bricks workspaces break everyone out into their own little sandbox. You have some consultants, no problem. They can be over here. they have all separate views of the data, right? Delta live tables we're just scratching the surface on. There's so much more for us to really exploit there with our with our own pipelines.
Um, and then Sigma, our experience is really, I think, indicative that we really had all of our alt data folks had been longtime Tableau users and just switched to Sigma within a couple of weeks and have not looked back. So, I think for us that's a very powerful statement of how intuitive we feel it is to use. Um and then finally from my from my seat as as you know worrying about infrastructure all day long materialized views really solve for that whole pain point of the extract refreshes.
We're able to just have that again we're able to solve that problem of data movement right the data is no longer leaving the warehouse. You're just keeping it where the compute is and it's far more efficient and performs a whole lot better for our use cases. So with that, I'd like to turn it over to my colleague, who is our head of data, who will talk to us a little bit about the mandate and the need that he was brought on to solve and how he solved it with data bricks and sigma. overarching theme of this is that without, I don't have a job, right?
I think I need to be really clear about that. And I think that most people in my seat that are responsible for taking the data from the mid-level to the end user um oftentimes are we are actually completely dependent on and the folks in his role in his dev team. So if you are listening dev team back at home, thank you very much and to my team too. A big shout out to you guys as well. But uh it's really important that we understand that none of this is possible without what he's done and the architecture he's built.
And having a partner that understands this is extremely valuable. I've been at a bunch of different shops and oftentimes that mismatch or the understand the need the switching can take a long time. It's painful. There's oftentimes a reluctance to do that. So Carlos has been a phenomenal partner in that. And I think that part of the firm culture that we have at Jay Goldman is what has brought there and almost 20 years of experience.
So, thank you. Um, when I talk to LPs, people that are investing in our fund and giving us their capital for us to grow, I oftentimes start out with a story and I say to them, "Okay, you're sitting in our office in Midtown Manhattan right now. How many data points do you think you generated from the time you left your hotel room until you walked into this office?" And I see their eyes start to kind of get a little bit wide and they're thinking to themselves, well, not not very many, right?
And I say, 'Well, did you, you know, swipe in the elevator with your card in in, you know, in the middle of Midtown Manhattan? Use a card to swipe. I'm sure here, too. They don't let you just up and down the elevators. So, yeah, you know, there's there's one piece of data. And how many apps did you check on the way up and down the elevator? Well, you know, there's a thousand pieces of data. Did you go to Starbucks? Did you go buy breakfast?
There's three or four different credit card swipes. Did you take an Uber? There's mobility data. There's a thousand different pieces of information to give us a very robust sense of where the consumer is doing and how they're spending their money and frankly what they're interacting with. I tell them at this point their eyes are like, you know, gigantic and they're staring at me and they're like, "Oh my god, like this is totally crazy."
They're also thinking about what did they look at on their phone that they maybe should not have looked at and, you know, they're going through that little panic panic mode. Um I I think that there is a saying that I always convey to them which is that we pay for our convenience with the currency of data. In other words, data is the currency of our convenience now. And we all know that sitting in this room. This is our profession.
This is our livelihood. Like we've built our careers on this. We understand that and how vast it is. Most people don't. And some of the people that don't do that are oftentimes the end users at a hedge fund. And I'm going to kind of walk you through what that looks like. So by way of just introduction of who I am, I've I've done I've kind of progressed throughout the financial industry in a few different roles on both the investor side and the research side.
I started at Credit Swiss. I was a research analyst. I went to a small startup called Yippet Data which is now a very big startup and they are the largest I think reseller of alternative data in the marketplace. and then from there to the buy side. When I use the word buy side, just so we're all aware, the buy side is either a hedge fund or a mutual fund or some type of asset manager that is taking capital from outside investors and growing it and getting paid to do that job.
That's when I when I I just want to be very clear about terms and I'm going to define some of these. This is my favorite quote I've ever heard in this business and this is the reason why my job is so important. People lie, data doesn't. Okay? as investors and people that are constantly talking to companies and different people that are involved in a given name or stock that we're, you know, involved with or trading or thinking about trading, we're often going to get a lot of different stories from a lot of different people.
Everybody's going to talk their book. This is our saying. They talk their book, right? They're going to tell us stories. They're going to management always wants to sound great. There's reasons. There's incentives. But at the bottom bottom line is that people sometimes can get a little over their skis. I don't think anyone's doing anything malicious, but bottom line is people lie and the data doesn't. And that's why it's so important to have this read in real time all the time.
So let's go quick five minute primer on alternative data and what I do and how this works in the fundamental investment process. Okay, I don't think I need to define the terms. Everyone know what a stock is? Yes. Okay, let's move on. Okay, alternative data. These are generally data sets for the sake of what I'm going to talk about that are only available to the buy side and sold to institutional investors like myself.
It's not something that any, and off the street can go buy. A portfolio manager, as alluded to, he called a PM, and I might use that word also, is the seniormost member on the investment staff that generally has discretion to buy or sell stocks on behalf of the firm with a defined pool of capital. Okay, they are the trigger puller. They're also the revenue generator. They're the most important person in the firm because without them, we don't have a business.
We can't make money. The common types of alternative data that we generally tend to use and purchase breaks into three buckets. This is by no means exhaustive, but this is largely I think encompasses a lot of what we're talking about. So transaction data, those are things like credit card transactions, email receipts, point of sale data, and you go into your local grocery store or, you know, Macy's, you swipe your credit card, that's the point of sale.
We have data that comes from there. Loyalty program spent. Every time you go into CVS and you type in your credit card or, you know, your phone number to get 20% off, that is captured. Medical device purchase orders. You know, I hope none of you have been to the hospital recently, but if you have and you had to have any type of, you know, uh, medical device, oftentimes those are generally ordered through a group purchasing organization to get better economics and economies of scale, that data is anonymized and resold.
So, we can see things across the medical landscape. How many you know consumables is intuitive surgical selling through to investors. Scrape data everyone here is probably very familiar with scrape data. This was generally how the alternative data thing really started. I think largely there was you know this worldwide web and some engineer was like hey we can actually go to eBay's website and with their HTML see P and Q on every single item that's listed there and from there go ahead and forecast revenue.
Right? That is a great use case. Another great use case was Groupon. I love this story because I think it's a great story, but Yipet at the time when I joined was actually a consumer product that was focused on daily deals. Now, most of you you in this room might look like you're too young for this, but I'm an old person. And about 15 years ago, there was this or 10 years ago, I guess, there was this thing called, you know, the daily deal, right? you would go on to this website and it would be like you can get a massage in Midtown Manhattan for 50% off or you can get five restaurant meals for 20% off the aggregate and the group on was the largest player in the market but there was hundreds of these so Yep came along and what they did was they actually scraped all of these put them into one place and as the story goes and you know I think this is folklore but uh maybe it's true as the story goes a big hedge fund came to them and said hey we are going to uh start trading group on when it goes public, would you be willing to sell us the data?
And Yipet was like, maybe. How much are you gonna pay? And they were like, a lot of money. And Yipet was like, we got you, you know. Okay. So, that then led to eBay and Etsy and these products where you could scrape data and use it to actually forecast actual KPIs with very high precision. Then there's usage data, right? This is something we're all familiar with. We see it all the time. Webclick data, app usage and downloads, and advertising.
Okay? We like we these are things that most of us have probably worked with or are working with. Why is alternative data so important? So a lot of you are sitting here and you're like that's great. You've got all this information. So explain to me why it's important. How does a public company life cycle work? Okay. Every 90 days, every quarter, they are responsible to report to the world how much revenue they're making, any business updates.
It's they file something called a 10Q or a 10K depending on when it's happening. And when they do that, stocks will move and rerate. A lot of you here might work at startups where your valuation is pegged to a funding round or something like that or potentially a revenue stream and you're marking that in your mind. We in the public markets that are trading public stocks, that valuation is changing daily and it's a function of the price, right?
So at the end of the day, every 90 days, we're getting a new piece of information that is going to meaningfully change the price of the stock. And that's how we get paid, right? We want to buy Google when it's low and sell it when it's high or Amazon or Palanteer. None of those are investment recommendations. Those are just examples to be very clear. Okay, so what does this mean? Well, this means that stocks are going to trade for 65 days between each one of these earnings periods when there is no new information.
Okay, but they trade every day. So the prices are moving and it's really weird when the prices move because they're like, "What is the price is moving on?" Imagine you work at a startup and like you walk in day one and they're like, "Here's your equity pegged at a dollar." And then day three they're like your equity is now 95 cents. You're like what changed? We didn't raise any money. We didn't do anything. Something must have changed.
But what? So that doesn't happen in the private markets. In the in the public markets it generally it does every single day. Alternative data is the only real time view of a company's business that investors can lean on to make fact-based decisions. That's what this means right. So if I am trading for example let's say Amazon. Okay. If I'm trading Amazon and they have told the world that next quarter we expect to do 10% revenue growth and my data is telling me, well, they're tracking nowhere near 10%.
In fact, it's 15%. What am I going to do? Well, I want to be long. I want to buy the stock. In other words, be long the stock as soon as possible so that I can take advantage of that when they do announce it. Okay. Now, this is happening in real time. We're not the only ones that have this data source. All of our competitors do. So that means every one of these 65 days becomes an earnings event essentially where the KPI is no longer what the company's telling us, it's what the data is telling us.
The data becomes the KPI. Okay, so that's why this is so important to get this right. The primary users of this is the portfolio manager and the investment team. Okay, they I work for them. My job as the head of data is to help them make investment decisions. We are not here to have fun with numbers. We are not here to do really wild fun projects. Look at raw credit card data and be like hash 37049 order Domino's pizza 75 times last year.
That's fun and it's funny. It's something you can do at night time when you're exhausted and working on a big project. But like that is not the that is not the modus operandi. Our job is to help the portfolio manager make money today, tomorrow, and into the into the future. Okay. Now, here's the catch. Most portfolio managers are non-technical. They are Excel wizards and they are industry experts in their coverage. But I want to focus on the first one.
They are non-technical. Okay, they I have never worked with the portfolio manager that has written a line of SQL code and I've been doing this for 10 years in this specific type of role. So that means I have to translate essentially there's some really great tools by the way in the expo if you've checked them out that do this in real time now. So my job is probably obsolete at this point. But what these guys can will do is say here is a question I have. can you help me translate this?
Okay. And I have to know how to be able to do that. The other thing alluded to about Sigma and Tableau that I want to really harp on and I'll get to why it's it's very um preient to be using Sigma in this market environment is most of these guys use Excel and they've grown up on Excel. A lot of the portfolio managers that we work with were analysts or um manage or you know highlevel ranking folks in investment banks.
That means they grew up modeling in Excel and they know how to use they they can do formulas that will put your code to shame as Python programmers. I promise you that. But what they don't realize is that Excel works different than code, right? Excel goes both diagonal. It goes diagonally, horizontal and vertical at the same time. When we write code, we go sequential, right? And oftentimes these folks don't necessarily realize that.
And this is why it's very important to be able to translate and understand these things. the most we we laugh about this as tech people in a hedge fund, but the question you get oftentimes is how hard would it be to do X? In which point you're like, oh my god, I'm about to get the most awful question ever that in Excel would take three minutes and in code is going to take three months. Oftentimes, something in Excel that might take three months is going to take me three minutes because it's essentially a wear clause in a SQL query.
Okay, so like these are the dynamics that we're constantly competing with and trying to deal with and understand. How do these guys make decisions once they have the information? Well, they generally use something we call the mosaic approach. And what does that mean? That means that what they are doing is taking a lot of different pieces of information and building them into one picture. A mosaic for lack of a better term.
But this is the word we use. This is fundamental investments process where you're modeling a company, understanding revenue, margins, all these type of things. Data, which is the last thing that I'm going to talk about, the macro environment and secular super cycles, right? Those would be things like is the you know in the AI boom for example right Nvidia was a very different investment maybe five years ago than it is today for all these reasons.
So conceptually speaking this is the framework that we think about and data is a piece of that process but it's not the only part of that process. A lot of you in the room may be familiar with quant funds and how quant funds operate. Um if you have talked to if you have friends that are programmers there in quant funds data is the signal the thesis and the trade. There's some human intervention that goes into basically writing the code and making sense of how to interpret the data.
But at the end of the day, the data is what's driving things. Now, in my side of the world, data is an input to the portfolio manager to either differentiate a thesis, to corroborate something, maybe to measure an unappreciated but quickly growing component of a business. But that's conceptually how it works. Okay. The use cases are, as we've talked about, KPI tracking like we've gone through, momentum indicators. This is a case where there's a business that has something growing very quickly.
Um, a great example of this would have been Popeye's chicken sandwich from like five years ago, if you guys remember, seven years ago. All everyone wanted to know is when the hell was this thing going to stop growing as fast. We didn't care if it was going to grow at 130 or 170%. We just wanted to see the line stop and go the other direction. And then cross-sectional analysis, right? This would be a question like, hey, can you help me understand how many different stores people go to when they leave for a shopping trip? very relevant during COVID.
Can you help us understand what people that buy Chewy products the pet store also buy with respect to retail? Do they go to Lululemon or do they go to Fiblettics? These type of questions. The challenges inherent in this setup obviously are that portfolio managers need a data scientist who have much less experience in the market and contextual information but have the coding knowhow. and vice versa when you flip it the other way around.
The common challenges that arise obviously are that non-technical portfolio managers can't necessarily communicate the need as fast as much as we'd like them to. And technical data scientists also have trouble simplifying key takeaways and analysis into actionable insights. Okay, we're all data folks. We're coders. When we put a PR in, when we do anything with respect to code, we are very explicit about what could be right and what is going to likely be wrong.
Okay, a portfolio manager does not care and does not have the time to do that. They want the number and they don't want all all the baggage around that. They're obviously not going to make a trade because you give them one number, but like they can't read 17 paragraphs about why your number might be 20 basis points off. Okay, that's something we need to be mindful of as data scientists in this side of the world. So, the key conclusions at the end of the day, I'm just going to go to this last one.
Simplicity, clarity, crucial for the impact of data delivery. are like the most important things and we have to know that we're dealing with folks that are not technical in nature. Now, we're going to talk about what I do and how I make sense of all this. How Sigma helps bridge the gap. Okay, talked a lot about the BI tools that we used previously and we're using now. So, previously Tableau, I've used Tableau and a number of shops in the past.
I've used Excel extensively. Sigma is an amazing tool and I this is really, I guess, a sales pitch for them to some degree and it's not intentional. They're not paying me to say this, but Sigma is incredible because it allows you to interact with data in a BI tool that is web- based and where you can use Excel functionality. Okay? So, like if you have users, I'm sure all of you have users in your businesses that prefer things in Excel.
Just drop this in a CSV for me, give it to me in Excel, I'll work with it, manipulate it. Sigma does both very, very well. And it's it's extremely impressive how it does it. And I'm going to walk through a couple examples. How big is the data set I'm working with? Okay, I was actually kind of shocked when I got this answer. I asked I I work there's two folks on my team, and Jayun, and I asked to run this query for me and tell me how much we have here.
We have 2. 2 trillion rows of data in our data bricks instance and 82 terabytes of data. I'm sure some of this could be there's some copies of stuff that we've done on purpose. There's some legacy stuff, but it's a heck of a lot of information. Okay, we're talking about 30 distinct vendors, multiple levels of post-level mapping and aggregation, and multiple forecasting modules. Right? Again, the goal is to get to a simple state, but in order to get there, we have to go through a lot of different steps to do that.
The competitive advantages we have with all of this, because again, remember like we're one of many hedge funds. We're not the only ones getting this data. Mapping and aggregation. Everyone laughs at me when I tell them this and I they ask me like, "What's your job?" I say, "I'm the world's greatest mapper." That's that's what I do best. Okay. If I asked you guys, who knows TJ Maxx, the business? Everyone here know about TJ Maxx?
Anyone ever been to TJ Maxx? Yes. Okay. Okay. So, mapping TJ Maxx for an investment use case is not simple, right? I don't know if anyone in the room knows what they actually report. They don't report TJ Maxx. They report something called Marmax, which is a combination of TJ Maxx and Marshalls. Both businesses they own. But if I'm dealing with the raw data coming in from 30 different vendors, I need each one to ensure that when an investor looks at Marmax comparables or revenue, they are getting that mapped and aggregated appropriately, which means I need to have an entire taxonomy that I've built, I control, and I own calendars.
You guys are going to think I'm totally crazy. The noise that companies create with their calendars is outrageous. Okay, this is the nature of the beast. It's a pain in the neck. I know more about company calendars and retail calendars and extra weeks and Mother's Day and Father's Day and blah blah blah than you care to ever hear anyone talk about. Obviously forecasts we have to do. We have to build accurate forecasts of both the raw data and the KPIs.
We want to have an accurate read on revenue. In order to do that, we oftentimes need to forecast raw data to get there. That's all wonderful. Most of that stuff never gets seen by the portfolio manager. All they care about is the last mile. Okay, the last mile, the dissemination and the visual visualization, in which case we partner with Sigma to get that done. Well, we actually pay Sigma, but yes, we partner with them, too.
So, how does this work? We need to have a robust push process. We need to be able to send information to investors without them asking me for it. I don't want to get an email that says, "Hey, what do you think Five Below is comping today?" or "What do you think about match. com's uh active users?" That should be in someone's inbox. I want it to be easy to schedule. I want them to be I want there to be an iterable front end where they can go and click on something and drop it down and see it and understand what it looks like.
And I want the takeaways to be unbelievably clear, perfect. Like there should be no ambiguity in anything that they're looking at. It shouldn't be like, is this a beat, is this the miss, is this good, is this bad. Should be obvious and clear and direct. The three tenants of a strong data product like I just talked about relevancy, simplicity, and it's got to be actionable. Actionable does not mean the person needs to look at it and know I'm going to make I'm going to make a trade with this.
From the perspective of a fundamental business, what they need to be able to say is does the portfolio manager know what the next step is? Can they see this and say, okay, I can see that there is strength or weakness here. Does this apply everywhere? Or there is strength and weakness here. I want to buy or sell this stock. It's very nerve-wracking as as an aside to be on um you know in the in a data seat where you get a question from a PM that says I what is there anything wrong with this data and you say no and then the next day you look and you realize there is a nine figure position on the book because of your data right that's a very nerve-wracking place to be but it means that there's really no room for error so what does this look like right at the end of the day how do I what do I want to get to um my compliance is not super cool with be showing you our internal tools and giving you a demo of them because then you can kind of do what we do.
So, I'm going to show you some pictures where I don't think you'll be able to get back into it. But what I want to get to at the end of the day is I want to be able to give a PM something called a oneliner. That's what I think is is the is the gold the golden goose, the holy grail, whatever you want to call it. What that means is can I give them a picture? Take all this data that we have, all 2. 2 2 trillion rows, valuation data, sentiment data, alternative credit card spend data, usage data, market positioning data, pricing data, and give them one number.
Is that possible? Right? So, we ask our three questions. What makes a good data product? Is it relevant? Is it simple? And is it actionable? If I take all these things, I can't compare them all. They're orthogonal. They don't one is not related to the other necessarily. They're different metrics. They're different units. As mathematicians, like most of us in this room are, we we can't just slap them together, right? What we have to do is we have to normalize them.
We have to create some type of a, you know, score, so to speak. Once we've created the score and we've done that using appropriate math, the next thing we can do is we can put them all together. And in this case here, so I'm looking at the sporting goods sector and I'm tell and the stock that I'm looking at in this case here basically falls smack dab in the middle of that sector. Now, there's a number of names in there. if depending on how you what you call sporting goods but the specific ticker that I'm looking at right there on the left sorry your right hand side is the is that one single stock and the score that we give it relative to its sector by itself okay so if somebody says what do you think about XYZ that's actually a ticker what do you think about CBA you know I can tell them well based on every single thing we have in every piece of data it's actually stacking up right in the middle there's nothing exceptional about it and let's Maybe we move on.
Then they say, "Well, what does it look like relative to the rest of the universe?" And then I say, "Okay, great. Look at the chart right next to it, next to the entire universe, we can see that in fact it's also pretty much smack dab in the middle, maybe a little bit trending better than than average." This is this is unambiguous, right? It's very clear and it takes everything and puts it all in one place. What about this idea of having an iterable front end where we can give people use Sigma to give people ability to answer their own questions?
Okay, I'm not going to tell you what business this is, but it is h it is a restaurant chain that had a very very viral campaign not so long ago. And what you can see here, look at that red line, just how outrageous it has exploded. If I'm an investor, I might want to actually go ahead and understand a bunch of different components of this narrative, right? So right now what I'm looking at is I'm looking at the trailing 28 day spend versus the same time from a year ago.
But I might want to look at that versus seven days, maybe 14 days, maybe the cumulative quartertoate number. What about the merchant name? What what if we wanted to look at that one merchant or the entire business as a whole? What kind of calculation are we looking at? Is it a year-over-year calculation? Is it a two-year overyear calc? These are all things we worry about in finance to make sure everything is like for like the data set that we're looking at is this is a credit card spend, but maybe we want to look at foot traffic, right?
I want to give this power to the investor and I don't want them to have to worry about coming to me to ask for something. Okay? And then I obviously the last point here is I want to be able to give them ability to look at what periods they need to be looking at. The last thing that we want to be able to do with Sigma obviously is make very intuitive charts where we put a lot of information together in one place to talk about macro themes or bigger things that are bigger than a one company.
So during the height of the tariffs the question was what is happening with the data right I'm not this is not political Trump came out he announced tariffs what happened to people's spend what did that look like let's take this 2. 2 two trillion rows and turned it into one line and make it very clear when it started and stopped. Sigma gives me the power to do that. This took me about 20 minutes to spin up because of the fact that I had a lot done a lot of the work on the back end and now I can easily import it to this right here.
So like this tool has changed the way that I function and and essentially work with my investment team because it gives I can build something quickly in in this morning in 20 minutes I actually did something very quickly that was a brand new dashboard for somebody based on a question I got in an email using sigma and being here. So like it's an unbelievably powerful tool and as in a data scientist and you know my job is to get insights out to the team as quickly as possible.
This changes the game for me. So, um, I can take any questions if anybody has them or, um, I can let you guys go to next session.

Learn more about the Databricks Data and AI platform.

The information provided herein is for general informational purposes only and may not reflect the most current product capabilities or configurations.