Pitch to Player Insights: Cleveland Guardians AI Platform on Databricks

Summary

The Cleveland Guardians migrated from a legacy MS SQL Server environment—where all workflows, analytics, ETL, and production code ran on a single box with no source control or CI/CD—to a cloud-native platform on Databricks that scores every pitch with neural networks and delivers AI-powered scouting insights.
Databricks Asset Bundles standardized deployment across modular jobs, shared clusters, and coordinated builds, while Unity Catalog eliminated environment fragmentation and enabled seamless data access across both R and Python workflows.
This video shows how LLM-powered player analysis and biomechanics data at scale are giving baseball decision-makers competitive intelligence that was not possible with the team's previous infrastructure.

Pitch to Player Insights: Cleveland Guardians AI Platform on Databricks

In professional baseball, every competitive advantage comes from data and AI. The Cleveland Guardians built a real-time platform that ingests live game feeds, scores every pitch with neural networks, and delivers AI-powered scouting insights to analysts. The challenge: moving from legacy SQL Server systems to a cloud-native platform that scales with biomechanics data while supporting innovation across dev, QA, and production.
Learn how Databricks Asset Bundles standardized deployment across modular jobs, shared clusters, and coordinated builds. Discover how Unity Catalog eliminated environment fragmentation, enabling seamless data access across R and Python workflows. See practical approaches to streaming ingestion, real-time model serving, and LLM-powered player analysis. Gain insights into the evolution from manual deployments to production-grade data engineering.
🤝

Chapters

FAQs

How are the Cleveland Guardians using Databricks for baseball analytics?

The Cleveland Guardians built a cloud-native platform on Databricks that ingests live game feeds, scores every pitch using neural networks, and delivers AI-powered scouting insights to analysts. The platform also supports biomechanics data at scale and LLM-powered player analysis, giving baseball decision-makers real-time intelligence to inform competitive strategy.

What were the main problems with the Guardians' legacy data infrastructure?

The team previously ran all workflows—analytics, production, and ETL—on a single MS SQL Server box with manual job scheduling on Windows Server and no source control or CI/CD. Senior data engineer Mike Halverson describes copying solution files from C# and working with deprecated Windows Forms code, with no ability to recover changes accidentally lost from the SQL server window.

How do Databricks Asset Bundles help the Cleveland Guardians manage deployments?

Databricks Asset Bundles allow the Guardians to standardize deployments across modular jobs, shared clusters, and coordinated builds, replacing the previous manual deployment processes. This video explains how Asset Bundles brought source control, CI/CD practices, and reproducible deployment to a data team that previously had none of those capabilities.

How does Unity Catalog support mixed R and Python workflows at the Guardians?

Unity Catalog provides a unified data governance and discovery layer that allows both R and Python users to access the same tables and assets without environment fragmentation. The Guardians use this capability to support analytics teams working in R alongside data engineering and AI teams working in Python, enabling seamless collaboration across the organization.

Full transcript

[00:07] Uh we are here today to talk about how the Cleveland Guardians are using Databricks, um pitch data, player insights, how we're using their platform to uh help us win baseball games. So, I am Mike Halverson. I'm a senior data engineer with the team. I've been working with the organization for 8
[00:25] years. Um was the second data engineer at that time on the the baseball side. Matt Peters here in the front row. Uh raise your hand, Matt, please. Um was the first. Um and in that time, the 8 years I've been part of the organization, I've seen our
[00:40] team take huge jumps in terms of our technologies, our capabilities, and the services we provide to baseball decision-makers. Our first software developer, Matt Tag, started in 1997. We've come a long way since then. And yes, your name does have to start with an M to work for our team. So, if that
[00:56] happens, please reach out afterwards. Um Um yeah, so as I mentioned, uh one of the first two software developers in baseball started with us. Um Dan Quail with the hosting SF Giants were the other ones.
[01:11] Um and as you can see, we have Jonah Hill as Peter Brand, who is actually just Paul DePodesta on the screen. Um you can see there with the arrow. Um we have a small Cleveland uh name to not be mentioned bobblehead on the uh on the
[01:27] screen. Um because uh the like Billy Bean's right-hand man started out with the uh the Cleveland team at the time. Um so, we didn't just get fleeced for Ricardo Rincon later in the movie. We were also there at the start. Um
[01:43] And yeah, Paul, we're not going to talk about what he did to the Cleveland Browns, but you guys are all familiar. Um but yeah, so we had a singular MS SQL driver for basically all of our workflows, analytics, production, ETL, everything was just on one box.
[01:59] Um, we were, you know, manually creating jobs on a Windows server or through the MS SQL scheduler. There was no um, really source control or CI/CD for that. Um, a lot of copying over solution files from C#, which is not fun. We have a lot of like legacy, very old,
[02:15] deprecated things. If there's any, uh, Windows Forms heads out there, please let yourself be known. Um, but, um, and yeah, the code lived on the SQL server, so um, if you accidentally made a change, you had better not have, uh, exited that SQL server window cuz it would have just
[02:30] stuck with you and you'd lost what what your work you'd done. Um, but yeah, so we needed to make a change. Um, we were in C# and we needed to modernize to help bring in new talent, which isn't really using that anymore, and also to get the latest technology.
[02:51] Um, so yeah, with Moneyball, I think we're that's kind of the start of everybody with baseball and, uh, it was a revolutionary concept that, um, getting on first base was more important than how you did it. Um, but as time has gone on and everyone's kind of caught on and everyone read that book, um, most of us are evaluating player performances in
[03:08] roughly the same way. Um, and so once that spread, the advantages got harder to find and, uh, started to flow more into player development space and then the expected outcome space and the strictly how did you perform this season space. Um, so for example, we have that,
[03:23] um, graphic from the New York Times, um, of of like a launch angle, the the way the ball comes off the bat. Um, so, you know, in 2002, it was fine to just know how many home runs you hit, what your slugging percentage was. Um, but now we're looking for the
[03:38] optimal angle from which you to hit the ball in order to generate power. Um, so that, you know, in the cases if you get 10 home runs robbed in a year, that's probably not your fault, probably won't happen again. We'll probably expect you to do better the next year. Um, so being the first in a space is an
[03:55] advantage at the time, but also can be a disadvantage in some ways, as I'm sure uh some of you are familiar with. Um as we have legacy systems to maintain, customers to keep happy, rather than just building the latest and greatest system to meet the needs that arise. Um and so kind of the first
[04:10] example of that um was in biomechanics. Um we have the the hard numbers up there of you know, we're tracking 29 joints at 300 frames per second for 2 seconds before ball release up to 1 second after ball release. Um so 29 joints, that's 52,000
[04:27] observations per play just for the pitcher and the batter. Um there's also uh fielder stuff, but that's less important in this use case. Um but when you start to add up 162 games per team, uh 30 teams in a league, there's five to six levels of baseball,
[04:43] um you're starting to far exceed what you can do on a Microsoft SQL Server. Um and so in the space, there's still some teams that are like, we started out with the Microsoft SQL Server just doing things on that server cuz we just had kind of our own
[04:59] um games on there, and so that's a much smaller universe. Um and some teams are still kind of in that position of processing just their own data, and then if they're interested in a player, they'll begin to do the pipelines, but we want to be able to evaluate and make decisions and you know, know what the
[05:14] rest of the league has going on um at all times. Um and even before we got biomech, um we do like centers of where players are on the field, and that data like we couldn't actually materialize, so we were doing views, reading like variant columns, um which was very slow and uh tended to
[05:32] slow down the work that we could do there and really tax our SQL Server's uh temporary DB and and memory. So with Sorry about that. Uh with that, we needed to move to the cloud. Um and with the team of two people at the time
[05:48] fighting for cloud approval and just any time at all to develop, we tended to take the easiest solution presented us out of the box by GCP. Um, I can distinctly remember being in spring training with a the season looming over us trying to race to get um the solution out. And so, we didn't
[06:04] always end up with the best solution, but one the proof of concept that usually did okay once we uh fine-tuned it. Um, we had moved on from manually building uh C# solutions and copying them over, which was nice. We could have jobs running while we changed code, which was our old solution.
[06:21] Um, but we were still building Docker containers locally and then uploading them and then eventually uh got CI/CD working. Um, but the only way to really test in the environment as it was on the environment was via Docker on your local machine. Um, and we we ran a lot of analytics on a our Docker image, which
[06:39] at times had different package versions than uh the what the developers were developing on, uh which led to a lot of frustration between us as the data engineers and the R&D analysts as we'd get an error, they would run the same thing on the same machine, they wouldn't get the error. Uh, we kind of end up at
[06:54] an impasse there. Um, and Airflow um we're orchestrating the cloud, which is a big upgrade, but it was difficult to work with at times, especially when we needed to scale up and run like large-scale reprocesses. Um, getting the pods to not die instantly and having all the IPs was something we were kind of
[07:11] manually managing, and it was a huge pain in the butt. Um, and the metrics and health metrics like the UI was pretty good, but it was tucked away um and we weren't really easy to access. Um, and then kind of in terms of how we were still processing data, it
[07:26] was still almost all on that SQL Server, which led to all sorts of blocking transactions, slowdowns, memory run outs, and R&D analysts writes their own query, and all of a sudden your entire database is choking. Um, so um it's a better solution than where we
[07:43] were when I started, but we still weren't quite there yet. And so, enter Databricks. Um the status quo could not continue. We needed to move fully to the cloud and get an enhanced unified data platform, and we made our switch. Um so, where we are Oh. I'm sorry. I I
[08:00] got two slides I got to update here, so um but yeah, the current state um we have around 120 jobs in Databricks. The majority of the processing is done there um from vendor import streaming to data downloads to um gold medallion tables.
[08:18] Um so, all game processing is on there. All biomech processing is on there. Um we just completed a full reprocess. Like, we get a new model from Hawkeye every year, and we just completed a new reprocess of our 2025 biomech data. It took a week, when in the past in Airflow
[08:34] it would have taken 3 weeks of me hand-holding compute and IPs, um which was a huge pain, and this was super painless. The compute just works. We were able to scale up and just get things done very easily. Um so, we have live data coming in during
[08:49] our Major League and Minor League affiliate games um both from web sockets and from API calls. We were quickly able to apply our analytical models to them uh in game. Our users can see both what's happening and also the more advanced metrics to get a sense of how our players are performing and how they're doing. Um
[09:06] uh one of the projects in spring training, I'm sure you guys um heard about the new ABS system of uh batters and pitchers being able to chan- challenge pitch calls of balls and strikes. Um and so, one of the things we were able to do is quickly get that to our users, so they can know
[09:22] both the challenges that did occur and and what the result was, and also kind of where we were missing opportunities to challenge. Um so, our SQL Server is now much more of just like a serving layer for our front end to use than um bearing the full weight of all our ETL.
[09:38] We do write like you can see the code sample up there. We are writing a decent amount of stuff back to SQL Server for that kind of front end layer which we're excited to get rid of at some point because for example, if we're doing a bulk read process in Databricks, it's
[09:53] very easy in Databricks. It can handle it. You know, we're scaling but our SQL Server does not scale quite as easily. So trying to write that back and the overload the server and we still have some of those blocking transactions and problems that we had in the old way.
[10:09] The after this presentation, you know, Databricks has their own product announcements but after this, you know, we are pulling the plug on BigQuery. It's going off life support. So we'll be saving a lot of money there. We're very excited about it. Airflow, we still have around 71 jobs left.
[10:26] So maybe this month is a little overly optimistic but many are simple stored procedure runs or our script runs. We could hypothetically just transfer them over but I think we want to do it a little bit better. Um
[10:42] So yeah, some of the benefits we've seen with the switch to Databricks with the Databricks CLI, we've kind of for a long time, we were kind of a middle layer between R&D and their jobs going to production that we had to do put a lot of time and work and testing into getting their jobs into production
[10:59] but with our our asset bundles, they're easily able to create compute that they need and create jobs without having to write a bunch of code or know much at all. We have their packages that they need all the R packages. If they're using Python, the Python packages, their internally developed R packages all just
[11:15] ready to go with setup variables. So they're just able to very easily they know what schedule they want. They can basically get a job going in you know, 2 minutes or less. Um In terms of yeah, we're no longer copying code, worrying about Docker images, or All
[11:31] that's handled by Databricks. We have GitHub as actions auto deploying our code to prod and to dev. So, if we create a pull request, that code's going up and we're able to test it in the dev environment um very quickly. Um in terms of performance and tracking, um
[11:46] you can see our little Slack message up there. We're much easily able to get access to um what's going on, what's wrong with our system, what kind of alerts we need. Whereas with Airflow, it was a little bit more of a pain living in email, which isn't really a place where much of our work in the organization lives. Um
[12:02] so, that's made things a lot easier for us to know what's going on. Um and then with the Unity Catalog, there's no more shuffling data around to different environments, no more linked servers, no more strange connections um to get like our on-prem data to BigQuery. It was like just a pain in the
[12:17] butt and like wasn't always in sync with what was going on in prod. Um so, to get that up there was it was huge for us. And having that all living the large-scale biomech and the small-scale pitch-by-pitch data, um all that living in the same place and seamlessly integrated was uh really big.
[12:33] And then having just an actual dev environment that we can quickly bundle deploy a job and test it up there is it was really really big. Um and then um with the the cloud storage, we're doing a lot of reading and downloading files. And then uh in addition, another kind of
[12:49] use case we found is um we'll talk a little bit more about R later, but like parquet files, um if you're trying to lar- write large-scale Spark data from R, it takes up a huge amount of memory and you'll just kill your cluster more often than not. Um
[13:05] so, what the solution we found was just saving it to one of our volumes and then we would just read it from it later um with with the Unity Catalog. So, that was a huge upgrade for us as well. Um
[13:23] So, analytics, um some challenges, some successes. Um I think Did Keith go home? Is Keith here? I don't know. He was He was here earlier. Um but yeah, so we have Keith Woolner, who I don't know if you guys have heard of VORP. Um it was value over
[13:38] replacement player, which then evolved into WAR, wins above replacement, which is kind of the single measuring stick, if there is one, of uh what a how good a player is that every outlet uses. You'll see it on the scoreboard at games. Um we have Max Marchi, who literally
[13:53] wrote the book analyzing baseball data with R. Um so even though we have our preferences on, you know, what platform, what language you're going to use, um our job is not to tell these these legends of the game how to do their jobs, but to work with them on how to make solutions work.
[14:09] Um In some ways, I think that we don't necessarily always appreciate. We are blessed that R is their solution. I've heard of teams like having to put MATLAB code into production, which um I'm glad is not my problem. Uh So, um in terms of performance and
[14:26] Databricks and R, you know, that we can't have serverless compute. Um I would say there's not necessarily as much time and energy put into R as there is to Python. Um but sparklyr does allow for us to use Spark and get some parallelization and
[14:41] work on some of those slower jobs that we have. Um And then, when we're kind of need to run like a lot of jobs like a big scale reprocess, um like if we need to shoot off 100 like pods basically running our code, we can do that. The scaling isn't always, you
[14:57] know, good to do it in all in one like you would get with Spark normally, but um there is that option, which we found super helpful in our large scale reprocesses. Um And yeah, more on the R front, I think we have some solutions we wouldn't call perfect, but do work in the environment.
[15:13] Um so some we're using like rpy2 to just execute R D S files and like the analytics files that they're giving us, um but still run it in Python, so we can keep all the benefits of Databricks in Python. Sometimes we're just running straight-up R scripts, and then
[15:28] sometimes there's a merger between the two. Um, but uh, and then another, uh, I think big benefit we're starting to realize is with the notebooks and the debugging process, like what we're doing when we need to fix a problem with our, uh, R&D
[15:44] analysts, it's much easier to just go through a notebook step-by-step, figure out where we're losing data or where something's going wrong. Whereas before with the different environments that they were running on versus we were running on, and like us having our stuff in Airflow, uh, it was a little bit tough for us to kind of sync up and figure out our
[16:00] problems together. But now it's, you know, it can be what was would be a couple hours, a lot of Slack messages back and forth, is now just a 15-minute debugging session sharing a screen with a notebook.
[16:18] Uh, AI workflows, this is something that, uh, we've been spending a lot of time recently on, um, that we're super, super excited about. Um, so we have all sorts of people doing reporting and writing about our players. There's sports science reports on the health of a player, there's scouting reports on what they could be, there's
[16:34] details from meetings with them, there's people watching video and writing what they're finding. Um, and that's just us doing it. Um, there's also, as I'm sure some of you are aware, there's a whole wealth of people writing about baseball, um, writing scouting reports, doing all sorts of analysis out there. Um, and so
[16:51] to actually synthesize and take all that would require a ton of time on each individual player. Um, we have a hard cap, but the hard cap is 165 players in our organization, so you can imagine if you're, you know, a high-level person who's overseeing every
[17:06] single affiliate, um, it can be very difficult to read every single report on every single player. Um, we have people that are spending every single day with our players, um, that are writing some of these reports, but then we'll have somebody that maybe get a two-day visit with an affiliate over the course of the year and that's
[17:22] kind of their big opportunity for the player to learn and grow. Um, and so our goal is to, you know, use these LLMs to sort of synthesize and create, you know, a single place to learn about a player and know what's going on with them. Um,
[17:39] as I'm sure with uh, uh, you know, any industry, we have a lot of jargon, a lot of acronyms. Um, if only you knew the acronyms, I could spend 5 minutes talking about our acronyms, but um, they're not always intuitive for uh, all the users. You know, the analytics teams
[17:55] know what their acronyms mean, but maybe the scouts don't. Um, and so in order to kind of, I think we we see the LLMs as a way to kind of create one common vocabulary to bridge those gaps, to create a source where you can say, "This is what this word means." Um, and kind of come together and learn.
[18:13] Um, another thing we have is like a lot of like manual tagging of text. If we're reading about like transactions, there's usually somebody like manually uploading information. Um, and so we want to allow our baseball people to focus more on baseball analysis than that sort of workflow,
[18:28] which is a lot of what they're doing currently. Um, and so I think we see a lot of a lot of potential in terms of using LLMs to figure out what text applies to what player and then do that kind of thing manualize manually. Um, on the software side, all our code lives
[18:44] in a mono repo. Um, we're a big cursor shop, so it's not necessarily using Genie as much as um, maybe this conference has you uh, talking about, but um, I think we're going to explore more there, but it makes asking questions across the database really easy. Um,
[18:59] in terms of like analytical workflows, we want to get those in the mono repo as well at some point. Um, but in terms of like front end, I get a somebody's talking to me about like a number they see on the front end that doesn't look right. Before I'd have to sort through, you know, 45 minutes of front end code
[19:15] or bother somebody in the middle of their day to figure out what was going on, which is I've never written front end code, so it was was difficult for me, but now I just ask Curser, and with everything in that mono repo, it's really easy. Um But yeah, in a in a broader sense, I think like I think Moneyball has has
[19:30] popularized somewhat of a view of like very players as like fixed fixed variables that can't improve, but like their potential is is static. Um and I think what we see is between lines buried in this text and what people are saying, um there's information that can kind of elevate these numbers, that can access potential
[19:46] that isn't maybe readily available or easy to see based on what the, you know, box score stats say. Um you know, this is before this sort of thing, but our MVP candidate, Jose Ramirez, uh was a middling prospect. I think he's 5'7" on a good day. Um not hyped when he arrived in the
[20:01] majors at all, but he met like the right player in the clubhouse at the right time and had got the right coaching, and he's been, you know, one of the best players in baseball for 10 years, and so I think we see some of this stuff is trying to access more of that untapped potential. Um
[20:18] Future goals. Um what's next for the Guardians? Um We would like to win the World Series, of course. We have not done so since 1954. If you are a Chicago Cubs fan or less likely a Florida Marlins fan, uh please see yourself out the back. Um
[20:35] But uh yeah, I think so we're we're on GCP still, so we um just got access to Lake Base preview, um which we're very, very excited about. Um we would like for our everything to be unified in one platform, um and so Lake Base is kind of our way
[20:51] to get uh get data to the front end with uh performance and latency at the required levels. Um that would remove the like right back problem that we have. Um this is actually inaccurate information as of 2 days ago because we can do uh right backs to the
[21:07] serverless compute. We got the networking figured out, so that's exciting. But I think still we would like to avoid that if possible, so that when we do a bulk reprocess, it's not blocking the database and ruining things at all. So, we want to get R&D and analytics more
[21:23] integrated into into Databricks. We got posit that I think there's some work to do there. And then kind of get them more familiar with how to run notebooks and that sort of thing, so we can kind of get all on the same page. I think we we talked a lot about AI in the last slide and I think we're we're building tools
[21:39] and kind of proof of concepting, but getting that organizational buy-in and kind of figuring out more directly what the, you know, people are actually wanting and needing is something that we're kind of still working to. But something we're really excited about. Our business side, shout out Kenny in
[21:55] the front row, is starting to get on boarded. They're moving their dynamic pricing model into Python/Databricks, which is exciting. But he tells me the 2013 R code is not quite a direct analog, maybe. But and they're yeah, they're doing some
[22:11] more large-scale stuff with the more web traffic data that MLB provides, larger scale of data. Um So, I think in a in a broader sense than kind of what we're talking about on this slide of direct goals, I think we see data engineering and what we do in
[22:26] Databricks as a as a competitive advantage for us. And just as our R&D analysts have had to get more and more creative to carve out advantages in how we evaluate players, so do we as data engineering, especially as the league becomes more sophisticated. The league, if there's, you know, any any fans out
[22:43] there, I'm sure you've you've played around with Baseball Savant and seen kind of the wealth of information that they have there. And they're, you know, we're getting more on our end than than they're providing to the public, so they're continuing to evolve in the way they serve us data and in doing so kind of starting to level the playing field
[22:58] across teams. Vendors are increasingly making deals with the league rather than the teams themselves. It used to be sort of you pay the money and you get the data and you have the data engineering. That's like an advantage in and of itself. Um, but now that the league's providing
[23:13] it and giving it in a much more easier format, that's kind of gone away. Um, some of the more thorny problems that we've we've tried to conquer um, the league has taken on. Um, so an example of that is in amateur player matching. Um, finding out who's playing what game,
[23:29] especially at the high school level, is a surprisingly difficult problem. It's very messy data. Um, you're not really getting like bio information. You're just getting like a name and he played in this game. Um, and not really an ID that corresponds to anything. So, and these guys are playing on a different
[23:45] team every week over the summer. So, there's not like you can't rely on a team. Um, names aren't always spelled right. They're using nicknames, you know, it's just a very messy process without like a you know, league overseeing it to make sure that data's clean. Um, so we had a lot of like work and time
[24:01] and apps that were built to support that um, process. Um, I'm not sure we thought we were good at it or the best at it, but I think we were better than some teams and so we saw it as somewhat of an advantage. Um, but now the league's taking that on and making a lot of that easy. So, um, they're they're telling you who's
[24:17] playing in what game, which yeah, it makes our jobs a lot easier, but also um, gives us less to do and it kind of takes away our advantage. Um, but we feel, especially with DataBricks as a partner and with, you know, the latest in technology to work with um, that there's still a trove of information and opportunity out there in
[24:33] video and text and more speed and better practices available. Um, and we are continuing to look for that advantage and and using DataBricks to do so. Um, that is all I got for the presentation.

Learn more about the Databricks Data and AI platform.

The information provided herein is for general informational purposes only and may not reflect the most current product capabilities or configurations.