How Blue Origin Accelerates Innovation With Databricks and AWS GovCloud

Summary

• FedRAMP compliance: Blue Origin uses Databricks on AWS GovCloud to meet ITAR and FedRAMP High compliance requirements.
• Predictive maintenance: Real-time IoT analytics and AI-driven tools predict equipment failures and boost operational efficiency.
• AI-to-manufacturing: Advanced features like Unity Catalog and Structured Streaming transform CAD designs into factory instructions.

How Blue Origin Accelerates Innovation With Databricks and AWS GovCloud

Talk by Filippo Seracini, Staff Product Manager, Databricks ; Seths Sethuraman, Data Platform Leader, Blue Origin
Blue Origin is revolutionizing space exploration with a mission-critical data strategy powered by Databricks on AWS GovCloud. Learn how they leverage Databricks to meet ITAR and FedRAMP High compliance, streamline manufacturing and accelerate their vision of a 24/7 factory. Key use cases include predictive maintenance, real-time IoT insights and AI-driven tools that transform CAD designs into factory instructions. Discover how Delta Lake, Structured Streaming and advanced Databricks functionalities like Unity Catalog enable real-time analytics and future-ready infrastructure, helping Blue Origin stay ahead in the race to adopt generative AI and serverless solutions.
Here's more to explore:
Databricks named a leader in the 2024 Gartner® Magic Quadrant™for Cloud DBMS: https://www.databricks.com/resources/analyst-paper/databricks-named-leader-by-gartner
An open, unified approach to your data, BI and AI workloads: https://www.databricks.com/product/databricks-sql
See all the product announcements from Data + AI Summit: https://www.databricks.com/events/dataaisummit-2025-announcements

FAQs

What is FedRAMP compliance?

FedRAMP (Federal Risk and Authorization Management Program) certifies cloud systems meet security standards for federal agencies. Blue Origin uses AWS GovCloud, which is FedRAMP High authorized, to protect sensitive space technology and meet government requirements.

How does predictive maintenance work in manufacturing?

Databricks analyzes IoT sensor data from equipment to identify failure patterns. Blue Origin uses these insights to predict failures before they happen, reducing unplanned downtime and maintenance costs in manufacturing operations.

What does ITAR compliance mean?

ITAR (International Traffic in Arms Regulations) restricts access to technical data on space and defense equipment. AWS GovCloud provides the isolated environment needed to handle ITAR-controlled data securely on the Databricks Data and AI platform.

How does AI help design-to-manufacturing?

AI-driven tools analyze CAD design data to generate optimized factory instructions. This automation accelerates manufacturing planning, reduces manual effort, and ensures designs translate efficiently to production.

Full transcript

Good afternoon everyone. Uh sorry for the slight delay for some of the technical issues that obviously they always happen in uh presentation. I'm the uh product manager at data bricks for compliance and all the special clouds like AWS cloud and I have here today s please s ramen u I'm from blue origin I've been with blue for a little over two years and I'm part of uh what we call as the enterprise data management and analytics team at blue so uh happy to be here and uh show us show what uh we are doing with uh data bricks awesome thank you sets So first of all, we'll try to keep you up uh since it's lunch break.
Maybe some of you already had it. So uh we'll do our best. But one thing I can tell you, you are at the presentation then the whole data summit has the best slide that you can possibly imagine. So we win the award hands down. So let's get started with that. Um so obviously this one probably you already seen. Whatever we say here, don't kill us. These are roadmap dates. So we'll try our best but things may change. Uh don't forget the survey at the end you'll receive it.
So please give us your feedback. And now let's get into the content. So let me introduce a little bit of data bricks on AWS cloud. Uh I'll talk about a little bit the journey. We started about two and a half years ago has been one of the companywide biggest priorities. Massive effort. Uh really a lot of lot of engineering hours went through it. Um we hit GA milestone uh in the fall last year and uh in the uh in the first iteration this okay awesome you want me okay uh thank you so in the first iteration we deliver on the classic infrastructure think like the classic compute where you own and your own um clusters is it designed to have feature parity with our commercial environment We're not there yet in all the aspect but I'll talk to you in a second on how we're getting there.
It also some of two things that I want to call out. Uh it has Unity catalog and DBSQL already available for you in terms of certification which in a regulated environment as a golf cloud environment is very important. We have the Fed high atto we have the uh department of defense impact level five uh provisional authorization. We have we support heap workloads and obviously by being in GC cloud also your ITAR uh workloads or ITAR requirements can be met uh by using data bricks on AWS GC cloud.
Where going next? I know a lot of you are going to interested in this slide. So in the next couple of months uh we just launched the system tables. I think that was uh towards the end of May are now available. Not all of the tables but at least the first batch of tables and we keep going bring more and more. Aback UCA Unity catalog Aback is coming at the end of the quarter for us. The quarter ends in July uh in public preview same shipping with the public preview on the commercial side.
In a couple of weeks give or take we're going to open up the private preview of serverless SQL. We are finishing up the last uh bits of or last round of testing and then we'll start on boarding customer on the private preview that will be followed by the preview of serverless DBR which means notebooks jobs and delta life table which should happen towards the end of July August time frame. This quarter we also bringing uh the ability to transact uh WSO cloud the data bricks on the WSCO cloud through the Pago marketplace that should happen somewhere in towards the end of June sorry July.
Uh so those are all things that are coming now starting in Q3 which for us starts in August to October and Q4 is November to January next year. We're going to start focusing on building the infrastructure to support all the Genai or Mosaici features. We are going to start with uh model survey which will start landing the first uh few components in the Q3 time frame and then we'll complete it by Q4. That's the current road map.
And now one thing I want to um want you to take away from this is that the serverless SQL DT specifically model serving and in the future also coming vector search those are the major pillars of all the geni mosa features we have. So we are basically building the foundation of uh for us to be able to offer and bring you all the serverless and geni uh features to go cloud. So once we have these pieces done you'll see a lot of more momentum building because all the other feature will not have infrastructure dependency anymore uh sorry all the infrastructure dependency will be uh met so they can just come to GC cloud. our northstar the target that we're working towards is to actually seam ship the features on commercial side as well as in the Gulf cloud at the same time obviously for the compliance uh when the features ship to compliance workloads.
So in other words when they reach maturity from a compliance perspective. Okay. So you could see still some delta from when they ship uh just release but when they are made available to com uh customers on regulated workloads in the commercial side we want to also make them available at the same time on the golf cloud side. So that's sort of our north uh northstar where we are marching towards and I think we're going to start getting very close to that towards the end of our year fiscal year which is uh end of January next year.
And with this overview, I now pass uh the mic to my friend sets here, which actually gonna show you the cool slides I was talking about. There you go. Thank you, All right. Uh how many of you here know about uh what Blue Origin does? Uh show hands. Okay, that's good. Uh again I've been there two and a half years and I don't know everything that we do about um but u you know I wanted to start off here talking about um the big picture of what blue origin does and then we'll go into how uh data bricks is helping us.
So that being said, let me move on here. Again, we are uh going with the vision of uh building a road to space uh for the benefit of Earth, right? So that's really what's driving everything that we do. Every product that you uh see today will be tied into that v mission. Um and again, our intent is to see millions of people living and working in space. That's really what we are uh working towards. So that being said, um fast facts about Blue Origin.
Um started in 2000. It's been a while, but uh predominantly being an R&D organization in the first decade. Um and then eventually getting into a lot of uh rockets that we have today. Um we are spread across right from Seattle to Huntsville to Florida. And I met somebody just a few minutes ago who uh worked with Blue Origin at uh Florida. So uh really a lot of u uh enthusiasm in what we do. Um 10,000 employees, right?
That's uh you know when in 2021 we were just 3,000 employees and we've grown significantly. We're continuing to grow. Um and we are sending more people to space right um we were um at the beginning of May it was 58 people that we were able to send to space through new new rockets and it's already uh up to 64 six more people and we're continuing to add more people to that. So that being said let me move on to the next slide and talk about New Shepard.
So New Shepard is one of our first rockets. It's uh the smallest one as well. Like it's 17 the size of a New Glenn rocket. Um it seats six people and really um the purpose of New Shepard is for us to understand the whole um flying to space uh workflow, right? So um this enables somebody to go to space uh experience weightlessness for 3 minutes and that's really um more a learning platform for us. uh and also enables a lot a lot of people to travel to space.
Um let me move on to the next interesting thing. Luna permanence blue moon uh we are partnering with NASA on it on our MK2 u to have a crude land mission to uh moon. The purpose of these landers is once we are in moon how do we move from one place to another. So that's really the focus of uh uh blue moon. Um and really you would need something like that when you're talking about living and working in moon, right? So uh really cool project.
We are currently uh in the design and development phase of this. Moving on, New Glenn. This is one of our biggest commercial projects. Um one unique thing about New Glenn is the 7 m fairing, right? The diameter of the fairing is 7 m which and 72 ft tall. Uh that's unique. um and that you know the industry standard is like five meters. Majority of rocket companies uh plan for that and kind of work towards that standard.
We are plan you know we are able to have twice the volume of a a standard uh payload. So that's really a unique thing about u new glenn. So that being said we had a very successful launch on Jan 16th this year. um we were able to deliver our payload successfully in in in our first launch. So that's uh pretty incredible. Um so that's really what drives the organization, right? Everything that we do, we try to do it uh right so we can get um success the first time we do it.
That being said, data bricks, data bricks plays a big role in what we do, right? Especially with New Glenn, um we have a ton of analysts on New England who use the data that uh is generated by New Glenn in terms of the telemetry data, the image data and you know the supply chain data everything that um is required to build new and you know launch and operated is captured and you know a lot of analysis happens on data bricks right I would say a few hundred gigs of data is generated on every test or flight and that's captured in uh data bricks and analyze in data bricks.
Um so that being said, let's move on to the next one. Blue ring. Blur ring is uh again um think of it as a mobile infrastructure within space. So if uh space computing, right? If you want to have host a data center in space, you would be able to do that through Blu-ray um and also um move payloads within uh orbit LO destination. So again this is a space station that we are building right now. Um we all know about the international space station and how old it is.
This is our attempt on building a a space station in the lower earth orbit. To power all of this what we have is the rocket engines right this is again we have a variety of rocket engines from B3PM to B4 to B7s. Uh and each of these rockets have a different purpose. Right? uh some take uh our propulsion modules, some are uh enabling us enabling New Glenn to go into space. We also sell these engines to other uh companies like ULA uh they had the Vulcan launch last year that was the first time B4s were deployed and they had a very successful launch during their first attempt as well.
So a lot of uh uh great work happens in the engine space and um you know we are cranking engines and all of the data is also fed into data bricks for analysis and enabling us get to red. So now let me talk about data bricks at blue. Um we started this journey uh 3 years ago with data bricks. Um we initially were using a varrow data varrows and we wanted to start looking looking at a lakehouse right like I said the variety of data we have with telemetry with images and also structured data we wanted a platform that can process all of this and provide intelligence right so we learn a lot right we all of this telemetry data is provides us a lot of inputs into making sure that we are operating efficiently and are able to uh deploy rockets without any uh failure or minimize failure.
You can never be perfect. Um so that being said, we built data bricks as the centralized data platform for making all of this happen and also ensuring that we can get our uh manufacturing running 24 by7. Right? So that's really the goal that we are working towards in terms of uh leveraging data bricks. Um one last thing I want to talk about is again uh we are in a operating in a ITAR compliant environment right so you rockets and spacecraft are part of uh is controlled by department of defense um so given that our data is supposed to be in a system that is fed high compliant or uh ITAR compliant right so that being said um when we first engaged with data bricks they did not have a version that can that is deployed in G cloud.
So we had to uh choose the route of something called as a private virtual cloud PVC where we install data bricks in our AWS GC cloud environment and manage the infrastructure behind it and again it was a lot of effort in in terms of maintaining the platform. So what we wanted to do was we wanted to we were want we have always been uh hounding our account team uh database account team on like how quickly can we get to go cloud we really don't you know want to use a lot of the features because every time we come to this conference we hear about a lot of features that are being built but we are not able to leverage it because we are on a PVC um so we were pushing them real hard to get go cloud enabled um And we wanted to be the first and they they like uh talked about on 8:25 or 24 August 24 they went uh genga on uh AWS go cloud we are we wanted to be one of the first.
So we started engaging with uh databicks account team and figuring out how we can um you know get this going. So primarily if you look at the features that we were looking forward to uh databick SQL we were predominantly using redash for a long period of time. So we wanted to get rid of redash because it's out of support and we wanted to le build uh um or use data brick SQL right. Um the second thing was unity catalog again it our intent was to leverage Unity catalog to kind of uh simplify the governance that we were doing on our data models and um those are the two features that are now being leveraged internally at blue.
We still waiting on like said we are still waiting on serverless SQL and enhanced machine learning but you know it was going to come we wanted to be first and we went in as soon as G um data brick workload cloud was G. So that being said let me move on to our approach that's actually a nice picture of new um ready I think it was in the test stand ready to go so that's the picture from that. So our planning we work with data bricks closely and really um through them engaged with lolytics as a partner to enable us do this migration.
We wanted to do go fast as fast as we can. Um so we really closely partnered with lolytics and then obviously we had daily standup calls with them. So we um were ensuring that they were getting every uh everything that they needed to be successful. At the same time we decided that we wanted to take over the validation and UAT part of it as well as the cut over. Right? The reason is um they're really good at automation.
They had automation scripts that can you know convert the assets from PVC to E2. We wanted to leverage them for that. We had a good partnership internally within our our teams. So we wanted to take over the testing and UAT aspects of it. Um eventually we did a broad parallel period with um where we gave our users access to both PVC and E2 and eventually um we shut down u PVC and used E2 as our primary. So what went well with this right because obviously uh this is a a effort that could have taken months more than what we had like we started off in um August September time frame litics competed their work in like mid October end of uh uh October time frame and then we started doing our UAT during November and we actually went live with D2 in December right like people um you know wouldn't be doing that but right before holidays we went live with E2 and we got most of our assets migrated into E2 and we ran this broad parallel period between um December and January and January is when we we shut down um very painful like uh some of my colleagues would attest here um but it was a lot of learning as well and we'll go into that next um one thing I want to highlight is the partnership with data bricks because that really enabled us to do this um the second thing is there motivation within blue and our user community.
So one of the things I will talk about uh here is we are a more uh federated organization in the sense that we own the central platform and we enable the various business units to do their own analytics. So they build their models, they build their uh Tableau reports, they build their uh all everything that they need from um you know machine learning to analytics, they self-s serve. We provide the platform from a central team perspective.
So that being said, they also wanted to leverage all the new features on E2. That was a motivator, a common motivator that enabled us to be successful in getting our users as well as data bricks also is motivated for us to migrate over to cloud. And that's really why I think we were able to do the successful migration. We also did uh datab bricks also did a lot of uh custom training sessions for our users on data bricks SQL because they were moving from redash to data brick SQL and u in fact we have a um a lot of LLM usage to migrate over some of the data models as well.
So that helped. All right. So what did we learn? What are the potential challenges that you all should be looking out for? Again, if you are in still in the early stages of migrating over from PVC into E2, um these are three things that came to my mind as uh top things that we ran into. Um we'll talk with invited gaps again in terms of access control and permission management in PVC and E2. They were different. So we had to make sure that the policies that we uh and the permissions for assets that and notebooks we had in E2 when we migrated over uh sorry in PVC when we migrated over E2 were carrying over and there were some gaps and that was uh there were some challenges around that.
Um, second thing is compatibility, right? This is a nuanced issue wherein the spa same spark version uh worked well for a specific notebook in PVC but did not that uh same job did not work well in E2 and we realized uh it was some kind of a bug that we had to run um that we ran into because as you are the early uh users of the platform obviously we run into things like that. Hopefully you won't run into the same thing.
The last thing and the most important thing is the policy right. So when it comes to unity catalog they pick the policy of zero trust right. So when an asset is created nobody has access to it. So if somebody needs access to it they have to be explicitly provided grants for it. Whereas in when we were using ranger everybody has access to all assets and we had to specifically deny um permissioning. So in in in essence we had to invert our rules like for instance if you uh if you have a home and you have a door with the lock you let open one uh door and you get you know all the data right like you can enter and you're having access to all of it whereas um with uni catalog with zero trust it was more like a turn style in offices wherein you badge in your identity is validated and you're allowed to go only one person is allowed to go so that is the kind of analogy I would use to um the pain that we had because we had to rewrite literally everything and we had to you know uh redo some of our DBT workflows as well because they were u built in such a way that these permissioning was not set up um and it was the most painful thing for us but we got through it.
So that being said I want to summarize this as the key learnings. The first two things on this list we were aware of right which is we ensured that we communicated with our users with the help of loyalics we were able to automate the migration. The first two things we did well the second two things security posture and the hypercap period. So security posture was something that we thought we could handle but then we had a lot more issues than we anticipated.
The hypercat period was almost buck right like because when we started off and we talking to lolytics and planning this whole thing it was almost like we had to do a hard cut over because you're moving from PVC into E2 and your notebooks and everything is has to be migrated over into E2 and you cut over on one specific day but that is hard to do when you have you know hundreds and thousands of users on the platform. So we really needed a way to do a a pro parallel.
Then came up this idea of all of our data assets are in S3 buckets irrespective of whether it's Unity or um PVC. So unless the schema changes on PVC, you don't really have to uh change anything. It'll continue to work, right? If the schema changes, Unity would be updated, but PVC would not be. And that's when you'll have a break and you'll have problems. So again being it's December don't expect a lot of changes. We did let our users know that hey this if something breaks then that's an indicator for you to migrate over into E2 otherwise you're good to go.
You have a you know prod parallel period till end of the year or something like that and uh you know it it that helped that was uh a saving grace and it helped us to do the prod parallel and do the cut over. Um lastly, one of the things again this was a boon uh which we didn't plan for going in. As we started rewriting policies and started building out um the Unity catalog, we realized how easy governance can be like a traditional database, right?
Like an Oracle or whatever, right? Like you can manage everything through uh the catalog. um that was eye- openening and that in fact enabled us to get rid of a lot of bespoke tools that we had and we are you know continuing to invest in it to make sure that becomes our governance platform eventually. So uh like I said it it saved us a lot of time cost and effort in in doing that with that I'll get to the enabled use cases.
This picture is my favorite from our New Glenn flight on Jan 16 this year. Um the seven engines, B4 engines, they create these beautiful diamonds in purple here. Um and I I think this is one of the best shots of seeing all the engines in their glory and doing their job. So um that being said, how what did this enable? Right. So it enabled a ton of use cases for us. um we wanted to leverage data bricks SQL which is you know what we are doing today. uh data sharing was enabled. we were able to um share assets like at this point you know we are working with a lot of government organizations and we need to share data with them and this has really been uh helping with that um and we are also researching some of these like from a predictive maintenance perspective we have a lot of IoT data that's coming in um we are able to run analysis on data bricks using data bricks compute to u p prevent any unplanned downtime Um that is important because for us um there's only a window a launch window on which you can launch a rocket.
If you miss that launch window sometimes it takes months before you can get the same launch window as say if you're sending some payload to moon. Um so given that we can't miss timeline right so it's it's something that is very very important and we need to ensure that the factory is always operational. data bricks is a great support in bringing all the data together and doing analysis and making sure we can prepare for that.
So we doing a lot of use cases to enable that. Um the other said one question on this one how often do you launch a rocket up in space? What's your frequency you guys are going towards? So we are again we are doing a lot of launches uh this year um you know with uh New Shepard. We already had three or four launches this year. We continue to keep the same cadence. Um same thing with New Glenn. We had one in January. We didn't uh you know capture the booster then.
So our goal is on August 15th, that's the next date we planning to launch New Glenn. We want to capture the booster. Um so that's going to enable us to repeat it because then it becomes a reusable booster that we can do more often. So that's um yeah. So basically between the launches that's when you have to analyze all the data and be ready for the next right. So we we have a short window and we're trying to shrink that window even further um that you know especially with IoT insights right look uh we have um today as the day the rocket flies in near real time we are able to stream that data into data bricks and use it for uh doing analysis.
So we are able to do the postflight analysis really really quickly within a day or so right that's that's very important um and that input is used for planning the next flight. So lastly this is another thing that we are experimenting with this as well. Um so we have a lot of CAD diagrams and we have manufacturing engineers and responsible engineers interpreting the CAD diagrams and creating work orders for uh execution.
Um and we are experimenting with how can we automate the whole thing right like how can a 3D picture be uh read and um work order instructions created so doing a lot of uh research activities there right that being said want to switch the topic a little bit um club for the future again like I started off we are doing a lot of these um in to meet the mission of having millions of people live and work in space. We I don't know if I'm going to see it uh in my lifetime, but this is definitely something I would think our our kids would see.
Um so with that in focus to encourage uh the next generation to be more engaged here and again the summer break for most kids at this time um and this might be a fun thing for them to uh uh be inspired about. So this is a program wherein you're able to send us uh postcard to space. We have um you know if you go to this url draw. club forthe future. org and have have your kid submit a picture. Uh it uh flies to space in our New Shepard flight and then it returns with the stamp.
Um couple of colleagues have done that and it's uh it's incredible to encourage your kids to do this. That's some serious bragging rights at school, huh? All right, so some fun activity for summer. Um moving on, we are continuing to hire and grow. Um one of the things we have realized with uh the advent of LLMs and AI and VIP coding and all that. We don't specifically call our engineers as data engineers or BI engineers or uh with any kind of specialization.
We are going we are calling our engineers as software engineers, right? So they they are able to develop front end back end the whole uh spectrum can be developed by an engineer. So like we tal you know it was referenced in the keynote this morning as well which is great to see. So if you go to this site and you are interested in making these rockets happen you know being part of a lot of the things that we do. Um for instance we are now working on um you know this morning it talked they talked about lake base and lake house. uh how zero ETL can bring data from lake base into lakehouse.
We want to do the same thing in our space as well. So we are building tools internally at blue to enable our data producers on a single click publish all the data into uh the uh the silver layer and enable the whole medallion layer through that process. Right? So working a lot of cool stuff. We're working on um stuff which can automate our uh data cataloges. Um so if you want to be a part of it, look for data software engineer titles.
There won't be any data engineer or um BI engineer roles. It'll be software engineer. But if you read the role, it'll it'll specialize and talk about data. So uh if you want to be part of this journey, uh those are some roles that are available. And there are also data scientist roles that are available. Do you get to go to space with a signing bonus as a signing bonus? Well, at some point I guess we will all be able to go.
Awesome. Thank you so much. So I wanted to ask you a couple of question. Uh first of all I mentioned that we had cool slides. I think uh we deliver on that promise. Uh thank you for sharing these beautiful images. So um actually how many of you have used are have used or are still on PVC or how many of you are familiar with PVC at all? Okay three hands too many but at least uh hopefully we it's a product it's end of sale and uh we are trying uh helping our customer they are using PVC in their own cloud environment to actually migrate to the new uh E2 environment of data bricks.
So now that you went through the journey, thank you for sharing uh the experience and the pain points obviously um now that you have done this journey and now you have the full new data bricks platform in your hands because PVC was sort of stuck in 2020 at best um how is your usage of data bricks and or what can you do now that you have the new platform right so one of the things that we have always wanted to do is leverage everything that data data bricks can provide, right?
Like we it it didn't make sense for us to build bespoke tools for capabilities that are already available in data bricks. Um so that being said like when Redash was out of support, we didn't want to look at another tool that can replace Redash. We really were pushing data bricks to get us data bricks SQL, right? So data bricks SQL and Unity catalog are the two major things that we were looking for when we migrated over into go cloud.
That being said, we are also looking forward to using a lot of the serverless capabilities, right? Um there's a talk about, you know, Genie this morning and I'm like, we talked about Genie two, three years ago in this conference and it's being used by every non-go cloud customer. It's just that we haven't really gotten it. Um we'll get it. We'll get it. So the our intent is to be part of um this journey to enable more use cases on data bricks specifically um you know machine learning use cases that we are doing outside the platform today um and there is um you know when I say machine learning not the traditional machine learning we're able to do a lot of data science problems within data bricks it's the LLMs and genai that we uh are not able to leverage uh data bricks for uh so So hopefully we'll have an opportunity to do that.
That being said, it also enabled data sharing, right? One of the things we you know like I talked about many of these projects we partner with governments and other organizations right I mean rocket uh it's hard to build and in some cases we need to have these partnerships to be u accelerating these efforts. Um so delta sharing has been a boon in that space because we tried you know FTPs and other things in the past to uh share data and that doesn't scale right.
So uh really delta sharing has been uh something that we have leveraged. Awesome. Thank you. So folks uh want to open the floor uh to you. So if you have question either for me on Golf Cloud or for sets and their experience uh the blue experience with Golf Cloud please. So

Learn more about the Databricks Data and AI platform.