A Practical Roadmap to Becoming an Expert Databricks Data Engineer

Summary

• Core tech stack: This video outlines the essential Databricks tech stack including Python, SQL, Git version control, testing, and data governance skills needed for modern data engineering.
• Skills progression: This video provides a structured learning roadmap from foundational concepts through advanced capabilities, supported by live code examples and real-world demonstrations on the Databricks Data and AI platform.
• Career development: Attendees learn about certifications, hands-on experience opportunities, and reference materials to accelerate their journey toward becoming expert Databricks data engineers.

A Practical Roadmap to Becoming an Expert Databricks Data Engineer

Talk by Derar Alhussein, CEO, Acadford
The demand for skilled Databricks data engineers continues to rise as enterprises accelerate their adoption of the Databricks platform. However, navigating the complex ecosystem of data engineering tools, frameworks and best practices can be overwhelming. this video provides a structured roadmap to becoming an expert Databricks data engineer, offering a clear progression from foundational skills to advanced capabilities. Acadford, a leading training provider, has successfully trained thousands of data engineers on Databricks, equipping them with the skills needed to excel in their careers and obtain professional certifications. Drawing on this experience, we will guide attendees through the most in-demand skills and knowledge areas through a combination of structured learning and practical insights. Key takeaways: Understand the core tech stack in Databricks Explore real-world code examples and live demonstrations Receive an actionable learning path with recommended resources
Here’s more to explore:
Production ready data pipelines for analytics and AI: https://www.databricks.com/solutions/data-engineering
The Big Book of Data Engineering: https://www.databricks.com/resources/ebook/big-book-data-engineering-2nd-edition
See all the product announcements from Data + AI Summit: https://www.databricks.com/events/dataaisummit-2025-announcements

Chapters

• 0:00 Introduction and agenda
• 1:46 Data engineer roles
• 6:25 Engineering on Databricks
• 9:49 Learning roadmap steps
• 16:07 Pipeline automation
• 19:33 Resources and career tips

FAQs

What are the core skills needed for Databricks data engineering?

Key skills include Python and SQL for data processing, Git for version control, testing for code quality, and knowledge of data operations, maintenance, and governance practices across the Databricks Data and AI platform.

What learning path does this video recommend?

This video presents a structured progression from foundational concepts through advanced capabilities, combining theory with live code examples, demonstrations, and hands-on experience to build practical expertise.

Are there certifications available for Databricks data engineers?

Yes, Databricks offers professional certifications. This video covers how to pursue these certifications as part of your career development and provides guidance on preparation resources.

What reference materials are recommended for ongoing learning?

This video recommends resources including "The Big Book of Data Engineering," official documentation, and hands-on projects on the Databricks Data and AI platform to deepen your expertise.

Full transcript

0:00 Introduction and agenda

Hi everyone, can you hear me? Okay, perfect. Um, so yeah, uh, thank you for joining. Um, we are here today to, um, to present a practical road map, um, to becoming an expert, uh, data bricks data engineer. Um so here's our agenda for today. Um we are going to start by just presenting um uh the data engineering uh role. Um then how data engineering uh looks like on data bricks. Um then we are giving some you know learning bats and certifications. uh how to get hands-on experience and uh finally some uh reference materials.
Uh first of all um my name is uh. Uh I am a datab bricks MVP. Um also um I am uh founder and CEO of AAT for which is an AI based uh e-learning platform um that helps in um learning um data bricks. Okay. Um so uh some of you maybe know me from my Udemy courses. Now we are like uh more than 80 uh,000 student on Udemy. Um also maybe uh you know me from uh this book uh from uh my or book uh so it just released uh three months ago uh in February and uh yeah so and we have some you know data bricks certifications.

1:46 Data engineer roles

Great. So first of all let us uh understand uh the data engineer role. Okay. Um so data engineering is um the practice of uh designing and building data processing systems. Okay. So the goal is actually to uh prepare data to be ready uh for use uh for different use cases. Uh so here for example uh we are ingesting data. Okay. uh we are uh preparing these data, we are transforming these data and finally load it uh to be available for different use cases.
Um so for example maybe for uh business intelligence, maybe for uh AI or machine learning. Okay. So the things that here that to build such a system you need um a mix of skills. Okay. Um so these skills usually can be classified in four uh categories u which are software engineering uh data ops or data operations uh maintenance and data governance. So let us start by data engineer by software engineering. Um so since we are uh talking about data processing system so it's a system it's a software okay so we need to build this system we need to develop these system so we need to master code development okay um so mainly uh the main language today for uh for modern data engineering is Python okay and since we are um working with data we need also to master the language of data which is SQL um Since it's a software, you need also to track all the changes uh all the um um versions of your uh code using git.
So we have also the version control. Okay. Uh next we have testing. So to apply different testing like unit testing, integration testing and lastly you need to store your data. Okay. So um this is why you need to um to master also the different uh types of uh data storage formatting. Okay. Um, next uh we have uh data ops or data operations. So because in fact the goal is not just to build the system but we want to um start uh deploying this system uh to production and automate this process.
Okay. So this is why here we we see like also um plenty of uh skills that need to master actually. So starting by CICD which is uh continuous integration and continuous uh delivery which allow us to automate uh the deployment process of our uh pipeline of our data processing system into uh production. Okay. Um next uh we have orchestration uh because uh usually we don't want actually to our pipeline or system to run manually.
So we would like to orchestrate orchestrate these. So um it's run in uh like for example on a scheduled basis. Okay. Um also we need to control the data quality of uh of our data uh of our data pipeline. We need also to monitor all if there is any issue during the during the pipeline execution. Okay great. Uh next you after running your pipeline you may see that okay there is need for optimizing this pipeline. Uh there is maybe need to make a cleanup for the files system for uh any uh files related or issues um related with your uh pipeline.
Okay. And lastly uh we have the data governance where you need to ensure that your data is governed secured um in order to um to make sure that uh your uh pipeline is actually um secure and also um fully um uh like in term of security in term who have the access to these uh act who have access to these data when and um um when and which exactly uh type of data. Okay, great. So now how all these uh services or skills can be uh applied in uh data bricks how data engineering look like looks like on uh data bricks.

6:25 Engineering on Databricks

Um so first of all if you look here in the uh in the middle actually uh we have spark and uh DT um in fact in order to uh build uh data uh pipelines or data processing system in data bricks uh you need to use u apache spark or now uh recently we have also the DT platform uh which is another way uh a modern declarative way to build also uh data processing pip pipelines. Um so yeah we have these two options. However all the other skills remain the same actually.
So when when talking about uh software engineering uh we we need to uh master again code development and here we we are precising that we need we need to learn pi spark uh which is uh where uh we are going to use the engine the spark engine in python. Okay. Uh using pi spark data frames. uh you need also to uh work with um spark sql as well to work with the with the data in the spark. Okay. Uh in data bricks we have git folders in order to manage the uh git uh operations.
Uh you can use different libraries like piest for testing your uh your your pipeline. Um we have also uh delta lake which is the default table format on data bricks um in order to store your data. Okay great. Uh next we have data ops data operations. So you can you can deploy your pipelines to data bricks platform using CI/CD. Um the idea is to run your uh jobs to deploy your jobs. So on data bricks the orchestration is possible using datab bricks jobs where you can um automate and schedule your uh the run of your pipelines.
Um we can also monitor the data quality using uh lakehouse monitoring. We can also troubleshooting um our pipeline using different uh services notification logging uh spark UI. So we have um okay uh okay next we have maintenance. So we can um we need also to master some of these uh you know uh tools like um commands the optimize command Z order indexing and uh liquid clustering. Okay. And also we have the vacuum command for maintenance.
And finally all your data is managed all your data assets is managed using unity catalog. Okay. So yeah so as you can see uh we have plenty of things to learn plenty of shows of uh things actually that you need to master in order to um to work with u to be a data engineer actually. Okay. So this is why uh having you know a sort of structured uh road map uh may help actually or should help in um in learning these old skills in a progressive way actually okay so here's we suggest a six steps uh road map okay where uh we start from SQL till we um we master the advanced data engineering skills Okay.

9:49 Learning roadmap steps

So, first of all, uh we need to start by SQL. This is this is very essential. This is the fundamental when you want to be a data engineer. Okay. Uh SQL is the language of data in order to uh to query your data. Uh to retrieve these data um you need to master SQL. Okay. You can learn SQL on different levels. uh starting from like starting by the fun fundamental skills uh then more inter intermediate skills and finally to the advanced skills.
Okay. Um so like learning the how to retrieve data, how to filter them, how to aggregate them. Next with the intermediate level you can um master like advanced joins, self join, uh how to do subqueries. Okay. Okay. And lastly the advanced level where you have the optimize um how to optimize the performance of your queries. Okay. So we see different resources uh available for these uh which is like you can go to the W3 schools the on Udemy you can find great courses and boot camps.
Leits code can also uh helpful in these case uh to learn uh and solve exercises. Okay great. In the same way we need to also learn Python programming. Uh so we we the same idea you can learn it on different uh uh levels from fundamental intermediate and advanced starting from working with variables data types control flow um then uh object-oriented programming packaging then even applying decorators and generators. Okay.
Um so on the data bricks academy they have a great course uh which called introduction to python uh for data engineering. Okay so this will um will help you in learning these kind of skills also on Udemy uh you can find great uh courses and boot camps uh on these. Okay. Um next you need to learn Apache Spark as we said that Spark is the engine of uh of uh big data uh engineering in data bricks. Okay. So you need to learn like the architecture of sparks, sparksql, p by spark data frames uh more about also structured streaming, spark connect, pandas api on spark and spark ui.
Okay. Uh so there is a great uh book actually from it's called learning spark. Um also there is a course on data bricks academy uh for apache spark programming. Okay. and it will help you also um prepare for the certification uh to be a spark developer associate. Okay. Um next uh we start uh here we can start learning more about the data engineering. Okay. So uh we going to start by the fundamental of data engineering.
Okay. um like to learn more about the data bricks uh lakehouse um more about ETL with SparkSQL and Python processing data incrementally production pipelines data bricks SQL and data object uh privileges. Okay. Uh so again here there is also a dedicated certification for this uh level of uh of skills. um on datab bricks um academy also there is a uh a great course uh to learn the skills if you have if you already like a partner or client for data bricks you can access this this one uh also I am helping on Udemy like with my course uh to get all these kind of skills um so yeah you have different resources at this level okay um next you can go to more intermediate skills in your data engineering uh um journey.
Uh so like now about learning about data modeling um change data capture, security and governance, rest API and command line interface monitoring, logging and testing. Okay. And at this level you can get the data engineer professional certification. Okay. Um again on data bricks academy you have a course for this and uh and I help also with this uh with this level at this level with my course also on Udemy. Okay. Um great.
Lastly um you will once you master all these skills actually you will see that there is still need actually to learn to learn more and more about uh uh about data engineering actually to master this role. Okay. Um so one what it's very important actually um after like um working with um uh when working with the data engineering is actually to apply what we call it the data engineering uh design patterns. Okay. Uh for this there is a great book actually uh which is like just it was released two months ago.
Um so it help you um to learn like a different design patterns uh maybe for data ingestions for data quality. Uh so yeah you have plenty of data design pattern that is very important for you as a data engineer. Okay. Um learning also having the fundamental of cloud services because most of the time you need to learn uh to to configure multiple cloud services. Learning Kfka if you are working with the streaming is very important on Confluent CFKA training. to have also a great course um CI/CD packaging and publishing to PI u to Python um package index and lastly also in infrastructure as code okay so if you remember we started by a very simple uh uh pipeline okay where we have just the row data the data processing system and our readytouse data okay we would like to uh to reach to a level actually where we can have everything automated.

16:07 Pipeline automation

Uh we have like different services where we can build pipeline our pipeline in a more you know uh reliable way and trying to build it even even faster actually using different uh automation service. Okay. So first of all uh you will see that when working with in data engineering you need to work on the different environments like uh dev environment product environment and this will contain for example your data bricks work space um your like the storage in the cloud uh for example.
So for this you can use um here at the bottom you we have like terraform or any uh infrastructure as code service okay to deploy this environment these different environment automatically. Okay. Um next you will have your code in uh in a g uh provider like for example on github. Okay. Once it is in a g provider you can now uh publish this code uh like to a package uh index. So at the top we have the um Python package index where we are going to uh deploy or publish our code as a package to um to this index.
Okay. Uh and during this process we can also apply any you know uh like um dependency check any um data uh or unit test for example during the CI process. Next we have the CD process where we can now deploy the job. Okay. So as you can see we have a job uh around our uh data processing system in order to schedule our uh system to run uh on uh like scheduled basis uh automatically. Okay. And at the end once our pipeline is running on a production now we can uh track uh the like monitor this uh the results uh for example using spark UI using any logging system uh for example um and notification.
Okay. So yeah this is uh where we would like to um to arrive. Okay. So yeah the goal at the end is actually all these process actually in order just to build or to deliver these data actually which is ready to use data. Okay. So the idea to make these data available for different use cases like for business intelligence or for example for machine learning and AI. This is the ultimate goal of building pipeline. We are not just building pipeline because we like building pipelines. uh a data engineering uh role is not just about uh building pipelines.
You will not be like evaluated by the number of pipelines you can uh build uh for example in a month. Okay. Uh so you will be evaluated by the value that this uh pipeline will uh bring to the business. Okay. As you can see there is a cost behind these. Okay. So these pipeline they should deliver uh a really an impact positive impact on the business in order uh to um to be to say that I build a successful pipeline. Okay.
It's not just building a pipeline that is uh running successfully. Okay. Uh great. It's great to have like um always real um like real world project. If not, you can also check these resources for hands-on experience on the uh data bricks demo uh center. Okay, where you can install some uh you know uh exercises and demos and you can now do it with the free uh data bricks uh work space. Uh you have also the datab bricks solution accelerator which is uh like fully functional notebooks for different use cases and this is will help you like understand how you can build different solutions.

19:33 Resources and career tips

Okay. Um you have also if you like to enjoy further readings you have like delta lake the definitive guide practical uh lakehouse architecture and also recently uh soon we will have the data governance with unity catalog on data bricks. Um so we are almost done. Lastly, uh here is try also to be to engage with the community um to participate uh you know in the community forum to in meet in meetup and workshop. Um and my last advice is that as you can see that a data engineering um a data engineer role um it needs um plenty of skills.
Okay. So sometime um getting um a junior a junior data engineer role could be tough actually with all these skills. So my advice is to like to adapt a more like a career uh progression uh pathway where you start with any uh data related job. Okay. Like for example you could start with a data analyst with ETL developer uh with a BI engineer. Any of those role will give you the fundamental actually um skills like for example to learn how to perform data you know manipulation how you will see the problems that you usually have with data like uh data duplication um data that is dirty how to clean them you know even at uh even it's not at a um at a scale um like in big data okay uh next you can transition to a data engineer role it will be like more smooth um and easier process.
Okay, great. Uh that's all from me. Um here is my link in as well. If you would like to connect with me with me on LinkedIn, uh it will be a pleasure. Um and yeah, that's it for me. Thank you.

Learn more about the Databricks Data and AI platform.