Faster, Leaner, and Easier to Debug: PySpark UDFs in 2026

Summary

Native Arrow UDFs and Arrow UDTFs eliminate the Pandas conversion overhead of previous Python UDFs by operating directly on columnar Arrow data, delivering 3–7x faster execution in benchmarks.
Arrow optimization is also applied automatically to existing Python and pandas UDFs, so teams can gain performance improvements without rewriting any code.
New debuggability tools include built-in faulthandler integration for crash diagnostics, CPU and memory profilers, and structured UDF logging that surfaces worker logs as queryable DataFrames.

Faster, Leaner, and Easier to Debug: PySpark UDFs in 2026

PySpark has long provided one of the most powerful and flexible ways to bring Python logic to large-scale distributed data processing. In this talk, we focus on two major advances in the PySpark UDF ecosystem: performance optimizations and improved debuggability. First, we introduce Arrow-based execution for Python, including Native Arrow UDFs and Arrow UDTFs, which operate directly on columnar Arrow data without Pandas conversion overhead. This design reduces memory usage, improves support for complex types, and delivers faster execution compared to earlier Python UDF approaches. Second, we present new debuggability enhancements for Python running inside Spark tasks, including built-in faulthandler integration for clearer crash diagnostics and improved profiling support to better understand CPU and memory behavior of UDFs. Together, these improvements make PySpark UDFs not only faster and more scalable, but also significantly easier to diagnose and optimize in production environments.
Talk By: Tian Gao, Senior Software Engineer, Databricks ; Yicong Huang, Software Engineer, Databricks ;

Chapters

FAQs

What are Native Arrow UDFs in PySpark?

Native Arrow UDFs are a new UDF type in PySpark that operates directly on columnar Arrow data without converting to Pandas, reducing memory usage and double-copy overhead. Benchmarks show 3–7x faster execution compared to traditional pandas UDFs.

How does Arrow optimization work for existing PySpark UDFs?

Spark can automatically apply Arrow-based execution to existing Python and pandas UDFs without requiring code changes. This allows teams to benefit from columnar processing performance improvements in their current pipelines.

How can I debug a hanging PySpark UDF?

New stack trace utilities let you detect and diagnose hanging UDFs by inspecting the state of distributed worker processes. Built-in faulthandler integration provides clearer crash diagnostics to help identify the root cause faster.

What profiling tools are available for PySpark UDFs?

Spark now includes built-in CPU profiling and memory profiling for UDFs running inside Spark tasks. Worker logs can also be surfaced as queryable DataFrames, giving engineers structured visibility into the behavior of UDFs at scale.

Full transcript

[00:09] All right. Uh I guess we can start. Um hello everyone. Welcome. Uh this is a very early morning session. I hope that we can use this right time to get those information into your mind like get you the greatest memory for now. And a short
[00:21] introduction, my name is Huang and today I'm talking speaking with my uh colleague Tien and we're both from data bricks working on uh spark or spark commters and we focus a lot on pi spark.
[00:33] Okay. So today our talk is all everything about the pi spark in 2026 and we know the the community has a lot of different people right like a lot of different users for spark. Uh there are
[00:45] data engineers who care about the ETL pipelines. There are data scientists running those ML workloads in the notebooks also analytics uh like having those SQL commands and also people who
[00:57] are building platforms right like the using the reusable Python APIs on Spark so we know there are different people different needs but there is one thing is we know people want to use the Python's productivity together with
[01:10] Spark's scal scalability so our focus for the past few uh like a few years this is actually not uh our own work, it's the community work on the uh spark is to focus on those two aspects. One,
[01:23] make it more performant, right? Like using arrow for pispark. That's the first part of the talk. We'll talk about how we use that to make the entire system uh pispark faster and linear. And
[01:35] the second part we'll talk about how we make the pispark more easily to debug so it can help you to understand your workload easier. And the first part arrow for pispark. So
[01:48] first of all I think this is uh well known like uh nowadays like we have a row based and columner formats and for analytical workloads the columner format is the winner and uh it's uh it's better
[02:01] for cache uh the efficiency like for example you you don't need to fetch the entire row just for analyze a single field instead you can get the sing the whole column out right uh it's because
[02:14] it has the whole column and share the same type the same schema you can do better compression and that's good for storage, good for uh data transportation anyway and they are also better for like a vectorzed execution. Uh it's uh better
[02:28] for you to hit your CPU's pipeline and also the R1 L2 caches. So Apache arrow is a in-memory uh format like a column format now it's trending I think it's one of the uh
[02:40] state-of-the-art formats um uh it has those continuous typed columns and also the arrow IPC which is you can use the uh to transfer data between different languages and different processes and
[02:53] besides that arrow also has this uh called arrow compute right like they have their uh vectorzed computation uh libraries on top of those uh uh vectorzed uh formats. So, and a lot of
[03:06] uh like big companies and big uh projects are supporting and using arrow to power them. Um and pispark is not the first time using arrow like we actually go back in
[03:20] 2018 like since spark 2.3 we introduced a pandas udf uh that time we used the arrow to transfer data uh to support pandas udf already and throughout the years we have multiple versions of panda
[03:33] udf we expanded the the APIs to support different workloads on pandas uh but the arrow stays there as the transportation layer it's hidden it's not shown to user Uh as you can see here uh this is how
[03:46] pennis UDF works right like uh uh the the Scala GVM will send those arrow batches into the Python side and the Python side receiving it it will convert it into a pennis data frame and then you
[04:01] can invoke the pennis udf on top of it and the results will also be pennis data frame and you have to do another copy another transformation to get back to the arrow batch so you can send back to the Java side and to the X operator. So
[04:14] we see every invocation here has two copies and this is not ideal. So what if we can do one thing is we can just squash it into we get rid of those
[04:26] copies and make it like arrow UDF. This is the idea right if you already having the arrow data formats going into Python why don't you operate on it? So that's why Pispark uh we introduced this arrow
[04:38] UDF concept. We want people to write code directly in uh arrow to compute arrow formats. Uh in this way you can get zero in zero out like you don't have any copy.
[04:51] So we're introducing the arrow udf. There are two ways to define it. The first way is the like we have this new annotation like at uh udf. Oh shoot my bad. uh the at uh arrow udf you can
[05:04] decorate it on on top of your function and then the pispark will understand this is a arrow udf and the other way is you can use the traditional at udf uh decorator but you can provide those uh
[05:17] typings directly then the pispark will be able to automatically detect oh this is a par uh uh arrow udf and then we will go ahead and execute it there are also multiple ways to use
[05:29] those udf once you define them right you can use it in the data frame you can just put the uh this is already in Python so you can go ahead invoke the function or you can register as a SQL function and invoke it with a query both
[05:44] will give you the same result as a table okay and we want provide the same shape of the UDFs uh so previously we had a lot of uh different APIs in pandas and in
[05:56] Python pure Python UDF we want to bridge the gap here so But we want to provide everything uh in arrow as well. So what do we have? We have those scalar uh like uh uh udf types. So they will do like a
[06:10] uh give you a column of data. It will return you a column data as well. In pandas it will take a panda series and return you a panda series. You can do iterators as well. But for arrow side we will do using the arrow pi arrow array.
[06:23] So you get a pyro array input you can get a pyro array output. We also support aggregations, right? Like if you have multiple series or multiple columns, you can aggregate them into one scalar a scalar uh uh value which you can do like
[06:37] aggregations on count uh min max those things and you can do uh co- groupouped ones uh if you have multiple batches. We also support those mapping pandas and
[06:49] mapping uh applying pandas uh APIs. We also provide the same thing like a mapping arrow and mapping uh applying arrow directly. So all the APIs we have a uh a version that's ready in arrow to
[07:02] use. Besides that, we also have this native arrow use userdefined table function or in short udtf where instead of taking one column at a time, you can take a
[07:14] take a table and we support batches like a support uh pyro batch uh record batch and also you can supply it to multiple columns like using the second way to invoke the multiple oh sorry multiple pi
[07:29] arrays right and for the output you can just directly use a pi table or you can yield multiple tables or yield multiple record batches. So this is ideal for you to
[07:41] customize your table functions. Uh and uh if you want to like some training it can also be done this way.
[07:53] All right. So back this let's see the timeline. So uh we see previously before 3.3 everything was uh like in the hidden layer using arrows as the transportation. But back in uh 2022 uh in spark 3.3 we first released the first
[08:07] um the userf facing arrow udf it's a mapping arrow and I think this got really good adoption and then on on the roll out we are like for those those many years like uh four years now we
[08:19] have been bridging the gap to provide all those different types in arrow UDF and arrow UDTF and now we we can uh confidently say that all the APIs are matched. So this is now we can support
[08:32] uh aeronative computation directly on pispark. It's now not just the backend uh supporting layer but the user user seeing uh interface
[08:44] and the speed or the performance of arrow UDF is phenomenon right like pandas because you have to do those additional copies um the arrow udf doesn't need those so we actually run
[08:56] those small mini benchmarks and we see the performance is three to seven times uh faster than the pennis udf on local machines and it's also reduced the memory usage by 40% because you don't
[09:08] need to have those copies in the memory. And if you put arrow UDF into the uh spectrum of all the UDFs comparing with other like SQL UDFs and the regular Python UDF, you will see the Arrow UDF
[09:22] is actually much more closer to the native uh runtime speed. So it's way better than the Python UDF and it's much better than pennis UDF. So we recommend you to use it if you really care about the speed.
[09:39] Besides that uh not only on classic this also supports uh spark connect. So for arrow udf we shipped the both the classic and spark connected together. So this is day one is in spark connect. Um how it works it can have a client uh in
[09:54] your for example in your local machine and you can start to define your arrow udf and arrow UDF in the client and you can submit the plan through protobuff to a spark server which can stand on anywhere. Um then the spark server will
[10:08] go ahead execute the arrow udf and uh with the result without it will use the arrow IPC to send it back to the client. So you'll be able to see your results. All all the uh uh APIs we mentioned
[10:20] above like arrow UDF and arrow UDTF are available here. Now we introduced this new arrow udf a native way to execute arrow. Uh how about the old ones like we have those
[10:32] Python UDF and pandas udf. Um we cannot get rid of them right. There's tons of work running on them or and we we need to optimize them as well. Um for example for pandas udf I believe there are 446
[10:45] million panups udf executed just in 2025 so we have to do something to them for python udf we actually did one thing it's called arrow optimize the python
[10:57] udf where it's a it's a decorator configuration you can just use use arrow equal to true when you declare the udf then uh once you opt in you will have the native speed of the uh not not
[11:10] native speed the arrow uh IPC to transfer data to serve your Python UDF. This was uh open in uh uh Spark 3.5 and recently in Spark 4.2 we
[11:23] have enabled this by default. So if you come to the platform it's already there and for pandas is a little bit more interesting. So we realize the pandas community is starting to support arrow
[11:36] as a backend uh data type. For example in pennas 2.0 Z they supported the optional PI arrow back end. Uh what you can do is you can declare this DT type.
[11:48] Oh shoot. You can declare this DT type when you're reading the pocket and it will read the uh the data type as the pyro uh backend data type
[12:01] that was optional. And in pandanda 3.0 there start to shifting the like the default value on string. So the string type by default it's already arrowbacked
[12:13] and it's not uh confirmed yet but I we believe the pendas community is moving towards using arrow as the default backend for more data types and more computes
[12:25] and for that for pispark to follow uh we want also have this uh like a capability to run so-called like a zero copy or arrow back to pandas udf so for now like
[12:37] for all the pendas The reason there is a copy between arrow and the pandas data frame is because the data pandas data frame are backed by the numpy uh types. So we have to do the copy do the transformation
[12:49] but in the future if they can support fully back uh fully support the arrow back to pandas we don't need to do any transformation anymore right we get the arrow batches and we can run pandas udfm on top of it we put this into a test and
[13:03] we run those uh with the uh the available uh arrow data type for now and we see for even for panelist udf we can have like two two times of the uh speed boost and 25% % of less memory.
[13:24] So, so far we covered uh like a native arrow UDF that we bridge the full API uh par and uh it will have a give you the faster execution. It will have a less memory consumption and we're also moving on to more type of uh eval types for
[13:38] example userdefined aggregate function uh and other like more iterative uh uh eval types in the future. And for existing ones like a python udf
[13:50] we have the arrow optimized enabled by default and for pandas udf we are working with the community to move to the arrowbacked pandas udf in the future and there are some other features uh
[14:03] I'll not elaborate too much but for example uh that we are trying to do compression on the arrow IPC to reduce the memory consumption uh transferring data and we're also evaluating a flight
[14:15] to replace the current uh like a socket I io uh to transfer data through gRPC and the arrow IPC and we uh there's also a collaboration right now to move uh move towards the language agnostic UDF
[14:29] the idea is that you don't need to stick with pispark you can declare like a spec uh spectation of the worker and you can launch those workers in whatever language you want so we want to move
[14:41] pispark onto this language agnostic udf uh for for a short term. Um and this will be a community work with Hayang sitting here. Yeah. All right. So uh that's the performance
[14:55] part. So next part I will uh welcome our uh uh colleague Tianga to introduce you some more fun things about debugging.
[15:11] >> Okay. Thank you. So let's talk a little bit about debugging because um debugging UDFs has always been a pain and uh the reason that UDFs are very difficult to debug is that your function runs in
[15:25] another process and probably on another machine remote. So your normal debugging techniques probably will not work. Uh well this is like a spark spark
[15:37] structure which you are probably familiar with. So you have a driver and your notebook runs on it. That's where you write your code. But your driver will distribute the workload to different exectors and each exector
[15:50] probably will spawn uh multiple Python workers. Each runs in its own process. So it's very difficult to collect information from those processes. It's
[16:04] almost impossible to attach debuggers to all those processes and just like think about it there might be hundreds of workers running running simultaneously. Uh because of this structure there are
[16:19] very there there is very limited uh communication channel. So there is no control signal that you can send from the driver to the UDF directly except for like killing it and uh just only a
[16:33] very small piece of data metadata uh is sent back when all the Python workers finishes and uh all the prints you added in your UDF goes directly to the exector lock instead of driver lock because
[16:46] there are just too many of them like running you cannot handle that much in your driver and uh the information you generated or we generated is also very limited. Basically this trace back when
[16:59] you hit an exception and your raw user print uh when you add in your UDF and everything is unstructured uh everything is basically raw string and exception is basically sent back from the UDF as raw
[17:12] strings and all your prints is in in exector lock is raw string. Uh I'm going to go through a few common pain points when you are using UDF and
[17:25] uh one of the very common scenario that we hit is when a UDF hands
[17:37] like that locked or it's just slow and the only thing you can do is wait and sometimes customers wait for hours and realize okay this is probably a deadlock. uh there's no exception,
[17:49] right? It's not going to report that, oh, I'm in a deadlock or I I hand or some something like that. Uh but it will never finish either. So there's nothing thrown to catch and uh you don't know
[18:01] whether it's a real that log, a real hand somewhere or it's just takes a lot a lot of time. It's a long running batch. And the worst thing about that is
[18:13] there's no postmortem. So you can kill it. Sure. you can just like say oh I don't want to run this anymore but even after you kill it you probably don't know that okay whether this is a hen or
[18:25] like what what happened and what is the last state of this uh process you have very little to debug with and uh we have developed a few techniques in uh spark
[18:38] to deal with this kind of issues before everything I want to introduce a utility that we use on python ite to deal with uh uh uh Python uh UDF handings and it's a fault handler. It's
[18:52] actually part of the Python standard library modules. So it is available on all Python versions and uh the major the major capability of it is to sorry
[19:05] dump a Python stack trace. It can dump a trace on a fatal signal uh from the uh OS like uh sigboard or something like that and it can dump the trace back
[19:18] periodically at a certain rate and it can dump a trace back anytime uh you ask uh which means it can deal with uh user signals that you defined. So this is
[19:31] what we use currently uh for pi spark to uh thumb stack traces and uh there are a few ways that you can
[19:43] collect much more information when you're udf kind of hands. One of them very common is socket related handing. There are some configs that you can use for your uh Python workers and say okay
[19:58] if a socket between the JVM and Python has not been active for a certain period uh of time we just kill the idle worker and of course before killing it we try
[20:11] to do a trace stop. Uh as you can tell you can set your idle timeout seconds you can enable this feature or disable it. Basically, it operates as a uh OS
[20:23] level u monitoring for a certain socket and if the socket is inactive for a while, the OS will report and uh JVM will send a kill signal to the uh Python
[20:35] worker and try to uh dump a trace back. And that is not the only way to figure out what happens for a Hen UDF. You can also do periodic stack trace stop
[20:49] because well sometimes it's not happening on the socket and sometimes you don't know whether it's a real uh deadlock or a real hand and periodically dumping means you know what is your udf
[21:03] doing. Uh that is also a very simple config to set up and you can basically dump it while udf is running and it it is in the exeactor the dump is in the
[21:15] exector lock. So when you go back and check what's happening, you'll see okay this uh stack trace is like this this and that and if it's really a deadlock uh the probability is that you will see
[21:27] the same uh trace back for a long period of time and you know okay my code stuck in it and for data bricks uh uh uh spark this
[21:42] is data bricks only uh we also have a active hanging task detector which runs on the JVM side. So if your Python somehow did not work perfectly, we also
[21:55] have a hanging task detector on JVM side and basically poking the Python uh process. So it will monitor whether a certain task is active and sending back
[22:08] like uh reasonable data and if it's not then it will just poke it and see okay what are doing and are you hand and try to send a user signal to the process and
[22:20] ask it to dump some trace back and then probably kill it. there are more than the this threshold minutes configuration for this specific active uh hanging task
[22:32] detector to uh debug hanging UDFs.
[22:44] are curious about your UDF. you often want to collect more information about the function itself like how's it going in the process right so we want to we want all everyone to
[22:57] understand your UDF and uh the very common request is where does the time and memory actually go why is my UDF using a lot of memory or why is my UDF
[23:11] slow uh so we uh provided two profilers for Python UDF specifically to answer that and each one with the same oneline uh switch or configuration.
[23:25] CPU profiler is the way to tell you why your UDF is slow. Uh currently we have a CP profilebased profiler for your UDFs and CProfile is the uh Python standard
[23:38] library profiler uh which also uh is available for every single Python version and you can basically just set the uh this configuration to Perf and it
[23:50] will automatically instrument your UDF and when your UDF runs it will basically collect data and send the accumulated data back to the driver through
[24:02] accumulator. So every single UDF provides some kind of data and the exector collects it and send back back to the driver. And this profile is visible on the driver side. So as you
[24:14] can tell after you run your UDF, you can just do a uh uh do a spark.profile.show type equalsperf and it basically prints
[24:26] out exactly what zprofile prints out. So if you're familiar with C profile, great. If you're not, it may look a little bit weird to you. Uh basically, it tells you uh at a uh functional
[24:40] granularity uh for each specific specific function, how many cal how how many calls are there for each function? What is the total time? What is the cumulative cumulative time for each
[24:54] function? So that from this uh uh profile result you can tell like okay which function takes the most of the time in my UDF. Uh sometimes there are unexpected results there so you can
[25:06] understand uh the CPU related uh performance about your uh UDF better and we also have a memory profiler. Uh
[25:18] memory profiler is well it's based on a thirdparty library called memory profiler uh and it's a line by line memory profiler. So it's as a setup is
[25:31] also very simple. You also set the same configuration to memory. We use the same configuration string because you cannot enable both memory and uh CPU profiler
[25:44] at this time simultaneously because they use the same Python mechanism to uh to trigger uh the event and uh collect data. So if you set the profiler to
[25:56] memory and the rest is very similar, you will see a print of what my profiler provided. It's like this. So you will have your UDF on the right side. It's
[26:09] like basically on the line contents part and you will be able to tell how much memory usage uh uh is on that specific line. How much increment of course the
[26:21] first line you see the increments is really huge because it's the initial line. So it starts with 900 megabytes and after that you will know okay how many uh extra megabytes I'm using to execute this line. So this is helpful
[26:34] for you to understand okay which line specifically for my uh UDF am I using about memory. Uh these are the profiling
[26:47] tools that we provide uh provided uh for Spark and uh I mentioned above that all of your existing prints um even though they
[27:01] go to uh the exector log they're unstructured. Uh and by that basically you have like a huge text file and you need to like find your logs from it and
[27:13] figure out what it means. Uh and also you need to open your uh like spark UI or whatever and open your exector lock. So what we also want to provide is a easier way to log stuff from your UDF
[27:26] because print is what people want from their their UDFs. And what we have now is like the uh UDF logging. Uh it's also very easy to enable it.
[27:38] It's also spark config. And after you en enable it, you can use the logging module in your UDF to log stuff. This is also very commonly used uh Python
[27:50] standard library and also prints actually work. Uh it will collect prints as well. Uh, and to retrieve the logs, you just need to do a spark.tvf.python
[28:04] worker logs and you can get the log as a data frame. So it's an actual data frame where you can do all kinds of things that you want to do for a data frame like you can filter, you can group, you
[28:17] can join and the table is a little bit look like this. So it has the uh the the the the logging level. It has uh which logger it comes from. It also has the message of the uh uh the the uh the the
[28:31] the logging that it actually printed. It uh it actually also uh provides a context which is where uh which function this logging comes from.
[28:44] So yeah this is the uh structured uh UDF logging that we provided.
[28:56] But that is not the end of story, right? Uh what is the next thing to expect? What are the things that we can do better uh for PI Spark that make everyone's lives even more uh even
[29:08] easier? Let's talk about tracebacks because trace back is something we actually sent back from the UDF uh to the driver where you can see what happening uh in the
[29:22] driver and I know some people actually using um are using trace backs are are using exceptions to debug. So they cannot easily print stuff. they raise an specific exception with a string and
[29:36] saying okay this is happening because this is the thing that automatically gave transfer uh uh transferred back to your driver but before we the only thing we have is
[29:49] the call stack so which function you are in what about a trace back with local variables if you take a look at the right of course there's a zero division error so you probably can guess that okay balance is zero
[30:02] But what if we print that in the trace back so it's visible in your driver? If a UDF raise an exception, we can attach all the local variables in that specific
[30:15] frame and you will understand uh your current status of your Python program. You can see all the local variables for every single function, for
[30:27] every single frame and that is much more helpful for your debugging. This is uh kind of merged already uh in the uh in mainline but it's not released yet.
[30:44] And uh when I mentioned profiler uh I said two things well I said one thing that member profiler cannot be enabled together with the CPU profiler but there's another thing that I did not tell you is that profiler has a high
[30:57] cost. Well, C profiler has a high cost in memory uh uh in the third party memory profiler. That's just because they need to instrument every single call or every single line. There will be a Python function or Python callbacks ex
[31:11] executed. So enabling profilers on your code actually will makes your code run much slower. Uh and that is not good. That means you can only enable the
[31:23] profiler when you really want to understand what is happening. You cannot in uh have it on by default because it's just so slow that will impact your uh production. Uh and uh the overhead
[31:37] actually grows with the call column. Like I said, every uh it will instrument every single call. So like more calls you make the more time you will spend on actually triggering the call back and uh
[31:50] recording the uh time. Uh and a good way to solve uh both things that I mentioned above is a sampling profiler. So a sampling profiler basically samples stack on a
[32:03] timer. So it periodically samples your program and the sampling rate can be defined by the users. So if your program runs for let's say four hours, you can
[32:15] easily sample your program uh at a 10 second or 30 second interval which will introduce almost zero overhead to your program in general and you will still
[32:27] have a lot of samples which uh will make sense to you uh for your program. You will also have you will you will still have a very uh good sampling for your program and you can change that and uh
[32:42] uh that will basically uh give you a very good uh statistical result uh for your uh for your code. And if you think about it, uh the member profiler and the
[32:56] uh uh uh uh uh CPU profiler can run together on sampling because for every simp sample point I can just easily uh record the memory and record CPU usage,
[33:10] right? Uh so that is uh helpful and that introduces much much less overhead which means you can probably uh have this always on. So just enable it on your uh
[33:23] uh UDF and set a very low sampling frequency and you will always have some profiling when you say oh my production code is slow what happened. uh this not
[33:35] implemented yet but we do have plan on this uh sampling profiler and it probably will you'll probably see it I don't know next year or so and if you think about the mechanisms of
[33:48] a sampling profiler you realize that a sampling profiler is actually a hanging detector right a sampling profiler basically samples your programs and if your Python gill is if your gill is
[34:00] still working if your Python code is somehow still It's just a Python level deadlock. It actually samples all the threads and you will understand that if you you you realize that your program
[34:14] stucks on a certain thread waiting for a certain clock. So a sampling profiler is really a hand detector that is safe to live on. uh other than the fact that it
[34:27] actually samples uh the uh uh uh threads to give you enough information it will be at Python level. One of the things we had for fa fault handler is fault
[34:39] handler is designed to work in faulted states. So it does not care too much about if it can keep your Python program working properly. It cares about dumping
[34:51] some information that is useful for debugging. So we have rare cases where it conflicts with other uh seale native libraries and uh uh so it's normally
[35:04] designed at bad state but for sampling profiler you don't have that issue so it can always be enabled and safely run with all of your uh python udf pro uh uh uh python udfs
[35:17] so uh this will be much safer to enable uh by default and those are the potential futures that we'll see uh uh later this year or even next year.
[35:32] And that is everything that I want to introduce. And uh if you have any questions, feel free to ask.

Learn more about the Databricks Data and AI platform.

The information provided herein is for general informational purposes only and may not reflect the most current product capabilities or configurations.