Kafka Forwarder: Simplifying Kafka Consumption at OpenAI
Summary
- Consumer proxy pattern: Kafka Forwarder offloads partition management, offset tracking, retries, and error handling from client applications, ensuring reliable at-least-once message delivery.
- DLQ visibility: Failed messages and retries automatically route to Dead Letter Queues in Databricks, making it easy to analyze failures and debug production issues at scale.
- Self-service infrastructure: OpenAI's teams can now onboard to Kafka without managing partition complexity, enabling scalable, high-reliability consumption for data warehouse ingestion and real-time processing.
Kafka Forwarder: Simplifying Kafka Consumption at OpenAI

Talk by Jigar Bhati, Member of Technical Staff, Open AI
At OpenAI, Kafka fuels real-time data streaming at massive scale, but traditional consumers struggle under the burden of partition management, offset tracking, error handling, retries, Dead Letter Queues (DLQ), and dynamic scaling — all while racing to maintain ultra-high throughput. As deployments scale, complexity multiplies. Enter Kafka Forwarder — a game-changing Kafka Consumer Proxy that flips the script on traditional Kafka consumption. By offloading client-side complexity and pushing messages to consumers, it ensures at-least-once delivery, automated retries, and seamless DLQ management via Databricks. The result? Scalable, reliable and effortless Kafka consumption that lets teams focus on what truly matters. Curious how OpenAI simplified self-service, high-scale Kafka consumption? Join us as we walk through the motivation, architecture and challenges behind Kafka Forwarder, and share how we structured the pipeline to seamlessly route DLQ data into Databricks for analysis.
Here's more to explore:
Databricks named a leader in the 2024 Gartner® Magic Quadrant™for Cloud DBMS: https://www.databricks.com/resources/analyst-paper/databricks-named-leader-by-gartner
An open, unified approach to your data, BI and AI workloads: https://www.databricks.com/product/databricks-sql
See all the product announcements from Data + AI Summit: https://www.databricks.com/events/dataaisummit-2025-announcements
Chapters
0:00Introduction and agenda1:22Early infrastructure problems5:48Decoupling strategy7:43Producer and consumer proxies10:20uForwarder architecture15:48Kafka forwarder implementation18:15Consumer onboarding flow20:33Multi-region routing23:44Miscellaneous system improvements26:18Kafka to data lake system28:22Future work and conclusion
FAQs
What problems does Kafka Forwarder solve?
It abstracts away partition management, offset tracking, error handling, retries, and Dead Letter Queue management from client applications, enabling reliable consumption without complex client-side logic.
How does Kafka Forwarder ensure delivery reliability?
The proxy implements at-least-once delivery guarantees with automated retries and routes failed messages to Dead Letter Queues in Databricks for visibility and analysis.
Why did OpenAI build Kafka Forwarder?
OpenAI had 30+ non-standardized Kafka clusters with no unified platform, leading to reliability problems. Teams needed 5-9s or 6-9s reliability without managing complex consumer semantics.
How does Kafka Forwarder integrate with Databricks?
The proxy routes messages to consumer applications and automatically directs failed messages and Dead Letter Queue data into Databricks for analysis, enabling centralized debugging and monitoring.
Full transcript
0:00 Introduction and agenda
Before I start maybe let me introduce myself quickly. So I'm a member of technical staff at OpenAI. For the past two years I've been working in the real-time infrastructure team where we manage anything and everything that has to do with Kafka. Uh we build some abstraction and proxy layers on top of it. We also support APIs that our product teams can easily use and integrate their services on top of our Kafka ecosystem and uh before joining OpenAI I was working at Twitter for almost 7 years on distributed databases and today I'm going to be talking about uh Kafka forwarder and how that helps in simplifying the Kafka consumption at OpenAI.
So let's dive into it. Uh so this is the rough agenda of the talk. I'll start with talking about early streaming infrastructure at OpenAI when I joined back in 2024. Uh then I'll talk about how we decoupled users from Kafka infrastructure to improve the reliability. Later on I'll talk about how the Kafka forwarder architecture looks like at OpenAI and then I'll talk about uh the Kafka consumption how the new onboarding system looks like.
At the end I'll talk about the Kafka to data leak system that we use for some of our uh deadl topics and I'll end with the future work that we planned for the next few months. Okay, let's start into it. Uh so going back uh to H1 2024 u we had a lot of user demand for Kafka ecosystem in general. They wanted to use Kafka for data warehouse injection purposes so that data scientists can analyze the data. uh they also wanted to use Kafka ecosystem for real-time asynchronous processing uh and also interervice communication uh but many of this use cases they had reliability problems back in the days because of the immature platform that we had for Kafka ecosystem and we could see like there were lot of use cases that's going to come up in the near future that's going to need like 5 9s or 69s for reliability which was not existing back in the days. uh some of the problems uh that we saw I'll talk about that next.
1:22 Early infrastructure problems
So the first one is there was no standardization. Uh overall we had like 30 plus different kafa clusters and there was no dedicated data platform team as well and these kafa clusters were created by the product teams themselves just to make sure that products are launched uh as soon as possible. Uh so they were created in a non-standardized uh isolated manner and uh many of them had no like uniform configuration settings and uh because of this they were facing uh like any any new use case new use case that they wanted to onboard. uh they were not sure where the topic like which cluster the topic should be created to and that would cause like frequent confusion and many of these clusters were shared clusters with other uh industry experts and also many of them were single tenant clusters as well.
So there was a lot of confusion uh with regards to standardization and which clusters to use for onboarding new use cases. Uh the second one is there was no resilience to cluster outages. Uh so just to get an example the producer services which are the product teams uh they would directly interact with the Kafka cluster uh back in the days and uh if the Kafka cluster goes down or the region where the Kafka cluster is deployed goes down it will cause unavailability for entire Kafka publishes which was not not ideal at all and uh even if uh the Kafka the APIs that were built back in the days were such that if this communication path has problem it would also affect the reliability of the producer service itself.
So for example just if chat GPT wanted to use Kafka and if Kafka has problems then chat GP would get affected. Uh that was the reliability problems back in the days. Uh the next one is integration challenges. Uh because the product teams were directly interacting with Kafka clusters. They had to hook up their own secrets to communicate with the Kafka clusters and also set up their own firewall so that the network communication is secure.
And all of this required a bunch of Terraform scripting and also hooking up the secrets right into their services which was tedious and overall the onboarding type could take uh like days if not weeks for them to be able to see uh safely onboard onto the Kafka ecosystem. The next is scalability limitations. Again just to go to an example of CHPity if if it could be deployed in like multiple uh different uh regions and each of these region could have like tens and thousands of instances or pods deployed amongst themselves and uh since like OpenI operates primarily as a Python service.
Uh what could happen is typically all these Python pods will have multiple processes just to get around the G limitations and that would create like millions of connections to the Kafka cluster. uh so that would end up being a bottleneck on the coffer cluster like the connections itself rather than the ingress or the eress traffic which is typically uh what should be the case for the cafer clusters. So we had like many different tactical challenges like this but boiling down everything we had one big strategic issue that we wanted to solve was the infrastructure uh inflexibility that was the main problem because of tight coupling of clients to individual kafa clusters. made uh changes and maintenance is impossible.
Like uh just to get an example again, if if we want to create like more Kafka clusters or migrate one topping from one Kafka cluster to another, we would have to directly uh integrate our uh changes into the product services code which have their own deployment cycle and reliability risks. And if we have like thousands or hundreds of services, then we'll have to do these changes for all of them, which was not ideal. uh so all in all we wanted something which uh by the way we by which we can decouple the producers and consumers from the coffer clusters directly.
So we thought of uh like before we dive into like the architecture why should we decouple in the first place the first is simplified integration so that the product teams are not interacting directly with Kafka clusters at all and we hide all the abstraction of infrastructure behind some layers that we can build our build ourselves and uh expose those services to the product teams rather than the Kafka infrastructure.
5:48 Decoupling strategy
The second is high availability uh by routing around degraded clusters. uh this is primarily the uh single point of failure that I talked about. So if you have like multiple kafa clusters, you can route and uh automatically detect failed clusters and route the traffic appropriately so that we can achieve high availability and provide like five or six lines of availability for our customers. The next one is improved scalability by reducing connections.
Uh so so if you have the if you build a layer such a way that u the layer is interacting with the Kafka then we don't have to worry about all the producer services parts connecting directly to Kafka and then we can control the uh connections that are made to the Kafka broker much more uh easily and with a reduced uh count of connections per uh brokers. So the strategy of decoupling as I already hinted is we wanted to build something some layers before between the producers and the kafa clusters and also between the kafa clusters and the consumers so that we can simplify the entire architecture.
Uh and the producer and the consumer services which are the product names are interacting with this proxy layers rather than the kafa clusters directly. So we built exactly that. Uh so for the producer proxy we built something called prism which is the producer proxy layer. I'll dive uh at a high level into how that is built. And on the consumer side uh we adopted something called euoder which is an open source code for consumer proxy by uber that we adopted and evaluated and migrated most of our production use cases to and I'll dive deeper into the uh orderer and kafka orderer architecture.
So what is prism? uh as I said it's a Kafka producer proxy and it's a lightweight gRPC service which exposes only a single endpoint uh called produce batch uh by which the product teams will just call this API and produce messages to the Kafka ecosystem. As you can see like it's not directly interacting with Kafka APIs or anything along those lines. It's it's a custom gRPC endpoint that is exposed by the Prism service that the clients can use.
7:43 Producer and consumer proxies
The second is simple simple integration. Um so since since we they are interacting with an internal service there is no real uh secret management or firewall setting that's need to be hooked up we have some boiler plate code that gets integrated with every services that gets built at openi which has all this kafka related or prism configuration related settings already in built and they don't have to worry about like which topic is created where which clusters should be uh uh used to onboard those different topics and behind the scenes we take care of all of that.
Third is improved scalability um which is fewer connections to brokers. Now this prism proxy services would still be deployed in multiple regions across the globe but the number of parts that this would be running is within like less than 100 typically per region uh because it's not doing a lot of work. It's just interacting with Kafka and uh that way we are reducing the number of connections from millions to thousands uh if not hundreds.
And this is how the multicluster routing architecture looks like with the prism architecture. So let's say we create two different Kafka clusters. Uh a single topic will be onboarded into both of these different Kafka clusters and uh the prism will be responsible for load balancing all the requests that are incoming from the producer services onto this Kafka clusters uh in such a way that it it it will automatically detect the single point of failure as well that I'll talk about next. uh so let's say the kafa cluster one publish has issues uh then prism will automatically detect that and it will retry onto the next available cluster until it makes makes a successful response and that we have some retry logic tuned in within the prism service that will help us uh make sure that most of the requests are reliably served and even if there is a single cluster failure of Kafka or the region where the kafka cluster is deployed is failed then we have circuit breaker logic built within the prism service that will automatically detect that and route the traffic to the next available clusters as well.
And because of this uh we a bunch of issues that we had before regarding like cluster failures or region failures they were no longer existing and uh even the connection count went down from like millions to thousands as I said before. Now let's talk about the euphor which is the managed Kafka consumer platform that we have at openi. Uh so you folder in general as I said it's developed at Uber and we open source and we evaluated and migrated most of our use cases to it.
10:20 uForwarder architecture
Uh primarily it's a gRPC pushbased mechanism system. Uh so rather than uh the consumer services directly polling Kafka to get the messages. Uh this splits the script in such a way that the euphor components will push the messages to the consumer services and the consumer services are not directly interacting with Kafka at all. Uh the second is simplified integration. as before to consumer services don't need to worry about Kafka internal secrets, firewall settings, all of that is integrated within each service that gets built at OpenAI uh with the U forwarder platform.
Uh the third is powerful features. Uh so we we support uh like U forwarder supports retry and DLQ dead letter Q topic capabilities as a default mechanism. Uh so many of times like the consumer service uh might see bad messages or corrupted messages or some business business logic issues because of which the messages might be required to go into the dead letter queue. Uh so we support that as a first class feature as part of this consumer platform and it also has tiered retry architecture.
So for example, if the consumer service has a downstream which is not ready yet, they can say retribal error message and then we'll automatically put the messages into the retry queue which will be replayed at a later point in time. Um moreover as I said like OpenAI has a lot of Python dependency which has G limitations and uh back in the days whenever like the typical consumer service would directly interact with Kafka.
A typical way for them to scale up their throughput was to increase the number of partitions which was not scalable as we go like beyond the number of use cases that we wanted to onboard. Uh so with this u orderer architecture since it's built in Java it will uh make sure that the parallelis parallelism is existing per partition. It will route multiple messages per partition uh concurrently to the consumer services. So at a high level this is how it is on the left hand side you will see the Kafka cluster on the right hand side is the consumer service.
In between the U forwarder components exist. I'll dive deeper into U forwarder components expand on that later. But at high level what happens is U forwarder will read the messages from the Kafka topic. It will forward the messages over GRBC to the consumer services. The consumer service will respond with a success or failure message based on its business logic processing and uh if it's a success it will commit the offsets in the upper path and if it's not successful it will reenue the failed messages based on the retriable versus non-retriable error code that the consumer service will send and if it's a retribable error code it will route the message into the retry topic uh which is configured to retry at a later point in time which could be in seconds, minutes or days and in the DLQ topic if it lands there it's not replayed at all until the consumer service owners explicitly tell us to replay that data and it's all self-service mechanism so they can onboard uh themselves onto the system.
Uh now let let's talk about the cafer clusters aspect as well. Uh so u forwarder in this case if you have multiple kafa clusters since the topic is deployed to all the kafa clusters forwarder is also doing consumer job or consumption job for each of these different kafa cluster for each of the topic that is onboarded onto the system. So even if one Kafka cluster has problems uh then the other Kafka cluster since the prism will be routing the traffic to the other Kafka cluster will still be able to maintain high availability and whenever the first cluster Kafka cluster that was down comes back up it will be able to uh serve the data that was existing before.
Now let's dive deeper into the UOER components. So I've expanded on the U forwarder components. All the blue components are uh uh indicating the u orderer components. Um at the top you will see there are controller and zookeeper. Uh so controller is like the brain of the system uh which is orchestrating all the consumption job amongst all the workers of the system and the workers are actually doing the majority of the work here. uh talking to the Kafka cluster and also routing the messages to the Kafka consumer services and um all of that information regarding the workers and the job information is stored in Zookeeper uh so that the controller can use that state store management layer to orchestrate all the work amongst the workers and we get three uh important properties as part of that.
The first is at least once processing. So which basically means that uh the kafa consumer service has to be item potent. Uh it will always ensure that it will never lose the messages but if you want exactly one guarantees this is not the system for that. The second is out of order message delivery or no ordering guarantees. Again if you need ordering guarantees this is not the system for that. The reason being within a single partition we are parallely sending the messages to the consumer service.
So the messages ordering can be jumbled up and this is this is a good trade-off uh for us to make at OpenAI because most of our services don't care about ordering much. So this works pretty well for us. The third is the flow control or the rate limiting mechanism. Uh which basically is used by the kafa consumer service owners to indicate this is the max number of messages that my service can process and this is an indicator for us to say this is a max so that we are not overwhelming the kafka consumer service that gets deployed in the U orderer architecture.
Okay. Now let's talk about how we converted euphor to kafka folder. Uh so all all in all we adapted uh pretty much all the components of your fodder. Uh we deploy them in Azure Kubernetes and uh the controller and the zookeeper are deployed as stateful set components in Kubernetes and the worker is deployed as a deployment. As you can see there is a gray component that is added newly. Uh it's the engineext ingress or the envy gateway uh which is used as a routing mechanism and the load balancing layer between the workers and the Kafka consumer services.
15:48 Kafka forwarder implementation
Um and it allows us to have DLS communication automatically as part of that and it also load balances across all the pods of the cuffer consumer services um such that if they horizontally scale the number of pods they'll be able to get higher throughput for the services. So yeah it's pretty simple to do that as part of uh the Kubernetes ecosystem. Other than that we also added a job watcher component. Uh so talking about the onboarding flow like if if I'm a Kafka consumer service owner if I want to get onboarded onto the Kafka for a request system how do I do that?
Uh you write just a simple YAML YAML file of 78 lines and that gets merged into the git or some source control mechanism. From there the job watcher component will be watching for any changes. It will always compare the git checked code versus the Kafka forer uh uh running consumer jobs. it will see for any discrepancies between them and it will do crowd operations to be able to reconcile everything that um that should be existing in the Kafka cluster.
So if you delete a job it will detect that. If you create a job it will detect that. If you update the rate limiting parameters it will detect that and automatically update that in uh runtime. On top of that, for the DLQ data, we wanted to expose an ability so that the product team owners can dig into what's wrong with the DQ data and be able to analyze the data and make corrections. Even if there's a business logic bug, they'll be able to see that data uh and uh fix the business logic issues.
And moreover, we did that as part of uh data lake. Uh so the DLK traffic will be pulled via orchestrated by airflow and it will be put all the data will be put into data lake like delta table or iceberg periodically every 6 hours or so and once that lands into there the batch uh data job owners or the product team owners can analyze the data to figure out the issues with it. Uh in terms of scale, we are operating three Kafka fodder clusters uh each with 2,000 plus parts.
18:15 Consumer onboarding flow
Uh and currently we have like around four Kafka clusters that they subscribe to. We serve around 8 million gRPC requests per second to all the consumer service that we have deployed at OpenAI and we typically serve within less than 2 seconds uh consume to pub sorry publish to consume latency and uh this includes the latency measured at the workers. This does not include the latency measured at the consumer service level because each consumer service can have different business logic and that can have latency of seconds or up to minutes and that is what we support as part of this architecture.
Now let's see how the new cuffer consumption onboarding process looks like. So as I said like uh the product team owners just write a YAML script or YAML file uh and this is how it looks like. At the top we have the topic name which is streaming. events events and uh down below there are some other configuration. The key ones are client service name which is the consumer service name uh which uni uniquely identifies that specific Kafka consumer service so that the workers know uh which service to route the traffic to.
Uh the next one is the forwarding type. We support two different forwarding types. The first is load balanced and the second one is replicated. I'll dive deeper into load balance uh in a bit but the high level idea is uh if a consumer service is deployed in three different regions let's say uh then the load balanced forwarding type will let the workers know that the traffic should be distributed across all these different regions uh so 33% split in an ideal case and for the replicated types it's um if if if it's the same scenario then it will route 100% of traffic to each of these different regions so those are two of the typical use cases that we have ati um that we serve as part of this.
The next is unique identifier which is very closely mapped to the cafer consumer group ID concept but we use that internally to determine the retry topic name and the DLQ topic name automatically and we create all those topics as part of physical config generation. This is the logical config generation that is abstracted u very highly for our product teams. The next is the flow control mechanism that I talked about to re not overwhelm the produ uh consumer services.
Uh now before we dive into how the physical config generation works, let's look at how the region uh deployment strategy looks like at openi. Uh so we have Kafka clusters on the left as before. On the right we have the Kafka consumer service owners. Uh in the middle we have Kafka for clusters. So as you can see all of these services are deployed uh in multiple regions for for region failure tolerance and all of these regions can be completely disjoint with each other.
20:33 Multi-region routing
They need not be colloccated within the same region and we want to be able to support region failure across on any of these different components that we have. Uh so as part of physical config generation what we do is we create a separate consumer job for each of the main topic and the retry topics as well. Um apart from that for each of the Kafka cluster we create a separate job. So if you look at this we have three different Kafka clusters.
We'll create three different jobs for each of this Kafka cluster. And within each Kafka cluster we create the main topic job and the retry topic job. And for each job we automatically select the least loaded Kafka order cluster. So at any point in time like we have multiple Kafka order clusters. uh whenever the product team checks in the code, we'll automatically determine uh uh like which one is the least loaded based on some static configurations at the moment uh to select the least loaded Kafka order cluster and that job will be onloaded into that and for each job we automatically derive the DQ and the retry topic using the unique identifier that I talked about.
So all in all in this particular example if you look at it uh we create six different jobs for a single consumer service because we have three caf clusters and we have two uh main topic and the retry topic. Now let's talk about the load balanced uh setup in this architecture. So as I said like there are two different forwarding types. The first one is load balanced. uh in this case again using the same example we have the kafa consumer service deployed in three different regions in an ideal case all of that will have equal number of parts deployed in them so we have another component called service watcher which is continuously monitoring for pods availability uh within all of these different consumer services deployed in all of these different regions and it will update that information in SQL DB which is also multi-reion available uh down below you'll see there's a service config updator uh module which is continuously monitoring for any changes in this NoSQL DB and then it will update that information in the config map that is used by Kafka folder.
So the Kafka faer whenever it detects any changes it will dynamically load that routing information and it will route the uh route all the traffic to these different consumer services accordingly. Again in an ideal case 33% split will be happening roughly. Uh now let's say the region one uh consumer service pods has capacity issues or some issues and it goes down from four pods to two pods while the other regions are still existing at four parts and it will route all the information the Kafka for dynamically within a matter of minute and uh it will route 20% of traffic automatically to region one while the 40% traffic flows through other regions of these services.
Uh let's talk about what are some of the other miscellaneous improvements and setups that we did to convert you forwarder to Kafka fodder. The first one is consumer service side batching capability. Uh so you orderer traditionally only does a single message request and response uh between the consumer services. But many of our uh consumer service wanted batching pro batch processing capability as part of their asynchronous jobs.
23:44 Miscellaneous system improvements
Uh so what we did here is instead of building the layer within the U forwarder we did it on the consumer service library side that we own and so the way it works is we'll create um we'll wait for a batch of messages to be received from you forwarder to the consumer service and then we'll dispatch that from our library to the consumer services so that they can deal with the batch of messages and we'll not act back to the workers with success or failure until the entire batch is processed. uh that we sim simulate uh the batching capability on for the consumer services.
The next is uh we did some hard bit improvements and added jitter capabilities within the U4ER itself. The reason being as we scaled beyond 2,000 parts, we saw some reliability issues um with like whenever we would restart the workers, if a large number of workers are restarted, they would create create a large number of worker registration requests with the controller which would all go to Zookeeper to store the information and that could lead to Zookeeper right overwhelming uh issues and because of that we added some heartbeat improvements added jitter so that not all workers are registering themselves to the Zookeeper at the same time.
We added some uh self-service DLQ main topic backfill capabilities. Uh so let's say there is a consumer service business object bug and uh because of which all the successful messages are being marked as retribable or non-retriable and all of that messages lands into DQ data or DQ topic and now they want to replay that data. So we added that sales capability so that they can self-service onboard themselves onto the system and replay the DLQ data and the same thing happened for main topic as well.
So we also add it for the main topic. Now Uber uses M3 telly for observability but we use data dog for the most part. So we added data dog adapter so that we can get observability and logging and everything along those lines. Um, OpenAI also has specific communication mechanisms that we uh use secret providers and different form of secrets uh that we need to plum you forwarder uh with so that they can efficiently and correctly connect with all the different Kafka clusters that we have.
So we added that capability within it as well. Let's talk about the Kafka to data lake system for a bit. Uh this is owned primarily by my sister team but we have integrated the DLQ topic inte routing through this entire ecosystem as well as before the ML configuration is checked into git uh and all of that is abstracted as part of the physical config generation. So all the consumer services that gets onboarded onto Kofka orderer they'll get this capability for free.
26:18 Kafka to data lake system
So we'll create the DLQ topic uh to data lakeink uh job as part of uh every job job that gets onboarded into the system and once it's checked into git the airflow orchestrator is monitoring for any changes within that and it will trigger spark structure streaming apps or spark structure streaming queries that will be responsible for interacting with the kafa cluster and putting all of that dq topic data after d serializ d serialization into uh the datal link tables.
Uh now we have different kafa clusters. So we'll create like a different spark structure job for each of these different cafer cluster. Again as before if one cafera cluster has problem the streaming job will continue working for the other cafera clusters and once the cafa cluster is back up it will continue processing without any problems. And once it lands into the data lake tables uh the users or the products or the data scientists they can do some analysis on top of it to figure out what's wrong and fix the issues.
And this is how the DLQ physical config looks like. At the top we define uh the uh table automatically using the owner name the environment and the DQ topic name which is derived from the unique identifier that I talked about before. We have the Kafka related parameters which is the topic name what schemite corresponds to. Uh down below we have two different configurations. One is for the reader group to ensure the right reader right groups have the ability to read the data. uh and typically this is managed through Azure AD um and at the end we have uh the spark cluster or the data bricks cluster that you can configure for running the spark jobs.
28:22 Future work and conclusion
Now talking about the future work uh the first thing is we as I said like before we do consumer service library side batching of gRPC request which is not very optimal and efficient because it leads to a lot of overutilization of resources which is not necessary. So the ideal way to do it is to do batching on the Kafka orderer side or the EU order side so that we can utilize the ingress resources and the Kafka consumer service resources much efficiently.
So we plan to do that within the next few months. The next is we want to support in-house metadata management layer uh as a potential replacement for Zookeeper uh just because like most of our metadata will be managed through a unified ecosystem at OpenAI. So we want to be able to move and push towards that. The next is we plan to build a dedicated Kafka fer control plane. So right now everything is YAML based which is good but it's still tedious because you need to check in check in the code into git which can sometimes take 30 minutes 1 hour you don't know based on all the different CI checks unit test that runs for every jobs.
Uh so we want to be able to support that through a dedicated UI plus a dedicated control plane. uh and the control plane will have an additional benefit where we can um dynamically determine the least loaded Kafka cluster. Uh we can monitor each different each of these different Kafka for the cluster how much QPS they are serving and based on that we can decide rather than the static configurations or the uh that the consumer service gives us based on the rate limiting parameters.
Apart from that we can also automatically determine uh detect uh the failed Kafka foder cluster and route the traffic appropriately and dynamically to the other Kafka foder clusters and uh on top of that we can do automatic rebalancing uh between these Kafka fer clusters as well. The next is async taskuler. Um so if orderer has an ability to do delayed message delivery within a single topic but it cannot do delayed message delivery per message. uh but some of our use cases might require that.
So we plan to add a database on top of this euoder architecture so that we can orchestrate the delayed delivery per message rather than per topic. The next is priority scheduling of messages across topics. Um some of our use cases uh the consumer service logic uh business logic is required to integrate with GPU for model inferencing. Uh so for those kind of use cases they want to utilize uh the GPU as well. Um so and if they have like multiple topics they want to consume from this feature will help them prioritize high priority topics processing before the low priority topics so they can so that they can optimally utilize the downstream GPU resources.
So this is another feature request that we'll probably work on in the next few months. Yeah, that's it. Thank you all for listening. If you have any questions, let me know.
Learn more about the Databricks Data and AI platform.
The information provided herein is for general informational purposes only and may not reflect the most current product capabilities or configurations.