Sponsored by: NTT DATA | From Months to Days: Using AI DevKit to Democratize Data Development

Summary

NTT DATA extended the Databricks AI Dev Kit to transform natural language inputs—such as requirements documents and meeting transcripts—into production-ready Databricks artifacts including pipelines, Unity Catalog objects, AI/BI Dashboards, and Model Serving configurations.
The core challenge solved is that data correctness depends on lineage, schema, business definitions, and data freshness—context that cannot simply be passed to an LLM—so NTT DATA built a business ontology layer to bridge natural language intent and governed data outputs.
Projects that previously required months of specialized engineering effort can now be delivered in days, reducing dependency on scarce technical talent and enabling enterprise data platforms to scale without increasing headcount.

Chapters

FAQs

What is the Databricks AI Dev Kit and how did NTT DATA extend it?

The Databricks AI Dev Kit is a framework for AI-assisted development on the Databricks Data and AI platform. NTT DATA extended it to accept natural language inputs like requirements documents and meeting transcripts and automatically generate production-ready Databricks artifacts including pipelines, Unity Catalog objects, AI/BI Dashboards, and Model Serving configurations.

Why is AI-assisted development harder for data workloads than for application code?

Unlike application code—where correctness is self-contained and visible when a UI fails to render—data correctness depends on lineage, schema, business definitions, and data freshness. This means the full context cannot simply be passed to an LLM within a context window, requiring a separate business ontology layer to bridge intent and output.

How does NTT DATA's approach reduce engineering overhead?

By converting natural language inputs into governed, production-ready artifacts, NTT DATA's approach eliminates the repetitive pipeline building, mapping, and delivery work that previously required specialized engineers. This video shows how projects that took months can now ship in days, enabling data platforms to scale without scaling headcount.

What does NTT DATA's global Databricks practice look like?

NTT DATA operates with 17,000 data practitioners globally, including more than 6,000 data scientists and over 500 certified data engineers. The company runs Databricks services across more than nine countries and operates more than seven Databricks innovation centers worldwide.

Full transcript

[00:09] Let's get started. I want to start by introducing myself. My name is Anvesh. I've been with NTT for 4 years now. I'm a director of data strategy. For those of you that doesn't know NTT, we are a global company with 17,000 data
[00:23] practitioners across the globe. We have 6,000 plus data scientists, 500 plus certified data engineers, and 25 plus champions. We have nine plus countries where we are
[00:35] doing Databricks uh services right now, and we have more than seven innovation centers for Databricks. That's a little bit about NTT and myself.
[00:47] So, the aim of my talk today is going to be helping you understand how we can democratize data development and also agentify some of our data workloads. At
[01:01] the end of the session, I want you to leave with an impression that the months to days is not just a marketing line, but is going to be soon
[01:13] a reality. All right. Let's take a look at how things have been evolving with technology over the past few years, right? In the 1950s, we were
[01:26] all writing machine level code, binary code, ones and zeros of how you turn on a light or turn off a light. And in the 1960s and 70s, we were
[01:39] doing mnemonics or uh machine level code, uh assembly level programming, if you will. And then in the 70s, we got to see C++, Java, where you were able to
[01:51] define procedurally what needs to happen, and the compiler converted that to machine code. Later on came the frameworks like SQL and Python where you could declaratively say what you're
[02:03] trying to do and the intent gets converted into code. Fast forward to today, for the past couple of years, we're in a state where English is the most prevalent
[02:15] programming language, right? Most of the software today is being built using agents and using English as input. But the same didn't happen for data
[02:28] workloads. If you look at why that's the case, you would end up with three critical things, right? On the app side or the front-end development
[02:42] side, your correctness is self-contained. So, whenever you build a UI or a front-end interface or software code, it's very evident when
[02:54] it doesn't render properly or if a button is not working the way it should. The correctness is contained within the scope of the application and the context is something that you could pass to an
[03:07] LLM within the context window. But that's not the case with the data workloads. The correctness depends on lineage, correctness depends on the schema, the business definition, and the
[03:20] freshness of data, right? So, you can't do the same thing with data. The other thing was LLMs had vast input of GitHub repositories
[03:32] with Java code, C# code, and all of the front-end code that it trained on. So, it's really good at emulating and validating that code from all of the past examples.
[03:44] That's not the same case with data and also because the correctness is dependent on context with data workloads, your context doesn't live within the boundaries of your code when it comes to
[03:56] data workloads. It is defined by the business. Revenue could mean one thing for one business and a different thing altogether for another business. How do you give LLMs the same input and the
[04:10] same context to be successful with building these data workloads? That's a gap that existed. Um And whenever failures happen on the front-end side, it's fairly evident. You
[04:25] can It's loud and it's clear. You can look at it right away that something is not working the way it should. That's not the case with your data workloads. If you got something wrong, you might
[04:37] not find out about it until weeks, months, or sometimes years later where you have a yearly report that was run which has the wrong values or you realize that data was not getting
[04:50] updated for a particular table that it was dependent on. So, with respect to the data workloads, again, the failures are silent and wide because failure on a given table can impact
[05:04] downstream tables and the dashboards and models that are dependent on it. So, that was the primary reason why data lagged with respect to agentic adoption
[05:17] of SDLC. So, let's see how we can bridge that gap. Most of you have attended the keynotes and you've heard the announcement from Databricks about AI
[05:29] Gateway, Genie Ontology, bringing in the context to where you already have your data workloads running, right? So, what it's going to mean is now you have the
[05:41] same place where you're building your data workloads, a clearly defined business ontology of what revenue means, what each KPI means. You have your semantic layer built out there. You have
[05:54] your lineage available within the same ecosystem. You have governance and access everything defined in the same ecosystem, which is going to help agents interact and build this code more
[06:06] efficiently. Databricks also has open source AI dev kit, which has skills, agents built into it, which
[06:18] help agents understand how to build Databricks artifacts, like pipelines, declarative pipelines, models, dashboards, and things of that nature. So, that is really helping bridge the
[06:32] gap that existed previously with automating data workloads. So, what we did at NTT is we took some of these fundamental elements, the
[06:44] skills that are available through the AI dev kit, the MCP tools that exist today, uh and the agents that can interact with these, right? So, for our use case, we
[06:56] primarily used Claude Code and Codex, but you could essentially use any large language model, like Gemini or Cursor or any of the popular ones to do
[07:09] the same thing. Uh the skills are natively available through the dev kit. The skills are going to teach the large language model the guidelines for how to build a
[07:22] declarative pipeline or a model or an artifact within the Databricks ecosystem. The tools are going to enable the created artifacts to be deployed to the Databricks ecosystem. Agents are
[07:35] going to take the skills and tools together to get them deployed, right? So, we need an agentic harness uh to be able to orchestrate all of this together. And
[07:47] you've heard from Databricks about Omnigent. We were thinking along the same lines. Omnigent announcement came over the weekend, but we were understanding that
[07:59] the need for a tool like Omnigent and what we built at NTT is an extension on top of the AI Dev Kit, which captures your intent of what you're trying to do
[08:11] with your data workloads from meeting transcripts, technical specs, and is able to understand and decompose that into features, right? So, if you have a
[08:25] predictive use case for modeling your churn, it can understand that it has to ingest data from source A, source B, load it into a table, create those
[08:37] tables, then create the pipelines for that, create a AI BI dashboard to visualize the forecasting. So, it's able to capture the intent from a meeting transcript or a technical spec and break
[08:49] that down into different features to be able to accomplish your end goal. And what we've also baked into the solution is the governance aspect of it.
[09:02] You don't want agents to run rogue. You've heard about agents going in and deleting databases. So, it inherently works with AI Gateway and the permissions that you have within AI
[09:14] Gateway to be able to ensure that you're working within the guardrails of what you're allowed to do. We've also augmented our solution with our industry templates and our assets
[09:27] that we have at NTT DATA to make it more comprehensive. So, what that means at the end of the day is you would be able to define your
[09:39] intent in natural language in simple terms for the machine to then correlate with your ontology, your business context, and then be able to take that and
[09:52] execute it into an action plan. So, I'm going to step away here for a bit because I have a narrated demo that is going to showcase the tool for you.
[10:05] >> This is the NTT data databricks devkit, an AI harness built on the databricks AI devkit. It supports the full delivery team around a data product managing requirements, tasks, code artifacts,
[10:18] execution, and validation results. We sign in as a platform engineer. Every action from here is tied to a real identity and fully audited. This is the home surface. Projects,
[10:30] active features, approvals, and running tasks all in one place. Let's build a demand forecasting lakehouse starting from nothing but a conversation. The new project modal captures the basic
[10:43] setup, title, summary, and workflow mode. Guided mode pauses at the major review gates, while fast-forward mode fast-tracks execution using configured approvals. We select guided for this
[10:56] demo. We begin with the raw material every data team already has, a meeting transcript. The planning team talked through what they need, five source feeds landing in a unity catalog volume,
[11:08] a medallion flow from bronze through silver to gold, a forecast model, a dashboard orchestration, and packaging. No tickets, no specs written by hand, just the conversation captured as the
[11:21] source of truth. We click create project and a background process starts breaking down the requirements. The kit applies spectrum and development orchestrated by the AI harness. AI interprets the workshop,
[11:34] deterministic coding forces schema contracts, routing, persistence, validation, and audit. The intake completes in a few minutes, and the kit has decomposed the conversation into five governed features spanning the full
[11:47] medallion stack, bronze through gold, plus forecast scoring, a dashboard, and bundle delivery. Each feature arrives with its own requirements baseline, type tasks, and predecessor relationships,
[11:59] ready for review. Let's review the AI-generated constitution, the project's governing blueprint. It records what the system is allowed to build, how tables are named, which feature owns each table, and how
[12:12] data flows from source through bronze, silver, gold, model, and dashboard layers. This gives every later AI call a shared memory instead of forcing it to re-reason the domain. The audit
[12:24] scorecard checks schema integrity, execution contracts, and lineage. Ensuring the project is execution-ready before a single line of code is generated, and a human stays in the loop to approve before anything runs.
[12:39] Now, we execute. Feature one creates the Unity Catalog schema and bronze table structures, then ingests five source CSV files from the governed volume landing path into bronze Delta tables. The tasks
[12:51] tab shows two sequence tasks, DDL first, and then ingestion. The predecessor relationship is enforced. Ingestion runs only after DDL has completed. The agent works against real Unity Catalog
[13:04] context. It can see the schema, the volume, the existing tables, and lineage, so it builds against the truth instead of guessing. It reasons over the task content, project constitution,
[13:16] feature context, and schema-bound scaffold to create the required code artifacts. The kit then deterministically resolves the right skills, fills templates, routes execution through Databricks APIs to the
[13:28] correct service, pushes the code, runs the remote action, executes validation gates, and records the run manifest. Every artifact you see, the DDL SQL, the quality gates, the pipeline source, is
[13:41] generated, written to the project bundle, deployed to the Databricks workspace, and validated against the platform. The validation queries confirm that the schema and tables exist, and the columns match the contract. The task
[13:54] succeeds only after the Databricks action completes and the quality gates pass. Local assets created, remote assets deployed, executed, and validated. The ingestion task creates
[14:07] the bronze pipeline using Spark declarative pipelines, reading the five CSV folders, adding file lineage metadata and ingestion timestamps, and applying expectations for required keys.
[14:19] The backend materializes a serverless LakeFlow pipeline, starts the update, and reconciles the resulting bronze tables as validated Unity Catalog assets. The same governed loop now drives
[14:31] through the remaining features. Feature two standardizes bronze data into conformed silver dimensions and facts. Feature three creates curated gold outputs, demand history, forecast versus
[14:43] actual, and forecast accuracy, materialized as Spark declarative pipelines. Feature four adds SQL forecast scoring with a linear trend model and validation queries. Feature
[14:55] five packages the consumable and operational side. An AI/BI LakeView dashboard depicting the weekly forecasted and predicted demand and calculated KPIs, a master orchestration job tying the full sequence together,
[15:09] and the bundle ready project configuration. Each task in the job waits for its predecessor, so a failure can be rerun from that exact stage while completed stages remain intact. This is the loop that matters. The agent doesn't
[15:22] just write code, it ships it and confirms it ran. And the whole time, the work is governed. Naming conventions, access, and lineage are enforced as the artifacts are generated, not bolted on
[15:34] afterward. In the Databricks workspace, every pipeline, catalog object, job, and dashboard is deployed as a real asset with run history. The Databricks review tab closes the loop. It shows what was
[15:47] generated, where it was deployed, and how to open it in Databricks. This is not throwaway code. It's packaged as a version Databricks asset bundle, ready to promote across environments.
[15:59] Everything is accountable. The audit trail captures every step. What prompt was assembled, which skills were loaded, which action plan was used, what model ran, what artifacts were produced, what
[16:11] tokens were consumed, and which validation gates passed or failed. Intake, requirements, approvals, execution. Each tied to an operator and a timestamp. A human approves, the
[16:23] system records. Speed without losing control. Skills are the platform knowledge layer. First-class managed and versioned assets that teach the agent how to use Databricks capabilities correctly. Over
[16:36] 20 skills selected by deterministic task type mapping, so the model receives the right knowledge package for each task. Prompt management exposes the managed prompt fragments. Teams can review or
[16:49] adjust the standing instructions that shape execution. The code assembles the prompt, but the editable fragments keep the operating model transparent. Workspace profiles define the Databricks
[17:01] host, catalog, schema, SQL warehouse, and target paths. This is how the kit avoids AI inventing workspace details. Configuration owns the facts, and execution routes to the correct
[17:13] services. Here's where we land. AI balanced with code determinism. A meeting transcript became an execution ready portfolio, pipelines, catalog objects, a dashboard,
[17:26] and a model governed end-to-end in a fraction of the time it used to take, from months to days.
[17:38] >> Right. As you saw, we've been able to take a meeting transcript and turn that into artifacts on the Databricks ecosystem, right? This would have taken developers weeks, if not months, to do that, but with the
[17:52] tool that we have built, we've been able to reduce what typically takes months into days, and this would free up your developers' time to focus on what is important and not trivial boilerplate
[18:06] code that they have to focus on. Think about being in a meeting with a project manager or a business analyst or a client stakeholder. By the time you go back to your desk, you have a prototype
[18:18] of the code that you require for the project ready for the developer to then go review and act on top of it. That's the future that we're living in. That's the future that we're envisioning. So,
[18:30] you can have the same team being able to take the same team, but be able to do more with the same team and the same specialty that you already have within your organization. The same same skills,
[18:43] same team, but better outcomes for the business. That's our end goal. So, this slide is not a recap of what we have discussed today, but this is more
[18:56] of a diagnostic for your organizations, right? For this to be successful, you need to have your ontology defined in a single place. So, if you don't have
[19:08] your schemas, semantics, lineage, access, and everything in a governed place, you're not going to be successful with agent defying your workloads. And
[19:21] I want you to critically think about what your team is doing manually today and where is where lies an opportunity for automation. And the third thing is governance. If
[19:35] if you don't design your solutions with a governance first approach in this agent equal, you're bound to fail. Governance shouldn't be an afterthought. It needs to be baked into your delivery.
[19:49] With that, I'm going to conclude my session. You can come find me at booth 620 for additional questions. I know it's a short 20-minute session, but appreciate you all being here this morning. Uh happy to talk to you about
[20:02] any questions.

Learn more about the Databricks Data and AI platform.

The information provided herein is for general informational purposes only and may not reflect the most current product capabilities or configurations.