Build Fresh Data View for Agents

Summary

CocoIndex is a live context engine for AI agents that maintains an always-fresh, incrementally synchronized view over large and rapidly changing enterprise data sources — including 100GB+ monorepos where approximately 1% of files change daily — by reprocessing only what actually changed rather than reindexing the entire corpus.
The engine supports declarative transformations written in Python where users define the target state as a function of the source, and CocoIndex handles incremental sync, pipeline lineage, version tracking, and processing at scale automatically — producing relational, vector, or graph database outputs for agent consumption.
Beyond semantic search, CocoIndex can build multi-level code structure views for navigability, extract cross-file dependency maps, construct knowledge graphs from entities and relationships across documents, and provide coding agents with the global context they need to reason about a large codebase without fitting the entire thing into a context window.

Build Fresh Data View for Agents

When you have a large enterprise monorepo or corpus with 1% changing every day, how do you build a data pipeline that effectively surfaces fresh insights to your coding agents continuously? Agents run ~50x faster than humans and create massive unstructured data. You cannot fit the entire thing into the agent; it requires building a view over a large unstructured corpus. CocoIndex is a context engine for agents: a new computing paradigm over evolving unstructured assets and multi-stage processing. You declare the state of your target as a function of the source; the engine keeps it in sync with minimal reprocessing, across time horizons, at scale. Demo: building context for coding agents on 100GB+ monorepos, ~1% changing daily.
Talk By: Linghua Jin, Cofounder and CEO, CocoIndex ;

Chapters

FAQs

What is CocoIndex and what problem does it solve for AI coding agents?

CocoIndex is a context engine for AI agents that solves the problem of keeping agent context fresh when enterprise data — code, Slack messages, logs, PRs, design documents — changes constantly and cannot fit into a context window. It works like a materialized view system for unstructured data: users declare the target representation they want as a function of the source, and CocoIndex keeps that representation synchronized with minimal reprocessing as source files change. This video demonstrates this on a 100GB+ enterprise monorepo where approximately 1% of files change daily, showing how agents can access accurate, up-to-date context without full reindexes.

How does incremental indexing work in CocoIndex?

CocoIndex tracks which source files have changed since the last indexing run and reprocesses only those files and any dependent views that need to be updated, rather than rerunning the entire pipeline from scratch on every change. The system maintains pipeline lineage and version tracking so it knows exactly which downstream artifacts depend on each source file and can propagate updates efficiently. This incremental approach is what makes it practical to keep a large enterprise codebase indexed for agents in a production environment where files change continuously throughout the day.

What kinds of views can CocoIndex generate over a codebase for agents?

CocoIndex can generate several types of views for agent consumption, as described in this video: structured extractions from unstructured documents, hierarchical summaries of code structure for navigability, semantic search indexes that allow agents to search by meaning rather than just text matching, and knowledge graphs built from entity and relationship extraction across documents. These different view types serve different agent reasoning needs — navigating large codebases, finding relevant code by concept, understanding cross-file dependencies, and discovering correlations across the enterprise data corpus. Users declare which views they need in Python and CocoIndex maintains all of them in sync.

How is CocoIndex different from a standard vector database or RAG pipeline?

A standard RAG pipeline typically runs a full indexing job when the data changes and does not automatically track which parts of the index need updating when individual source files are modified. CocoIndex adds a control plane that handles incremental processing, pipeline lineage, and version tracking automatically, so the index stays current with the source without manual orchestration. It also supports multiple output target types — relational databases, vector databases, graph databases, and message queues — from a single declarative pipeline, rather than requiring separate pipelines for each downstream store an agent might query.

Full transcript

[00:08] Good afternoon everybody. How is everyone? I just want to make sure you can all hear me. You hear me? Well, we're Coco Index and uh we build fresh data view for agents. >> So the problem we try to solve is most
[00:21] enterprise data is unstructured. your code, your design docs, your meeting notes, they all live outside a database and it provides the most valuable signal. It present how the organization
[00:34] actually syncs and operates. However, these are the kind of data that are hard to index. This data comes in any format. It changes constantly and it's hard for
[00:46] agent for to access. So, how do we help agent access this data efficiently? get started. >> So the way we think about the problem today is agent is part of the data loop.
[01:00] >> Yeah. >> The agent has to access the code, your slack, the logs, the PRs. The agents will reason and it generates code traces, plans and decisions. Everybody
[01:12] here is probably already familiar with materialized view. And in materialized view, you define the query once in circle and the engine keeps the result in sync with the source changes. But in
[01:26] unstructured world, what is a view where sources like codebase, slacks, PRs, right? What kind of view are we generating here? It can be structured
[01:39] extraction of some unstructured data. It can be summarizations from hierarchical code bases. You can build search indexes from your codebase, your PRs, right? And
[01:54] you can also build knowledge graphs. You do the entities and relations triple extraction and you try to find the correlations of all of these documents. So you provide agent something to reason about.
[02:12] Koko index provides a live context engine for AI. The sources are code bases, meeting nodes, databases and images or voices. And the targets are relational database, vector
[02:25] database, graph database, message cues and user writes declarative transformations in Python. And it automatically comes with incremental and
[02:37] builds this materialized view from unstructured data. We work well with many projects here in the ecosystem. And >> Coco index itself provides a control
[02:49] plane that take cares of all the heavy lifting work for you. Pipelog lineage incremental processing version tracking everything you need to take your agents
[03:01] data to production. So how does it actually work? We have talked about this kind of stuff which is a high level. So we can drill in by example. >> Sure. Sure.
[03:14] >> So coco index code is a one clean example about how do I help the agent to understand the global context of a large codebase. enterprises runs on large monor repos
[03:27] that enter hundreds gigabytes >> or even thousands of ripples and you probably have like 5% of the corpus changes daily. If you use clot or code you're probably familiar with a pattern
[03:39] where you keep guessing about the names and you're gripping through. On the left side, you can see example where the cloud tries to grab through, but grab can be slow and it flats the context
[03:52] window and it it misses the code that uses different names and it never sees how the pieces connected. I remember when I was working at Google many years ago, a lot of people from the
[04:04] organization will catch the same block of code and at some point later like the function name is no longer what the function actually does. So how what kind of view can we build to
[04:16] help agent understand this kind of very rich source of large unstructured data data dynamically changing every day.
[04:31] So when we build this Coco index code, we think about using Coco index to build the three kinds of materialized view materialized and incre incremental view for the coding agents that we think they
[04:43] need. The first thing we think the coding agents will need is this high level understanding. How can we help the coding agents understand the intent and the structure without reading and gripping through every file? We think
[04:56] about a hierarchical summarization of your repos, your modules, your files and agent will have the access to the resolution it needs. How can we help agents to navigate through the codebase?
[05:09] The second is we think about search beyond grip. How do we search by meaning and not just the keywords? So this goes back to the scenario I talked about
[05:21] earlier where you have enterprise a lot of people touch the same amount of the code at some point your function name may not be the same thing as your code is actually doing. So how do we build this semantic search about it and the
[05:34] last one is you you have a high level of view of it. You you have a local view of different functions and stuff like that. How do we understand help the agent to understand the interconnections
[05:47] on your corpus? We think about the graphs like call graphs or how do we build course file dependencies or build shared concepts about a module that is being used in many places so your agents
[06:00] don't end up rewriting the same code again. And on top of all of this think about we are trying to build a view z is over a large codebase in real time for the enterprise data. So it has to be
[06:13] able to go to produ to go to production in the end to handle these hundreds gigs of monor ripples that is dynamically changing every day. So the first kind of view we think about
[06:26] building for the coding agents is you cannot feed a codebase with gigabytes in the context. So if we think about Google map, you have San Francisco, you have different
[06:38] blocks, you should be able to zoom in and zoom out whenever you need it. How can we provide agent a navigational view like this? Then we think about a multi-resolution index. You over the
[06:50] ripple, you can over the module, the file and the symbol. And every level will carry a summary at its own level of detail and agent can choose to drill in or drill out whenever it needs it.
[07:04] So here is a very concrete example about what does the actual presentation look like. So CCC stands for Coco index code and in this project we provide the interface called CCC describe where it
[07:17] reads a summary of a codebase at different levels of a detail. For example, this is a codebase of open claw. It's only like one gigs and so when we do on the on the on the left
[07:29] side when we do CCC describe and the root folder you can see the high level you can help the agent so this is really the data for the agent you can give it the high level architecture summary and with like what are some of things it
[07:41] needs to know when it wants to do a navigational drill in the agent decides okay I'm going to take a look at the source folder then it will probably do a CCC describe the source folder and you drill in with similar amount of details
[07:55] where it describe what does a source folder do with different kind of mounted gio similar like similar like when you're zooming and zoom out in a map over a large corpus.
[08:10] So with coco index the way you think about it is a declarative transformation. So the way to do it is you have a code folder and you define different components your summarize files differently and you can control how do you want to roll up and create
[08:22] this kind of mini map row wherever you need it. So next once we have established the global view the next we think about is
[08:34] how do we help the agent to find the code by meaning you don't always know exactly which functions to search for pure grap floods the context window and misses anything that name differently so
[08:47] if we take a look at this example you may have like a source py that implement some like o functions with logging and verify and you may want to split it with like different functions, module or
[08:59] class and you build a semantic index on top of it. In the end, you should be able to have the agent when it tries to when you try to ask a question like where do we check the token, you should be able to have the agent know okay this
[09:13] verify function is where I should take it from there. So in general a codebase is hard to keep well index and coco index builds the
[09:26] best engine for it to begin with you want to build to build a search for the co code right first you want a good segmentation of the codebase you don't want to chunk the code in the middle otherwise you won't be able to have a
[09:39] comprehensive understanding about what does your function do so you need to have a good segmentation of the codebase by the natur semantic boundaries like functions class and Coco index provides aware chunking of on the semantic
[09:52] boundaries and the second is data freshness. Think about you are feeding the data to your coding agent or you're feeding it to coding review agent. If you give them stale context then they're going to build suggestions based on the
[10:04] stale corpus and if it's wrong so you are basically wasting developers time. So Koko index has builtin data freshness. Whenever your source data is changed or your indexing logic is changed, it will automatically fresh
[10:17] your index with minimal recomputation needed at a minimal cost. Okay. So we have like a global view right then K level tells you where does
[10:30] everything sits. We have covered like how do we build like a retrieval view to help you understand the localized view of functions but it doesn't helps to answer like when you
[10:43] change the something what will break. So we think about the graph and the interconnections like what breaks if you touch it. So then on top of this we can think about building call graphs like
[10:56] who calls who a function or method or we think about cross file dependencies if a imports b and b import c and how do we understand this kind of a transitive chance and lastly how do we understand
[11:10] the shared concepts and types that are used in between the modules and once you have this kind of view that you can feed it to the agent the agent will see the blast radius of a change
[11:23] and we'll stop re reinventing the same function again and again. And there's a bonus view we can think about is to build con concept maps. For
[11:35] example, how do you how is your codebase organized around Oflow or indexing pipeline or target sync? The kind of characteristic of concept map is is goofed per area and the core concepts
[11:49] and how they relate. And the second is it is semantic and not structural. Compared with a hierarchical view we talked about earlier. These concepts may span across many files and beyond the
[12:01] directory tree structure. And once you are able to achieve a view like this, you should be able to answer beyond the traditional search. You can actually be able to answer like
[12:13] what does this system actually do beyond where do I find zent chunk or something like that. So in general a codebase key concept spans many locations and KCO index gathers the knowledge around each
[12:25] concept into a kind of wiki for AI which is built for the codebase and it will keep it fresh. So in general building is easy but we're talking about how do we do a production
[12:39] context for AI and the real challenge here is how do you keep all of this cog graph fresh. If you think about it in this graph you see here AD imports B and B imports C right and if you edit the C
[12:54] interface every edge touching C will have to change the every import C and use the change signature transitively is now run. So how can you keep everything fresh and you give that AI to
[13:08] AI and all the time. So then we end up like do you have to reindex everything on every change which is absurd
[13:20] and you'll always be behind. Think about how fast the coding agent are changing your code every day at a large corpus right and then you may end up like doing some of this manual thing which is very easy to break. You need to do change
[13:33] detection, chrome job, CDC, selective recomputee. When you change your codebase, you swap the prompt, recmp compute only what it touches. How do you roll out the changes and reindex
[13:46] across these branches without the downtime? And how do you do the orchestration and lineage and keep your target always in sync? This is where the info attacks leaves.
[14:00] And Koko index ships all of this in the engine. So this side is what you actually write write in your codebase. And the Koko index as a framework itself will take care of the CDC lineage
[14:13] tracking change propagation. How do you maintain and evolve schema automatically and roll outs. So with all of this said, we can relax a little bit and uh see a demo.
[14:33] So this is an example that I tried to ask open call to help me understand how do I create a environment global variable that disallow some functions functionalities and as you can see we uh use CCC search
[14:47] and it didn't show up in the corpus so we end up to write a function that creates this global allow list for open claw
[15:05] And once we have updated the corpus, we go back to the CCC search
[15:17] and we can immediately see the functions were showing up in the index. So your agent will have immediate access to it.
[15:31] And the next thing is now we added this new agent like this new function what is the area that it touched. So we run CCC call graphs
[15:45] it will brings up all the call sites then transitively this will influence and your agent will have a holistic view of where actual components it may break down the line.
[15:58] So now we can do a sideby-side comparison running on the open core corpus. On the left side we will disable the cocoa index search. So as you can see we cannot find the CCC search on the left side and on the right side we have the uh on the other side we have the
[16:12] coco index. So uh yeah this side is showing the um without CCC search and our side has a CCC search
[16:36] and you can see uh CCC was able to return some results find some results already and is able to complete the task with the semantic understandings while the
[16:48] other side without any view built. This is very classical that the cloud will just grip and batch through grapes through your your ripple and tries to guess and find
[17:02] the related code block.
[17:29] I think in general um we find like providing the coding agent with a good view of the um coding based on codebased understanding will helps it per performs the task at a higher quality and and with a with a faster speed.
[17:43] So, and and it's almost done.
[18:00] Cool. That's um all I have for today and um I'll be around if you have any questions. We can like chat about it. You can grab me. It's very easy to find find
[18:13] Thank you.

Learn more about the Databricks Data and AI platform.

The information provided herein is for general informational purposes only and may not reflect the most current product capabilities or configurations.