Own the knowledge, rent the thinking: building organizational memory for AI agents
You can give an AI agent access to your company's wiki, Slack, CRM and project tracker, and it still won't know who owns a decision, which information is current, or what it's allowed to share. That isn't a model problem. It's a memory problem.
In this episode, recorded live at AI DevCon in Brooklyn, Ran Aroussi shares what VarOps learned from a year of building organizational memory for AI agents. The findings were tested against three public benchmarks, and the failures and negative results are published too.
Three architectural decisions that changed everything
- Store claims, not chunks. Distill information into structured knowledge, with evidence, timestamps and provenance attached.
- Make time and absence explicit. Track what changed, keep the history, and teach agents when to say "I don't know."
- Enforce permissions before inference. Models should never receive information that the person asking isn't authorized to see.
The results
- 99.8% evidence retrieval on LongMemEval
- 35 million tokens ingested for $8.24
- Fully local models reaching 84–96% of all-frontier performance
- Local memory paired with rented frontier reasoning reaching 96–99%
- 45 of 45 adversarial isolation probes refused or contained
The bigger lesson
AI agents don't need bigger context windows or endless conversation histories. They need a governed understanding of the organization they work for. The model is a seat we rent. The memory is the asset we own.
Links
- Read the full research paper, Own the Knowledge, Rent the Thinking
- Learn more about VarOps and Resident AI
Transcript
So today I want to talk about something that, we've been working on for the past year or so at, VarOps. it's about how an agent can have access to everything that a company's doing and still be wrong when you ask it stuff. Quickly about me, I started coding and sorts of entrepreneurship thirty-five years ago.
I've been doing open source. I wrote two books on agentic AI, which you can download at our website. And I run Automaze, a company that, provides managed, software factories, and building Resident AI at VarOps . So all this to say that this is something that we've been really working hard on. We have the, the experience and we, we're gonna share some, some interesting stuff about that.
And everything that I'm going to do-- to cover today is linkable at the end of this session in the slide, the last slide. You can download the the materials, the white papers, the, the proofs and everything.
So imagine that you have an agent that's connected to everything that's happening in the organization, right?
So it's connected to your CRM, it's connected to your, OneDrive (it's corporate after all), your Slack, emails, everything. And then you ask it a simple question. You ask "What's the payment terms of our contract with Acme?" All right? So the agent have, these two information that he, he can see.
He can see that the contract draft was talking about, net thirty, and there was an email thread that is also, saying net thirty, and it comes back with an answer. It says, "Okay, that's net thirty. Payment is due within thirty days of invoice." And, that's the answer, and it sounds completely reasonable, except that it's wrong the reason that it's wrong is because the final negotiation, happened on WhatsApp.
The final negotiation happened on WhatsApp. There was a quick chat between the, client and the sales rep. The client asked for sixty days, and the sales rep agreed, and then he casually mentioned it at the office the next day, and it was- wasn't logged anywhere, and so the agent never got to know this information.
So that's what was actually agreed. We agreed on net sixty, and the agent said, it's it's, only net thirty. now, the problem is that it would probably be better off if the agent didn't have access to anything, because then it would've said, "Okay, I don't know. Can you share the contract?
Let me answer that." And by having just access to some of the information, it's, very confidently being wrong, and it's sort of like the Dunning-Kruger effect, but for agents. they don't know what they don't know, and they're very confident about what they do know. and w- with people, hopefully, at some point we'll get some h- humility.
With agents, we're, we're still working that out. So you could of course say, "Okay, just hook up WhatsApp and, we'll have everything," but that's not really what's, what's going on. we can do that obviously, and I do recommend that, organizations will connect pretty much everything that's happening in the org.
but what about a phone call or a conversation or a decision that, was made before we connected any of these tools? So we are never going to be able to capture everything, okay? That, that's, that's, something that we need to know from the start. And also, even if we capture everything, was the thirty-day, was the sixty-day net, was that a proposal? Was that actually agreed on? did the sales director have the authority to even approve that? So all of that, w- what exactly is, is going on? So that's the three questions that we need to make sure that the agent knows. It needs to know what was actually agreed, which evidence are authoritative, and how complete is the information.
Without answering these three, questions, we cannot trust what the agents are saying, and that's very important, especially in organizational environment to-- so we can build that trust with our agents, so we can let them in, on more parts of the organization. So If you take this qu-- like, little example and you multiply it by, you know, hundreds of clients, thousands of employees, it gets messy real quick.
There is a difference between just dumping everything on the agent and hoping that it will-- it, it's able to sort things out. but it's, it's just never gonna happen. We've tried it. we've tried it a lot, different models, different methodologies.
It just doesn't work because the model, not only does it not need to know the company itself, it shouldn't, I think. but even, even if, if we, ignore my personal preferences of whether or not it should or shouldn't, I think that this job belongs in the system, and I think that the company needs to own, the, the...
its DNA, its knowledge, what makes a company a company. 'Cause if you keep dumping everything to a model, at the end of the day, on the next version of that model, without you ever men-- planning for that, the model is going to give away pieces of your company's DNA to, other companies or, or asking.
Maybe not by name, but that's what's going to happen. So we need to make sure that the organizational understanding is something that we own. at least that's what I believe. And we can use external model and frontier models to reason over the data, but we need to make sure that they only have glimpses into, what's happening in the organization.
Now Instead of thinking about, agent memory, because when we started building this, th-this platform, which we call Acropolis, we started building an agent memory, and we-- The, the more we kept on going, the more we reali-realized that's not what we're actually, building. I mean, an agent memory, usually, you know, people think about, my past conversations with the agents, what was said, the, the types of information and tools that I'm giving it, and that's useful.
I mean, I'm not knocking that. But, think about a new employee that's joining the company. They don't arrive with, skills and plugins that you can install onto the employee, right? So they need a way to access and learn about the company and that's not just happening from memory. So what we set up to do, we ended up building something that we call the living organizational model which is...
something that actually understand how a company actually runs because it is connected to everything. It monitors, not in a creepy way, but it monitors conversations over Slack.
it's connected to the CRM, to the ERPs, to the accounting software, to the file system. It knows everything that's happening, so it is able to make the connection between, okay, these two employees are having a chat on, Slack, and it relates to this record in the CRM, which relates to these three invoices in our invoicing software, and it's able to really identify where the bottlenecks are and, not, see the, the About Us version of the company that's very pretty and polished.
So today we are, today the system, works, we-we're plugging it in with MUXI, which is, an AI application server that, we've developed. But our plans is really to make sure that any agent, will be able to connect to it, and we're weeks away from that. So and especially now with, Anthropic just, opened, their, their ability to manage tools per employee group, that would make things a lot easier for everyone.
So the idea is that use whichever agent you want, but make sure that it has, for a lack of better word, a middleware that it uses the company's memories and not your own memories, only when it tries to resolve questions about the company. So if I'm asking, "Hey, what about that client, and what's, what's falling through the cracks?"
It's able to go to that memory store and, reason over it. Now RAG. Let's talk about RAG because i-it's, it's been around and it's great, but it's great for static data, and that's what... why we're seeing it working, really, really well on help centers, support site, documentation websites. It is really good at static information.
But a company is not a set of static documents. A company is something that's changing on a per second level at times and, if not policies, but stuff happening in the company on a per second level and a RAG model just cannot keep up. More than that, the, the problem with RAG is that it's based on chunks.
So if I'm, having all the information loaded with the draft and with the final pro-- proposal and everything, depends on how the human talking to the agent, depends on how they ask the question maybe the similarity is, gets a higher score towards the draft rather to-- towards the, the final decision.
So we need to make sure that the prompt does not skip all these parts. Okay? We... because the LLMs, whenever they see gaps in knowledge, they would happily fill in those gaps with something that sounds probable, and that's not good for, an organizational environment.
Three decisions that we've made when we built this thing, and apparently none of them turned out to be new. So instead of just using RAG and dumping everything on an LLM, just like, oh, I know I, I, I've done that, and we are building based on structured records. So we are building claims with provenance.
We make sure that everything that we store in the company's, memory has history, h- is version-controlled, has the, the full chain of custody to every claim that's happening, and we are making sure that we are enforcing permissions before distilling the information and before accessing the information.
And it may sounds boring, and that's because it is boring. It's stuff that, we've been doing with software for decades now. That's how, the best practices for use of pretty much any database. So when data comes, the first thing that we do is we distill it, and that's a really important part of, how we work.
You might say that, okay, you know, we decided that we don't wanna remember transcripts because we want to kind of mimic how people remember things. So when none of you is going to remember what I said here verbatim, but you may have some takeaways. So you, you'll remember, you know, that a meeting, a week ago you know, you'll remember that there were three options, that, Kate didn't like, one, and that the boss, decided to go with three.
But you will definitely not remember every single word that, was said. And we were trying to mimic this behavior. So instead of just hoarding information, we are distilling the information as it, comes through. And I know what you're thinking. You know, we're taking all this information, we're summarizing it, we're making sure that we are taking the, key points orthe key takeaway from that.
it will, by design, supposed to be very lossy because we're not storing everything, right? It turns out that, it's not. we went to check exactly that. So when we distill information, what we do is, we look at the conversation, we look at the terms, we look at the email threads, we look at what in WhatsApp, and then we build a claim.
And that claim is something that the organization can trace back on. any agent answer can trace back onto that claim, and that claim can trace back to the evidence of what made us believe in that. Now, as I mentioned before, compared to RAG when it's working, on chunks of text and chunks of data again, based on how I'm gonna ask a question, RAG will either point me to the draft or to, to the final decision.
It-it's really a matter of similarity. With claims, because we have full provenance and a chain of custody of anything that happened in the company, we are able to skip all of that. Now- As you, as I mentioned before, it is lossy, and it's lossy by design. So doesn't this hurt the performance? Well, as it turned out, it does not.
Because of the way the contracts, how we call the con-the contracts of the claims, of, and how they are stored, specifically with the LongMemEval, we were able to... LongMemEval, just to, make sure that everyone knows what I'm talking about. It's a, a memory benchmark that teststhe memory over long, sessions or long ter- horizon conversations.
And it has five hundred questions, and sessions, and we were able to retrieve the right information at four hundred and ninety-nine, times out of five hundred, which is an excellent score, especially when you consider that, yes, we do store the data, lossy as a distilled information. Another test that we ran, it's a different benchmark.
We ran the BEAM, one million, ingestions. What happened is that we've ingested, I have the numbers here, we've ingested thirty-five million tokens coming in, and it turned out to be sevent- seventeen thousand, accepted claims. Now, it cost us only eight dollars working on, local embeddings with no GPU on a CPU.
So that was another thing that we set out to, to test, is whether or not organizations will be able to afford that. 'Cause not every company can now start dumping millions of dollars on, building, their own data center. So we wanted to see if the, kind of the upper ceiling of fifty thousand dollars of hard- worth of hardware for an organization of a hundred or so people will be enough to be able to give them that governance and ownership over their information.
Another important thing, and that was a bug that I'm, I, I actually, was happy that we had, during one of those tests the, claims went into the storage, with human-readable dates rather than, compute... machine-readable dates. And what happened is that five thousand, and seven hundred, claims were quarantined, so they didn't enter the, storage at all.
So we fixed that, and then we run another test and essentially everything went through and now we know that nothing, really is getting lost during ingestion. So the way that we do this, we look at the observations. We have validation gates, so we wanna know who said what, who owns what document or what claim or what file or what record in the, the, company systems.
We wanna know what-- which part of the organization they b- they belong to. That, dictates who will have access to this information. Of course, they will always have access to information that they've created, but and we're gonna see an example in a bit. Not everyone in the organizations should be privy to that information.
All right? There is something that is a need-to-know basis
Another very important thing is we decided to adopt kind of a Git, methodology where, we never delete data. So when n-new information comes in, it supersedes the latest information, but it does not delete it. So if I'm asking the agent, "Okay, when did we agree on net sixty?" Because our standard contract says net thirty, there is a full trace of evidence, oh, you've changed it on this date, for that, specific client only, and we have that, information.
Now, we're gonna have that information if we capture that information. Okay? So And that's, that's a really important part, because if we do not have the information, we can not conjure it into existence. So there's another type of answer that a lot of benchmarks and a lot of memory systems that we've tested just skip, which is not knowing, right?
So if I'm asking, did Acme accept the NET60 terms, there could be three types of of not answering of reasons of why not answering. First, obviously, we can, we can answer it because the evidence exists. We have searched and found nothing, or w- it's, completely outside of our coverage, okay? So we know that something happened elsewhere or...
and, it, it... just something that we don't cover. So these are three different, very, very different ways of, three very different ways of telling the agent, "We don't have access to the information that you're looking for." Because if, y- if a company needs to rely on data, it needs to be able to trust the machine.
And what we did it, when we, when we told the agents is one of the, one of our tests, we told, "Okay, never say that you don't have the information, okay? Just never say." So the benchmark went from, eighty-eight percent, which was overall, and then it went to, eighty-seven point two. I mean, if you're gonna look at the benchmark, you're ga- barely gonna notice this difference, okay?
But that means that we had thirty fabricated answers. Because we told the agent, "Never say that you don't have information," or, "Never say I don't know," then it just generated thirty fabricated answer. And these are, th- these are not small things, right? Th- these are, questions that organizations are relying on.
Think about a, a financial institution or a health institution. You don't want it to, hallucinate thirty, questions. Even though percentage-wise it's a very, very small number, you don't want, thirty times as many errors. So this is something that was, really important to us, to make sure that the, the data that comes in is trustworthy.
And not only that, when we only ranked answerable questions, we ha- had the benchmark climb from, eighty-eight to ninety-three percent. And, if anyone want to hear me rant about benchmarks, I'll be available later, because the, the amount of information that don't get disclosed in those benchmark is mind-boggling.
Okay. Some things didn't work. Okay? It's not all great news. Some things didn't work. Specifically, stuff that was hard for us was contradiction. not because the models weren't able to to bridge those contradictions. Is because sometimes the evidence for those contradictions didn't come up at the same ranking.
So this is a ranking problem rather than a reasoning problem, and that's what we are working on resolving right now. Another thing, very important thing, is security. A prompt is not a gate. We don't want to tell the agents, "Okay, here's all the information that you need to answer this question just don't reveal certain things to certain people."
This is not how security needs to work. this is not trustworth-- Forget about prompt injection. Let's assume that one day m-models will be immune to prompt injections. That's not a, a, a way to ensure, boundaries in an organization. So what we do is first we're starting with who's asking, okay?
We, we wanna know who you are. We wanna m-make sure that you have authorization, to that andaccess to th-this information, and then we're providing the model itself with filtered answers. So before the evidence even reaches the model, it gets filtered. And here's a quick example. Okay, we have two people, same question.
What was, our, EMEA revenue over the last twelve months? We have the CFO, and we have, Cleo, which is from, sales. So the first thing, who's asking? So we see thatwe see that, Ada, belongs to the group executives and finance, whereas Cleo only belongs to the groups, to the group sales.
That would dictate on how they're gonna receive the answer to different, to the basically the same information. Two different people, same question. So, Ada's receiving that the revenue, for the trailing twelve months was seventy million and whatever number, covering January first till, thirty-first of the December.
The figures come from the, um Q4 general ledger, And Cleo will get this. The envelope doesn't give the exact revenue figure. It only provides a band. A trailing twelve months, were between fifteen and twenty million. Current currency is not stated, and it's an overall upward trend. Now, it, it may sounds very s-looks very simple. We just tell the agent, "Hey, don't provide information to, Chloe." But what happens is that if you're looking at the evidence for, for Chloe, that's what the model received. The, the model received information with a, a band, and it receives, the kind of the de-declassification.
So if we need more information, she needs to be she needs to have a higher permission levels. And what wasn't in the evidence are the two numbers. What was actually, the, the revenue and what was the revenue in theprevious period. So if we're looking at the different evidence, we can see that this is what, Chloe receives.
This is the envelope from Ada's, and these are completely different types of evidence that even reaches the model. another e-example of if Chloe's gonna ask about the figure's claim directly Cleo will receive a claim or outside of the asserted stop or unknown claim. It really depends on what she wants.
Now, there are going to be attempts of trying to attack the system, and we've, we've tested this. We, we've tested it on r- nine types o-of attacks. that these includes missing, credentials, using different foreign keys, tr- using another person's, credentials, after they've logged in. So w-we tried attacking the system, and we were able to beat those attacks forty-five out of forty-five times.
So I don't s-- I, I don't claim that we will, that these will never be breached, but from the test that we've ran, it w- clearlyprovided us with the proof that we needed to show that there are boundaries within the organization. So one organization different models. I mentioned that we started, with, our own, we, we start with our own oracle and, that acts as the living organization model, but we're opening up the platform so any agent can c- can talk to to this system What I would want you, and I have two more slides we believe that an organization needs to own their knowledge, and I mentioned this before.
And one of the tests that we've done, as I mentioned, we went with, kind of the fifty thousand dollars, the, as the higher mark of what an organization would spend on hardware to run local models. So we took, Gemma, which is a truly unremarkable model not in a bad way.
It's good for what it was designed to do. It's, a thirty-one billion parameter model, and we've m- we've used it to distill the information and access the claims. And Gemma actually generated sixty percent only sixty percent of the claim number that frontier models have generated. But if we're looking at the baseline with a hybrid approach, I'll talk about it in a second, but f- first with a lo- fully local, nothing's rented, just pure Gemma, we were able to achieve eighty-four to ninety-six percent of frontier quality. And once we had Gemma dis- sort of know when it's not smart enough and consult a frontier model only on some questions using, some classification, then we would, we were able to climb to, ninety-six to ninety-nine percent of all frontier model, and that's with the very, very locals, where all the embeddings are happening on a CPU and Gemma is running on its own, GPU machine.
So the things that I want you to take away from this talk is first distill information. don't just hoard it. Keep the history because you need that chain of evidence, and, when you don't know, just say you don't know. Sometimes this could be the, the most valuable and useful answer an agent can tell you, is "I don't know."
So humans as well. And, authenticate and authorize before any sort of, inference that you do over the data. So that was kind of, I try, I tried to run as fast as I can. you can download the white paper and all the evidence that are s- that are attached to it with pretty much everything that, we did in order to get these numbers.
My X, LinkedIn, same one. Uh, so that's it. Thank you.