The Case for Resident AI in Enterprises | S2E10

Hello everyone, and welcome to "Old School / New Tech". I'm, uh, back after more than a month that I didn't make any videos or any podcasts. And, the reason that I wasn't, creating more episodes for this podcast was that I finally launched a s- project that I've been working on very hard over the past six months or so, a little bit more actually, and it's, VarOps.

You can see right over there kind of the logo. So you're more than welcome to check it out, It's called varops.com. And, it's a new company, and I'm going to be discussing kind of the reasons of, why the company was came to be, why I decided to start it.

But maybe the thing that will explain it more than anything else would be this, three-minute documentary that I've created, and I really wanted to share it with you. So I'm just gonna let it play and, Hopefully, this will be a good intro for what I want to talk about today.

Here we go

Here in the temperate business districts, we find the modern enterprise, a remarkably complex organism. Hundreds, sometimes thousands of individuals, and almost as many ways of sharing information. Its primary information-sharing ritual is known as the meeting. The enterprise has since evolved a more efficient method, allowing it to attend considerably more meetings each day.

Curiously, many participants can still be found in the same building Much of the organism's knowledge is carefully preserved in documents, messages, and systems. But its most valuable knowledge and many of its decision-making mechanisms are often found somewhere else. This is Maria. Maria has been part of the colony for nineteen years.

She knows which rules can be bent and for whom, which clients require special handling, and why that spreadsheet must never be touched. The spreadsheet is well documented. Why it must never be touched is not. But Maria is on vacation. The organism continues, apparently unaffected, until someone needs to know why a particular client was promised something rather unusual.

The answer exists. Somebody was in the meeting. Somebody remembers the decision. Unfortunately, that somebody is still on vacation. Recently, the organism has introduced a powerful new tool: artificial intelligence. It is remarkably capable. But like an external advisor, it can only work with what the organism provides.

What it receives may be perfectly accurate and still not be the whole story. And like any external advisor, what happens next is no longer up to the colony. In some enterprises, something different is emerging, an intelligence that lives within the organism. Its knowledge stays there. It remembers what happened, understands how decisions are made, and knows what it is allowed to do, even when Maria goes on vacation.

AI that lives inside the business, built around the systems already there and owned outright. Resident AI. That's what we're building at Varops.

All right. So, that's the video, and the big premise is that, organizations use AI, I think, wrong. And, it's not just me saying. Actually, there are a lot of, different sources and researchers from MITs and McKinsey and, and, other research companies that detected a dissonance between the investments that organizations are making in AI adoption and the actual ROI that comes with it.

And, my theory this is what I'm betting and I've betted the previous six months on, is that the, the problem is with the actual adoption of AI. And this all is gonna tie in pr- quite nicely to the fact that we got four new models, within a span of a week. And, I'm gonna I'm gonna circle back to that in, in a few minutes.

But the problem that I see with a lot of organizations is that th- they bring in the-- let's start off with the problem. They have information scattered throughout the organization. Some information is only stored in the, in the minds of some employees, in, in the-- this video's case wa- it was Maria.

And, it's not because it's not necessarily documented, it's because they don't necessarily know what to document. So a lot of the times what happens is that there's a, a different version of the company that's actually running from the one that's being documented, or from the one that the organization chart shows.

And I'm not talking about startups that are kind of young companies and, they're trying to, Or they, they can afford to adopt pretty much anything new under the sun. But I'm talking about companies that have been around for decades. They already have processes, and try to mold them around the new tools is bound to fail.

The-- It, it should be the other way around. It should have an AI that's molded around the business, and it all starts with the understanding. So in the video, I kind of, jokingly mentioned that, the spreadsheet is well documented, but why it should never be touched is not. And every organization have, those, problems where some decisions have been happening during a conversation kind of a during the, the hall-- in the hallway or over a WhatsApp message or any sort of, kind of non-official communication method that's going on within the company.

And s- it makes things fall through the cracks. And building a company brain by just having kind of an elaborate an elaborate RAG system where you just dump a bunch of documents into the AI and hoping that it will Be able to sort it out and understand what is relevant, what's the connections between the different records in the company, between the different individuals in the company, between the, different clients and the company.

And not only that, it would need to, needs to know what the decisions have been preceded by new decisions. So there it's becoming a mess. And another issue, of course, is security. Most of the the, the way that organizations have been implementing AI comes with a lot of security issues.

They say things like in the, the kind of prompting things like, "Do not expose sensitive information to, employee XYZ," instead of having it enforced in a deterministic way. For example, if the company has made, I don't know, $1,000,010 this year compared to, compared to $1 million last year, so some people in the company needs to know the exact dollar amount.

Some people in the company needs to just know that the company has been in a slight uptrend, if you can call $10 an up- an uptrend, and others should not even be privy to that. And letting the AI segment different individuals and kind of segregate that, that information based on who's talking to it, that's all reliant on a prompt, is very insecure.

And what I've been building over the past six months is, and I can share it, this is a research paper that I recently, published just, just about a week ago. It's a governed o- operational model of an organization. You can think about it as, And you can-- You're more than welcome to go and check it out.

It's on vops.com/acropolis. And w- we were able to prove basically that a company needs its own, we ca- I call it kind of L- LOM, a large organization model. It needs to have a model that's tailored fitted to the company that has a full version control of decisions and people, and is being able to understand everything that the company has been doing.

But I think that the most interesting part here, and this takes me back to the title of this podcast, is, the research proved... Where is that? I was looking for the benchmark. The research proved that using a very unremarkable model like Gemma 4, that has, like 30 billion parameters or 31 billion pa- parameters we-- I was able to to reach, when I say I, I mean doing the research that we've done and spent quite a lot, of, time and effort and GPU power on it, we were able to reach about 96 to 99% of what frontier models were capable of reaching, with the-- just by the harness alone.

And we're able to, to, kind of close the gap by leveraging frontier models. In this case, we've used Gemini, which I'm gonna talk about why Gemini and not, smarter models. But in that case, we were able to use Gemini on top of, several types of questions to kind of close the gap. And this whole research paper kind of goes to show that you don't need frontier models to f- to achieve company understanding.

So the whole title of, this, this research paper is "Own the Knowledge and Rent the Thinking." And this takes me straight to to the new models that have been surfacing up, you know. Immediately after everyone was talking about let's pace the AI and let's, try to, ease into slow down the, the pace of advancement.

Immediately in the following week, we got four new models three from, Frontier Labs. So we got just yesterday, we got Claude Opus 5.5 and GPT-6o. And a few days before that, we received a Grok 4.7. They are all, representing a major leap forward in thinking, and this is exactly what I think we're going to continue seeing.

We're going to continue seeing models that are cheaper, more efficient, and smarter. So you can see that OpenAI dropped the the cost of GPT-4o compared to 5 point, 4, sorry, 5 point, 6. They dro- they dropped it by 50%, and Anthropic did something that I don't recall them ever doing, which is reducing the price, as well if only by 20%, which is quite a lot, and they also gave a free reset.

So it's, it's quite cool to see. Grok didn't need to drop their prices even more. They were very, very easy on the, on the, on the, on the pocket, on, on the wallet even before that. And all these models essentially provide a much longer, working sessions. So they can work autonomously for a longer period of times.

They can make much better decision, and they can reason over the data given to them. And I'm not going to even, go through which one is better for what, and th- that that's besides the point. What I wanted to tell you is that we were able to achieve for understanding, for building a model on how the company works, we were able to achieve superior results with local models and kind of finishing it off with Gemini.

And my theory, at least with Gemini, is that that's what Google was training it for. Retrieving information and try to find the relevant information from mountains of data. So in retrieval, it was able to get the best results out of all the models that, that we've tested, and we've tested all of them in all sorts of different, efforts and, and thinking efforts.

And we got the best results with, with Gemma and Gemini, and you're more than welcome to read the, the research here. And, where is it? Here it is. You're more than welcome to read the research. It has the full methodology of, what was used and, how, we ran, the experiments and all the different kind of outputs that we were able to achieve.

And as you can see, by using the rent-- the thinking we were able to be on par or slightly better than Frontier in every seat. And let, let me explain what I mean by frontier in every seat. So we ran this, benchmark in several different ways. One was, let's just use, and maybe I should go a step a step back.

We have a component within the system that which we call Acropolis. We have a component called Varys, kind of the Master of Secrets from, Game of Thrones. And it has connectors that it sits pretty much everywhere the company wants it to sit. It can-- sits on the company's email server and Slack Microsoft Teams call conference calls specific contacts in WhatsApp and iMessage and all that stuff.

So it basically have access to everything that's happening that is a company context. Obviously the company's ERPs and, systems and databases and code, and so it knows everything. So it just a huge ingestions application and it distills information. So we don't save raw data. Raw data is only being saved as reference, but we distill information in a way that makes it a lot efficient for retrieval.

And What we've realized that once we have this data and we tried first distilling the information using frontier models and retrieving the information with frontier models, and then tried to, reason over this information using frontier models, and we've, set it up as kind of the maximum that, the, the ceiling of what can be achieved on that data.

When we tried it with, other models as I mentioned, with a combination of just local models, we were able to reach about, as you can see, we were able to reach about 70% or maybe 80% of what we were able to achieve on frontier. But when we added, okay, let's use the, local models to retrieve the data, but add frontier models like Gemini to reason over the data, we were able to pretty much be on par with full frontier models.

And that kind of goes to show that you don't necessarily need not the pricing nor the full power of Opus type of models and, GPT types of models in order to achieve a model that's, that can reason over your company's information.

Um, and that, that was very, a, a very interesting discovery because of the way... i'm also gonna take some credit here because of the way we've created the contract and how we store the information. So if we j- if you're just gonna take, your company brain that's based on markdown files or a database and you you tr- you're gonna try to compare local models or Gemini to, something like, Opus you're not gonna get anywhere close.

So- it's, really about building the right harness and the right way of storing information. So that's kind of the, the overall results that we were able to achieve. So we were able to achieve based on the about one to two points change from swapping the models that, build the records.

And when we rented kind of frontier models only to answer, not to retrieve data, then we were able basically to, to achieve 96% to 99% of the the frontier capabilities. And it really only took us about $8 doing just 35 million tokens of conversations into our records, and we've done that specifically on CPU.

Now- Why did we end up doing that? It all started with a few months ago, Fable came out, and then a week later, it was, pulled out, from under our feet. And there was a period of time where, we weren't sure that we, I specifically as a non-US citizen, will ever get a chance to use it. If you remember the, the whole saga that was there.

And this kind of started-- I was already in the middle of creating this enterprise level company brain. But really that e-event, that Fable saga is what drove me to really focus on local models. I wanted to reshape everything that I was doing to fit local models that enterprises can use.

Because an enterprise can easily spend $100,000 to $200,000 on commodity-ish hardware. They can buy device with, three or four H100 or H200 GPU cards. They can afford it, no problem. And what they cannot afford is keep sharing their company's gold with frontier labs and, have that risk of a model suddenly being deprecated or an access being taken away from them, or Fable decides deciding not to answer because, somebody used the word hack.

So I've noticed that when I use Fable to to develop software, I often try to come up with, a prompt. At some point, I'm, I'm gonna say something like, "Let's try to hack this," because I wanna test the, security, levels that I was able to achieve on that software. And just because I'm using the word hack Fable refuses because it's, it thinks that I'm trying to attack someone even though it was the, the one that just wrote the software.

Anyway, so I didn't want to have enterprises be in a position where they are not in control. They don't have model sovereignty, data sovereignty, and kind of AI, control over how they're gonna implement AI. So that's why I decided to lean in really hard on local models and I was able to achieve-- I keep saying I because I was really head down with with lots of, number crunching over the past, few months.

And out of the whole thing came VarOps, which is, how we're going to implement this in our organizations. We've already running pilots and proving to work quite nicely. So if anyone wants to check it out, you're more than welcome to visit this mega project that I've been working on.

Go to varops.com. It's, decided-- With the site, I decided w- when you're gonna visit this, the website, you're gonna kind of notice that it looks like a magazine, which it is. And the reason is that I decided to reverse how most websites, look. So instead of having kind of a landing page and then have a resource tab where, or a blog where, you can, read cool information and, kind of, e-expand your knowledge, I decided just to put that front and center.

And if you wanna learn more about what the company's doing, then you have the About page and a whole thesis on resident AI, and the research is here. So you're, yeah, you're more than welcome, and you can see the columns and the podcast. They all have a new home. And all the columnists in this, in this magazine, by the way, aside from myself who, who writes the Founder Mode column, everyone are bots.

So this can also be a cool first thing is to, and I'm gonna link it in the show notes, is how build a magazine that's filled with, AI agents, and I have an AI editor-in-chief, and I have kind of the full breakdown in how it works and how the workflow works and how the whole curation process.

So I'm gonna link that in the show notes. It's just kind of a a proof of concept of to what can be achieved. And it's been kind of running hidden from the public eye for a few months now, just to make sure that when, when I launch, it's, it's not empty. So it has quite, quite, quite a lot of information.

So you're more than welcome to go and check it out. Now, I promised that I also mention Jev. I mentioned three models that came out, and Jev is, the new kid on the block. And it's, it's a really interesting entry because it's a new type of model, and it's really good at making quick decisions.

So I really like the way that they've, positioned it as kind of system one thinking versus system two. And if, anyone here read Thinking Fast and Thinking Slow or the other way around by Kahneman, you know that he refers to system one as that part of the brain that is more instinctive i- in the way that it re- acts and reacts.

And that's what Jev is really is really good is really good in. Let me just pull it up so I can share this. let me share this at this stage. So this is what basically they've done, and they have this whole system one thinking, and JeV is its-- is a new model in that regard. and it's really, really good for...

It's-- You cannot chat with it. It's not a chatting model. It's not really an LLM. I, I call it, an FLM, so it's basically really good at, at classification. A large language classification model, if you will. y- the way that it works, it... Let me see if they have an example here of the structure. No.

But the way that it works, you just dump a chunk of, text or information, whatever you want and some options. So the options would be like, is it a yes or a no question? So you can dump, like a picture, and you can ask, "Is it a safe for, for work type of image?" And it will return yes or no with confidence.

Or you can ask it to classify, is this picture of, Go back to Silicon Valley. Is this a hot dog or not hot dog? And it will classify it, and stuff like that. So it's really, really good. It's really fast, and it's also the f- the, um... You can see how people reacted to it by the sheer number of alternatives that came out within a span of a week.

Some of them are really good, some of them are not as much, but there's no, there's no doubt in my mind that this type of model is really needed both in terms of the ability to basically use an LLM style, and I'm saying style, not type, an LLM style of o- of model to make a more deterministic decisions and responses, much more than having a full thinking process and that, that dumps a bunch of, words to to their response.

That being said, when I tested it on, MUXI on MUXI.org so MUXI has a bunch of about 11 different gates every time someone sends a a request. So we have different gates like, hey, is this s-somebody trying to give us a credential, like an API key or a password or something, just so we know we're not gonna, we're not gonna process it.

Is the person even asking a question? Maybe they just responded with a thumbs up to a previous response. Is, this an attempt for prompt injection or all sorts of different gates that, that we have. And for those specific gates JeV didn't perform well. And I'm saying this, we, we were actually being, we were able to achieve quite good results, and by good I mean 100% passing on 11 different gates, with a 95 megabytes of an embedding model that's fully local in trust and memory.

And I'm able to say this while also saying that JeV is amazing because the point here is not whether or not you should use JeV or not, or JeV-like models like Laia or, Cleaner and there are different other models that are kind of in the same spirit is because it really is a matter of finding the right model for what it is that you're trying to achieve.

I've said this before, but I'm gonna say it again. Models are not interchangeable. So the fact that you had some sort of a workflow where you used, Sonnet and it performed great, you cannot just put Opus on it and expect the same results but smarter. Or, you cannot replace it with, GPT and expect similar results but cheaper or whatever it may be.

Every model will behave completely different, and you will have to fine-tune the, prompts that you're using and any sort of deterministic gates that you have before and after to make sure that they fit that particular model. Okay? So that's, that, that kind of what I wanted to say about Jev. I started by saying it's a really, really good model.

Not, not only it's really good model, it's a much needed new thesis and, and a, a new paradigm that is m- the, the market really, really needs. But it's n- it's not a drop-in replacement for if statements or for any sort of, other gates that that you have. Okay? So that's kind of what I wanted to say.

And yeah, so I have I had so many things I co- I feel that I've covered today, but the, the whole-- it will take me back to what I started this with, is you have to figure out which model works for your use case, and especially if you are an, or an organization where the, the problem is in the error rate.

So if you feel like, okay, this model was successful 97% of the time, and this model, was successful I don't know, 92% of the time, it may look like the difference is not a lot. But in an environmental, i-in in an enterprise environment where things need to be mission critical, difference is between 3% error rate and 8% error rate.

So it's more than double. It's more than two and a half times the error rate. So that means that you're gonna have more than two and a half times issues and un- and, dissatisfied customers and wrong diagnosis or different accounting errors. So 92 to 96 or 97 doesn't seem like it's very far apart, but look at it from the other side.

The error and, the bad decisions, just jumped from just jumped from, 3%, which was the 100 into more than double that. Hopefully this, this made- makes sense. I never know how to end these, these, broadcasts, so I'm just gonna end it by inviting all of you to go to VarOps. Check it out and sign up for a newsletter, for the da- dai- daily newsletter.

It's, you can obviously unsubscribe at any time if you feel that the content is not for you. And Old School New Tech will be going back to its, usual schedule of once a week. Was a pleasure being back. I feel like I have, still have a lot of things that I wanna talk about, but we'll save some, some things for future episodes.

Thank you very much for joining me, and I'll see you in the next one.