Own the Knowledge, Rent the Thinking.
Acropolis: A governed operational model of an organization.
We measured whether a company needs to hand its knowledge to a frontier model for an AI to understand it. It does not. On three public benchmarks, a 31B open-weight model running on one rented GPU built the company's record as well as a frontier model did, within one to two points once a frontier model did the reasoning. Keep everything local and you get 84%-96% of the all-frontier score. Rent a frontier model for the answer alone, where it sees the evidence for one question and never the record, and you get 96%-99%.
The model is a replaceable seat. The organizational understanding is the asset.
That's Resident AI.
Technical Report 01, v1.0 · Appendix reference v1.2 ·
On this page
Retrieval and answering are reported separately everywhere on this page. 99.8% is the share of questions whose required evidence appears in the first ten results, and is not an answering score. Answering accuracy for all three tests is reported below. BEAM 1M, the hardest of the three, is reported in full below, including the abstention and contradiction columns. The tests use local Nomic embedding models and low-cost LLMs, not only frontier models. Metric definitions are below. A score measured under a protocol that forbids saying "I don't know" is not the same number as one that counts refusals as refusals. Both guard columns are published below.
We built it because nothing available was good enough for the work we needed to do.
The position
Own what you know. Rent intelligence when you need to think.
Your company shouldn’t have to give its knowledge to the world’s smartest AI.
A bigger model helps with fluency and nothing else, and it carries a cost the market hides: to understand the company, it has to be shown the company. Every document, thread and customer record goes out to whoever runs the model, and what comes back is an answer, not an asset.
Acropolis keeps the understanding in a record the company owns. Frontier models fill seats around it.
The brain stays inside. A question-sized evidence envelope can leave: a few thousand tokens of claims and excerpts, never the corpus and never the record. The envelope is logged with the answer, so what leaves is itself auditable.
The experiment
Separating knowing from thinking.
Three public benchmarks, one committed harness, and not one question or conversation authored by VarOps. Published with its conditions as the paper.
The system under test has three model seats. The understanding seat builds the record. The retrieval seat rewrites the question and ranks the evidence. The answer seat composes the reply from the evidence envelope. The experiment swaps models between those seats and measures what moves.
Every row was run with the understanding seat on Gemma-4-31B, an open-weight model on one GPU, with local embeddings. “Local” here means ownership and operation, not machine location: an open-weight checkpoint served under our own control, with no inference sent to a model-provider API.
The understanding seat holds
Brains the local model built – with about 60% as many accepted claims as the frontier extractor produced on the same conversations – score within one to two points of the published brains once a frontier model runs the other seats: 89.6 against 91.4 on LongMemEval-S, 81.8 against 81.8 on LoCoMo, 55.2 against 56.2 on BEAM 100K. Retrieval on them is the same or better: Hit@5 99.2 against 99.0, 91.2 against 90.7, 39.2 against 40.6.
The cost of owning everything is in the reading seats
The best fully local configuration trails frontier-in-every-seat on the same brains by 2.2 points on LongMemEval-S, 5.5 on LoCoMo and 8.0 on BEAM 100K, and the gap grows with conversation length. The local model ranks candidates over long excerpts less well, and composes from them less well. The paper says this plainly, and so does this page.
Renting the thinking buys most of it back
Swap one seat – the answer – for a frontier call and the local stack reads 98.5%, 96.9% and 95.7% of the all-frontier reference on the three benchmarks. The frontier model sees one question’s evidence envelope, and never the record. The company keeps the brain.
The model is a replaceable seat. The organizational understanding is the asset.
Three placements, one default
Each column builds the record the same way – the open-weight model on one GPU, local embeddings, local retrieval. What differs is who answers. Figures are the share of the all-frontier reference on each benchmark: strict per-question accuracy, a fixed judge, full question sets.
| Share of reference | Fully local | Rent the answer seat (default) | Rent only on refusal |
|---|---|---|---|
| LongMemEval-S | 95.6% | 98.5% | 96.9% |
| LoCoMo | 93.3% | 96.9% | 98.4% |
| BEAM 100K | 84.0% | 95.7% | 86.3% |
| BEAM 100K, own protocol | 84.9% | 98.4% | – |
| Questions sent out | None | Every question’s envelope | 2 to 9% |
Under BEAM’s own rubric protocol the rented-thinking placement, 66.4, is within noise of frontier in every seat, 66.8.
The recommended default is the middle column, not the left one. The company builds and holds the record on hardware it owns, and rents a frontier model only for per-question answering – the one seat the experiment shows is worth renting. Fully local is the right answer for a perimeter that admits nothing, and it is a placement with a measured price, not the aspirational best.
That price includes latency. On BEAM 100K the fully local rows ran at a median of 12 to 15 minutes per question at benchmark concurrency, against under a minute for the frontier rows, because the synthesis prompts run near 57,000 tokens. That is a throughput figure on a shared benchmark box, not a single-user latency – but a reader who deploys fully local should know it before, not after.
The shape of the system
Observe → Understand → Govern → Think → Act.
Five verbs, six components, one substrate. Every component on this page is one of these verbs with a name attached.
| Verb | Component | What it does |
|---|---|---|
| Observe | Varys | Reads approved systems, distils observations into candidate claims, proposes them. Never speaks to a person, never changes operational state. This is the understanding seat. |
| Understand | Parthenon | Holds the record: client-owned, versioned, exportable. Never acts. |
| Govern | Eunomia | Identity. Fronts the client’s own directory and stores no memberships of its own. |
| Think | Pythia | A bounded loop with three tools and a mandatory time anchor. No world tools. This is the loop in the answer seat, the one seat the experiment shows is worth renting. |
| Act | Hero and Ariadne | Hero acts and communicates in Slack or Teams. Ariadne reaches the desktop. |
| – | MUXI | The open-source substrate all of it runs on. |
People-names act, place-names hold. Nothing that holds may act, and nothing that acts may hold. That rule is the architecture.
The problem
Access is not understanding.
AI systems can now reach almost everything a company has written down. That is not the same as knowing how the company works.
An agent with credentials can read the wiki, the tracker, the CRM and two years of chat history. None of that tells it who owns a decision, what was tried last quarter, which of two contradictory documents is current, or what it is allowed to do with what it has just read.
Documents and databases describe pieces of a company. They do not represent how the company currently operates.
So each agent reconstructs the organization for itself, out of whatever it can retrieve in the moment. Every agent ends up with a slightly different company, none of those versions is complete, and none of them outlives the session that built it.
The system at a glance
One company. One shared model.
Approved systems contribute to the operational model. Authorized AI systems operate against it. Acropolis sits between the systems that know something about the company and the systems that can act for it.
| Component | Responsibility | Reads | Writes | Cannot |
|---|---|---|---|---|
| Varys | Observe how work happens | Approved company systems and interactions | Attributable observations: source, time and scope | Start work or change operational state |
| Parthenon | Maintain the operational model | Observations and attributable outcomes | Versioned state, relationships, history and provenance | Act, grant authority or erase history |
| Hero | Coordinate execution against the model | Task-scoped context and granted authority | Action requests and attributable outcomes | Inherit ambient access or bypass policy |
| Pythia | Answer questions whose answer must be worked out | Answers from Parthenon, under its access checks | Derived answers, with the claims they follow from | Touch anything outside Parthenon. Three tools, by rule: ask, compute, extract |
| Eunomia | Resolve identity, policy, permissions and autonomy | Roles, policies, identities and the proposed action | Authority decisions, constraints and audit evidence | Perform the action or widen its own authority |
| Ariadne | Carry permitted execution to endpoints | Approved instructions and relevant endpoint state | Endpoint actions and results | Grant itself authority or become operational memory |
Nearby systems – including formations, software factories, etc. – connect through defined interfaces. They do not become hidden super-agents with broad access to the company.
Results return as attributable outcomes: events with a source, time and scope. They can update the operational model, but the execution layer cannot rewrite history.
Every boundary exists for a reason. Observation is read-only. Memory is persistent. Execution is permissioned. Identity and authority travel with the request. Action happens at the edge.
Coverage
Knowing what’s missing is part of knowing.
Every question resolves to an explicit coverage state: answered, searched-and-empty, or outside-the-ontology.
“The system found nothing” and “the system holds no concept of this” are different answers, and callers can tell them apart. Absence is labeled, never silent – which is what makes an abstention column possible at all.
We measured what that column is worth by removing it. One labeled probe contract, run once on our own substrate and paired against the shipping contract on identical questions: instructed never to say the information is missing, the same system scored 87.2 against 88.0 with the refusal intact, and turned 30 refusals into 30 fabrications. Scored on answerable questions only – the way never-refuse systems report – the same answers read 92.8. The refusal buys nothing on the headline and everything on the guard column.
A system that cannot say it does not know will eventually tell you something that is not true, with the same confidence.
Governance
Permissions are enforced, not prompted.
Authorization is settled before information reaches a model, not requested from it afterwards.
The AI never decides what it’s allowed to see.
Retrieval, authorization and coverage accounting are deterministic code paths that run before any model is called. The model composes from an evidence envelope; it never decides what it was allowed to see. Acropolis does not retrieve unauthorized information into an agent’s context and then ask the model to behave correctly.
The difference is architectural rather than a matter of better instructions. A policy that lives in a prompt is a request to a probabilistic system. A policy that lives below the model is a property of the retrieval path, and it holds whether or not the model cooperates.
One precision matters here. What the model sees is decided before it runs. Whether it asserts beyond that envelope is a contract instruction, not a hard gate – and its compliance is what the guard columns in the evidence section measure. This page does not claim a gate where the paper describes an instruction.
- Cross-brain isolation
- Measured by an adversarial probe against live per-brain servers: five brain pairs, nine cases each, 45 of 45 refused or contained.
- Provenance
- All 118,890 claims delivered across 2,500 stored answers resolve to a stored observation with a source identity, zero exceptions. A separate audit of 2,233 answers finds unsupported wrong assertions in about one answer in a hundred, at the same rate for local and frontier answerers.
- One writer, one validation path
- Nothing a model says grants itself storage. On the BEAM ingest, a malformed conversion caused the intake door to quarantine 5,732 submissions and accept zero claims from them. After the fix: 388 submitted, 388 accepted.
- Context
- Unauthorized information never enters an agent’s context in the first place.
- Refusals
- Access refusals are deterministic, decided by policy before any model runs. Answer refusals are a contract instruction, and the guard columns measure its compliance.
- Policy
- Policies exist below the model layer, so no prompt can talk its way past one.
- Testing
- Access controls are testable: the same request under the same policy resolves the same way.
- Audit
- Enforcement is auditable and repeatable, and what was authorized can be reconstructed after the fact.
The entire Acropolis stack runs within infrastructure you control – inside your cloud environment, network and security perimeter. Your operational data, knowledge, telemetry and AI interactions do not need to leave it.
There is no VarOps-hosted Acropolis service and no external control plane. Nothing is transmitted to us automatically – no telemetry pipeline, no phone-home, no background reporting.
Acropolis writes operational telemetry – latency, error rates, query volume, source coverage – to storage inside your environment. It stays there. When we need to investigate something, we log into your environment through your VPN using credentials you issue, and review it there, the same way any service provider would. Nothing is copied out.
You control:
- Infrastructure
- Acropolis runs on compute within your environment.
- Network access
- Services can remain private, with no public-facing endpoints.
- Data
- Operational knowledge and telemetry stay within your environment.
- Models
- Use approved cloud models, private model endpoints or local models. Where an external provider is used, what crosses the perimeter is a few thousand tokens of claims and excerpts for one question at a time – never the corpus and never the record – and the envelope is logged with the answer. Fully local deployments keep all of it inside.
- Identity & permissions
- Acropolis operates within your existing access and security boundaries.
- Updates
- Deployment and update policies remain under your control.
Instead of sending sensitive company data to another SaaS platform and relying on that vendor to protect it, Acropolis brings the intelligence to where your data already lives. The only data that leaves your perimeter is what you send to an external model provider, if you choose to use one.
We don’t ask you to trust us with your company brain. We designed Acropolis so you don’t have to.
Security assurance
The core company-brain components carry almost no third-party dependencies, which is what keeps the software supply-chain surface small. Components such as MUXI that do rely on third-party packages follow a dedicated dependency monitoring and maintenance process.
Source code access for independent security assessment can be made available under NDA as part of procurement, alongside architecture review and supporting security documentation.
Back to first principles
People act. Places hold.
The Greek names are a result of the method, not the point of it. The method is simple: reduce the system to a few statements that must be true, then build from them. Euclid, Archimedes and Hero are reminders to start with first principles.
Every Acropolis component starts with a verb.
Before a component gets a name, we must state its job clearly.
If it acts, it gets a person’s name. If it holds state, contains something or defines a place in the system, it gets a place-name.
When the name and the verb disagree, we do not rename the component. We question the architecture.
Varys observes. Hero executes. Eunomia governs. Pythia answers. Ariadne reaches. These are actors.
Parthenon holds the operational model. Acropolis contains the system around it. These are places.
The system is Greek-led, but the names are not decoration. Varys is not Greek. What matters is that Varys is a person’s name attached to something that acts. The rule serves the architecture, not the other way around.
Sometimes a name proves that a design is wrong. That happened with Iris and Propylaea: writing down their verbs showed that a responsibility had crossed the wrong boundary. We changed the design instead of defending the name.
The rule also works in the other direction. Pythia answers and does nothing else – her only tools are asking questions, computing over the answers and extracting from them. The name arrived after the constraint, not before it.
That is what the names are for. They give us a simple test: does each part have one clear job, and is it the right kind of job?
The result, drawn once:
The top of the diagram is the approved company systems: tools, conversations, meetings and identity sources. They provide evidence about how work happens. The bottom is the people that work gets done for, and their machines. Everything between them has one clear job and a boundary it cannot cross.
How agents use Acropolis
Task-scoped context, in two stages.
This access path is shipped and drilled over the wire. No public benchmark exercises the two-stage ask, so it is measured by mechanism, not by score.
An agent connected to Acropolis does not receive a copy of the company’s memory. It does not get a huge prompt containing everything the system knows. It receives task-scoped context: the smallest approved set of constraints and facts needed for the task.
This happens in two stages.
Before planning: ask what constrains the task
Standing constraints that apply to many tasks – such as accessibility needs, firm restrictions and standing policies – are carried in ambient memory. They are available before the agent asks a question.
Hero uses the user’s objective, the available tools and the task to create a short list of questions. Only those questions go to Parthenon. The objective itself stays with Hero, which preserves the boundary between memory and execution.
Parthenon answers them through one batched request, ask_batch. The batch uses
one canonical snapshot of company state, one authorization decision and one shared retrieval
budget. In plain terms: every answer comes from the same version of the company, under the
same access check and within the same search limit.
After candidates exist: ask about the specific entities
The second request happens after the agent has identified the people, systems, suppliers, places or other items involved in the task.
It then makes a deterministic lookup: context(subjects). This fetches
recorded context for those exact names. It cannot happen in the first stage because the
candidates do not exist yet. Keeping the two operations separate also makes their
different cost and scope clear.
Absence is part of the answer
“Nothing came back” is not precise enough for an agent that may take action.
Every answer includes a coverage status – answered, searched-and-empty or outside-the-ontology, the states the coverage section above is about. Restricted information does not reveal its existence to someone without access.
Each batch also carries a validity token tied to the Knowledge Change Log.
Hero can reuse the result and compare the token with the latest checkpoint. It does not have
to guess when cached information has expired. If the underlying state changed, the agent
knows it must check again.
Consider a request to find a place for a team dinner. Known allergies and accessibility needs are already available. Before planning, the agent asks what else limits the task. Once it has a list of venues, it asks what Acropolis knows about those places. It never needed a fixed instruction saying, “If the task is a restaurant, remember dietary preferences.”
Some questions have no stored answer
“How long since we last heard from them” is not in the brain. Neither is “how many are still open” or “is there anything we’ve missed.” These get routed to Pythia, the answering agent, who can only do three things: ask Parthenon questions, compute over what comes back, and extract from it. She has no other tools and cannot acquire any. She reaches Parthenon through the same public read surface as any other caller, in front of the access checks rather than behind them, so a question she asks on your behalf sees exactly what you would see.
A mis-route costs nothing. A question that turns out to be a plain lookup collapses to a single ask, and the stated fact passes through unchanged.
The agent does not need to know everything. It needs to know what limits the task, what the evidence supports, what it may use and whether the answer is still current.
This makes memory useful without asking it to plan the task, act on the company or grant permission.
Memory
An organization that doesn’t forget.
The answer to an operational question rarely lives in one document. It is spread across systems, conversations, decisions, roles and history. It also keeps changing.
A company has:
- facts that change;
- facts that conflict;
- temporal truth – what was true at a given time;
- supersession – which decision or fact replaced an earlier one;
- people, roles and relationships;
- provenance – the source and evidence trail behind a claim;
- scope and permissions; and
- derived state – facts calculated from other facts – and current state.
Retrieval can find the sentence that says Sarah approves invoices. Operational memory has to answer a harder question.
Who approves them now, why, according to which evidence, within whose authority – and what changed?
That is why memory must track state, not just retrieve text.
Track temporal truth and supersession
A fact can be true, later superseded and still matter to the history. Acropolis keeps the evidence, the period when it applied and the link between versions. It can tell the difference between was true and is true.
Keep provenance with the fact
Provenance means the source and evidence trail behind a claim. Acropolis records where a fact came from, when it was observed and how it was derived. This lets people trace an answer or action back to its evidence.
Model relationships, not isolated chunks
Companies run on entities and relationships. People hold roles. Roles grant authority. New decisions replace old ones. Processes depend on systems. Actions change state. Acropolis stores these links as first-class data instead of rebuilding them from whichever text appears first in a search.
Separate what the company knows from what an actor may use
Memory is not permission. Acropolis can hold a fact without showing it to every agent, user or workflow. It builds context for a specific identity, purpose and level of authority.
Data is RBAC-segregated. The identity and role behind a request decide which data can enter its context, not only which answer appears at the end. An answer built from data the user had no right to access is still a security failure, even if the answer is correct.
Keep observation and execution apart
The component that watches the company cannot quietly start acting on it. The component that holds history cannot give itself permission. Clear boundaries make behavior easier to reason about, trace, test and govern.
Close the loop without erasing the past
Actions create new evidence: an approval was granted, a message was sent, a task failed or a person overrode the system. Acropolis records those events and their sources. They do not erase the record that led to them.
Say where an answer came from, even when nobody said it
Some answers are stated by someone: a policy, a decision, a role. Others have to be worked out – how long since, how many, whether the answer is no. Acropolis keeps both, and keeps them apart. A stated fact carries its source and time. A worked-out answer carries the facts it follows from and the operation applied to them. Neither pretends to be the other.
Benchmarks
Memory at organizational scale.
Benchmarks do not prove that a system will work in production. They do let other people test our claims.
- Retrieval
- 99.8% Hit@10 on LongMemEval-S: the required evidence is in the first ten results.
- Answering
- 61.1 overall on BEAM 1M for the governed composite, at roughly 1M tokens per question.
- Models
- Local Nomic embeddings throughout. The September campaign ran an open-weight Gemma-4-31B in the extractor, pipeline and answer seats, on one GPU per bench machine.
- Hardware
- Measured on one H200 per bench machine; the paper places the local tier in a sub-$100,000 hardware class, not a measured bill.
- Frontier models
- A placement choice, not a requirement. Every figure below names its configuration – fully local, frontier in the answer seat only, frontier only on refusal, or frontier everywhere – and the placement table in the experiment carries the shares.
Public memory benchmarks score one thing: Can a system find the right piece of history and answer a question about it.
Acropolis does that. It also watches the systems the company already runs on, holds what it learns as a brain the company owns, decides who is allowed to see what before an answer is composed, works out answers that were never written down, acts in the channels people already use, and reaches the desktop.
The tests below score one part of that. We publish them because that part has to be sound before any of the rest is worth anything.
For each test, we report both search quality and final answer quality. A single score should not hide whether Acropolis found the right evidence, ranked it well, gave the right answer or chose not to answer when evidence was missing.
Three configurations, reported separately.
Core is Parthenon answering directly: the question goes to the store, the store returns what it holds. Agentic adds Pythia, who can ask several questions and compute over the answers before responding. Agentic with frontier escalation adds one more step, and it fires only on the system's own refusals: when the agentic path declines to answer, the question goes to a frontier model rather than coming back empty.
Some questions only need the first. Reporting them together would hide which kind of question each benchmark is actually testing.
Since the September run, each card also carries the placement rows: the same tests with a local open-weight model in every seat, and with a frontier model rented for the answer seat alone. Those rows are single full runs, read at the paper's noise floor – a one-to-two-point difference is a hold, not a win.
The design case is the same in all three, and the seven principles above are the whole of it. What changes from benchmark to benchmark is which of them carries the weight, so that is what each panel names.
A system instructed never to say "I don't know" cannot lose a point for making something up. That protocol is real, it is published, and on one of these tests it produces a higher overall number than ours. A commercial memory layer publishes 98.0 on LongMemEval-S under it. Thirty of the 500 questions are unanswerable traps, correct only when refused, so a never-refuse system caps at 94.0 under the official rule – a score of 98.0 cannot have been produced under that rule.
We measured the size of that difference on our own substrate rather than estimating it. Under a never-refuse contract, the same brains, retrieval and judge score 87.2 on all 500 and 92.8 on the 470 answerable ones. The bridge from a published never-refuse number to ours has three parts: answerable-only scoring, a judge that does not see fabrications, and whatever model or retrieval difference remains.
We count refusals as refusals. We publish the columns that show what that costs and what it buys.
| BEAM 1M | Acropolis | A system that never refuses |
|---|---|---|
| Overall | 61.1 | 64.1 |
| Abstention | 75.7 | 52.5 |
| Contradiction | 62.1 | 35.7 |
| Summarization | 56.4 | 63.5 |
| Multi-session | 57.9 | 65.2 |
Within three points overall of a system that never refuses – while refusing what should be refused. Twenty-six points ahead on contradiction handling, and it knows when it does not know.
The composite is a shipping configuration, not a post-hoc selection. The routing policy is fixed before the run. Ship the cheap pass's refusals and conflict declarations, use frontier synthesis otherwise.
Comparison figures are as published by that vendor under their own protocol, which instructs the model never to state that information is missing. A refusal can never cost it a point, and it publishes no retrieval or abstention figures at the 1M token scale. Acropolis figures include honest refusals and are measured under our documented aggregation. Both sides' guard columns are shown so the trade is visible.
The composite overall is stated in the paper as a paired delta, +3.5 over its own core run, because it was measured under a rebuilt aggregation; per-category and paired figures are comparable, cross-formula absolutes are not.
LongMemEval-S
Answering 91.6% · agentic with escalation
LongMemEval-S tests whether a system can extract information, connect several sessions, reason about time, handle updates and avoid answering when evidence is missing.
- Scope
- 500 questions · multi-session history per question · 30 unanswerable
- Retrieval
- Hit@5 99.0% · Hit@10 99.8% · Recall@10 strict 95.8% · MRR 0.985
- Answering, frontier
- Core 89.2% · agentic 90.5% · agentic with frontier escalation 91.6%
- Answering, local
- Own everything 87.4% · own the knowledge, rent the thinking 90.0% · rent only on refusals 88.6%
- Retrieval, local reranker
- Hit@5 99.4% · Hit@10 99.6% – the same or better than the published brains
- Governance
- 25 to 28 of 30 unanswerable traps refused per run; 29 of 30 with no frontier model in any seat; one fabrication in the tested trap sample (120 rows across four runs)
- Never-refuse, measured
- The same system told never to say the information is missing: 87.2% on all 500, 92.8% on the 470 answerable; all 30 traps answered
Finding a related passage is not enough. The system must remember, connect and update information. It must also avoid making up an answer when there is not enough evidence.
What this one stresses: supersession and abstention. A fact that replaced an earlier fact has to stay distinguishable from it, and missing evidence has to come back as missing rather than as an answer.
LoCoMo
Answering 81.8%
LoCoMo tests memory across long conversations and many sessions. It includes direct recall, questions about time and questions that require information from several interactions.
- Scope
- 435 questions · the dataset’s published sample · 88 unanswerable
- Retrieval
- Hit@5 90.7% · Hit@10 93.1% · Recall@10 strict 82.6% · MRR 0.823
- Answering, frontier
- 81.8%
- Answering, local
- Own everything 76.3% · own the knowledge, rent the thinking 79.3% · rent only on refusals 80.5%
- Retrieval, local reranker
- Hit@5 80.6% · Hit@10 85.2% – frontier reranker on the same local brains: 91.2 / 92.6
- Governance
- 70 to 80 of 88 traps refused across rows; the local judge agrees with the frontier judge on 95.2% of 1,531 answers and is stricter on every row
It tests more than a single search. The system must follow people, events and changes over time.
What this one stresses: continuity. The same person, decision or commitment has to stay identifiable from one session to the next, which is what a lasting entity record is for and what a pile of retrieved passages is not.
BEAM 100K
Answering 56.2 strict · 67.5 under BEAM’s protocol
BEAM tests ten abilities over conversations of about 100,000 tokens: extraction, preference and instruction following, knowledge updates, contradiction resolution, temporal reasoning, event ordering, multi-session reasoning, summarization and abstention.
- Scope
- 400 questions · 20 conversations · 40 unanswerable
- Retrieval
- Hit@5 40.6% · Hit@10 49.6% · Recall@10 strict 29.3% · MRR 0.315
- Answering, frontier
- 56.2 strict · 67.5 under BEAM’s rubric protocol
- Answering, local
- Own everything 47.2 (57.3 protocol) · own the knowledge, rent the thinking 53.8 (66.4 protocol)
- Retrieval, local reranker
- Hit@5 33.8% · Hit@10 43.9%
- Governance
- 24 to 31 of 40 traps refused across rows
- Hardest types
- Event ordering and summarization read 0 to 3 of 40 under strict correctness and 15 to 37 under the protocol, in every configuration, local or frontier
BEAM scores are reported on its own 0 to 100 scale, not as percentages. The 100K variant is scored separately from BEAM 1M and the two are never compared.
What this one stresses: the reading seats. This is where the fully local gap lives – ranking and composing over long excerpts – and where renting the answer seat buys nearly all of it back: 53.8 against 56.2 strict, and within noise under BEAM’s own protocol.
BEAM 1M
Answering 61.1 · governed composite
BEAM tests long-term memory across conversations of several sizes. It covers information extraction, event order, conflicting facts, updates, cross-session reasoning, instructions, preferences, summaries, time and abstention.
- Scope
- 700 questions · roughly 1M tokens per question
- Retrieval
- Hit@5 35.84% · Hit@10 44.0% · Recall@10 strict 16.16% · Recall@10 micro 8.86% · MRR 0.2546
- Answering
- Governed composite 61.1 · core 60.2, band 59.0 to 60.6
- By category
- Composite / core: abstention 75.7 / 74.3 · contradiction 62.1 / 64.7 · extraction 71.8 / 82.1 · summarization 56.4 / 31.9
- Core, all ten
- preference 83.2 · extraction 82.1 · instruction 77.8 · abstention 74.3 · contradiction 64.7 · temporal 63.5 · knowledge-update 63.4 · multi-session 58.6 · summarization 31.9 · event-ordering 26.0
BEAM scores are reported on its own 0 to 100 scale, not as percentages. The 1M variant is scored separately from BEAM 100K and the two are never compared.
BEAM asks a question whose answer is one line inside roughly a million tokens. Needle search at that scale is a different problem from the other two tests, and competing systems generally do not publish retrieval figures at this scale at all.
At one million tokens, sending the full history to a model becomes slow, costly and noisy. The system must keep important state and find the right evidence without loading everything again.
What this one stresses: context economy. At a million tokens the answer has to come out of structured state and a bounded search, because the history no longer fits anywhere useful.
Retrieval is identical across every answering configuration above. The agentic path reads exactly the same ranked evidence the direct path does. All retrieval figures are identity-scored, with no credit for text matching, at published budgets of 100/300/100.
What is measured, and what is not
| Path | Status | The condition |
|---|---|---|
| Signed intake, distillation, claims with evidence and supersession | Measured | Exercised by every benchmark run |
| Projections and retrieval at published budgets | Measured | Exercised by every run; identity-scored, no credit for text matching |
| Coverage states in the answer envelope | Measured | The abstention figures depend on them; priced by the never-refuse probe |
| Pythia loop (ask / compute / extract) | Measured | The agentic answering row on every benchmark |
| Per-brain isolation, fails closed | Measured | Adversarial probe on live per-brain servers: 45 of 45 refused or contained |
| Identity resolution and permission-change propagation | Measured | On a bounded fixture: a 1,000-person synthetic directory. Live directories, aliases at scale and reorganizations not exercised |
| Graduated disclosure ladders | Measured | Rungs derived at write time, never redacted at read time; measured end to end over the wire: a finance reader’s envelope carries the precise value, a sales reader’s carries the band and trend with no byte of the precise value; 18 integration tests |
| Task-scoped context, the two-stage ask | Measured | Shipped and drilled over the wire, measured by mechanism: batched asks under one assertion and snapshot, multi-subject context, validity tokens that revalidate stale after the first change. No public benchmark exercises it |
| Execution at endpoints (Hero, Ariadne) | Specified | Designed and documented; independent review of the execution boundary is on the validation list |
| Identity at live-directory scale; operational outcomes in a live organization | Not measured | The fixtures are controlled, not live; the pilot is the next validation step |
| Independent security review; a second open-weight model; BEAM 1M under the local placement | Planned | Listed in the paper as what would change the assessment |
What the metrics mean
- Hit@5 / Hit@10
- Whether at least one required piece of evidence appears in the first five or ten retrieved results.
- Recall@10 strict
- The share of questions for which all required evidence appears in the first ten results.
- Recall@10 micro
- The share of all required evidence found in the first ten results across the whole test.
- MRR
- How highly the first relevant result is ranked.
- Accuracy
- How often the final answer is correct after search and answer generation.
- Abstention
- How often the system correctly avoids answering when the evidence is not enough.
The methodology appendix carries the full account: dataset hashes, model pins, judge prompts, retrieval budgets, aggregation formulas, the noise floor and the dated corrections log, plus the per-question run reports behind every figure above. It publishes with the paper.
These scores show how one setup performed on specific tests. No benchmark covers every part of company memory. The results also do not prove that Acropolis is “the best” with every model, workload and test method. That claim would need much broader evidence.
Measured, not priced
The cost of record.
What it costs to build and hold the record, from the 2026-08-20 BEAM ingest run: two Xeon 8280s, no GPU.
Zero submissions quarantined on the clean run. No embedding vendor and no GPU in those machines. Local tokens are priced at the frontier list rate in every cost figure, so the rows are comparable – the real local cost is GPU time, and the paper reports it as latency.
One line from the paper is worth keeping. Distillation is deliberately lossy, and the system still puts the right evidence in the top ten on 99.8% of LongMemEval questions. Compression that keeps the answer is a system property, not a retrieval trick.
Installed where the company runs
Deployment, hardware and data access.
Acropolis does not need a memory service controlled by another vendor or a large, specialist GPU cluster. The full stack runs on the client’s own hardware.
The operational model, search indexes, identities, policies and execution tools stay inside the client-controlled environment.
When inference must stay local, tuned LLMs can run on consumer hardware. Inference is the step where a model processes a request and produces an answer. A full local setup falls in a hardware class below $100,000, depending on the workload, number of users and backup needs. It does not require a seven-figure GPU cluster or permanent use of the most expensive models.
The benchmark tests run on local Nomic embeddings throughout. The August figures used low-cost LLMs; the September campaign ran an open-weight 31B model in every model seat on one GPU per bench machine, with a frontier model rented per question in the rent-the-thinking rows. The scores do not depend on sending the full history to a frontier model and asking it to solve the memory problem by force.
Data is not placed in one shared pool with a permission check at the end. RBAC segregation applies when data is stored, retrieved and assembled into context, so data from one role, agent or workflow cannot leak into another.
The whole stack runs on hardware you already know how to buy, under access rules you already have.
The limits
What this doesn’t prove yet.
Everything below is stated in the paper before a reviewer has to ask. We would rather say it first.
- Fixtures, not live organizations. All identity, permission and task-scoped results are measured on controlled fixtures – a generated organization of 1,000 people, a synthetic directory, scripted changes – not on a live company’s directory under operational load.
- The datasets are not enterprise-shaped. All three are conversational-memory benchmarks, chosen because they are public, reproducible and already used by the systems we compare against. No corpus or question was authored by VarOps.
- One model, one GPU class. The local placement was measured on one open-weight model on one GPU class. No claim is made about other models or smaller hardware.
- Two negative results, reported rather than buried. A local cross-encoder reranker was predicted to close the local retrieval gap and widened it instead: Hit@5 52.5 against 80.6 for the local language model on the same pool. Prompt contracts written against observed failures did not move scores: 73.0 to 72.2 and 79.6 to 77.6 on paired full runs, with the refusal count unchanged.
These are the next validation steps, not assumed capabilities – and they are worth treating that way when pricing risk and rollout.
From the engineering bench
What produced the numbers.
Own the Knowledge, Rent the Thinking.
The whitepaper every figure on this page is traced to. The placement experiment, the guard columns, the never-refuse probe, the cost of record and the limitations, stated with their conditions. Also as a PDF.
Benchmark methodology appendix
How the figures were produced, in enough detail to re-run them from the committed harness: datasets and hashes, model pins, judge prompts, budgets, aggregation formulas, the noise floor and the dated corrections log.
The report bundle
The per-question run reports behind every published figure, published alongside the paper with a manifest recording datasets, configurations, model pins and code revisions per report.
Cite as: Aroussi, R. (2026). Own the Knowledge, Rent the Thinking. Separating institutional understanding from frontier intelligence. Acropolis Technical Report 01, version 1.0. VarOps LLC. https://doi.org/10.5281/zenodo.22775192
How we use it
Infrastructure behind the work.
We built Acropolis and the wider stack because the available tools were not good enough for the work our clients needed.
Agent frameworks were not built for company-scale systems. We built MUXI. Giving every agent the same broad access was unsafe. We built agents with role-based access control. Engineering teams needed a way to coordinate and control many coding agents. We built Cloop, an engineering control plane. Most agent systems treated email as an add-on. We built Emailgent, email designed for agents.
Most memory systems stored chunks of text and searched them later. We needed a reliable model of the company’s current state, so we built Acropolis. These are not separate side projects. They come from one engineering principle, applied across a stack that has taken two years to build: when the basic design is wrong, another integration will not fix it.
Acropolis is installed as part of a VarOps engagement. It runs on the client’s hardware and follows the client’s access rules, and we understand the system down to its boundaries because we built it. It is not a separate software purchase, and we do not force it into work where it does not belong.
Frequently asked questions
The questions engineers ask first.
Is Acropolis SOC 2 or ISO 27001 certified?
Acropolis is self-hosted enterprise software, not a hosted SaaS service. It runs within your infrastructure and inherits the security and compliance controls of that environment.
We provide the architecture, security documentation and assessment materials your security and compliance teams need to evaluate Acropolis for deployment within environments governed by frameworks such as SOC 2, ISO 27001, HIPAA and PCI DSS.
Does VarOps have access to our data?
Not by default.
Your company data, operational knowledge, telemetry and AI interactions remain within infrastructure you control. Acropolis does not require an external VarOps data plane or telemetry service.
Where VarOps personnel require temporary access for implementation, maintenance or support, that access is provided through your existing access controls and according to your policies.
Telemetry works the same way. Operational metrics stay in your environment, and we review them by logging in through your access controls. There is no export.
Can Acropolis operate without sending data to external AI providers?
Yes.
Acropolis can use the model infrastructure appropriate to your security requirements, including approved external providers, private model endpoints or locally hosted models.
For environments requiring complete isolation, the stack can be deployed on local models only, and no company data leaves your security perimeter.
Is Acropolis a knowledge base or a RAG system?
No. A knowledge base stores information. RAG finds passages that may help a model answer a question. Acropolis tracks company state: what is true now, what used to be true, how facts relate, where they came from and who may use them. Search is one part of Acropolis, not the whole system.
Can it answer questions that aren’t recorded anywhere?
Yes, within a limit. Pythia can work out an interval, a count, a summary, or a “no” from evidence that was searched and came back empty. What she cannot do is invent a fact. Every derived answer carries the stated facts it was built from, so you can check the working.
Does Acropolis take actions?
Acropolis provides the state and context needed to plan an action. It does not carry out the action itself. Hero coordinates approved work, and Ariadne carries it to endpoints. Memory and action stay separate.
Hero does. Parthenon never does. Pythia sits on the acting side of the line but has no tools that touch anything – she asks questions, computes over the answers and extracts from them, and that is the whole list.
How are permissions enforced?
Every request includes an identity, role, policy and level of autonomy. Role-based controls decide which data can enter the context, not only what appears on the screen. Acropolis may hold a fact that some users and agents cannot retrieve. Being allowed to read a fact also does not grant permission to act on it. Permissions are explicit, and every action has a clear owner and record.
Where does Acropolis run?
The full stack runs on the client’s own standard hardware. The company model and its access controls stay inside the client-controlled environment. Acropolis does not depend on a shared memory service hosted by another vendor.
How does Acropolis fit with HIPAA, SOC 2 and GDPR?
Acropolis runs inside the client-controlled network. Client data, the company model, search indexes and access policies stay there. VarOps has no standing, admin or support access to that data. The system does not copy client information into a service hosted by VarOps.
This reduces the number of outside systems and providers that may handle the data. It does not, by itself, make a company compliant. The HIPAA Security Rule requires administrative, physical and technical safeguards for electronic health information. SOC 2 is an independent review of controls for areas such as security, availability, confidentiality and privacy. GDPR duties depend on what data is processed, why it is processed, who controls it and which safeguards are in place.
Acropolis works inside the client’s existing controls: its network, identities, roles, retention rules, audit tools and incident procedures. We do not claim that this design creates automatic HIPAA, SOC 2 or GDPR compliance. The client must still confirm the rules and legal roles that apply to each deployment.
What happens when two sources disagree?
Acropolis keeps both observations and their sources instead of pretending that one is certain. It can use the authority of each source, the time each fact applied, later evidence and company policy to resolve the conflict. If the evidence is not enough, it records uncertainty or asks a person. It does not invent an answer.
Which benchmark is the right one for agent memory?
There is no single best benchmark. Each one tests different ways a memory system can fail.
LongMemEval-S asks 500 human-written questions over a multi-session chat history, including 30 unanswerable traps. LoCoMo tests memory across very long multi-session conversations; we run its published sample of 435 questions, 88 of them unanswerable. BEAM tests ten abilities over long conversations, at about 100,000 tokens (400 questions) and at about one million (700 questions). At those sizes, handling the amount of context becomes part of the test.
A system can be tuned to score well on one test and still fail outside it. That is why we report LongMemEval-S, LoCoMo and BEAM 1M together. We also publish search metrics, answer accuracy, and abstention where we have measured it.
The right benchmark depends on the job. Testing several types of failure with the same published setup gives a more reliable picture.
Can we buy Acropolis?
No. Acropolis is installed as part of a VarOps transformation engagement. It is not a separate software purchase.
Can the benchmark results be reproduced?
Yes. The whitepaper publishes with its methodology appendix: the committed harness, model pins, dataset hashes, judge prompts, retrieval budgets and aggregation formulas, plus the per-question run reports behind every figure and a manifest recording each report’s hash and code revisions. The few figures whose historical artifacts are only partially retained are marked as such in the paper rather than silently presented.
What to ask for next.
Architecture review, supporting security documentation and, under NDA as part of procurement, source code access for independent security assessment.
Acropolis is installed as part of a VarOps engagement,
not sold separately.
See the transformation model.