Write-Time Trust Gating for LLM Memory Systems

Abstract

Most memory systems built for LLM agents trust whatever the agent or a tool call reports and store it as-is, relying on retrieval-time ranking or manual cleanup to manage errors later. This paper evaluates a different design point: gating writes before they enter memory, using a combination of source provenance, evidence corroboration, and conflict detection against already-established facts. We build a 113-case labeled evaluation set covering twelve distinct failure modes and measure a rule-based gating system across four configurations. The baseline system achieves 69.9% accuracy at correctly accepting or rejecting a candidate memory write. Adding a source-kind-aware evidence floor raises this to 87.6%. Adding a bi-temporal conflict check (lexical, then embedding-based) raises it further to 98.2%. We report inter-annotator agreement on our ground truth (Cohen's kappa = 0.475, moderate agreement) and find that agreement is high on process-based judgments and meaningfully lower on interpretive judgments — a finding we treat as substantive rather than a flaw in the methodology. We report every failure mode we found, including two dataset construction bugs discovered during development and one case that remains unresolved against a pre-registered threshold.

1. Introduction

Systems like mem0 and supermemory manage what an LLM-based agent remembers by ranking or consolidating memories after they've already been stored. The agent (or a tool it calls) asserts something, and the memory layer's job is to retrieve the right thing later, not to question whether the assertion should have been stored in the first place. This works reasonably well when memory contents are mostly correct, but it puts the entire burden of catching bad, contradictory, or stale information on retrieval-time mechanisms, which have to work harder the longer a memory store accumulates errors.

This paper evaluates the alternative: filtering writes before they happen. A write-time gate has less context than a retrieval-time ranker — it sees one candidate memory and whatever provenance accompanies it, not the full set of things already known — but it has one advantage a ranker doesn't: the chance to simply refuse to store something in the first place, rather than trying to rank around it forever.

We make three contributions:

1. A write-time gating architecture combining source-provenance weighting, evidence-count corroboration, and bi-temporal conflict detection, built as an extension to an existing rule-based memory system (TIMPS memory-core).
2. A 113-case labeled evaluation set spanning twelve failure categories, with ground truth assigned via documented, reproducible rules rather than ad hoc judgment, and inter-annotator agreement measured across two rounds.
3. A measured, incremental improvement path — three independent fixes, each targeting a distinct diagnosed failure mode, raising overall accuracy from 69.9% to 98.2% without changing recall on cases that should be accepted.

2. System Design

2.1 ConstitutionalGuard — The Base Gate

The gate is deterministic and rule-based, not learned. A candidate write is rejected if any of the following hold: no provenance is attached; stated confidence falls below a threshold; the source kind's evidence count falls below a floor (see 2.3); or the write has already been flagged as contradicting three or more other entries. Otherwise, it is accepted. This simplicity is deliberate — a rule-based gate is auditable and its failure modes are traceable to a specific rule, which matters for a system meant to be trusted with deciding what gets remembered.

2.2 ProvenanceForge — Where Trust Inputs Come From

Each candidate write carries a provenance record: a source kind (user_direct, doc_reference, git_history, tool_output, agent_pattern, web_search, cross_project, or agent_inference), an evidence count, an age, and a stated confidence. Source kinds are not treated as equally trustworthy — a directly stated user fact and an unsupported agent inference do not clear the same bar.

2.3 Source-Kind-Aware Evidence Floors

The original gate applied a single evidence-count minimum regardless of source kind. A single web search result and a single directly-stated user fact were held to the same standard. We introduced per-source-kind floors: web_search requires at least two corroborating instances, cross_project requires at least three. This single change resolved three of our twelve evaluation categories almost completely (Section 4.3), at a documented cost of one new false negative discussed in Section 6.

2.4 ChronosForge.checkConflict() — Detecting Conflicts with Established Facts

The base gate has no mechanism for recognizing that a candidate write restates something already superseded by a more current, more reliable fact. We added checkConflict(), which compares a candidate write against currently-valid memory nodes in the same domain and flags a conflict when both (a) textual or semantic similarity exceeds a threshold and (b) the existing node is scored more reliable than the incoming write. We implemented and evaluated three similarity signals, in increasing order of sophistication:

• Character-trigram Jaccard similarity — catches near-identical restatements.
• Word-level Jaccard similarity on meaning-bearing tokens — catches paraphrases that share vocabulary despite different sentence structure.
• Embedding cosine similarity, using the system's existing embedding pipeline (Ollama, nomic-embed-text) — catches paraphrases that share meaning but not vocabulary.

Each signal is checked in order; if any one exceeds its threshold against a more-reliable existing fact, the write is flagged as conflicting.

3. Evaluation

3.1 Dataset

We built a 113-case dataset across twelve categories, each targeting a distinct failure mode: well-sourced direct claims, single-source web claims, unsupported agent inference, stale facts, contradicted facts, claims with no provenance, claims from a weak source kind but with heavy corroboration, confidently-phrased but unsupported claims, cross-project claims, claims at the numeric confidence boundary, well-corroborated inferences, and claims that restate a recently superseded fact using different wording. Ground truth for each case was assigned according to an explicit, documented rule per category, not individual judgment call by call.

3.2 Inter-Annotator Agreement

A second labeler judged all 113 cases independently, first without a written rubric, then again after receiving one. Round 1 produced Cohen's kappa = 0.292 ("fair agreement"), with disagreement concentrated almost entirely in two categories: claims with no provenance (86% disagreement) and stale facts (80% disagreement). The pattern indicated the second labeler was judging claims on face-value plausibility rather than the process-based question the task asks — a rubric-communication failure rather than evidence the underlying criteria were incoherent.

Round 2, with a written rubric, produced Cohen's kappa = 0.475 ("moderate agreement," raw agreement 73.5%). This is the number we report. Improvement was not uniform: categories governed by checkable process rules (no provenance, evidence-count floors, contradiction counts) resolved almost completely (from 6/7 and 4/7 disagreement down to 0/7 and 1/7 respectively). Categories requiring interpretive judgment — is a claim stale relative to a prior fact, is confident phrasing sufficient evidence on its own — remained contested even with the rubric provided (stale-fact disagreement actually rose slightly, from 8/10 to 9/10). We report this as a substantive finding: process-based trust criteria are learnable from a short written rule; judgment-based criteria about staleness and confidence are not fully resolved by writing more rules down, and may require either worked examples or accepting some irreducible disagreement.

We also discovered, while reviewing round 2, that our own rubric (specifically its evidence-floor rule) directly contradicted a deliberate exception in our own ground truth for one case (a single-source government data feed). The rubric was corrected after the fact but not re-tested; we note this as a limitation of the rubric-writing process itself.

3.3 Results

We measured four configurations against the identical 113-case dataset:

The overall improvement, 69.9% to 98.2% (+28.3 points), reflects three independent, additive fixes, each targeting a distinct, previously-diagnosed failure mode, rather than a single tuned change.

3.4 The Evidence-Floor Fix Traded One False Negative for Three Fixed Categories

Recall dropped slightly (100% to 98%) between the baseline and the evidence-floor configuration. The cause: a single-source government weather data feed, which our evidence floor treats identically to a single-source opinion blog, both being web_search. We consider this an honest trade-off — three categories (single-source web claims, cross-project claims, and confidence-boundary claims) improved substantially, at the cost of one specific, now-documented false negative — rather than a bug to silently patch before publication.

3.5 Closing the Paraphrase Gap: Lexical Methods, Then Embeddings

Lexical conflict detection alone (trigram plus word-level Jaccard) raised accuracy to 96.5%, resolving most, but not all, of the "stale fact" and "recently superseded" categories. Two cases remained unresolved after lexical fixes, both requiring genuine semantic understanding rather than vocabulary overlap: their word-level Jaccard scores (0.222 and 0.167) fell below our calibrated 0.25 threshold because the conflicting statements shared too few actual words despite expressing the same underlying contradiction.

We wired a real embedding-based similarity check into the same conflict detector, using the memory system's existing embedding pipeline (Ollama, nomic-embed-text) rather than a synthetic or mocked similarity function. Cosine similarity was computed between each candidate write and its conflicting prior fact. We pre-registered a similarity threshold of 0.75 before running this evaluation. One case scored 0.761 and was correctly caught. The other scored 0.724 — below threshold, and left unresolved.

We deliberately did not lower the threshold to catch this second case after seeing its score. Doing so would mean selecting an evaluation criterion after observing which specific known case it needed to flip, which fits the metric to this particular test set rather than evaluating against a criterion set independently of the data. We report 98.2% as the honest number rather than a higher one obtained by post-hoc threshold adjustment. A properly tuned threshold would require a held-out validation set distinct from the 113 cases reported here.

During development of this fix, we also discovered and corrected two dataset construction bugs unrelated to the detection algorithm itself: three cases were missing the seeded "prior fact" needed for any conflict-detection method to have something to compare against, and one seeded prior fact was assigned a lower reliability score than the incoming (stale) claim's own stated confidence, causing the conflict rule to correctly, but unhelpfully, treat the incoming claim as the more trustworthy one. We note this explicitly because an eval failure that looks like an algorithm limitation can, on inspection, turn out to be a dataset error — a distinction worth checking before drawing conclusions from a low score in either direction.

4. Related Work

Retrieval- and consolidation-focused memory systems. Mem0 (Chhikara et al., 2025) extracts salient facts from conversation turns and maintains them in a vector store; for each candidate fact, an LLM-based controller decides whether to add, update, delete, or discard it relative to similar existing memories, with an optional graph-based variant for relational structure. This places the correctness burden on an LLM call made at write time, without an explicit, auditable rule set governing that decision. Supermemory positions itself similarly — automatically extracting facts, building user profiles, and explicitly advertising that it "handles knowledge updates and contradictions" and "forgets expired information" — but, like Mem0, the mechanism by which conflicting or stale information is detected is not published as an evaluable, rule-based procedure; both systems report benchmark standing on retrieval-quality benchmarks (LongMemEval, LoCoMo, ConvoMem) rather than on write-time correctness specifically. MemGPT/Letta (Packer et al., 2023) treats the LLM's context window as an OS-style virtual memory problem with tiered storage, but — as later work notes (Wei et al., 2025) — provides no principled mechanism for confidence estimation or write-time rejection. A-Mem (Xu et al., 2025) is closer in spirit to our conflict detection design: it links related memory notes at insertion time and performs a form of "memory evolution" that can retroactively revise related entries, which is a different mechanism for handling supersession than the explicit, threshold-based conflict check we evaluate here, but addresses a similar underlying problem.

Learned memory-management policies. A separate line of work trains the memory-management decision itself via reinforcement learning rather than fixing it as a rule set. MEM1 (Zhou et al., 2025) and MemAgent (Yu et al., 2025) train models to rewrite a running text memory as new information arrives. Memory-R1 (Yan et al., 2025) trains a memory manager to choose between add/update/delete operations, reporting that RL training corrects cases where a naive manager would fragment a fact into contradictory entries rather than issuing a single consolidating update. Mem-α (Wang et al., 2025) extends this line of work to more complex, heterogeneous memory schemas, training an agent to construct and maintain multiple memory types jointly via a reward tied to downstream question-answering accuracy. These systems address a genuinely different problem than the one evaluated in this paper: they learn how to update memory well, given that a write will happen, whereas our gate asks a prior question — whether a write should happen at all, before any update policy is invoked. The two approaches are compatible rather than competing; a learned update policy could sit behind our write-time gate rather than instead of it.

Bi-temporal data modeling. ChronosForge's valid-time/transaction-time structure is not a novel invention of this system — it is a direct application of bi-temporal database theory, originating with Snodgrass and Ahn's (1985) formal distinction between valid time (when a fact was true in the world) and transaction time (when a fact was recorded in the system), later standardized in SQL:2011 and elaborated at length in Snodgrass's own work on temporal database design. Our contribution here is narrow relative to that literature: applying the same valid-time/transaction-time distinction to individual LLM-agent memory writes, and using it as an input to a write-time trust decision, rather than proposing a new temporal data model.

Where this paper sits relative to this work. To our knowledge, no existing published system evaluates write-time rejection specifically — as a distinct, measurable decision separate from retrieval ranking or a learned update policy — against a labeled ground-truth set with inter-annotator agreement reported. That is the gap this paper's evaluation methodology, more than its specific architecture, is intended to fill.

5. Limitations

We list these directly rather than deferring them to a brief closing paragraph, since several of them materially affect how the headline number in Section 3.3 should be read.

1. 113 cases, not the 100+ per category we initially aimed for. The dataset is real and hand-reasoned, but its size should not be rounded up in how it's described.
2. Inter-annotator kappa of 0.475 is moderate, not strong. It reflects near-total agreement on process-based rules and real, unresolved disagreement on interpretive ones (Section 3.2). We report this pattern explicitly rather than treating the aggregate kappa as the whole story.
3. One case (of 113) remains unresolved against a pre-registered embedding similarity threshold (0.724 versus 0.75). See Section 3.5 for why we did not adjust the threshold to resolve it.
4. Source-kind-aware evidence floors do not yet distinguish source authority within a kind. A single-source government data feed and a single-source opinion blog are currently treated identically under web_search. Documented in Section 3.4 as a deliberate, known trade-off.
5. The gating system is entirely deterministic and rule-based, not learned. This is a design choice made for auditability, not a limitation to be apologized for, but it should not be described in terms that imply machine-learning sophistication the system doesn't have.
6. Both the system and its initial ground truth were designed by the same author (with AI assistance). The second-labeler process in Section 3.2 partially addresses this, but does not eliminate the underlying bias of one person having defined both what is correct and what the system should do.

6. Future Work

• A properly held-out validation set for tuning the embedding-similarity threshold, distinct from the 113 cases used for the headline evaluation.
• Source-authority-aware evidence floors, distinguishing (for example) institutional data feeds from opinion sources within the same source kind.
• A calibration process — beyond a written rubric alone — for the interpretive judgment categories (staleness, confidence-versus-evidence) where inter-annotator agreement remained low even after Section 3.2's rubric was introduced.
• Extending conflict detection to multi-agent write contention, where multiple agents may write near-simultaneously to a shared memory store — a distinct evaluation problem from the single-writer setting studied here.

7. Conclusion

A deterministic, write-time gate combining source provenance, evidence corroboration, and bi-temporal conflict detection improved write-acceptance accuracy from 69.9% to 98.2% on a 113-case evaluation set, through three independently diagnosed and independently fixed failure modes. Agreement between two independent labelers on the underlying ground truth was moderate (kappa = 0.475), stronger on process-based judgments than interpretive ones. One case remains unresolved against a threshold set before the result was known, and one deliberate trade-off remains documented rather than patched. We report these as the actual, current state of the system, not as issues resolved for the purpose of a cleaner result.

References

Chhikara, P., Khant, D., Aryan, S., Singh, T., & Yadav, D. (2025). Mem0: Building Production-Ready AI Agents with Scalable Long-Term Memory. arXiv preprint arXiv:2504.19413.

Packer, C., Wooders, S., Lin, K., Fang, V., Patil, S. G., Stoica, I., & Gonzalez, J. E. (2023). MemGPT: Towards LLMs as Operating Systems. arXiv preprint arXiv:2310.08560.

Snodgrass, R. T., & Ahn, I. (1985). A taxonomy of time in databases. Proceedings of the 1985 ACM SIGMOD International Conference on Management of Data.

Wang, Y., Takanobu, R., Liang, Z., Mao, Y., Hu, Y., McAuley, J., & Wu, X. (2025). Mem-α: Learning Memory Construction via Reinforcement Learning. arXiv preprint arXiv:2509.25911.

Xu, W., Liang, Z., Mei, K., Gao, H., Tan, J., & Zhang, Y. (2025). A-MEM: Agentic Memory for LLM Agents. arXiv preprint arXiv:2502.12110.

Yan, S., Yang, X., Huang, Z., Nie, E., Ding, Z., Li, Z., Ma, X., Bi, J., Kersting, K., Pan, J. Z., Schütze, H., Tresp, V., & Ma, Y. (2025). Memory-R1: Enhancing Large Language Model Agents to Manage and Utilize Memories via Reinforcement Learning. arXiv preprint arXiv:2508.19828.

Yu, H., Chen, T., Feng, J., Chen, J., Dai, W., Yu, Q., Zhang, Y.-Q., Ma, W.-Y., Liu, J., Wang, M., et al. (2025). MemAgent: Reshaping Long-Context LLM with Multi-Conv RL-based Memory Agent. arXiv preprint arXiv:2507.02259.

Zhou, Z., Qu, A., Wu, Z., Kim, S., Prakash, A., Rus, D., Zhao, J., Low, B. K. H., & Liang, P. (2025). MEM1: Learning to Synergize Memory and Reasoning for Efficient Long-Horizon Agents. arXiv preprint arXiv:2506.15841.

Supermemory (2026). Supermemory: Memory API for the AI era. Product documentation, https://supermemory.ai/docs.

Citation

Published by TIMPS. For questions about this research or the TIMPS memory system, contact timps.ai090@gmail.com .