Calling it memory made agents better at memory

What Napkin is

Napkin is a local-first, file-based knowledge system for AI agents. It stores notes as markdown in an Obsidian-compatible vault and gives agents a CLI and SDK to search, read, and write them.

The design principle is progressive disclosure: instead of dumping the full vault into context, the agent starts with a short overview (~200 tokens), then searches, then reads specific files. This keeps token usage low while still giving the agent access to everything.

On LongMemEval — a long-term memory benchmark from ICLR 2025 — Napkin with a frontier model already matches or beats the best published systems (91.0% on the S split, vs 86% prior best). It does this with zero preprocessing: no embeddings, no graph database, no fact extraction pipeline. Just BM25 search on markdown files.

That's the baseline. The question for this post is whether a surface rename could push it further.

Napkin scored 87.6% accuracy on the full LongMemEval_S split with 89.0% session recall, running a local Qwen3.6 model.

But I wondered whether the name was working against the model. "Napkin" is a fine project name, but it doesn't describe what the agent is doing. The agent is using memory.

I ran a test:

If every model-visible surface says memory instead of napkin, does the agent perform better on a memory benchmark?

Same model, seed, dataset, and concurrency. The answer was yes.

This isn't a rename announcement. It's an interface experiment.

The largest gains came from multi-session reasoning, temporal reasoning, and knowledge updates — the parts that most look like memory.

What LongMemEval_S tests

LongMemEval is a long-term chat assistant memory benchmark. The S split gives each question a sizeable history, roughly dozens of sessions and around 115k tokens of conversation. The agent has to retrieve the right pieces and answer questions about old user interactions.

The benchmark covers five abilities:

A memory system can look good on simple retrieval and still fail in practice. Users don't ask "find note 2023-11-30/round-6." They ask "how many postcards have I added since I started collecting again?"

That needs search, reading, temporal context, and sometimes arithmetic. The model has to treat the archive as memory, not a folder of markdown files.

The baseline was already strong

The baseline was the existing Napkin surface:

npx tsx bench/longmemeval-eval.ts \
  --dataset s \
  --n 500 \
  --concurrency 2 \
  --seed 42 \
  --json \
  --model "cyankiwi/Qwen3.6-35B-A3B-AWQ-4bit:xhigh"

That run finished with:

That's already strong for a local agentic memory system — no embeddings, no graph database, no fact extraction pipeline. The benchmark creates markdown notes, injects an overview, and lets the agent use search and read tools.

The question was whether the interface name was making the model work against a confusing metaphor.

The memory variant changed names, not retrieval

For the variant, I changed the model-visible surface to memory:

The underlying retrieval stayed the same — same BM25, note splitting, dataset, model, and seed. The memory extension still calls the same Napkin library methods for overview, search, and read. For compatibility, the benchmark creates the model-facing .memory directory and keeps a .napkin symlink for the unchanged library path.

I changed what the agent sees.

The memory run used:

npx tsx bench/longmemeval-eval.ts \
  --dataset s \
  --n 500 \
  --concurrency 2 \
  --seed 42 \
  --surface memory \
  --json \
  --model "cyankiwi/Qwen3.6-35B-A3B-AWQ-4bit:xhigh"

The memory surface scored 90.6%

The memory run finished with:

Memory gained 3.0 accuracy points and 2.5 recall points.

Session F1 dropped, though. Memory found the answer more often but wasn't more precise about which sessions it pulled evidence from. It retrieves more useful context, not a tighter evidence set.

Retrieval metrics aren't the product metric. Users care whether the answer is right. But engineers should still watch evidence quality — broad retrieval can hide fragility.

The gains showed up where semantics should matter

The strongest evidence for the hypothesis is the ability breakdown.

Information extraction dipped slightly. When the task is just "find the fact," the interface name probably doesn't matter.

The big gains were multi-session and temporal reasoning — where the model has to treat the corpus as personal history, not a pile of documents.

The matched-question flips tell the same story:

The 35 memory wins were concentrated in multi-session reasoning:

About as clean as you can get from a single noisy benchmark run.

Why a name can change behavior

LLMs aren't API clients. They bring a lot of associations into every decision, not just function signatures.

"Napkin" works as a product name, but inside the model's action loop it's abstract. A human learns that Napkin means memory. The model gets a short prompt, a tool list, some snippets, and a few turns. If the interface says "napkin vault," the model has to figure out what that means.

"Memory" does not require that translation.

This shows up in small ways:

• the model may search with different terms
• it may decide to read more context before answering
• it may treat conflicts as memory updates instead of document contradictions
• it may be more willing to answer from prior user history
• it may be more careful with temporal references

No magic keyword makes an agent smart. But naming is part of the prompt. Tool names are part of the prompt. File paths are part of the prompt. If the agent reads it, it steers.

Same reason memory_search is a better tool name than kb_search for a memory benchmark. kb_search is generic. memory_search tells the model what the tool is for.

What changed operationally

The memory variant changed these model-visible surfaces:

The search and read implementation stayed the same. The memory extension still uses the same Napkin library. The benchmark kept the same question sample with --seed 42.

This was a surface rename for the model, not a product rename.

The caveats are real

This is one model:

cyankiwi/Qwen3.6-35B-A3B-AWQ-4bit:xhigh

Different models may respond differently. Frontier models may infer the metaphor easily. Smaller models may benefit more from literal names. Tool-trained models may be especially sensitive to function names.

The judge isn't independent either. LongMemEval uses the same model for the agent and the judge, so read the absolute accuracy as "accuracy under this judge," not ground truth.

For this A/B, the same judge scored both surfaces, so the relative comparison holds. Before making stronger claims, I'd rejudge both logs with a separate model.

Both runs completed all 500 questions with zero errors, so the accuracy comparison is clean — no confound from different sample sizes.

Latency is noisy. The memory run used less total agent time (5592s vs 7342s across questions), but local model server conditions vary. I wouldn't claim the speedup until it reproduces.

The accuracy gain is the signal worth tracking.

What I would test next

First: rejudge with a stronger independent model. The logs already have the question, gold answer, and agent answer, so the expensive agent part doesn't need to rerun.

Second, run the same A/B across more models:

Third: isolate the pieces. The memory variant changed the whole surface at once, which was right for a first test. Next experiments should split it apart:

1. prompt wording only
2. tool descriptions only
3. tool names only
4. .memory and MEMORY.md path only
5. full memory surface

That would tell us whether the gain came from the word "memory" in the system prompt, the tool names, the path, or the combined consistency.

The practical lesson

Interface names are prompts.

A tool called search_archive and a tool called memory_search may call the same function, but they mean different things to the model. The model is always interpreting the interface.

Napkin is still a good product name — short, unusual, memorable. But inside the agent loop, the word "memory" is a better steering surface, at least for this model on LongMemEval_S.

Memory systems should look like memory to the model.

Keep the project name. I'm keeping mine. But when the agent reads the tool list, the vault path, and the instructions — call the thing what it is.