How I lost my agents' memory to four locks, one manual and a bug. And how to get it back.
Investigation
Bernhard Rieder/
2026-09-13/
5 MIN READ
|
It's NemoClaw's fault.
I asked my agent a simple question this afternoon: "What did we decide about the render farm?"
It had the answer. It had written it down itself, two days ago, in its own notebook. And it could not find it. Because the note said "GPU cluster" and I said "render farm". The agent that NVIDIA sells me as "remembers everything" searches its memory like a 1998 text editor: exact words or nothing.
That is not a bug I introduced. That is the state NVIDIA ships. Let me show you the crime scene.
Every claim below was reproduced today on this Spark. The raw commands and outputs are attached to
the GitHub issues I filed with both projects.
What smart memory needs, in one breath
An agent that really remembers needs two things. A notebook it can write into. And a librarian who turns every page into a fingerprint, so "render farm" and "GPU cluster" land next to each other. The librarian is an embedding model. Ours is qwen3-embedding:4b, it is tiny, it runs in Ollama right next to the chat model, and it is installed and loaded on this Spark right now. Idle. Waiting. Unreachable.
So: how do you make smart memory work with NemoClaw, OpenClaw and Ollama with vector embeddings? Short answer, as of today: you don't. Not on the path NVIDIA documents. Here is why, lock by lock.
The house
The house: one room for the agent, a chat-only hallway, a welded kitchen door, and a librarian
who cannot be reached.
NemoClaw builds a house around the agent and owns every door. The agent sits in one room. The models cook in the kitchen: Ollama on the host, with the chef (gemma4:26b) and the librarian (the embedding model).
From the room to the kitchen there used to be two ways. A hallway that NVIDIA built, inference.local, guarded by a proxy with a password. And a kitchen door, the direct line to Ollama on port 11434.
>
The house, the room, the hallway and the locked kitchen door — NemoClaw owns every door. The
librarian sits in the kitchen. The agent cannot reach him.
Lock 1: they welded the kitchen door shut
The 0.0.123 update rewrote the Ollama settings on my machine. OLLAMA_HOST=0.0.0.0 became OLLAMA_HOST=127.0.0.1. From inside the room the door now answers upstream_unreachable. And it is not a one-time slip: every time NemoClaw touches the sandbox, it writes that line again. You cannot even leave the door open on purpose.
Lock 2: the hallway carries food, not fingerprints
Fine, use the hallway. Ask it for a chat completion: works, fast, cached. Ask it for an embedding: 501 This server does not support embeddings. Ask it the Ollama way, /api/embed: connection not allowed by policy. NVIDIA built one corridor and made it chat-only.
Lock 3: the librarian's phone is switched off
Even with a corridor, OpenClaw needs its bundled Ollama plugin to talk to an embedding model. NemoClaw's image excludes it from the plugin allow list. The gateway says so itself, in its log, every start:
text
1no loaded plugin registered a memory embedding provider … falling back to keyword/FTS-only search.
It knew. It wrote it down. Nobody read it.
Lock 4: the filing cabinet was never delivered
The library that stores fingerprints in the database, sqlite-vec, is not in the image. require("sqlite-vec") → Cannot find module. Without it, a vector table cannot exist. Not "does not work", cannot exist.
The punchline: the manual
NemoClaw 0.0.123 still ships a documentation page titled "Configure Memory Search". I followed it today. It says: reach Ollama from the sandbox at http://host.openshell.internal:11434. That is Lock 1. It says: write the provider with nemoclaw <sandbox> config set. That command was removed in 0.0.120.
>
NVIDIA wrote a manual for a door, then welded the door shut, then removed the key, and shipped the
manual anyway.
Four independent decisions, in four different places, each one alone enough to kill the feature. That is not an accident. That is a feature nobody owns.
And OpenClaw, don't smile
Suppose NVIDIA opened everything tomorrow. Run the one command that builds the fingerprint index, openclaw memory index --force, and OpenClaw 2026.7.1 crashes: "unable to open database file." On a fresh, writable database, owned by the right user, in a writable folder, with /tmp writable. I opened that same file from node's own sqlite module by hand: fine. The crash is inside OpenClaw's reindex routine. So the open-source darling has its own dead bolt on the same door. Lock 5, honorary.
What the update took, and what it gave
Taken: the configuration lock ("Shields"), the validated config get/set, the local image build, restoring older snapshots, and the direct Ollama route. Not one of these is mentioned next to the memory-search page it invalidates.
Given, and I mean it: the agent's notebook is finally writable, changes apply in a second without a lock window, gateway restart behaves, the dashboard forward survives, the token survives a restart. The house got more comfortable. It just lost the library card.
How to actually make it work
The honest recipe. Five things must be true at the same time:
A route from the room to the librarian that carries embeddings
The Ollama plugin allowed
sqlite-vec present
memorySearch.provider pointed at that route
An index command that does not crash
Today: 0 of 5 on the sanctioned path.
What a brave admin can do this afternoon //
Against the design. Allow the plugin: the config is writable now, that is a two-line change.
Install sqlite-vec into the sandbox's own writable folder and point vector.extensionPath at
it. Then choose your poison for the route: give the agent the hallway guard's password so it can
use the side door on port 11435 (a host secret inside the agent's room, regenerated by NemoClaw
at every onboard), or open the kitchen door with OLLAMA_HOST=0.0.0.0 (NemoClaw closes it again
at the next lifecycle command, and the LAN can see your Ollama). Then hope OpenClaw's index
command works in the next release. Every one of these dies at the next update. I know, because I
did the equivalent dance under the old version, and this update erased it.
What NVIDIA has to do — and it is small
Let inference.local carry /v1/embeddings and /api/embed
Allow the bundled Ollama plugin, or ship your own embedding provider for the managed route
Put sqlite-vec in the image
Fix or delete the "Configure Memory Search" page
Raise the default context floor: onboarding wrote a 16k window for an agent whose prompt alone is 9k tokens, which produced "context too large" errors until I raised it by hand
Stop regenerating the gateway token at every container start; that complaint is a year old
What OpenClaw has to do
Fix memory index --force on a plain, writable SQLite file. One bug, one repo.
Why I am not just patching it quietly
Both projects are open source. I could fork NemoClaw, fork OpenClaw, patch five things and be done by Monday. I would also be alone with my fork forever, re-applying patches after every release, which is exactly the life I had under the previous version.
So this goes upstream: an issue with full reproduction for NVIDIA, a bug report for OpenClaw, and pull requests where I can, starting with the manual page. Until then my team's agents keep a notebook they cannot search, on a machine with a librarian sitting idle in the kitchen.
NVIDIA, here is the good news
You are one sprint away from the best local-agent story on the market.
The hardware is there: gemma4 and the embedding model sit side by side in 121 GB of unified memory, and the whole fingerprinting of a team's memory takes seconds on a Spark. The architecture is there: the hallway already exists, it already authenticates, it already caches, it already survives restarts. It only needs to carry one more kind of request. The plugin already exists in the image, it needs one line in an allow list. sqlite-vec is a single package. The manual page is a two-paragraph edit.
And 0.0.123 already did the hard part right: the notebook is open, config changes apply in a second, restarts are clean. Finish the job and the Spark becomes the machine where agents actually remember, locally, privately, without a cloud bill.
>
I want to write the follow-up post. The one titled "It's fixed." Open the door. I will bring the
coffee.
Latest Blog Posts
Opinion2026-09-06
When Are You Getting Absorbed?
Absorption is not a maybe. MCP-connected corporate agents empty seats floor by floor — and the Agent War is already lit.
Opinion2026-09-06
Is OpenClaw About to Become DeadClaw?
OpenClaw becomes DeadClaw when the Gateway stays too complex for real users. Consumer-ready defaults—not more plugins—are what get Agents to work out of the box.
Opinion2026-09-06
Your AI Agent Needs a Door. I Built One.
I use WebMCP because my sites are Agent-first. The product is the Agent Gateway — MCP, REST, notify, discovery — so YOUR AI AGENT connects to bernhardrieder.com without screenshot theater. WebMCP is the in-page door.
Field Report2026-09-02
Unreal Engine MCP Devin IDE: First PBR Material Test Worked
Unreal Engine MCP Devin IDE test passed on the first real job: standard PBR master material, texture map inputs, material instance with exposed parameters. Epic MCP plugin, Python extension, UE 5.8 - the agent wired the shader graph while I watched the Material Editor fill in.
A
B
C
This article is part of a Semantic Cluster. All links are managed by the Digital Architect AI.