While I was writing this text, the search over my knowledge store reported a coverage of 100 percent. And it was right. It counted notes, and every note was indexed. Only: of the 8.77 million characters in the store, 1.28 million had never been embedded — 14.6 percent, cut away in silence. Among them were the two files my agents read as their memory.
This is not an embarrassing one-off that I am owning up to here. It is the subject. A store that maintains itself fails quietly, and the number that watches over it is often exactly the one that hides the failure.
What is inside when you open it up
891 Markdown files, 8,773,809 characters between them. The store is divided by purpose, not by topic: an inbox for the raw material, a curated tree for what has been checked, plus a link collection, topic maps and a handful of templates. Every note begins with a header block and carries its identity with it — title, type, related notes — so that a section from the middle of a long dossier still knows what it is about.
It is read in three ways: a lookup before every request, a handover board in the morning and in the evening, and a web view for browsing that is rebuilt every thirty minutes. The build goes into a side directory and only swaps in once the build has succeeded, because otherwise a single broken note aborts the whole run — and a page that has been served does not tell you how old it is.
Eight cadences that on their own prove nothing
| Component | Cadence | How I can see that it ran |
|---|---|---|
| Sync into version control | every 15 minutes | timestamp of the last entry against today |
| Reindexing of the search | every 15 minutes | byte coverage, not note count |
| Cleanup service and report | every 5 minutes | status file with its own timestamp |
| Watchdog over the writing service | every 2 minutes | 671 runs in one day |
| Mirror onto two volumes | every 30 minutes | 365 good runs, checksum comparison |
| Encrypted off-site copy | daily | decrypt and check the file list |
| Health report, read-only | daily | result file with a date |
| Web view | every 30 minutes | page count and duration |
The third column is the actual work. No service proves through its exit code that it has done what it is there for. It proves that it did not crash. That is something else, and over the past weeks that difference has cost me weeks, more than once.
The find from this morning
The index builder splits long notes into chunks, because the embedding model has an upper limit. The line it comes down to is this one:
chunks = [text[i:i + body_room] for i in range(0, len(text), body_room)][:_MAX_CHUNKS]The slice at the end is a silent truncation. No error, no exception, no exit code other than zero. The note gets its vectors, counts as embedded, appears in the coverage. Only its tail does not exist for the search.
A chunk holds 4,000 characters, but every one of them carries the note’s identity header so that a section from the middle still knows what it is about — which leaves roughly 3,750 for the text. Twelve chunks therefore put the ceiling at 45,000 characters. Fifteen notes were above it. Two of them are the handover board (200,794 characters) and the operations log (175,995) — precisely the files that every session reads first. Their more recent half stood outside the search.
That this hurts is not asserted, it is measured. A verbatim quote from the embedded part of the board brings the note back at rank 1 with a similarity of 0.7212. A verbatim quote from the truncated part of the same note comes back at rank 3 with 0.5777 — beaten by an unrelated note. Positive control and proof of damage in a single run.
The bitter part: the builder does report gaps. Its source says, word for word, “Gaps must be a report, not a surprise: name every degraded/dropped note”, and underneath it names every note that failed to embed or is now findable only through its title. That discipline is right and it works. It just does not catch this one, because a truncated tail is not a failure. It looks like success.
Both are repaired. The ceiling now sits at 64 chunks, so 240,000 characters: board and log fit whole, and what is left over are two imported standards documents of 337,805 and 304,437 characters — reference material, not memory. Zero loss would take 96 chunks per note, which bloats the index and slows every query, because a search scores all the chunks. More important than the number is the second change: the truncation is now counted, every affected note named with its lost characters, and next to the note coverage there is now a byte coverage. The note count can report 100 percent while a seventh of the content is missing. The byte count cannot.
Fourteen days in which every success message was true
The service that writes the store into version control every fifteen minutes sat on a machine that was switched off during a migration. On the new one the files lay ready, registered and never switched on. Between the sixth and the nineteenth of August not a single entry was created. The catch-up entry then came to 463 files and 17,384 inserted lines.
On the same day and from the same cause the reindexing died. The search kept answering — it was seeing 489 of 707 notes, so it was blind to 31 percent, without a single error line. The gap was not found by a watchdog but by looking: hold the timestamp of the last entry against today’s date. That is the whole test, it takes ten seconds, and it is the only one that would have fired here.
A backup is measured by the freshness of its content, not by the exit code of its job. And a search that answers is no proof that it sees everything.
A watchdog that restarted the healthy
Another watchdog checks every two minutes whether the writing service answers. It asked it the wrong way. The endpoint sends its success line and then keeps the connection open — for that endpoint the normal case. So the query tool printed the success code and afterwards ran into its timeout, and my evaluation read that as a failure.
Service is up but not answering — restarting
STILL DEAD — needs a humanFourteen times in a row, within 28 minutes, against a perfectly healthy service. It probably tore apart exactly the connections it was meant to protect. Since the correction it has healed real outages and been quiet afterwards — a statement that is only worth anything because in that time it ran around 671 times a day. A watchdog that stays silent says nothing until you know how often it has looked.
Forty percent of the suggestions were wrong
A home-made tool proposes missing cross-references between notes. On the live pass, 470 raw suggestions became 283 links. The 187 that were rejected were not a matter of taste, among them these:
- 72 sat in a heading, 62 of those in the first one — and here that is the title of the note.
- 68 sat in a table row.
- 26 pointed at an everyday word: a project name that also happens to be an ordinary noun. Of its hits, practically none meant the project.
The result was not checked through the number of changed files — that would have lied, a tool can touch a file without changing anything in it — but through the difference in links across the whole store: exactly 283 more, and the number of dead links unchanged before and after. A suggestion tool only becomes a tool once somebody knows its error rate.
What stays open
The inbox fills up faster than anything empties it. Its high-water mark was 239 notes, today it is 135. The reduction appears in no log: 128 notes moved into the curated tree in one afternoon because somebody decided it, not because a service did it — a good two dozen new ones arrived in the same period, which is why the inbox stands at 135 today and not at 111. That is deliberate — the cleanup service repairs what can be decided without loss, and it promotes nothing. “Byte-identical” is decidable. “Does this note belong with the projects?” is not.
Two imported standards documents are still being truncated. They now appear by name in the report of every run, with the number of characters lost. That is not a solution, but it is the difference between a known hole and an unknown one.
And the pattern I take away from all of it is smaller than it sounds. It is not a tool and not an architecture. It is the habit of asking, at every zero and every green number, whether it would even be capable of showing anything else. A test that cannot fire looks exactly like one that passed.