The Graph Layer: Planning LightRAG

OpenViking gave Hermes semantic recall. Ask it what I wrote about something, and it finds the right notes. What it doesn’t do well is relationships. “How does this project connect to that person” isn’t a similarity question, it’s a graph question, and vector search alone keeps missing it. This is the design for adding LightRAG to close that gap, plus how I’m planning to ingest the vault into it without blowing my memory budget.

Nothing here is built yet. Writing it down before I touch config.yaml.

Splitting memory by query shape

The instinct is to merge everything into one index. I’m doing the opposite: OpenViking and LightRAG stay separate providers, and Hermes’ router decides which one a query actually needs.

Telegram message
      │
      ▼
  Hermes router (OpenRouter free model)
      │
      ├─▶ OpenViking   — semantic / episodic recall
      ├─▶ LightRAG     — relationship / multi-hop queries
      └─▶ Websearch    — external, current knowledge

LightRAG sits between plain vector RAG and something heavier like Microsoft’s GraphRAG. It builds entity-relationship pairs without the full community-detection layer, so it can answer “how are these connected” without the cost of a heavier graph pipeline. That’s the gap OpenViking leaves open, not a reason to replace it.

For the final answer, results from both (graph hops and vector chunks) get combined and passed through qwen3-reranker-0.6b before hitting the chat model. That reranker already sits in the retrieval group doing nothing most of the day, so this just gives it a second job.

Where this fits the memory budget

24GB of unified memory sounds like a lot until the model weights and KV caches start stacking up. In practice I keep about 14GB free for LLM use, which is why llama-cpp/config.yaml already splits models into groups instead of loading everything at once:

Model	Role	Group	Notes
qwen3.5-9b	interactive chat	interactive (swap)	thinking disabled, unloads after 3 min idle
qwen3-embedding-0.6b	embeddings	retrieval	unloads after 20s idle
qwen3-reranker-0.6b	reranking	retrieval	unloads after 20s idle

LightRAG’s embedding step slots straight into the existing embedding model, no change needed there. The expensive part is entity extraction during ingestion, which needs an actual LLM call per chunk, not just an embedding. That’s where a naive setup would blow the budget.

Two ways to handle it:

1. Send extraction to OpenRouter’s free model. Doesn’t touch local memory, but free-tier rate limits won’t survive a full-vault backfill, and it means shipping personal notes and conversations to a third party. Not something I want for vault content.
2. Run extraction locally, at night, on qwen3.5-9b. This is the better fit, and the config already leans this way. Thinking is disabled on that model already, which is exactly what extraction wants: fast, non-reasoning passes, not a slow deliberation loop. Off-peak means no Telegram traffic competing for the interactive slot, and the existing TTL unloads it automatically once the batch finishes. Data never leaves the machine.

One adjustment: LightRAG’s extraction step typically wants a bigger token budget than my chat context (--ctx-size 16384) gives it. Rather than touching the daytime profile, I’ll add a second llama-swap entry pointed at the same GGUF with a larger context, used only by the nightly ingestion job. Costs nothing during the day, since llama-swap loads on demand.

Fine-tuning LLM memory budget

I assumed the context bump would be the expensive part. I measured it instead of guessing, and I had it backwards.

Method: load each model with llama-server on a spare port, send one real request to page everything in, then read footprint -p <pid>. Two numbers matter. Clean is the mmap’d weights, dirty is KV cache plus compute buffers, which is the part that actually scales with your settings.

Starting with the 9B at four context sizes:

--ctx-size	Dirty (KV + compute)	Weights	Total
16384	762 MB	5.4 GB	6.1 GB
32768	1275 MB	5.4 GB	6.6 GB
40960	1531 MB	5.4 GB	6.8 GB
65536	2299 MB	5.4 GB	7.6 GB

Clean linear scaling at roughly 32 KiB per token. Worth noting because the naive calculation from the GGUF metadata says otherwise: 32 layers × 4 KV heads × 512 dims × 2 bytes works out to 128 KiB/token, four times what I actually measured. Qwen3.5 isn’t holding full KV on every layer.

Which means going from 16k to 40k context costs me 769 MB. That’s nothing. The context bump for extraction was never the problem.

Then I measured the two 0.6B retrieval models, expecting them to be rounding errors:

Model	Settings	Dirty	Weights
Embedding	ctx 20480, ubatch 512	2343 MB	624 MB
Embedding	ctx 8192, ubatch 512	998 MB	624 MB
Embedding	ctx 4096, ubatch 256	550 MB	624 MB
Reranker	ctx 20480, parallel 5	2435 MB	624 MB
Reranker	ctx 8192, parallel 2	1090 MB	624 MB
Reranker	ctx 4096, parallel 2	641 MB	624 MB

There it is. My two smallest models were sitting on 4.8 GB of dirty memory between them — more than the 9B’s entire KV cache — because I’d set ctx-size 20480 on both and parallel 5 on the reranker without thinking about what those actually cost. Each model was burning close to 4x its own weight in buffers.

Dropping both to ctx 8192 recovers about 2.7 GB. I’m not going to 4096 even though it saves more; LightRAG chunks run around 1200 tokens and OpenViking embeds longer documents, so 8192 leaves real headroom where 4096 starts cutting close. parallel 5 on the reranker also made no sense for a single-user Telegram bot, so that goes to 2.

The tuned config:

Model	Context	Other	Purpose
qwen3.5-9b	16384	unchanged	daytime chat
qwen3.5-9b-extract	40960	same GGUF, short TTL	nightly extraction
qwen3-embedding-0.6b	8192	ubatch 512	embeddings
qwen3-reranker-0.6b	8192	parallel 2	reranking

Loading the 9B at 40960 alongside both retrieval models at the tuned settings lands at 10.3 GB total (3.7 GB dirty, 6.6 GB weights), verified with all three actually serving requests. Roughly 3.7 GB of headroom against my 14 GB working ceiling, and no swap.

So the nightly extraction job fits with room to spare, and I get the memory back by fixing settings I’d never questioned rather than by giving anything up. The lesson I’ll actually remember: on a memory-constrained box, audit the small models first. They’re the ones nobody bothers to tune.

Ingestion: hash it, don’t trust timestamps

The vault syncs across devices, and file modification times drift for reasons that have nothing to do with content changes. So the ingestion pipeline hashes instead of watching mtimes. SHA-256 per note, tracked in a small manifest: path, hash, last-ingested time, LightRAG doc ID.

Each scheduled run:

flowchart TD
    A[Walk vault, hash every note] --> B{Diff against manifest}
    B -->|New file| C[Insert into LightRAG]
    B -->|Changed hash| D[Delete old doc, reinsert]
    B -->|File removed| E[Delete by doc ID]
    C --> F[Batch changes into one LightRAG run]
    D --> F
    F --> G[Write manifest only after success]

No partial patching. LightRAG treats any edit as a full document replace, and the whole batch runs through the extraction model once instead of loading it per file. If the run crashes before the manifest write, it just retries next cycle, no duplicate or missing state.

There’s already a watchdog pattern doing something similar for OpenViking indexing. LightRAG’s ingestion job should mirror that shape rather than invent a second one.

For storage, LightRAG’s default local backend (NetworkX for the graph, NanoVectorDB for vectors, JSON for KV) is fine at personal-vault scale. It fully reloads the graph into memory on startup, which would matter at a much bigger scale, but not at a few thousand notes. No need to stand up Postgres or Neo4j for this.

Scheduling around myself, not around cron

The one thing cron gets wrong on macOS: if the machine is asleep when a job is due, cron just skips it. launchd handles this properly. A job scheduled with StartCalendarInterval still runs shortly after wake if it was missed.

I’m pairing that with pmset repeat wakeorpoweron to wake the machine a few minutes early, and wrapping the ingestion command in caffeinate so it doesn’t fall back asleep mid-run. Somewhere around 2 to 4 AM looks like the right window, when there’s no realistic chance of a Telegram message competing for the interactive group.

Open questions

• Should daily_note_agent writes touch OpenViking immediately for same-day recall, and just let LightRAG’s graph view lag by up to a day? Leaning yes. Relationships between notes rarely need same-day freshness the way a direct lookup does.
• Manifest storage: a dedicated SQLite file next to LightRAG’s working directory, or reuse whatever state the OpenViking watchdog already tracks, if it already has something hash-shaped.

---

Next post is probably whichever one of these turns out to be wrong once it’s actually running.