200 lines
6.8 KiB
Markdown
200 lines
6.8 KiB
Markdown
# Knowledge Genome System
|
|
|
|
> A distributed, modular, and secure personal knowledge base — no vector database required.
|
|
|
|
The **Knowledge Genome System** implements the [LLM Wiki pattern](https://gist.github.com/karpathy/442a6bf555914893e9891c11519de94f)
|
|
by Andrej Karpathy, extended with a multi-domain submodule architecture, git-crypt
|
|
encryption for sensitive data, and a human-in-the-loop Git Flow for quality control.
|
|
|
|
---
|
|
|
|
## Core Philosophy
|
|
|
|
Most RAG systems make the LLM rediscover knowledge from scratch on every query.
|
|
This system is different: the LLM **incrementally builds and maintains a persistent wiki**
|
|
that sits between you and the raw sources. Knowledge is compiled once and kept current —
|
|
not re-derived on every question.
|
|
|
|
**This means: no vector database, no embedding pipeline, no external retrieval server.**
|
|
The `wiki/index.md` of each genome is the retrieval layer. At moderate scale
|
|
(~100 sources, hundreds of pages) this works better than RAG because cross-references,
|
|
contradictions, and syntheses are already resolved — the LLM doesn't have to piece
|
|
them together at query time.
|
|
|
|
If the wiki grows beyond what the index can navigate efficiently, the only recommended
|
|
search extension is [`qmd`](https://github.com/tobi/qmd) — a local, on-device
|
|
BM25 + vector search engine for markdown files with an MCP server interface.
|
|
No external infrastructure required.
|
|
|
|
---
|
|
|
|
## Architecture
|
|
|
|
```text
|
|
master-knowledge-genome/ ← Root orchestrator
|
|
├── core-karpathy/ ← LLM Wiki reference pattern (read-only submodule)
|
|
├── genome-dev/ ← Submodule: web dev, Angular, TUI
|
|
├── genome-finance/ ← Submodule: personal finance
|
|
├── genome-homelab/ ← Submodule: Keru infrastructure
|
|
└── AGENTS.md ← Global coordination schema
|
|
```
|
|
|
|
Each genome is an independent repository with this structure:
|
|
```text
|
|
genome-{name}/
|
|
├── raw/
|
|
│ ├── articles/ transcripts/ code-packs/ assets/ ← Plaintext, open to collaborators
|
|
│ └── private/ ← AES-256-CTR encrypted (git-crypt)
|
|
├── wiki/
|
|
│ ├── index.md log.md ← Navigation and audit trail
|
|
│ ├── sources/ entities/ concepts/ queries/ ← Agent-maintained knowledge
|
|
│ └── private/ ← AES-256-CTR encrypted (git-crypt)
|
|
└── AGENTS.md ← Per-genome agent contract
|
|
```
|
|
|
|
---
|
|
|
|
## Prerequisites
|
|
|
|
**Required:**
|
|
- `git`
|
|
- `git-crypt`
|
|
- `curl`
|
|
- `jq`
|
|
|
|
**Optional:**
|
|
- `bw` (Bitwarden CLI) — for runtime key injection from Vaultwarden without writing keys to disk
|
|
|
|
Install on Ubuntu/Debian:
|
|
```bash
|
|
sudo apt update && sudo apt install -y git git-crypt curl jq
|
|
```
|
|
|
|
---
|
|
|
|
## Quick Start
|
|
|
|
```bash
|
|
# 1. Clone this setup repository
|
|
git clone <setup-repo-url> knowledge-genome-setup
|
|
cd knowledge-genome-setup
|
|
|
|
# 2. Export your Forgejo token
|
|
export FORGEJO_TOKEN="your_token_here"
|
|
|
|
# 3. Run full setup
|
|
make setup
|
|
```
|
|
|
|
`make setup` will:
|
|
- Check all dependencies
|
|
- Create the master and genome repositories on Forgejo
|
|
- Scaffold the local directory structure with git-crypt active on `private/`
|
|
- Install the pre-commit security hook in each genome
|
|
- Export the symmetric git-crypt keys to `keys/`
|
|
|
|
---
|
|
|
|
## Management Commands
|
|
|
|
| Command | Description |
|
|
|---------|-------------|
|
|
| `make setup` | Full system initialisation (master + all genomes defined in `config.env`) |
|
|
| `make add-genome NAME=x DESC="y"` | Scaffold and register a new genome |
|
|
| `make lint` | Validate schema, privacy flags, and metadata across all genomes |
|
|
| `make status` | Show git submodule status and first 10 git-crypt encryption states |
|
|
| `make help` | Show all available targets |
|
|
|
|
**Adding a new genome example:**
|
|
```bash
|
|
make add-genome NAME=genome-research DESC="Academic papers, deep-dives, open research"
|
|
```
|
|
|
|
---
|
|
|
|
## Security Model
|
|
|
|
### Hybrid Privacy Architecture
|
|
|
|
Each genome has two layers:
|
|
|
|
| Layer | Directories | Access |
|
|
|-------|-------------|--------|
|
|
| Public | `raw/articles/`, `raw/transcripts/`, `wiki/sources/`, `wiki/concepts/` | Plaintext — safe for collaborators |
|
|
| Private | `raw/private/`, `wiki/private/` | AES-256-CTR via git-crypt — owner only |
|
|
|
|
On the remote (Forgejo), private files are opaque binary blobs.
|
|
Collaborators without the key can contribute normally to public directories
|
|
— git handles the encrypted files transparently with no errors.
|
|
|
|
### Runtime Key Injection
|
|
|
|
Encryption keys are never stored as persistent files on the AI server.
|
|
They are injected at session start via the Bitwarden CLI (`bw`) against
|
|
your self-hosted Vaultwarden instance, using process substitution:
|
|
|
|
```bash
|
|
# Key lives only in a kernel file descriptor — never touches disk
|
|
git-crypt unlock <(
|
|
bw get notes "genome-dev key" --session "$BW_SESSION" | base64 -d
|
|
)
|
|
```
|
|
|
|
**Use `bw` (standard Bitwarden CLI), not `bws`.**
|
|
`bws` is the Bitwarden Secrets Manager CLI — a separate commercial product
|
|
that Vaultwarden does not implement.
|
|
|
|
### Pre-commit Hook
|
|
|
|
A security hook is installed in every genome's `.git/hooks/pre-commit`.
|
|
It inspects every staged file: if any file in `raw/private/` or `wiki/private/`
|
|
is not encrypted by git-crypt, the commit is blocked with a clear error message
|
|
explaining how to fix the issue.
|
|
|
|
### Key Rotation
|
|
|
|
If a key is lost or compromised:
|
|
```bash
|
|
source lib/git-crypt.sh
|
|
cd ~/knowledge-genome-setup/genome-dev
|
|
gcrypt_rotate_key "genome-dev"
|
|
```
|
|
The function decrypts all private files, generates a new key, re-encrypts,
|
|
and prints instructions for updating Vaultwarden.
|
|
|
|
---
|
|
|
|
## Agent Interaction
|
|
|
|
At the start of every AI session, declare the privacy context explicitly:
|
|
|
|
```text
|
|
PRIVATE_CONTEXT: disabled
|
|
```
|
|
The agent ignores all `private/` directories. Outputs are safe to share.
|
|
|
|
```text
|
|
PRIVATE_CONTEXT: enabled
|
|
```
|
|
The agent processes encrypted data. Requires the genome to be unlocked.
|
|
All outputs referencing private data are prefixed with `[PRIVATE DATA INCLUDED]`.
|
|
|
|
---
|
|
|
|
## Knowledge Quality
|
|
|
|
The system includes three quality mechanisms drawn directly from the LLM Wiki pattern:
|
|
|
|
**Conflict Resolution** — when new evidence contradicts existing wiki content,
|
|
the agent creates a `wiki/queries/conflict-*.md` node instead of silently overwriting.
|
|
Human review required before merging.
|
|
|
|
**Knowledge Decay** — pages with `maturity: stable` not updated in 6 months,
|
|
and `maturity: draft` pages not updated in 3 months, are flagged during lint passes
|
|
with a `⚠️ STALE` callout. The agent proposes re-validation but does not change
|
|
maturity without new source evidence.
|
|
|
|
**Cross-Genome Lint** — once a month, a manual session passes the aggregated index
|
|
of all genomes to the agent to detect concept duplication and missing cross-references.
|
|
No automated LLM controller in CI/CD — the cost in tokens and complexity is not
|
|
justified at this scale.
|