LLM Wiki

A pattern for building personal knowledge bases using LLMs.

This is an idea file, it is designed to be copy pasted to your own LLM Agent (e.g. OpenAI Codex, Claude Code, OpenCode / Pi, or etc.). Its goal is to communicate the high level idea, but your agent will build out the specifics in collaboration with you.

The core idea

Most people’s experience with LLMs and documents looks like RAG: you upload a collection of files, the LLM retrieves relevant chunks at query time, and generates an answer. This works, but the LLM is rediscovering knowledge from scratch on every question. There’s no accumulation. Ask a subtle question that requires synthesizing five documents, and the LLM has to find and piece together the relevant fragments every time. Nothing is built up. NotebookLM, ChatGPT file uploads, and most RAG systems work this way.

The idea here is different. Instead of just retrieving from raw documents at query time, the LLM incrementally builds and maintains a persistent wiki — a structured, interlinked collection of markdown files that sits between you and the raw sources. When you add a new source, the LLM doesn’t just index it for later retrieval. It reads it, extracts the key information, and integrates it into the existing wiki — updating entity pages, revising topic summaries, noting where new data contradicts old claims, strengthening or challenging the evolving synthesis. The knowledge is compiled once and then kept current, not re-derived on every query.

This is the key difference: the wiki is a persistent, compounding artifact. The cross-references are already there. The contradictions have already been flagged. The synthesis already reflects everything you’ve read. The wiki keeps getting richer with every source you add and every question you ask.

You never (or rarely) write the wiki yourself — the LLM writes and maintains all of it. You’re in charge of sourcing, exploration, and asking the right questions. The LLM does all the grunt work — the summarizing, cross-referencing, filing, and bookkeeping that makes a knowledge base actually useful over time. In practice, I have the LLM agent open on one side and Obsidian open on the other. The LLM makes edits based on our conversation, and I browse the results in real time — following links, checking the graph view, reading the updated pages. Obsidian is the IDE; the LLM is the programmer; the wiki is the codebase.

This can apply to a lot of different contexts. A few examples:

• Personal: tracking your own goals, health, psychology, self-improvement — filing journal entries, articles, podcast notes, and building up a structured picture of yourself over time.
• Research: going deep on a topic over weeks or months — reading papers, articles, reports, and incrementally building a comprehensive wiki with an evolving thesis.
• Reading a book: filing each chapter as you go, building out pages for characters, themes, plot threads, and how they connect. By the end you have a rich companion wiki. Think of fan wikis like Tolkien Gateway — thousands of interlinked pages covering characters, places, events, languages, built by a community of volunteers over years. You could build something like that personally as you read, with the LLM doing all the cross-referencing and maintenance.
• Business/team: an internal wiki maintained by LLMs, fed by Slack threads, meeting transcripts, project documents, customer calls. Possibly with humans in the loop reviewing updates. The wiki stays current because the LLM does the maintenance that no one on the team wants to do.
• Competitive analysis, due diligence, trip planning, course notes, hobby deep-dives — anything where you’re accumulating knowledge over time and want it organized rather than scattered.

Architecture

There are three layers:

Raw sources — your curated collection of source documents. Articles, papers, images, data files. These are immutable — the LLM reads from them but never modifies them. This is your source of truth.

The wiki — a directory of LLM-generated markdown files. Summaries, entity pages, concept pages, comparisons, an overview, a synthesis. The LLM owns this layer entirely. It creates pages, updates them when new sources arrive, maintains cross-references, and keeps everything consistent. You read it; the LLM writes it.

The schema — a document (e.g. CLAUDE.md for Claude Code or AGENTS.md for Codex) that tells the LLM how the wiki is structured, what the conventions are, and what workflows to follow when ingesting sources, answering questions, or maintaining the wiki. This is the key configuration file — it’s what makes the LLM a disciplined wiki maintainer rather than a generic chatbot. You and the LLM co-evolve this over time as you figure out what works for your domain.

Operations

Ingest. You drop a new source into the raw collection and tell the LLM to process it. An example flow: the LLM reads the source, discusses key takeaways with you, writes a summary page in the wiki, updates the index, updates relevant entity and concept pages across the wiki, and appends an entry to the log. A single source might touch 10-15 wiki pages. Personally I prefer to ingest sources one at a time and stay involved — I read the summaries, check the updates, and guide the LLM on what to emphasize. But you could also batch-ingest many sources at once with less supervision. It’s up to you to develop the workflow that fits your style and document it in the schema for future sessions.

Query. You ask questions against the wiki. The LLM searches for relevant pages, reads them, and synthesizes an answer with citations. Answers can take different forms depending on the question — a markdown page, a comparison table, a slide deck (Marp), a chart (matplotlib), a canvas. The important insight: good answers can be filed back into the wiki as new pages. A comparison you asked for, an analysis, a connection you discovered — these are valuable and shouldn’t disappear into chat history. This way your explorations compound in the knowledge base just like ingested sources do.

Lint. Periodically, ask the LLM to health-check the wiki. Look for: contradictions between pages, stale claims that newer sources have superseded, orphan pages with no inbound links, important concepts mentioned but lacking their own page, missing cross-references, data gaps that could be filled with a web search. The LLM is good at suggesting new questions to investigate and new sources to look for. This keeps the wiki healthy as it grows.

Indexing and logging

Two special files help the LLM (and you) navigate the wiki as it grows. They serve different purposes:

index.md is content-oriented. It’s a catalog of everything in the wiki — each page listed with a link, a one-line summary, and optionally metadata like date or source count. Organized by category (entities, concepts, sources, etc.). The LLM updates it on every ingest. When answering a query, the LLM reads the index first to find relevant pages, then drills into them. This works surprisingly well at moderate scale (~100 sources, ~hundreds of pages) and avoids the need for embedding-based RAG infrastructure.

log.md is chronological. It’s an append-only record of what happened and when — ingests, queries, lint passes. A useful tip: if each entry starts with a consistent prefix (e.g. ## [2026-04-02] ingest | Article Title), the log becomes parseable with simple unix tools — grep "^## \[" log.md | tail -5 gives you the last 5 entries. The log gives you a timeline of the wiki’s evolution and helps the LLM understand what’s been done recently.

Optional: CLI tools

At some point you may want to build small tools that help the LLM operate on the wiki more efficiently. A search engine over the wiki pages is the most obvious one — at small scale the index file is enough, but as the wiki grows you want proper search. qmd is a good option: it’s a local search engine for markdown files with hybrid BM25/vector search and LLM re-ranking, all on-device. It has both a CLI (so the LLM can shell out to it) and an MCP server (so the LLM can use it as a native tool). You could also build something simpler yourself — the LLM can help you vibe-code a naive search script as the need arises.

Tips and tricks

• Obsidian Web Clipper is a browser extension that converts web articles to markdown. Very useful for quickly getting sources into your raw collection.
• Download images locally. In Obsidian Settings → Files and links, set “Attachment folder path” to a fixed directory (e.g. raw/assets/). Then in Settings → Hotkeys, search for “Download” to find “Download attachments for current file” and bind it to a hotkey (e.g. Ctrl+Shift+D). After clipping an article, hit the hotkey and all images get downloaded to local disk. This is optional but useful — it lets the LLM view and reference images directly instead of relying on URLs that may break. Note that LLMs can’t natively read markdown with inline images in one pass — the workaround is to have the LLM read the text first, then view some or all of the referenced images separately to gain additional context. It’s a bit clunky but works well enough.
• Obsidian’s graph view is the best way to see the shape of your wiki — what’s connected to what, which pages are hubs, which are orphans.
• Marp is a markdown-based slide deck format. Obsidian has a plugin for it. Useful for generating presentations directly from wiki content.
• Dataview is an Obsidian plugin that runs queries over page frontmatter. If your LLM adds YAML frontmatter to wiki pages (tags, dates, source counts), Dataview can generate dynamic tables and lists.
• The wiki is just a git repo of markdown files. You get version history, branching, and collaboration for free.

Why this works

The tedious part of maintaining a knowledge base is not the reading or the thinking — it’s the bookkeeping. Updating cross-references, keeping summaries current, noting when new data contradicts old claims, maintaining consistency across dozens of pages. Humans abandon wikis because the maintenance burden grows faster than the value. LLMs don’t get bored, don’t forget to update a cross-reference, and can touch 15 files in one pass. The wiki stays maintained because the cost of maintenance is near zero.

The human’s job is to curate sources, direct the analysis, ask good questions, and think about what it all means. The LLM’s job is everything else.

The idea is related in spirit to Vannevar Bush’s Memex (1945) — a personal, curated knowledge store with associative trails between documents. Bush’s vision was closer to this than to what the web became: private, actively curated, with the connections between documents as valuable as the documents themselves. The part he couldn’t solve was who does the maintenance. The LLM handles that.

Note

This document is intentionally abstract. It describes the idea, not a specific implementation. The exact directory structure, the schema conventions, the page formats, the tooling — all of that will depend on your domain, your preferences, and your LLM of choice. Everything mentioned above is optional and modular — pick what’s useful, ignore what isn’t. For example: your sources might be text-only, so you don’t need image handling at all. Your wiki might be small enough that the index file is all you need, no search engine required. You might not care about slide decks and just want markdown pages. You might want a completely different set of output formats. The right way to use this is to share it with your LLM agent and work together to instantiate a version that fits your needs. The document’s only job is to communicate the pattern. Your LLM can figure out the rest.

LLM Wiki

一种使用大语言模型构建个人知识库的模式。

这是一份思路文档，可以直接复制给你的 AI Agent，例如 OpenAI Codex、Claude Code、OpenCode、Pi 等。它的目的是传达整体思路；具体实现则由你的 Agent 与你一起制定。

核心思路

大多数人使用大语言模型处理文档的方式类似于 RAG：上传一批文件，提问时模型检索相关片段，再生成回答。这种方法可行，但每次提问，模型都要重新从资料中寻找知识，无法形成积累。如果提出一个需要综合五份文档才能回答的复杂问题，模型每次都得重新寻找并拼接相关片段，不会留下已经整理好的成果。NotebookLM、ChatGPT 文件上传和大多数 RAG 系统都采用类似方式。

这里的思路有所不同：不是仅在提问时从原始文档中检索，而是让大语言模型逐步构建并维护一套持久的 Wiki。它由结构化、互相链接的 Markdown 文件组成，位于你和原始资料之间。当你加入新资料时，模型不只是建立索引供日后检索；它会阅读资料、提取关键信息，并将其融入已有 Wiki：更新实体页面、修订主题摘要、记录新旧信息之间的矛盾，让不断发展的综合认识得到加强或受到挑战。知识经过整理后持续更新，而不是每次提问都重新推导。

关键区别在于：**Wiki 是一份持续存在、不断积累的成果。**交叉引用已经建立，矛盾已经被标记，综合内容也已经反映你读过的资料。每加入一份资料、提出一个问题，Wiki 都有机会变得更加丰富。

你很少需要亲自编写 Wiki，甚至可以完全不写——由大语言模型负责撰写和维护。你负责挑选资料、探索和提出合适的问题；模型负责总结、交叉引用、归档和记录等繁琐工作，让知识库长期保持可用。在实际使用中，我会一边打开 LLM Agent，一边打开 Obsidian：模型根据我们的对话修改文件，我实时浏览结果，跟随链接、查看关系图谱、阅读更新后的页面。Obsidian 是 IDE，模型是程序员，Wiki 则是代码库。

这种方法可以用于很多场景，例如：

• 个人生活：追踪自己的目标、健康、心理状态和自我提升；归档日记、文章及播客笔记，逐步形成关于自己的结构化认识。
• 研究：围绕一个主题持续数周或数月深入研究，阅读论文、文章和报告，逐步构建全面的 Wiki，并不断完善自己的论点。
• 阅读一本书：边读边归档各章节，逐步建立人物、主题、情节及其联系的页面。读完后，你会拥有一份丰富的阅读指南。可以想想 Tolkien Gateway 这样的粉丝 Wiki：志愿者社区历时多年建立了数千个互相关联的人物、地点、事件和语言页面。你也可以自己建立类似的知识库，由大语言模型处理交叉引用和维护。
• 企业或团队：建立由大语言模型维护的内部 Wiki，摄入 Slack 讨论、会议转录、项目文档和客户通话等资料。必要时由人参与审核更新，模型则负责大家不愿反复处理的维护工作，让 Wiki 保持更新。
• 竞争分析、尽职调查、旅行规划、课程笔记、深入研究爱好：任何需要长期积累知识、又希望内容井井有条而不是四处散落的场景，都可以使用。

架构

知识库分为三层：

原始资料（Raw sources）：由你挑选收集的源文档，包括文章、论文、图片和数据文件。这一层是不可变的：大语言模型可以读取，但不修改。它是核对信息的原始依据。

Wiki：由大语言模型生成的 Markdown 文件目录，包括摘要、实体页面、概念页面、对比、概览及综合分析。模型负责这一层：创建页面，在新资料到来时更新内容，维护交叉引用，并尽量保持整体一致。你负责阅读，模型负责编写。

规则（Schema）：一份告诉模型 Wiki 如何组织、遵循什么约定，以及摄入资料、回答问题和维护时如何操作的文档。例如 Claude Code 可以使用 CLAUDE.md，Codex 可以使用 AGENTS.md。这是关键的配置文件：它让模型从一个通用聊天机器人，变成有章法的 Wiki 维护者。随着你逐渐了解哪些做法适合自己的领域，可以和模型一起不断完善规则。

操作

**摄入（Ingest）。**把一份新资料放进原始资料区，然后告诉模型处理。一个示例流程是：模型阅读资料，和你讨论关键收获，在 Wiki 中写一篇摘要，更新目录，更新相关的实体与概念页面，并在日志中追加一条记录。一份资料可能涉及 10–15 个 Wiki 页面。作者个人更喜欢一次摄入一份资料并保持参与：阅读摘要、检查更新、引导模型关注重点。你也可以减少监督，一次批量摄入多份资料。关键是找到适合自己的流程，并记录在规则中供后续会话使用。

**提问（Query）。**基于 Wiki 提出问题。模型查找并阅读相关页面，综合回答并附上引用。回答可以根据问题采取不同形式：Markdown 页面、对比表、幻灯片（Marp）、图表（matplotlib）或画布。重要的是：**有价值的回答可以作为新页面放回 Wiki。**你要求的对比、得到的分析、发现的联系，都不应该消失在聊天记录中。这样，你探索问题的成果也能像摄入资料一样在知识库中不断积累。

**体检（Lint）。**定期让模型检查 Wiki 的健康状况，包括：页面之间是否有矛盾、旧说法是否已被新资料取代、是否有没有入链的孤立页面、重要概念是否被提及却没有自己的页面、是否缺少交叉引用，以及哪些信息缺口可以通过网络搜索补充。模型也可以建议值得进一步研究的新问题和新资料。这样，Wiki 在不断增长时仍能保持健康。

目录与日志

两个特殊文件可以帮助你和模型在不断增长的 Wiki 中找到方向，它们的用途不同：

**index.md 面向内容。**它是整个 Wiki 的目录：每个页面都列出链接和一句话摘要，也可以附上日期、来源数量等元数据，再按实体、概念、来源等类别组织。模型每次摄入资料时都会更新它；回答问题时先读取目录，找到相关页面后再深入阅读。在中等规模下——约 100 份资料、数百个页面——这种方法效果相当不错，可以避免搭建基于向量嵌入的 RAG 基础设施。

**log.md 按时间组织。**它是一份只追加、不覆盖的记录，记下什么时候发生了什么，例如摄入、提问和体检。一个实用技巧是为每条记录使用统一的标题前缀，例如 ## [2026-04-02] ingest | 文章标题，这样就可以用简单的 Unix 工具解析：grep "^## \\[" log.md | tail -5 可以查看最近五条记录。日志展示 Wiki 的演变时间线，也帮助模型了解最近做过什么。

可选：命令行工具

随着知识库增长，你可能想建立一些小工具，让模型更高效地操作 Wiki。最直接的是搜索工具：规模小时目录文件已经够用，规模变大后则需要更完善的搜索。qmd 是一个选项：它是运行在本地的 Markdown 搜索工具，支持 BM25 与向量混合搜索以及大语言模型重排序。它既有 CLI，可以让模型通过命令调用；也有 MCP 服务器，可以作为模型的原生工具使用。你也可以从更简单的方案开始——需要时让模型帮你用自然语言驱动编程，写一个基础搜索脚本。

提示与技巧

• Obsidian Web Clipper 是浏览器扩展，可以把网页文章转换成 Markdown，很适合快速将资料放入原始资料区。
• **将图片下载到本地。**在 Obsidian 的「设置 → 文件与链接」中，将附件文件夹路径设为固定目录，例如 raw/assets/。然后在「设置 → 快捷键」中搜索 Download，找到「下载当前文件的附件」，为它设置快捷键，例如 Ctrl+Shift+D。剪藏文章后按下快捷键，就可以把图片下载到本地。这一步是可选的，但有助于模型直接查看和引用图片，而不必依赖可能失效的 URL。大语言模型不能在一次读取中自动看懂 Markdown 中嵌入的所有图片；可以先让模型读取文字，再分别查看部分或全部引用图片，以补充上下文。操作稍显笨拙，但效果不错。
• Obsidian 图谱视图可以展示 Wiki 的结构：哪些页面互相连接、哪些是中心节点、哪些是孤立页面。
• Marp 是一种基于 Markdown 的幻灯片格式，Obsidian 有对应插件，适合用知识库内容生成演示文稿。
• Dataview 是 Obsidian 插件，可以查询页面的 frontmatter。如果模型为 Wiki 页面添加 YAML frontmatter，例如标签、日期和来源数量，Dataview 可以据此生成动态表格和列表。
• Wiki 是由 Markdown 文件组成的 Git 仓库，可以利用 Git 的版本历史、分支和协作功能。

为什么这种方法有效

维护知识库最繁琐的部分不是阅读或思考，而是各种整理工作：更新交叉引用、让摘要保持最新、记录新资料与旧说法的矛盾，以及维持几十个页面之间的一致性。人们放弃维护 Wiki，往往是因为维护负担增长得比收益更快。大语言模型不会感到厌烦，可以一次处理多个文件，因此能够承担这些重复的维护工作。

人的工作是挑选资料、引导分析、提出好问题，并思考这些内容意味着什么；大语言模型则负责其余整理工作。

这个思路在精神上与 Vannevar Bush 的 Memex（1945）相关：一个由个人挑选、通过关联路径连接内容的知识存储系统。Bush 的设想比后来发展成的互联网更接近这种模式：私人持有、主动整理，文档之间的联系与文档本身同样有价值。他当时没能解决的是由谁来维护；这里由大语言模型承担维护工作。

说明

这份文档刻意保持抽象：它描述的是思路，而非某一种具体实现。目录结构、规则约定、页面格式和工具选择，都取决于你的领域、偏好和所用的大语言模型。上面提到的所有内容都是可选、可组合的：选择对自己有用的，忽略其余部分。例如，资料可能全部是文本，不需要处理图片；Wiki 的规模可能小到只用目录文件就够了，不需要搜索引擎；你可能不关心幻灯片，只想要 Markdown 页面；也可能希望采用完全不同的输出格式。

正确的使用方式是把这份文档交给你的 LLM Agent，一起将这套模式落实成适合你的知识库。文档的作用只是传达模式，剩下的细节可以交给模型协助完成。

补充要求：英文资料用中文融入知识库

**以下是我为这篇教程额外加入的要求，不属于 Karpathy 原文。**如果摄入的原始资料是英文，保留英文原件，不改写或删除；AI 在提取关键信息、生成摘要、更新概念页面并将知识融入 Wiki 时，使用中文撰写。专业名词、产品名等可以保留英文原文，同时保留来源链接或原始文件路径，方便回查。