openkb

OpenKB: Open LLM Knowledge Base

• Google Open Knowledge Format (OKF): Wiki pages follow the Google OKF specification for knowledge sharing.
• Entity Pages: People, orgs, places, and products as dedicated wiki pages, auto-extracted and kept in sync.

---

📑 What is OpenKB

OpenKB (Open Knowledge Base) is an open-source system (in CLI) that compiles raw documents into a structured, interlinked wiki-style knowledge base using LLMs, powered by PageIndex's vectorless, reasoning-based retrieval for long documents.

The idea is based on a concept described by Andrej Karpathy: LLMs generate summaries, concept pages, and cross-references, all maintained automatically. Knowledge compounds over time instead of being re-derived on every query.

Why not traditional RAG?

Traditional RAG rediscovers knowledge from scratch on every query. Nothing accumulates. OpenKB compiles knowledge once into a persistent wiki, then keeps it current. Cross-references already exist, contradictions are flagged, and synthesis reflects everything consumed.

OpenKB has two layers: a wiki foundation that compiles and maintains your knowledge, and generators (query / chat / Skill Factory) that turn it into useful output. See Usage for the full command list.

Features

• Broad format support: PDF, Word, Markdown, PowerPoint, HTML, Excel, CSV, text, URLs, and more.
• Scales to long documents: Long and complex documents are handled via PageIndex tree indexing, enabling accurate, vectorless, context-aware retrieval.
• Native multi-modality: Retrieves and understands figures, tables, and images, not just text.
• Compiled wiki: The LLM compiles your documents into summaries, concept pages, entity pages, and cross-links, all kept in sync.
• Query & chat: One-off questions or multi-turn conversations over your wiki, with persisted sessions to resume.
• Skill Factory: Distills redistributable agent skills from your wiki.
• OKF-ready: Wiki pages follow the Google OKF specification for knowledge sharing.
• Obsidian-compatible: The wiki is plain .md files with cross-links. Opens in Obsidian for graph view.
• Knowledge Workbench (Web UI): A bundled web UI served at / to browse the KB, upload and compile documents, and stream queries and chats — all in the browser.

🚀 Getting Started

Install

pip install openkb

• Latest from GitHub:

  pip install git+https://github.com/VectifyAI/OpenKB.git
  

• Install from source (editable, for development):

  git clone https://github.com/VectifyAI/OpenKB.git
  cd OpenKB
  pip install -e .
  

Quick Start

# 1. Create a directory for your knowledge base
mkdir my-kb && cd my-kb

# 2. Initialize the knowledge base
openkb init

# 3. Add documents
openkb add paper.pdf
openkb add ~/papers/                            # Add a whole directory
openkb add https://arxiv.org/pdf/2509.11420     # Or fetch from a URL

# 4. Ask a question
openkb query "What are the main findings?"

# 5. Or chat interactively
openkb chat

# (Optional) Turn the wiki into other outputs
openkb skill new my-expert "Reason like an expert on <your-topic>"   # a portable agent skill
openkb visualize                                                     # an interactive knowledge graph
openkb deck new my-deck "An intro deck on <your-topic>"              # slides — a single-file HTML deck

Set up your LLM

OpenKB supports multiple LLM providers (OpenAI, Claude, Gemini, etc.) via LiteLLM (pinned to a safe version).

Set your model during openkb init or in .openkb/config.yaml using the provider/model LiteLLM format (e.g. anthropic/claude-sonnet-4-6). OpenAI models can omit the prefix (e.g. gpt-5.4).

Create a .env file with your LLM API key:

LLM_API_KEY=your_llm_api_key

Subscription-based providers that authenticate via OAuth device flow (e.g. chatgpt/*, github_copilot/*) need no API key; OpenKB skips the missing-key warning for them.

Knowledge Workbench (Web UI)

OpenKB ships a bundled web UI, served by the REST API at /. Install the API extra and start the server — no configuration needed:

pip install "openkb[web]"
openkb-web                       # serves the API + Workbench at http://127.0.0.1:7566/

Open http://127.0.0.1:7566/ for the Workbench. Auth is off by default (local-first); set OPENKB_API_TOKEN to require a bearer token before exposing the server. See the full Web UI guide.

Working on the UI itself? Run the Vite dev server with cd frontend && npm install && npm run dev (it proxies /api to a running openkb-web), or npm run build to regenerate the bundled openkb/web/.

🧩 How OpenKB Works

Architecture

Short vs Long Document Handling

	Short documents	Long documents (PDF ≥ 20 pages)
Convert	markitdown → Markdown	PageIndex → tree index + summaries
Images	Extracted inline (pymupdf)	Extracted by PageIndex
LLM reads	Full text	Document trees
Result	summary + concepts	summary + concepts

Short documents are read in full by the LLM. Long PDFs are processed by PageIndex into a hierarchical tree index. The LLM reads the tree instead of the full text, enabling accurate and scalable retrieval for long documents.

Knowledge Compilation

When you add a document, the LLM:

1. Generates a summary page
2. Reads existing concept and entity pages
3. Creates or updates concepts with cross-document synthesis
4. Creates or updates entity pages (people, orgs, places, products)
5. Updates the index and log

A single source might touch 10--15 wiki pages. Knowledge accumulates: each document enriches the existing wiki rather than sitting in isolation.

⚙️ Usage

OpenKB commands fall into two layers: the wiki foundation (compile + manage your knowledge) and generators (turn that wiki into useful output). Each links to a concrete walkthrough — a real artifact OpenKB generated from one sample paper (browse them all in examples/).

Layer 1: 🧱 Wiki Foundation — compile and maintain

Command	Description
openkb init	Initialize a new knowledge base (interactive)
openkb add <file_or_dir_or_URL>	Add files, directories, or URLs and compile to wiki (URL content type is auto-detected)
openkb list	List indexed documents and concepts
openkb status	Show knowledge base stats
openkb watch	Watch raw/ and auto-compile new files
openkb lint	Run structural and knowledge health checks

Command	Description
openkb remove <doc>	Remove a document and clean up its wiki pages, images, registry, and PageIndex state (--dry-run to preview, --keep-raw / --keep-empty to retain artifacts)
openkb recompile [<doc>] [--all]	Re-run the compile pipeline on already-indexed docs without re-indexing. Regenerates summaries and rewrites concept pages; manual edits are overwritten (--dry-run to preview, --refresh-schema to also update wiki/AGENTS.md)
openkb feedback ["msg"]	File feedback by opening a prefilled GitHub issue (--type bug/feature/question to tag it)

→ Example: the everyday loop walked through end to end — examples/commands/.

Layer 2: 💡 Generators — turn the wiki into output

A "generator" reads from the compiled wiki and produces something usable: an answer, a conversation, a skill folder. The wiki is the substrate; generators are the surfaces.

Command	Output	Example
openkb query "question"	A grounded answer with citations (--save to persist to wiki/explorations/)	query & save