Indexing your own sources

docspack sync indexes documentation somebody published. docspack index indexes material this project already has — notes, decision records, an export, rows out of a query — and docspack recall answers from it.

docspack index --from ./notes
docspack recall "what did we decide about retries"

The reason is the one this tool exists for. A corpus larger than the context window cannot be read into it, and an agent handed a folder reads the wrong files or exhausts its budget. The index returns the passages that answer the question and nothing else, bounded by the same 3,000-token ceiling ask uses.

Indexing needs no model, so building the index costs no tokens at all.

A repository: its docs and a map of its code

docspack index --repo
docspack recall --name repo-code "where is a request for an unknown route served index.html"
docspack recall --name repo-docs "why are there two indexes"

--repo indexes the git repository you are in. It asks git which files belong to the project, so node_modules, build output and anything else .gitignore names are never read. Files git does not track yet are included, because the agent asking is often the one that just wrote them.

It writes two corpora:

• @local/repo-docs: every Markdown file, except changelogs and anything under tests/ or fixtures/.
• @local/repo-code: a map of the code. It has one entry per top-level declaration in TypeScript and JavaScript, holding its signature, its doc comment and its file:line, and never its body. There is also one entry per package.json, which lists the package’s entry points and source files.

Ask one of them with recall --name, with or without the @local/ prefix. They are kept apart because prose outranks the map. Asked together, the docs pushed the right declaration out of the top three answers for 8 of 42 questions about where code lives.

The map exists to tell you where to look. It answers “where is the thing that does X” in a few hundred tokens, and you then read that range of the file. Each entry also stores its name split into words (planAgentSetup as plan agent setup), so a question with no name in it can still find the name.

Indexing the docspack repository itself (about 180 source files and 70 Markdown files) takes under a second. When nothing has changed, a re-run takes about 0.2 seconds. That is cheap enough for a SessionStart hook.

An entry is a pointer, not the truth. Read the file before you edit it.

The index finds things. It does not, on its own, make a session cheaper. In Claude Code sessions measured on two repositories, an agent that searched with Grep and read a range of the file spent about the same as one that asked the index, because Grep returns matching lines, not whole files. What did lower the cost was asking the agent to use fewer, larger tool calls: see Setting up an agent to spend fewer tokens.

Records, not only files

Anything that can emit JSON can be indexed, so there is no database driver here:

sqlite3 -json shop.db 'select id, title, body as text from posts' | docspack index --from-json -
psql -At -c "select ..." --json | docspack index --from-json -
docspack index --from-json ./export.json

Each record is { "title", "text" }, with optional id, tags and entities. A record that carries an id becomes exactly one chunk under that id — a row’s identity is its key, and splitting it would either duplicate that key or discard it. A record without an id is split by heading and size like any other document.

tags are worth supplying. They weigh three times as much as prose in ranking, and for rows the useful ones are usually the columns you did not put in the text: an author, a status, a date.

What keeps it honest

The corpus changes underneath the index. Published documentation is immutable for the life of its version; your own files are not, and are often edited by the same agent that is asking about them. So index records each source’s size, modification time and hash, and recall leads with a warning when any of them no longer match:

NOTE: the corpus is out of date. 1 indexed source has changed since it was built:
notes/retries.md. The passages below may be superseded — run `docspack index` again
before relying on them.

A stale answer is quoted correctly and is still wrong, which is worse than no answer, so the warning comes before the passages rather than after them. Re-running index does nothing when no source has changed, so it is cheap to run often.

A corpus piped in through --from-json - has nothing on disk to compare against later, and an answer from one says so instead of implying it is current.

recall is not ask. They are separate commands on purpose. ask promises an answer from the versions this project installed, and that promise is kept by one filter in the query path. Your notes are never an installed version, so they cannot reach an answer about a dependency — and a dependency’s documentation cannot reach an answer about your notes. Neither --all nor pointing both at the same database changes that.

The index is a copy of your sources. It lives in .docspack/local.db as plaintext, so it is as sensitive as whatever went into it. docspack index writes a .gitignore beside it so it is not committed by accident.

Whether it is worth it

Not obvious, and measured rather than asserted. An agent already has glob, grep and read, and below some corpus size those are better than an index: exact, no setup, and incapable of going stale.

On one 269-chunk corpus of prose nobody wrote for retrieval, the index returned the answering passage for 15 of 17 held-out questions at about 1,170 tokens per answer. Reading a window around grep’s best hit managed 8 of 17 at a similar cost; reading grep’s top three files whole found all 17, but cost about 16,000 tokens — roughly fourteen times as much.

So the index is not more accurate than reading everything grep suggests. It is much cheaper at close to the same accuracy, and that gap is what compounds across a session. Where the crossover sits for a smaller corpus is not known.