Indexing your own sources

docspack sync indexes documentation somebody published. docspack index indexes material this project already has — notes, decision records, an export, rows out of a query — and docspack recall answers from it.

docspack index --from ./notes
docspack recall "what did we decide about retries"

The reason is the one this tool exists for. A corpus larger than the context window cannot be read into it, and an agent handed a folder reads the wrong files or exhausts its budget. The index returns the passages that answer the question and nothing else, bounded by the same 3,000-token ceiling ask uses.

Indexing needs no model, so building the index costs no tokens at all.

Records, not only files

Anything that can emit JSON can be indexed, so there is no database driver here:

sqlite3 -json shop.db 'select id, title, body as text from posts' | docspack index --from-json -
psql -At -c "select ..." --json | docspack index --from-json -
docspack index --from-json ./export.json

Each record is { "title", "text" }, with optional id, tags and entities. A record that carries an id becomes exactly one chunk under that id — a row’s identity is its key, and splitting it would either duplicate that key or discard it. A record without an id is split by heading and size like any other document.

tags are worth supplying. They weigh three times as much as prose in ranking, and for rows the useful ones are usually the columns you did not put in the text: an author, a status, a date.

What keeps it honest

The corpus changes underneath the index. Published documentation is immutable for the life of its version; your own files are not, and are often edited by the same agent that is asking about them. So index records each source’s size, modification time and hash, and recall leads with a warning when any of them no longer match:

NOTE: the corpus is out of date. 1 indexed source has changed since it was built:
notes/retries.md. The passages below may be superseded — run `docspack index` again
before relying on them.

A stale answer is quoted correctly and is still wrong, which is worse than no answer, so the warning comes before the passages rather than after them. Re-running index does nothing when no source has changed, so it is cheap to run often.

A corpus piped in through --from-json - has nothing on disk to compare against later, and an answer from one says so instead of implying it is current.

recall is not ask. They are separate commands on purpose. ask promises an answer from the versions this project installed, and that promise is kept by one filter in the query path. Your notes are never an installed version, so they cannot reach an answer about a dependency — and a dependency’s documentation cannot reach an answer about your notes. Neither --all nor pointing both at the same database changes that.

The index is a copy of your sources. It lives in .docspack/local.db as plaintext, so it is as sensitive as whatever went into it. docspack index writes a .gitignore beside it so it is not committed by accident.

Whether it is worth it

Not obvious, and measured rather than asserted. An agent already has glob, grep and read, and below some corpus size those are better than an index: exact, no setup, and incapable of going stale.

On one 269-chunk corpus of prose nobody wrote for retrieval, the index returned the answering passage for 15 of 17 held-out questions at about 1,170 tokens per answer. Reading a window around grep’s best hit managed 8 of 17 at a similar cost; reading grep’s top three files whole found all 17, but cost about 16,000 tokens — roughly fourteen times as much.

So the index is not more accurate than reading everything grep suggests. It is much cheaper at close to the same accuracy, and that gap is what compounds across a session. Where the crossover sits for a smaller corpus is not known.