Skip to main content
The same words name different things in the same library. LIBERO-90 is an evaluation suite in the LIBERO benchmark. It is also the set of 90 tasks the LIBERO paper uses as pretraining data. A kDB that tags results and training data with words cannot tell the two apart, so a question about one returns the other. To prevent this, conditions, dataset properties and model classes name ontology terms. Each term is one page under ontology/<ontology>/, with fields modelled on W3C SKOS: pref_label, alt_labels for the other ways papers write it, broader for the more general term, and close_match or related_match for the same idea in an established vocabulary, such as the SOMA activity ontology used in robotics. Each term also has an iri, and that IRI is what a row stores. The body is a short scope note, cited like any other claim. Ontologies are layered. A general ontology describes ideas that hold in any field. A specialized one refines it for one kind of data or work by pointing the broader of its terms at the general ones. The seed has this shape for datasets: vla-datasets extends datasets, so teleoperation is a kind of human-collected data and cross-embodiment pretraining is a kind of pretraining. A specialized ontology refines the one it extends and never redefines its terms. It may not reuse a label that ontology already has. Questions follow the layers, so a question about pretraining also finds models trained with cross-embodiment pretraining. Run okf-seed kdb once, after kdb lake init and before your first ingest, with the ten papers of What you are given in kdb/papers/. When a paper writes “put the bowl on the plate” or “pick-place”, your agent links the result to the pick-and-place term through its alt_labels. It reads the paper’s own context to choose the right term when a word could name more than one. When the library needs a term the seed lacks, add it in an ontology of your own that extends the seed ontology it belongs to. Write a seed spec in the same form as the course’s specs, with a cited scope note and a broader link, and run it with okf-seed kdb --spec <file>. okf-seed finds the quote in your copy of the paper, highlights it and writes the page through the SeedOntology action, so the page is logged like every other change. Well-chosen additions earn credit. This spec adds one task term under robot manipulation:
okf-seed does not check that your labels are new. Run okf-validate afterwards: a label that an existing term already uses is a project.term-label error. The seed ontology already lists “open drawer” as another name for articulated object manipulation, for example. Fix the spec and run okf-seed again, and the action rewrites the page. The ontology makes some questions answerable. “Show every pick-and-place result” is a query for every results row whose task is the pick-and-place term or a term narrower than it, grouped by benchmark, suite, embodiment, number of trials and perturbation. Results in different groups are not compared, so a clean result is never set against a perturbed one. “Which VLA models were trained on the same datasets?” is a query over the models rows whose model_class is the vision-language-action model term, joined with their trained_on rows and grouped by dataset, with the role each model gave the dataset. Every link in the answer is a row with its own evidence, so it can be checked like any other claim.

​
Incremental updates

Your agent must provide one command, ingest <pdf>, that adds a single paper to an existing kDB and lake. An incremental update changes only the rows the new paper affects. It writes them through one or more actions, and the log row of each action names every row it inserted, updated or deleted, with the values before and after. A row the new paper does not affect is not written at all. Because UpsertObjects is a patch, adding an alias to an existing model takes a row with only the model’s IRI and its aliases, and the rest of the row stays as it was. A list column is replaced as a whole, so send the full list of aliases, the old ones and the new one. Deleting rows and writing them again, or rebuilding the whole kDB to add one paper, does not count as an incremental update. Pages change only when the paper adds a term or another type. Every other page must stay byte-identical, including its frontmatter and its generated stamp. You can test this yourself at any time. Hold one of the 30 papers out, build the kDB from the other 29 and commit. Note the lake’s snapshot:
Ingest the paper you held out, commit again, note the new snapshot, and compare the two, with your own snapshot numbers in place of 18 and 34:
The first command lists, for each table and for the media table, the rows added, changed and removed between the two snapshots, each with the ids of the actions that made the change. A change that no logged action made is a diff.unlogged error. The second command lists the pages and files that changed between two commits of the kDB. It reports a page that was rewritten with no change to its content, and a page whose generated stamp does not match its change.

​
The command line

Your agent is used through one command-line program with at least these commands. The starter’s kdb command has a place for each of them. The rows that ingest writes, history, --show and --save are yours to write.
  1. ingest <pdf> adds one paper to the kDB, as described above. It registers the PDF with RegisterMedia, writes the rows with UpsertObjects, highlights the evidence and registers the annotated copy.
  2. ask "<question>" answers from the lake and the kDB alone. Each sentence of the answer cites the rows and the evidence entries it rests on. For each citation, the program prints the paper, the page and the quoted sentence or table value. --show also displays the cited region of the page with its highlight, inline in terminals that support images and as a PNG file otherwise. When the lake does not contain the answer, the program says so.
  3. history lists earlier questions with their answers and citations, and history --paper <arxiv-id> lists only those whose answers cite that paper.
  4. ask --save files the answer back as an analyses row, with the question, the answer, the rows it drew on in drew_on, and copies of the evidence it cites. The row must pass okf-validate and okf-ground. If it does not, nothing is saved. Later questions can then use it, so the kDB grows from the questions you ask, as Karpathy’s LLM Wiki proposes.
Keep the question history in a SQLite file next to the kDB. Store the question, the answer, every citation, the model, the token counts, the git commit of the kDB and the snapshot of the lake at the time, and the full conversation. The commit and the snapshot record which version an answer used. When a later ingest changes a row, you can see which earlier answers relied on the old version.
Found an error on this page? Report an issue.