Skip to main content
In this project you will build an agent that reads a library of 30 research papers on vision-language-action (VLA) models and compiles them into a Knowledge DB, or kDB. The kDB is an information source that stays up to date and grounded in the papers. Every claim in the kDB must point to the exact sentence or table cell in a paper that supports it, and that evidence must be highlighted in the PDF. The kDB is written for agents as well as people. At grading time, an agent of ours will answer questions using your kDB and nothing else. You may work alone or in a team of two. Teams larger than two are not allowed. This project spans five sections. This one covers the goal, what you submit and how you are graded. The other four give the details:
  • What you are given: the command-line tools, the starter, the seed ontology and the 31 papers of the library.
  • The system you build: the components your agent must use and the design of its ingest and ask agents.
  • The kDB format: the classes and tables, evidence, actions, worked examples and the rules every row must follow.
  • The ontology: how terms name conditions, how ingest updates the kDB one paper at a time, and the commands your program provides.

Models and agents you may use

You build the agent yourself, in Python, with Pydantic AI. That agent is what this project is about, so no other agent may do its work. Commercial agent products and hosted agent services are not accepted. These include the ChatGPT, Claude or Gemini apps, NotebookLM and hosted agent platforms. A framework that runs its own agent to write the kDB is not accepted either. The model behind your agent can come from one of three places:
  1. A local OpenAI-compatible server with an open-weight model, such as Ollama, llama.cpp, vLLM, SGLang or LM Studio.
  2. A free model on OpenRouter, whose model ids end in :free. Free models are rate limited, so ingest runs slower.
  3. A paid model on OpenRouter, with a strict credit limit set on your API key before you start. One build of the kDB from all 31 papers (the library and the paper you ingest last) took our reference agent about 3.6 million input tokens and 0.8 million output tokens, and you will build it many times while you develop.
No other hosted model API is allowed. Keep your OpenRouter key in an environment variable and never in a file you commit. The starter reads it from the variable named in kdb.toml. The papers in the library are public, which is why OpenRouter is allowed here. With private documents, the same agent would run on a local model.

Grounded answers for mission-critical work

A language model writes fluent, confident text whether or not a source supports it. In a chat about films a wrong answer costs nothing. Where a document decides what someone does next, the cost is real: a technician sets the torque value printed in a maintenance manual, and a pharmacist gives the dose on a drug label. Reactor procedures and motor datasheets are used the same way. An answer drawn from documents like these is useful only if it can be traced to the passage that states it, and someone has to check that trace before acting on it. Two court cases show what happens when nobody checks. In 2023 a federal court in New York sanctioned two lawyers who filed a brief citing court decisions that ChatGPT had invented (Mata v. Avianca). In 2024 a Canadian tribunal held Air Canada liable for a refund policy its website chatbot made up (Moffatt v. Air Canada). Both answers looked authoritative, and nobody compared them with a source. Retrieval-augmented generation (RAG) helps by putting relevant passages in front of the model, but the model can still combine fragments wrongly or cite a passage that does not support its answer. A citation only tells the reader where to look. We ran into this while building the reference kDB for this project. One paper prints two result tables side by side, and our extractor read each printed line as one row, so numbers from one benchmark were filed under another. Every number was on the cited page, so a check that only asks whether the text is in the document passed. We found the error when a number looked wrong and someone compared it with its table. We then changed the reader to split tables that share a line. In a kDB, each claim is written once into a row together with its evidence: the paper, the page, the exact sentence or table cell, and the box on the page where it appears. okf-ground checks that the evidence is where the row says it is, and the highlight in the annotated PDF lets a person confirm a claim in seconds. Results record their conditions as ontology terms, so numbers measured under different setups are not compared with each other. When two sources disagree, the kDB keeps both and records the dispute. The action log and okf-diff show what each new paper changed, and the question history records which version of the kDB each answer used. If the kDB does not contain an answer, your program should say so. Refusals are graded along with answers. Audits in regulated industries ask for the same things: a trace from each statement to its source, a record of what changed and when, and a plain answer when the sources say nothing. The EU AI Act requires high-risk AI systems to keep records that make their operation traceable. Many of these documents cannot leave the organization that holds them, for reasons such as patient privacy, export control or trade secrets, so an agent for them has to run on local models. The papers in this project are public, so you may also use OpenRouter (see Models and agents you may use). The papers in this library are about robot learning, but nothing in the method depends on that. With an aircraft’s maintenance manuals or a plant’s operating procedures in place of the papers, the same checks apply. In this project you build an agent whose output a tool can verify. You compare it with a RAG baseline on correctness and token use. You also teach it to say when the sources have no answer. The checks prove that a quoted sentence or value is where its source says it is. They cannot prove that the sentence supports the claim it is attached to, so a TA reads a sample of claims at grading time.

How a kDB differs from RAG

Most systems that answer questions about documents use retrieval-augmented generation (RAG). They split the documents into chunks, find the chunks that match a question, and generate an answer from them. The system keeps nothing between questions. A question that needs five papers therefore requires the model to find and combine the same fragments every time. Andrej Karpathy’s LLM Wiki proposes a different approach. When a source arrives, the model reads it once and adds what it learned to a persistent wiki of linked markdown pages. It updates the pages for the models, datasets and benchmarks the paper mentions. It also revises summaries and notes where the new paper contradicts an older one. The knowledge is compiled once and kept current. A kDB adds one requirement to that approach: every claim carries evidence that a tool can check. Your job is to test whether compiling produces better results than retrieving. Your kDB has to answer hard questions better than a RAG baseline running on the same model. VLA papers are hard material for this, because their important facts are often numbers in result tables, reported under different setups. The same model appears under several names (π0, pi-zero, pi_0). Newer papers revise the results of older ones.

Required and stretch goals

Required: the grounded kDB, ingest, ask with highlights, history, ask --save, every results row linked to the seed ontology, and a trained_on row for every dataset a model was trained on. Stretch, for credit beyond the required score: standing queries that are refreshed on every ingest (for example, a saved query of all pick-and-place results whose answer is brought up to date when a new paper adds one), extensions to the ontology, and a short write-up of how your agent reads a paper longer than its context window. The write-up should explain what it keeps between chunks and how it notices a contradiction with a row it can no longer see.

Deliverables

A submission has two parts: your kDB repository at a commit, and an export of your lake made at that commit.
  1. Submission 1: your agent’s code and command-line program, and the kDB built from the 30 papers, with the ontology, the papers and the annotated PDFs. Export the lake that goes with it. No row may cite the paper marked “ingest last”.
  2. Submission 2: the same kDB and lake after one ingest of the “ingest last” paper, its annotated PDF, a new export, and a findings report of at most two pages. The report lists every row and page that changed and why. It also lists every claim the new paper contradicts or supersedes with both sources, and every contested row you wrote.
Export the lake into your team’s shared Google Drive folder, using the copy of it that Google Drive keeps in sync on your machine:
kdb export writes one folder, named <team>-<snapshot>, with the catalog file, the lake’s data files, every registered media file and manifest.json, which holds the sha256 of each file. It exports the latest snapshot only, so export when the lake is in the state you submit. The kDB itself is not exported, because we read it from your repository. We restore the export with okf-restore, which checks every hash in the manifest and refuses the export if any file is missing or changed. You can check your own export the same way, restoring it into a new, empty folder:
Submit the kDB repository and your code through the course repository. We use the push times recorded by GitHub rather than timestamps in your files.

How you are graded

  1. Grounding. We run okf-validate and okf-ground on both submissions, with your restored lake. They check the pages, the rows, the evidence of every row, and that every page was written by a logged action. A TA also reads a sample of claims to judge whether each cited sentence really supports the claim attached to it. A claim whose evidence does not check out scores zero, even if it sounds plausible.
  2. The incremental update. We compare your findings report with the changes between your two submissions, row by row and page by page, and both with our own reference ingest of the same paper. We check that the right rows changed, that every change came from a logged action, and that the contradiction was caught and recorded. We also check that nothing else moved.
  3. Answering questions. Our question agent answers a set of questions we do not publish, using only your final kDB and lake, with a model we serve and a fixed token budget per question. Many questions need several papers to answer. Some cannot be answered from the library at all. For those, the correct response says so. We score correct answers, the evidence they cite, and correct refusals. We compare your score with a PaperQA2 baseline that runs on the same model over the raw PDFs. To do well, your kDB must beat that baseline and use fewer tokens per answer.
  4. The demo. In a short live session, a TA asks your program questions with ask, including at least one that needs several papers and one the library cannot answer. The TA looks at the highlights with --show, saves an answer with --save and checks that the saved row passes both tools, and reads the history. We grade whether the citations are right, whether the program refuses when it should, and whether the saved row is a correct, grounded summary.

Constraints

  1. Models: a local model, a free OpenRouter model, or a paid OpenRouter model with a credit limit on your key. You may not send document content, page images or extracted text to any other hosted model API.
  2. Use pre-trained models only. No fine-tuning is required or expected.
  3. Do not modify the original PDFs. Annotate copies.
  4. Your agent is your own Pydantic AI program. No commercial agent product or hosted agent service may do its work, and no other paid service is allowed.
  5. You may not use AI browsers, NotebookLM, alphaXiv or any other hosted tool that reads papers for you.
  6. Runtime is not graded, but ingest of one paper must finish in a reasonable time on your own hardware.
Found an error on this page? Report an issue.