Skip to main content
In this project you will build an agent that reads a library of 30 research papers on vision-language-action (VLA) models and compiles them into a Knowledge DB, or kDB. The kDB is a continuously updated, truth-grounded information source. Every claim in the kDB must point to the exact sentence or table cell in a paper that supports it, and that evidence must be highlighted in the PDF. The kDB is written for agents as well as people. At grading time, an agent of ours will answer questions using your kDB and nothing else. You may work alone or in a team of two. Teams larger than two are not allowed.

​
Models and agents you may use

You build the agent yourself, in Python, with Pydantic AI. That agent is what this project is about, so no other agent may do its work. Commercial agent products and hosted agent services, such as the ChatGPT, Claude or Gemini apps, NotebookLM or a hosted agent platform, are not accepted, and neither is a framework that runs its own agent to write the kDB. The model behind your agent can come from one of three places:
  1. A local OpenAI-compatible server with an open-weight model, such as Ollama, llama.cpp, vLLM, SGLang or LM Studio.
  2. A free model on OpenRouter, whose model ids end in :free. Free models are rate limited, so ingest runs slower.
  3. A paid model on OpenRouter, with a strict credit limit set on your API key before you start. One build of the kDB from the 31 papers took our reference agent about 3.6 million input tokens and 0.8 million output tokens, and you will build it many times while you develop.
No other hosted model API is allowed. Keep your OpenRouter key in an environment variable and never in a file you commit; the starter reads it from the variable named in kdb.toml. The papers in the library are public, which is why OpenRouter is allowed here. With private documents, the same agent would run on a local model.

​
Grounded answers for mission-critical work

A language model writes fluent, confident text whether or not a source supports it. In a chat about films a wrong answer costs nothing. Where a document decides what someone does next, the cost is real: a technician sets the torque value printed in a maintenance manual, and a pharmacist gives the dose on a drug label. Reactor procedures and motor datasheets are used the same way. An answer drawn from documents like these is useful only if it can be traced to the passage that states it, and someone has to check that trace before acting on it. Two court cases show what happens when nobody checks. In 2023 a federal court in New York sanctioned two lawyers who filed a brief citing court decisions that ChatGPT had invented (Mata v. Avianca). In 2024 a Canadian tribunal held Air Canada liable for a refund policy its website chatbot made up (Moffatt v. Air Canada). Both answers looked authoritative, and nobody compared them with a source. Retrieval-augmented generation (RAG) helps by putting relevant passages in front of the model, but the model can still combine fragments wrongly or cite a passage that does not support its answer. A citation only tells the reader where to look. We ran into this while building the reference kDB for this project. One paper prints two result tables side by side, and our extractor read each printed line as one row, so numbers from one benchmark were filed under another. Every number was on the cited page, so a check that only asks whether the text is in the document passed. We found the error when a number looked wrong and someone compared it with its table, and we then changed the reader to split tables that share a line. In a kDB, each claim is written once into a page together with its evidence: the paper, the page, the exact sentence or table cell, and the box on the page where it appears. okf-ground checks that the evidence is where the page says it is, and the highlight in the annotated PDF lets a person confirm a claim in seconds. Results record their conditions as ontology terms, so numbers measured under different setups are not compared with each other. When two sources disagree, the kDB keeps both and records the dispute. Git history and okf-diff show what each new paper changed, and the question history records which version of the kDB each answer used. If the kDB does not contain an answer, your program should say so; refusals are graded along with answers. Audits in regulated industries ask for the same things: a trace from each statement to its source, a record of what changed and when, and a plain answer when the sources say nothing. The EU AI Act requires high-risk AI systems to keep records that make their operation traceable. Many of these documents cannot leave the organization that holds them, for reasons such as patient privacy, export control or trade secrets, so an agent for them has to run on local models. The papers in this project are public, so you may also use OpenRouter (see the next section). The papers in this library are about robot learning, but nothing in the method depends on that. With an aircraft’s maintenance manuals or a plant’s operating procedures in place of the papers, the same checks apply. In this project you build an agent whose output a tool can verify, compare it with a RAG baseline on correctness and token use, and teach it to say when the sources have no answer. The checks prove that a quoted sentence or value is where its source says it is. They cannot prove that the sentence supports the claim it is attached to, so a TA reads a sample of claims at grading time.

​
How a kDB differs from RAG

Most systems that answer questions about documents use retrieval-augmented generation (RAG). They split the documents into chunks, find the chunks that match a question, and generate an answer from them. The system keeps nothing between questions. A question that needs five papers therefore requires the model to find and combine the same fragments every time. Andrej Karpathy’s LLM Wiki proposes a different approach. When a source arrives, the model reads it once and folds what it learned into a persistent wiki of linked markdown pages. It updates the pages for the models, datasets and benchmarks the paper mentions. It also revises summaries and notes where the new paper contradicts an older one. This compiles the knowledge once and keeps it current. A kDB adds one requirement to that approach: every claim carries evidence that a tool can check. Your job is to test whether compiling produces better results than retrieving. Your kDB has to answer hard questions better than a RAG baseline running on the same model. VLA papers are hard material for this, because their important facts are often numbers in result tables, reported under different setups. The same model appears under several names (π0, pi-zero, pi_0). Newer papers revise the results of older ones.

​
What you are given

  1. The library: 30 papers on VLA models, plus one more paper marked “ingest last”. The library section lists them. Save each PDF as papers/<arxiv-id>.pdf, for example papers/2306.03310.pdf, because the seed ontology and our grading refer to the papers by those names.
  2. The project repository, pantelis/physical-ai-kdb, with four command-line tools. We run the same tools at grading time.
    • okf-validate checks that your kDB is a valid OKF v0.2 bundle and follows the rules on this page.
    • okf-ground checks that every quoted sentence is on its cited page and inside its bbox. It also checks that every table value is inside its bbox and that every bbox matches a highlight in your annotated copy of the PDF.
    • okf-diff compares two versions of your kDB, as two directories or two git commits, and lists every page that changed.
    • okf-seed writes the seed ontology into your kDB (item 4).
  3. The same checks as a Python package, kdb_tools, so your agent can call them as functions instead of parsing command output. validate() and ground() return a report with one finding per problem. kdb_tools.locate.find_quote(page, text) returns the bbox of every place a quote appears on a PDF page. It uses the same matching as okf-ground, so your agent and the checker agree on where a quote is. The repository README explains how to install the package.
  4. A seed ontology in three parts. robotics covers tasks (such as pick-and-place, pushing and articulated object manipulation), benchmarks and their suites (LIBERO and its suites, LIBERO-Plus, CALVIN, SIMPLER), robots (such as the Franka Emika Panda and the WidowX 250), the seven LIBERO-Plus perturbation types and the vision-language-action model class. datasets covers how data is collected and what a model uses it for, for any kind of dataset. vla-datasets specializes datasets for robot learning data. Teleoperation is a kind of human-collected data, cross-embodiment pretraining is a kind of pretraining, and so on. okf-seed kdb writes one Term page per term under ontology/<ontology>/ and one page per ontology that says what it extends. Each term is defined by a sentence it quotes from a library paper and highlights in your copy of that paper. The ontology section explains how to use it.

​
The library

The library has 30 papers on vision-language-action (VLA) models, plus one paper to ingest last. Each paper links to its Semantic Scholar page and to one version of its PDF on arXiv. Download that version and no other. Several of these papers are still being revised, and a later version moves sentences and table values to other pages. The checks compare your sources with our copies of these versions. A correct source taken from another version therefore fails okf-ground. The papers report on the same benchmarks (LIBERO, SIMPLER, CALVIN), compare the same models (RT-2, Octo, OpenVLA, π0) and train on the same datasets (Open X-Embodiment, BridgeData V2, DROID). Because of that overlap, the kDB can answer questions that span several papers. Ingest last. Do not include this paper in Submission 1.

​
The system you build

Your agent must use the following components.
  1. Agent harness: you write it, in Python, with Pydantic AI (the pydantic-ai package). Your agent’s tools are typed Python functions, and their arguments and results are Pydantic models. A tool can reject a bad call, such as a quote that is not on the cited page. The model gets the error back and tries again. A central part of the project is deciding which tools the model gets, what each one checks, and when the agent stops.
  2. Model server: a local OpenAI-compatible server (llama.cpp, vLLM, SGLang, Ollama or LM Studio) with an open-weight model of your choice, or OpenRouter under the rules in Models and agents you may use. Your agent reads the base_url, the model name and the name of the API key’s environment variable from a config file. It must work unchanged against any such server because, at grading time, we run it against a model we serve. A reference setup that fits a 24 GB GPU is Ollama with gemma4:31b-it-qat. Plan for ingest to take hours. Our reference agent, running gemma4:12b-it-qat on a 16 GB GPU, took about 19 minutes per paper and almost 10 hours for the whole library. A prompt longer than the server’s context window is not rejected. Ollama drops the middle of the prompt, keeps the start and the end, and answers as if nothing happened. The only trace is a warning in the server’s own log. In our tests with a 32K context, a 45,000-token prompt was cut to 16,387 tokens and the model returned an empty answer with no error. One paper converted to Markdown is 25,000 to 35,000 tokens, so passing whole papers to the model will exceed the context window. Your harness should estimate the size of each prompt before sending it. It should also compare the usage.prompt_tokens that comes back with what it sent. A large gap means the server cut the prompt.
  3. PDF parsing: any open-source parser that keeps page positions, such as Docling or MinerU. You may read the PDF files directly. Watch the tables in two-column papers. Two tables often sit side by side, so one printed line holds a row of each. A parser that reads lines across the whole page joins the two rows into one under the first row’s label. The numbers of the second table are then filed under the first. okf-ground does not catch this because every number really is inside its bbox. The table, column and conditions are still wrong.
  4. Annotation: a highlight, with a comment, in a copy of each PDF for every source you cite. kdb_tools.annotate.highlight writes them with one box per line of text, so only the quoted words are shaded. PdfCraft and pypdf are also accepted.
  5. kDB engine: your own code, called by your Pydantic AI agent. A tool that runs its own agent to write the pages, such as OpenWiki, is not accepted.

​
Agent design

The project repository has a starter in starter/. Copy it into your own repository. It reads the server settings from kdb.toml, builds a Pydantic AI model for any OpenAI-compatible server, refuses prompts that would not fit the context, and has the two agents below with their output types and validators. It does none of the extraction. Its tests run offline, with a scripted model in place of the server. The checks are also available as tools your agent can call while it works. find_quote and find_value say where a sentence or a table value is on a page, the way okf-ground looks for it, and which page it is on when it is not where the model said. check_pages runs okf-validate and okf-ground on the pages you name, so your agent can check the pages it just wrote without failing on errors an earlier ingest left. kdb_tools.agent.toolset gives them to a Pydantic AI agent, and the starter’s ingest agent already has them. okf-mcp kdb serves the same tools over MCP. The ingest agent reads a few pages at a time and returns typed output: facts about models, datasets and benchmarks, each with a sentence in its own words and a quote copied from the page, and results read from table rows. Give it the text of the pages and the table rows with their positions, the ontology terms, and the names of entities that already have pages, so it links to them instead of creating new ones. Its output validator looks up every quote and every table value in the PDF with kdb_tools.locate.find_quote and sends back what it cannot find, with the reason. After a few retries, drop what is still wrong rather than writing it. Your code, not the model, then merges the facts into pages, writes the sources with their bboxes and the highlights, and updates the index and the log. The ask agent answers from the kDB through tools. A tool that lists every page works for a small kDB and stops fitting the context once there are a few hundred pages, mostly results. Give it tools that search instead: results by ontology term with the narrower terms included, by benchmark or suite, or a read-only query over a table you build from the result pages. Its output is a list of claims, each naming a page and the source ids on that page that support it, and its validator sends back any claim whose page or source id does not exist. Our reference agent ran into four problems that yours will meet too.
  1. A prompt longer than the context is cut without an error. A list of every page in a 450-page kDB took about 26,000 tokens, and every request after it was cut to a few hundred tokens. The model never saw its tool results and kept calling tools. Estimate each request’s size before sending it, and keep tool results small: return quotes only when they are asked for, and cap long lists with a note that says what was left out.
  2. Two tables printed side by side were read as one row per printed line, so numbers from one table were filed under the other. Read each column of a two-column page separately.
  3. Papers name their own method “Ours” or “Our best model” in their tables. Resolve that to the model the paper presents, or the kDB gets a page called “ours”.
  4. A model that appears only in a results table still needs its own page, or its results link to a page that does not exist.

​
The kDB format

Your kDB is stored as a bundle in Google’s Open Knowledge Format (OKF) v0.2. A bundle is a directory of markdown files with YAML frontmatter, kept in a git repository. Read the spec. It is one file and short. Lay the bundle out like this:
OKF lets a producer add its own frontmatter keys. This project uses its own: aliases, contested_by, conditions, the location fields of a source (page, quote, bbox, and table, row, column, value), the term fields pref_label, alt_labels, broader, close_match and related_match, and question on a saved answer. They are not part of the OKF spec, so this page defines them. Each page describes one concept: a model, a dataset, a benchmark, a robot platform, a method or a result. Each page lists its evidence in the sources frontmatter. Each claim in the body cites one of those sources with a markdown footnote whose label is the source’s id. A source points to one sentence or one table cell. The example below shows the shape; the content is made up.
A dataset is a page of its own. A model names the datasets it was trained on in trained_on, with the role each dataset played and the source that says so. This dataset example is real: the quotes and bboxes are from the Open X-Embodiment paper.
A result page states the conditions under which a number holds. A claim that a newer paper disputes points to the newer sources. The third example shows both. Two numbers contest each other only when they were measured under the same conditions. A newer paper that reruns a model under perturbed camera views reports a different result under a different perturbation, on a page of its own.
The rules are these.
  1. A quoted sentence must appear word for word on its cited page. When a sentence runs onto the next page, quote the part on each page as its own source, one per page. A quote must also stay within one column. Words taken from two columns are not a sentence of the paper. A table-cell source names the table, row, column and value. The bbox of every source must match a highlight in your annotated copy of that PDF, the file with the same name in annotated/. A bbox is [x0, y0, x1, y1] in PDF points, with the origin at the bottom-left corner of the page. That is the space PDF highlights use. Pages are counted from 1.
  2. Write every resource as a path from the bundle root that starts with /, such as /papers/2601.01234.pdf. OKF reads a path without the leading / as relative to the page that contains it, so it breaks when the page moves.
  3. Each model, dataset, benchmark and robot platform has exactly one page. List its other names under aliases. You lose points if you merge two different things into one page or leave one thing split across two pages.
  4. A result always states its conditions. A page of type Result carries conditions with the task, the benchmark, the task suite, the number of trials and the embodiment, and the perturbation when there is one. Every condition except trials is the path of a Term page rather than a word. Many tables do not say how many trials stand behind a number. In that case, write trials: not reported rather than a guess. A result without conditions cannot be used.
  5. A number you compute from other numbers, such as an average gain across benchmarks, is an OKF Attested Computation concept (spec section 10). It runs a short Python script that reads only the table cells it cites, so we can re-run it.
  6. When two papers disagree, keep both claims and both sources. In contested_by, map the source id of the older claim to the list of source ids that dispute it. Do not silently overwrite a claim.
  7. Keep index.md current and append one entry to log.md for every ingest.
  8. Every sentence of prose cites at least one source. This requirement does not apply to headings, lists, tables or code blocks.
  9. A Term page lives under ontology/<ontology>/ and has a pref_label. Its broader terms are in the same ontology or in one it extends. They must never loop back to it. Each ontology has a page, ontology/<ontology>/ontology.md, whose extends lists the ontologies it specializes. Ontologies never extend each other in a loop. No two terms share a label within an ontology and the ontologies it extends. Unrelated ontologies may share a label.
  10. A page of type Dataset lists in collection how its data was collected, as terms of the datasets ontology or one that extends it. It may also list the embodiments and tasks it covers as terms. A model lists the datasets it was trained on in trained_on. Each entry names a Dataset page, a role term from the same ontologies (such as pretraining or fine-tuning) and the source on the model’s page that says so. A model’s model_class, when given, is a term.

​
The ontology

The same words name different things in the same library. LIBERO-90 is an evaluation suite in the LIBERO benchmark. It is also the set of 90 tasks the LIBERO paper uses as pretraining data. A kDB that tags results and training data with words cannot tell the two apart, so a question about one returns the other. To prevent this, conditions, dataset properties and model classes name ontology terms. Each term is one page under ontology/<ontology>/, with fields modelled on W3C SKOS: pref_label, alt_labels for the other ways papers write it, broader for the more general term, and close_match or related_match for the same idea in an established vocabulary, such as the SOMA activity ontology used in robotics. The body is a short scope note, cited like any other prose. Ontologies are layered. A general ontology describes ideas that hold in any field. A specialized one refines it for one kind of data or work by pointing the broader of its terms at the general ones. The seed has this shape for datasets: vla-datasets extends datasets, so teleoperation is a kind of human-collected data and cross-embodiment pretraining is a kind of pretraining. A specialized ontology refines the one it extends and never redefines its terms. It may not reuse a label that ontology already has. Questions follow the layers, so a question about pretraining also finds models trained with cross-embodiment pretraining. Run okf-seed kdb before your first ingest. When a paper writes “put the bowl on the plate” or “pick-place”, your agent links the result to the pick-and-place term through its alt_labels. It reads the paper’s own context to choose the right term when a word could name more than one. When the library needs a term the seed lacks, add it with a cited scope note and a broader link in the most specific ontology it belongs to. Well-chosen additions earn credit. The ontology makes some questions answerable. “Show every pick-and-place result” is a query for every Result whose task is the pick-and-place term or a term narrower than it, grouped by benchmark, suite, embodiment, number of trials and perturbation. Results in different groups are not compared, so a clean result is never set against a perturbed one. “Which VLA models were trained on the same datasets?” is a query over the Model pages whose model_class is the vision-language-action model term. The results are grouped by the Dataset pages in their trained_on, with the role each model gave the dataset. Every link in the answer is a cited source, so it can be checked like any other claim.

​
Incremental updates

Your agent must provide one command, ingest <pdf>, that adds a single paper to an existing kDB. An incremental update changes only the pages the new paper affects. Every other page must stay byte-identical, including its frontmatter. OKF defines generated.at as the time of a page’s last meaningful change. A page the new paper did not change therefore keeps its old generated stamp. A page that gained a source, an alias, a trained_on entry or a contested_by entry gets a new one, even when its body reads the same. Rebuilding the whole kDB to add one paper does not count as an incremental update. You can test this yourself at any time. Hold one of the 30 papers out, build the kDB from the other 29, ingest the one you held out, and run okf-diff on the two commits.

​
The command line

Your agent is used through one command-line program with at least these commands.
  1. ingest <pdf> adds one paper to the kDB, as described above.
  2. ask "<question>" answers from the kDB alone. Each sentence of the answer cites the sources it rests on. For each citation, the program prints the paper, the page and the quoted sentence or table value. --show also displays the cited region of the page with its highlight, inline in terminals that support images and as a PNG file otherwise. When the kDB does not contain the answer, the program says so.
  3. history lists earlier questions with their answers and citations, and history --paper <arxiv-id> lists only those whose answers cite that paper.
  4. ask --save files the answer back into the kDB as a page of type Analysis, with copies of the sources it cites and links to the pages it drew on. The page must pass okf-validate and okf-ground. If it does not, nothing is saved. Later questions can then use it, so the kDB grows from the questions you ask, as Karpathy’s LLM Wiki proposes.
Keep the question history in a SQLite file next to the kDB. Store the question, the answer, every citation, the model, the token counts, the git commit of the kDB at the time, and the full conversation. The commit records which version an answer used. When a later ingest changes a page, you can see which earlier answers relied on the old version.

​
Required and stretch goals

Required: the grounded kDB, ingest, ask with highlights, history, ask --save, every Result linked to the seed ontology, and every model page that names its training datasets in trained_on. Stretch, for credit beyond the required score: standing queries that are refreshed on every ingest (for example, a page of all pick-and-place results that updates when a new paper adds one), extensions to the ontology, and a short write-up of how your agent reads a paper longer than its context window. The write-up should explain what it keeps between chunks and how it notices a contradiction with a page it can no longer see.

​
Deliverables

  1. Submission 1: your agent’s code and command-line program, the kDB built from the 30 papers, including the ontology, and the annotated PDFs. The paper marked “ingest last” must not appear in this bundle.
  2. Submission 2: the same bundle after one ingest of the “ingest last” paper, its annotated PDF, and a findings report of at most two pages. The report lists every page that changed and why. It also lists every claim the new paper contradicts or supersedes with both sources, and anything you marked as contested.
Submit both through the course repository. We use the push times recorded by GitHub rather than timestamps in your files.

​
How you are graded

  1. Grounding. We run okf-validate and okf-ground on both submissions. A TA also reads a sample of claims to judge whether each cited sentence really supports the claim attached to it. A claim whose evidence does not check out scores zero, even if it sounds plausible.
  2. The incremental update. We compare your findings report with the diff between your two submissions, and both with our own reference ingest of the same paper. We check that the right pages changed and that the contradiction was caught and recorded. We also check that nothing else moved.
  3. Answering questions. Our question agent answers a set of questions we do not publish, using only your final bundle, with a model we serve and a fixed token budget per question. Many questions need several papers to answer. Some cannot be answered from the library at all, in which case the correct response says so. We score correct answers, the evidence they cite, and correct refusals. We compare your score with a PaperQA2 baseline that runs on the same model over the raw PDFs. To do well, your kDB must beat that baseline and use fewer tokens per answer.
  4. The demo. In a short live session, a TA asks your program questions with ask, including at least one that needs several papers and one the library cannot answer. The TA looks at the highlights with --show, saves an answer with --save and checks that the saved page passes both tools, and reads the history. We grade whether the citations are right, whether the program refuses when it should, and whether the saved page is a correct, grounded summary.

​
Constraints

  1. Models: a local model, a free OpenRouter model, or a paid OpenRouter model with a credit limit on your key. You may not send document content, page images or extracted text to any other hosted model API.
  2. Use pre-trained models only. No fine-tuning is required or expected.
  3. Do not modify the original PDFs. Annotate copies.
  4. Your agent is your own Pydantic AI program. No commercial agent product or hosted agent service may do its work, and no other paid service is allowed.
  5. You may not use AI browsers, NotebookLM, alphaXiv or any other hosted tool that reads papers for you.
  6. Runtime is not graded, but ingest of one paper must finish in a reasonable time on your own hardware.
Found an error on this page? Report an issue.