> ## Documentation Index
> Fetch the complete documentation index at: https://aegean.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# The system you build

> The components your Physical AI kDB agent must use, and how its ingest and ask agents are designed.

Your agent must use the following components.

1. Agent harness: you write it, in Python, with [Pydantic AI](https://github.com/pydantic/pydantic-ai) (the `pydantic-ai` package). Your agent's tools are typed Python functions, and their arguments and results are Pydantic models. A tool can reject a bad call, such as a quote that is not on the cited page. The model gets the error back and tries again. A central part of the project is deciding which tools the model gets, what each one checks, and when the agent stops.
2. Model server: a local OpenAI-compatible server (llama.cpp, vLLM, SGLang, Ollama or LM Studio) with an open-weight model of your choice, or OpenRouter under the rules in [Models and agents you may use](https://aegean.ai/aiml-common/projects/nlp/physical-ai-kdb.md#models-and-agents-you-may-use). Your agent reads the `base_url`, the model name and the name of the API key's environment variable from a config file. It must work unchanged against any such server because, at grading time, we run it against a model we serve. A reference setup that fits a 24 GB GPU is Ollama with `gemma4:31b-it-qat`. Plan for ingest to take hours. Our reference agent, running `gemma4:12b-it-qat` on a 16 GB GPU, took about 19 minutes per paper and almost 10 hours for the whole library.

   A prompt longer than the server's context window is not rejected. Ollama drops the middle of the prompt, keeps the start and the end, and answers as if nothing happened. The only trace is a warning in the server's own log. In our tests with a 32K context, a 45,000-token prompt was cut to 16,387 tokens and the model returned an empty answer with no error. One paper converted to Markdown is 25,000 to 35,000 tokens, so passing whole papers to the model will exceed the context window. Your harness should estimate the size of each prompt before sending it. It should also compare the `usage.prompt_tokens` that comes back with what it sent. A large gap means the server cut the prompt.
3. PDF parsing: any open-source parser that keeps page positions, such as [Docling](https://github.com/docling-project/docling) or [MinerU](https://github.com/opendatalab/MinerU). You may read the PDF files directly.

   Watch the tables in two-column papers. Two tables often sit side by side, so one printed line holds a row of each. A parser that reads lines across the whole page joins the two rows into one under the first row's label. The numbers of the second table are then filed under the first. `okf-ground` does not catch this because every number really is inside its `bbox`. The table, column and conditions are still wrong.
4. Annotation: a highlight, with a comment, in a copy of each PDF for every quote and value you cite. `kdb_tools.annotate.highlight` writes them with one box per line of text, so only the quoted words are shaded. [PdfCraft](https://github.com/storytold/pdfcraft) and `pypdf` are also accepted.
5. kDB engine: your own code, called by your Pydantic AI agent, that writes the rows through actions. A tool that runs its own agent to write the kDB, such as [OpenWiki](https://github.com/langchain-ai/openwiki), is not accepted.

## Agent design

The project repository has a starter in `starter/`. Copy it into your own repository. It reads
the server settings from `kdb.toml`, builds a Pydantic AI model for any OpenAI-compatible server,
refuses prompts that would not fit the context, and has the two agents below with their output
types and validators. `kdb lake init` sets up your lake, and `kdb ingest` registers the PDF and
collects simple facts from it. Turning facts into rows is left to you: the starter's `write_rows`
raises `NotImplementedError` until you write it. Its tests run offline, with a scripted model in
place of the server and a local lake.

The checks are also available as tools your agent can call while it works. `find_quote` and
`find_value` say where a sentence or a table value is on a page, the way `okf-ground` looks for
it, and which page it is on when it is not where the model said. `check_pages` runs
`okf-validate` and `okf-ground` on the pages and rows you name, a row as `<table>/<iri>`, so your
agent can check the rows it just wrote without failing on errors an earlier ingest left. `kdb_tools.agent.toolset` gives them to
a Pydantic AI agent, and the starter's ingest agent already has them. `okf-mcp kdb` serves the
same tools over MCP.

The ingest agent reads a few pages at a time and returns typed output: facts about models,
datasets and benchmarks, each with a sentence in its own words and a quote copied from the page,
and results read from table rows. Give it the text of the pages and the table rows with their
positions, the ontology terms, and the names of entities that already have rows, so it links to
them instead of creating new ones. Its output validator looks up every quote and every table
value in the PDF with `kdb_tools.locate.find_quote` and sends back what it cannot find, with the
reason. After a few retries, drop what is still wrong rather than writing it. Your code, not the
model, then merges the facts into rows, writes the evidence with its bboxes and the highlights,
registers the annotated copy, and submits the rows with `UpsertObjects`.

The ask agent answers from the lake through tools. The starter gives it three: `list_classes`,
`query_objects`, which compares columns with plain equality and returns at most 50 rows unless
asked for more, and `get_object`. Listing every row works for a small lake and stops fitting the
context once there are a few hundred rows, mostly results. Give it tools that search instead:
results by ontology term with the narrower terms included, by benchmark or suite, or a read-only
SQL query over the tables. Its output is a list of claims, each naming a row and the entries of
that row's evidence that support it, and its validator sends back any claim whose row or evidence
entry does not exist. The ask agent has no action tools, so it cannot change the lake.

Our reference agent ran into four problems that yours will meet too.

1. A prompt longer than the context is cut without an error. One tool result that listed all 450
   entries of our reference kDB took about 26,000 tokens, and every request after it was cut to a few hundred tokens. The
   model never saw its tool results and kept calling tools. Estimate each request's size before
   sending it, and keep tool results small: return quotes only when they are asked for, and cap
   long lists with a note that says what was left out.
2. Two tables printed side by side were read as one row per printed line, so numbers from one
   table were filed under the other. Read each column of a two-column page separately.
3. Papers name their own method "Ours" or "Our best model" in their tables. Resolve that to the
   model the paper presents, or the kDB gets a model called "ours".
4. A model that appears only in a results table still needs its own row in `models`, or its
   results point at a model that does not exist.

***

Found an error on this page? [Report an issue](https://github.com/aegean-ai/eaia-feedback/issues/new?template=page-issue.yml\&page=/aiml-common/projects/nlp/physical-ai-kdb/system).


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.