<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0"><channel><title><![CDATA[Sylvie's AI Notes]]></title><description><![CDATA[Practical notes on building production AI agents, document ingestion pipelines, and shared memory workflows.]]></description><link>https://knowhere.hashnode.dev</link><generator>RSS for Node</generator><lastBuildDate>Sat, 21 Nov 2026 19:39:13 GMT</lastBuildDate><atom:link href="https://knowhere.hashnode.dev/rss.xml" rel="self" type="application/rss+xml"/><language><![CDATA[en]]></language><ttl>60</ttl><item><title><![CDATA[Why AI Agents Shouldn't Wait for Document Parsing: Building Progressive Ingestion in Knowhere]]></title><description><![CDATA[How Knowhere's Progressive Ingestion lets AI agents query massive PDFs while parsing—stress-tested on 11,945 papers from NeurIPS, ICML & ACL.]]></description><link>https://knowhere.hashnode.dev/why-ai-agents-shouldn-t-wait-for-document-parsing-building-progressive-ingestion-in-knowhere</link><guid isPermaLink="true">https://knowhere.hashnode.dev/why-ai-agents-shouldn-t-wait-for-document-parsing-building-progressive-ingestion-in-knowhere</guid><category><![CDATA[Artificial Intelligence]]></category><category><![CDATA[Open Source]]></category><category><![CDATA[llm]]></category><category><![CDATA[RAG ]]></category><category><![CDATA[Machine Learning]]></category><category><![CDATA[parsing]]></category><dc:creator><![CDATA[Sylvie]]></dc:creator><pubDate>Sat, 14 Nov 2026 17:40:17 GMT</pubDate><enclosure url="https://cdn.hashnode.com/uploads/covers/6ad7893a22765aa4ad62b9b2/1e9236fd-fae6-4db8-abc5-a940192b0305.jpg" length="0" type="image/jpeg"/><content:encoded><![CDATA[<p>Knowhere just crossed <strong>4,000 stars on GitHub</strong>.</p>
<img src="https://cdn.hashnode.com/uploads/covers/6ad7893a22765aa4ad62b9b2/f668218b-13c4-433c-aa21-775749b9bc00.png" alt="" style="display:block;margin:0 auto" />

<p>For an open-source project built specifically to wrangle messy PDFs, complex tables, engineering blueprints, and massive documents, hitting this milestone means a lot to our team. More importantly, it validates something we've suspected for a while: getting messy, real-world documents into AI agent workflows is a massive pain point across the industry.</p>
<p>Thank you to everyone who has starred the repo, deployed Knowhere, opened issues, or submitted PRs.</p>
<p>In this post, I want to share a look under the hood at our latest architectural update—<strong>Progressive Ingestion</strong>—and how we stress-tested Knowhere on all 11,945 papers from top 2025 ML conferences.</p>
<h2>The Bottleneck: Traditional Ingestion Works Like Boiling Water</h2>
<p>Until recently, document ingestion worked like boiling a pot of water: until it hits 100°C, you just sit there and wait.</p>
<p>When you pass a PDF into a standard parsing pipeline, a lot happens under the hood: text extraction, layout recognition, table and figure parsing, section tree reconstruction, summarization, and vector indexing. Historically, a document isn't queryable until every single one of those steps finishes.</p>
<p>For a 5-page PDF, waiting a few seconds is fine. But throw in a 500-page technical manual, an annual financial report, or a dense bundle of research papers, and the UX falls apart.</p>
<p>Even if the first 200 pages are already parsed, the entire file remains locked in a <code>"processing"</code> state as long as a few complex pages are still sitting in the queue. If a user—or an agent—just wants to look up a single part number on page 12, they are stuck waiting for page 480 to finish.</p>
<p>For AI agents, this all-or-nothing blocking is a dealbreaker. Agents operate iteratively: they search, read, and gather context on the fly. Forcing an agent to wait until a document is 100% pre-processed kills its real-time responsiveness before the workflow even starts.</p>
<h2>Introducing Progressive Ingestion (<code>ingestion_mode=progressive</code>)</h2>
<p>To eliminate this latency bottleneck, we built <strong>Progressive Ingestion</strong> into Knowhere.</p>
<p>When you enable <code>ingestion_mode=progressive</code>, Knowhere stops holding your documents hostage until the entire pipeline finishes. Instead, it streams context to your agent in three distinct stages:</p>
<img src="https://cdn.hashnode.com/uploads/covers/6ad7893a22765aa4ad62b9b2/7aaf3e37-8ec6-419e-af8f-ede51a5cb1e8.jpg" alt="" style="display:block;margin:0 auto" />

<ol>
<li><strong>Stage 1: Instant Baseline (Readable &amp; Searchable)</strong> Knowhere immediately publishes a baseline version of the document so your agent can open, read, and run searches within seconds.</li>
<li><strong>Stage 2: Structural Hierarchy (TOC &amp; Section Trees)</strong> Next, Knowhere reconstructs outlines and section hierarchies, transforming a flat pile of pages into a structured document tree that agents can navigate via a Table of Contents.</li>
<li><strong>Stage 3: On-Demand Background Enrichment</strong> Deep enrichment and heavy parsing continue asynchronously in the background. Crucially, fine-grained nodes are prioritized on-demand whenever an agent actively queries those specific sections.</li>
</ol>
<p>Think of it like dining at a restaurant: instead of starving at the table until every dish for the entire table is cooked, appetizers come out immediately while the main courses arrive as they’re ready.</p>
<p>To be clear, Knowhere isn't cutting corners or dropping processing steps—it simply slashes <strong>time-to-first-use</strong> while completing the full pipeline in the background.</p>
<blockquote>
<p><strong>When to use which mode?</strong> <strong>Progressive mode</strong> is built for latency-sensitive scenarios like live knowledge bases and interactive agent apps. Standard <strong>Full Ingestion</strong> remains the default mode for batch workloads where you only want documents published after 100% of processing is complete.</p>
</blockquote>
<h3>Other Notable Updates in This Release</h3>
<p>Alongside Progressive Ingestion, we also shipped a few developer experience upgrades:</p>
<ul>
<li><strong>MinerU is now optional:</strong> It is no longer a required default dependency (disabled by default), though you can still toggle it on for fine-grained text extraction.</li>
<li><strong>Expanded image format support:</strong> Added native support for <code>GIF</code>, <code>WebP</code>, <code>BMP</code>, <code>TIFF</code>, and <code>HEIC</code>.</li>
<li><strong>Public Document Navigation APIs:</strong> External agents can now call the exact same document navigation tools as Knowhere's built-in agents via <strong>MCP</strong> or <strong>REST API</strong>.</li>
</ul>
<h2>How Progressive Ingestion Fits Into the Knowhere 2.0 Architecture</h2>
<p>Back in Knowhere 2.0, we introduced two core capabilities for complex documents: <strong>VISION-MAP</strong> and <strong>Agent-Native Retrieval</strong>.</p>
<ul>
<li><strong>VISION-MAP (Visual Parsing):</strong> Visually dense files—like scanned PDFs, slide decks, and engineering blueprints—are routed directly to vision models as full pages to preserve layout fidelity. Clean, structured files like Word and Excel go through native text parsing. Both pipelines converge into a single unified document memory.</li>
<li><strong>Agent-Native Retrieval:</strong> Instead of relying on rigid, one-shot chunk retrieval, Knowhere gives the agent autonomy to inspect the document map, invoke navigation tools, and plan its own multi-step retrieval route based on task complexity.</li>
</ul>
<p>Both features were designed for deep, multi-step reasoning over long documents. However, the longer and more complex the file, the more noticeable the initial ingestion wait time became.</p>
<p>Progressive Ingestion completes the puzzle:</p>
<ul>
<li><strong>Ask while uploading:</strong> In conversational apps, users can drop a 400-page manual into the chat and start asking questions right away.</li>
<li><strong>Zero-blocking agent workflows:</strong> New reference documents can be added mid-workflow without stalling the agent.</li>
<li><strong>No single-page bottlenecks:</strong> A few complex, diagram-heavy pages won't block the rest of the document from being used.</li>
</ul>
<img src="https://cdn.hashnode.com/uploads/covers/6ad7893a22765aa4ad62b9b2/a10047cf-098f-4e27-a5d9-1c62583e5843.jpg" alt="" style="display:block;margin:0 auto" />

<h2>Stress-Testing on 11,945 Top AI Conference Papers (2025)</h2>
<p>You can't evaluate a document ingestion engine just by running demos on a few clean, single-column PDFs. We needed a benchmark with massive scale, structural complexity, and real-world utility for researchers.</p>
<p>Top-tier ML conference papers checked every box.</p>
<p>On the surface, academic papers look standardized. In practice, they are notorious for breaking parsers: double-column text easily scrambles reading order; equations, figures, and captions float across pages; and a casual <em>"see Appendix C for setup details"</em> might point to a table buried 15 pages away. With tens of thousands of new papers published every year, nobody can read them all cover-to-cover.</p>
<p>So we ran <strong>all 11,945 accepted papers from ACL, CVPR, ICML, and NeurIPS 2025</strong> through Knowhere.</p>
<img src="https://cdn.hashnode.com/uploads/covers/6ad7893a22765aa4ad62b9b2/f24984d2-8d8a-40eb-9976-88c54a46832d.png" alt="" style="display:block;margin:0 auto" />

<p>Instead of just converting PDFs into flat text and asking an LLM for a generic summary, our pipeline worked structurally:</p>
<ol>
<li>Knowhere parsed each paper into content chunks tagged with their exact section paths.</li>
<li>It reconstructed the full section hierarchy tree so every paragraph knows whether it belongs to the <em>Methodology</em>, <em>Experiments</em>, or <em>Appendix</em>.</li>
<li>With that structural context intact, an LLM classified the paper type and extracted <strong>28 structured fields</strong>—including the research problem, core innovations, methodology, baselines, and experimental results.</li>
<li>Finally, we packaged the structured fields, grounded source snippets, and section trees into a single dataset.</li>
</ol>
<p>Why does this structure matter? Because claims stay verifiable. If a record states <em>"Method X outperforms the baseline by 5%,"</em> a researcher—or a Research Agent—can follow the section path straight back to the original paragraph to check which dataset was used, whether the experimental conditions matched, and whether the number came from the main benchmark or an ablation study.</p>
<img src="https://cdn.hashnode.com/uploads/covers/6ad7893a22765aa4ad62b9b2/3b0a1ca6-4d83-4a89-929d-45433a894f9b.jpg" alt="" style="display:block;margin:0 auto" />

<p>For Research Agents, these aren't 11,945 isolated PDFs anymore—they are a structured, searchable, and citation-verifiable knowledge base. (Note: Since this dataset was extracted via an automated pipeline without manual paper-by-paper proofreading, we still recommend verifying against the original PDF before citing specific numbers in formal publications.)</p>
<h3>Open-Source Datasets on Hugging Face</h3>
<p>We’ve open-sourced the full 2025 dataset—along with our 2023, 2024, and Embodied AI collections—on Hugging Face:</p>
<ul>
<li><strong>2025 Top ML Conference Papers (11,945 papers):</strong> <a href="https://huggingface.co/datasets/JensCS/top-ml-conference-papers-2025">huggingface.co/datasets/JensCS/top-ml-conference-papers-2025</a></li>
<li><strong>2024 Edition:</strong> <a href="https://huggingface.co/datasets/JensCS/top-ml-conference-papers-2024">huggingface.co/datasets/JensCS/top-ml-conference-papers-2024</a></li>
<li><strong>2023 Edition:</strong> <a href="https://huggingface.co/datasets/JensCS/top-ml-conference-papers-2023">huggingface.co/datasets/JensCS/top-ml-conference-papers-2023</a></li>
<li><strong>Embodied AI Papers (2023–2025):</strong> <a href="https://huggingface.co/datasets/JensCS/embodied-ai-conference-papers-2023-2025">huggingface.co/datasets/JensCS/embodied-ai-conference-papers-2023-2025</a></li>
</ul>
<img src="https://cdn.hashnode.com/uploads/covers/6ad7893a22765aa4ad62b9b2/59a667dc-d23b-4fdb-8f9e-09d1793a33ca.jpg" alt="" style="display:block;margin:0 auto" />

<h2>What's Next</h2>
<p>From VISION-MAP and Agent-Native Retrieval in Knowhere 2.0 to Progressive Ingestion today, we’ve been chipping away at the same core problem: <strong>how to get messy, real-world documents into agent workflows without the friction</strong>.</p>
<p>Parsing is only step one. Documents need to retain their hierarchy, stay searchable, remain verifiable back to the source, and become usable the moment they enter your system. Running 11,945 conference papers through Knowhere was a great stress test, and it showed us plenty of edge cases we're excited to keep improving.</p>
<p>If you're building RAG pipelines, enterprise knowledge bases, or Research Agents—and dealing with gnarly PDFs, slide decks, blueprints, or long reports—give Knowhere a try and let us know what you think:</p>
<ul>
<li><strong>GitHub Repo:</strong> <a href="https://knowhereto.ai/github/?utm_source=hashnode">github.com/Ontos-AI/knowhere</a></li>
<li><strong>Website:</strong> <a href="https://knowhereto.ai/?utm_source=hashnode">knowhereto.ai</a></li>
</ul>
<p>Thanks again for every single star ⭐!</p>
]]></content:encoded></item></channel></rss>