ResearchPilot
Grounded question answering over research papers.
Problem. An LLM answering questions about papers is only useful if each answer comes from real passages and cites them correctly.
Design (Figure 2.1). A question arrives from the Next.js chat UI at a FastAPI endpoint. Vector search and BM25 each return their top 20 passages, reciprocal rank fusion (k = 60) merges the two lists, and a BGE cross-encoder reranks them down to the 5 strongest evidence chunks. Gemini answers from those chunks as structured output that names the chunk IDs it used, and citation validation keeps only IDs the retriever actually supplied. Models and indexes load once at server start, and every response reports retrieval, generation and total latency.
- Questionfrom Next.js chat via FastAPI
- Splits into parallel paths:
- Vector searchtop 20
- BM25 searchtop 20
- Rank fusionRRF, k = 60
- Cross-encoder rerankBGE, keeps top 5
- Geministructured answer + chunk IDs
- Citation validationanswer, citations, latency
Result (Table 1). I built a 40-question benchmark across 4 papers, with evidence labelled at document and page level. I tested 8 retrieval changes against it and kept the 2 that measurably helped. The final pipeline ranks the right evidence first for 62.5% of questions, and has it in the top 5 for 92.5%. Answer correctness isn’t benchmarked yet and is the next thing I’ll evaluate.
| Supporting evidence found | Share of questions |
|---|---|
| At rank 1 | 62.5% |
| In the top 3 | 90% |
| In the top 5 | 92.5% |
