Skip to content

Running Project

To keep examples from becoming disposable snippets, the course reuses one synthetic corpus and one question set from Part 1 through the capstone. It is called pharma-mini. It is graph-modeling practice, not medical knowledge or treatment advice.

1. Get the three files

After cloning the repository, all three live under examples/pharma-mini/. Each document contains Korean and English text; choose either text_ko or text_en.

git clone https://github.com/desty/study-graphrag.git
cd study-graphrag
head -n 2 examples/pharma-mini/documents.jsonl

2. Domain and ontology

The corpus contains two small connected structures.

Drug ─contains→ Ingredient       Paper ─cites→ Paper
  ├──treats→ Condition            ├──usesDataset→ Dataset
  └──manufacturedBy→ Organization └──evaluates→ Drug

There are six allowed entity types and six relation types.

Entities Relations Provenance
Drug, Ingredient, Condition, Organization, Paper, Dataset contains, treats, manufacturedBy, evaluates, cites, usesDataset every edge has source_id, source_text

The goal is not to memorize this list as the answer. In Ch 4, derive a schema from the questions yourself, then compare it with the supplied ontology.yaml.

3. Competency questions and retrieval paths

The question set deliberately mixes two of each kind.

Kind Example Required path
factual What ingredient does Aspirin contain? one document or Drug → Ingredient
multihop Which other drug contains the same ingredient as Aspirin? Drug → Ingredient ← Drug
global What are the corpus's two main themes? synthesize two community summaries

This is not a benchmark engineered for GraphRAG to win every row. A tuned vector retriever should be sufficient for factual questions. The test is whether the graph earns its extra cost on multihop and global questions.

Exact counts and totals are a separate class. For example, “which condition is treated by the most drugs?” belongs in a direct aggregate query such as Cypher count(DISTINCT ...), not generative global search. Use generation only to explain the query result.

4. Artifacts to keep after each part

Assume generated files live in a git-ignored work/pharma-mini/ directory.

Stage Work Artifact
Part 1 classify questions as factual/multihop/global question-analysis.md
Part 2 derive types and constraints from competency questions ontology.yaml
Part 3 extract, review, and load triples into Neo4j triples.jsonl, load.cypher, review.csv
Part 4 run vector/local/global retrieval on the same questions retrieval-results.jsonl
Part 5 compare correctness, grounding, latency, and cost results.csv, decision.md
Capstone reproduce from scratch and document limitations README.md plus runnable scripts

This is the minimum contract for triples.jsonl. Never trust an LLM-generated statement without retaining the source_id needed to inspect the original.

{"subj":"Aspirin","subj_type":"Drug","rel":"contains","obj":"Acetylsalicylic acid","obj_type":"Ingredient","source_id":"doc-001","source_text":"..."}

5. Definition of done

  • Run retrieval before looking at the expected answer and evidence for all six questions.
  • Every extracted triple passes the ontology's allowed-type checks.
  • Every answer can be traced from a used triple back to its original source_id.
  • Vector RAG and GraphRAG use the same corpus, questions, and generation model.
  • Record retrieval latency and model-call volume, not only correctness.
  • Write an evidence-backed routing decision in decision.md.

6. Observed local run

The repository includes results-local.csv and decision.md from a run with Neo4j 2026.06.0 and local SmolLM2 360M. This is a pipeline validation record, not a universal benchmark.

  • Schema-guided extraction: micro precision 0.923, micro recall 0.706
  • Factual: both vector and graph answer match 1.000
  • Multihop: vector 0.250, graph 1.000
  • Global: graph retrieved every evidence document, but the small generator still omitted one community on one question

The point is not that GraphRAG won every number. The same run shows that extraction recall, evidence retrieval, and answer completeness are separate failure surfaces. Reproduction commands are in examples/pharma-mini/README.md.

Now start at Ch 1 by classifying the six questions. Avoid opening the supplied answers until evaluation time.