Data Sculptures Artwork by Sean Patrick Morris
All Tools

Scholar Corpus Builder

Reproducible citation-enriched research corpora. Ingests an arXiv metadata snapshot, joins each paper to its OpenAlex record, constructs the citation graph, and emits a corpus pinned by a SHA-256 content hash so any result citing it can name exactly which corpus it used.

Scholar Corpus Builder on GitHub

  • Python
  • arXiv
  • OpenAlex
  • Citation Graph
  • Research Infrastructure
  • Reproducibility

What it does

The build code for a joined, citation-linked scholarly corpus. Given a scope (source, categories, date range), it runs a five-phase pipeline and produces three artifacts.

Five phases

  • 1. IngestScan the arXiv metadata catalogue, filter to scope, deduplicate
  • 2. JoinThree-strategy join to OpenAlex with a coverage report and gate
  • 3. Citation graphExtract in-scope citation edges (citing to cited, directed)
  • 4. HardenCheckpoint, verify, compute content SHA-256
  • 5. EmitWrite catalogue.db, edges.parquet, MANIFEST.json, build.log

What it does not do

It does not embed papers, fetch full text, or perform analysis. It produces a content-addressed catalogue and citation graph that downstream tools consume through a stable reader interface. The package is the build code. The corpus is the output.

Reference scale

The production corpus behind the Conceptual Predecessor Detector: 313,012 in-scope papers from 1.79 million ingested records, 4,720,309 citation edges, 93.9 percent join coverage on a pinned snapshot.

Two commitments

Resumability

Every long-running phase checkpoints after completing, with additional checkpoints within the scan stage at a configurable interval. Checkpoints are written atomically (temp file then rename), so a crash mid-write cannot leave a truncated checkpoint. A build that dies at hour two resumes from the last checkpoint and produces output identical to an uninterrupted run.

This was not a design preference. It was paid for in full by a build campaign that lost four hours of work to a mid-run crash with no checkpoint.

Reproducibility

A finished corpus is pinned by a SHA-256 computed over a canonical serialisation of catalogue content (rows read back in paper_id order, sorted keys, fixed separators). A parquet build and a csv.gz build of the same corpus share a content SHA-256 because the hash folds the edge set independently of file format.

Proven rather than claimed: two builds from independent installs against the same snapshots produced an identical content hash.

The join

The arXiv-to-OpenAlex join is the hard part. OpenAlex does not allow querying by arXiv ID directly. Strategies run in order of decreasing reliability; every matched pair records which strategy matched it and a confidence indicator.

  • DOIExact match, confidence 1.0. Applies when the source record carries a DOI.
  • External arXiv IDThe enrichment source's own arXiv ID field, confidence 0.99.
  • Normalised titleGuarded by agreement on author surname or publication year. Confidence 0.9 when both agree, 0.8 when one does.

Safeguards: unmatched records are retained and flagged, never dropped. A title match with more than one candidate is reported as ambiguous rather than guessed. A title match is never allowed to collapse two source papers onto one enrichment record without flagging and counting it. The coverage report states overall coverage and the breakdown by strategy.

The build exits non-zero when coverage falls below the configured gate (default 0.80), so a degraded corpus cannot silently become someone's foundation. The artifacts are still written for inspection.

Citation graph

The edge list is self-contained: an edge is emitted only when both endpoints are matched, in-scope papers. Direction is explicit: source_id cites target_id. References pointing outside the scope are counted (out_of_scope_references) rather than dropped silently. Self-citations are removed and counted (self_references).

Output format is edges.parquet by default (requires the [parquet] extra), or edges.csv.gz with no extra dependency.

Install

pip install scholar-corpus

# Parquet edge format (the default --edge-format):
pip install "scholar-corpus[parquet]"

Requires Python 3.11 or newer. The core has no third-party runtime dependencies. Use --edge-format csv.gz to write edges without the parquet extra.

Usage

# Build a corpus scoped to ML categories
scholar-corpus build \
  --source arxiv \
  --snapshot ~/data/arxiv-metadata-oai-snapshot.json \
  --categories cs.LG cs.CL cs.AI stat.ML \
  --date-from 2007-01-01 \
  --enrichment openalex \
  --enrichment-snapshot ~/data/openalex-works.jsonl \
  --coverage-gate 0.80 \
  --output ~/corpora/arxiv-ml

# Build without enrichment (catalogue only, no join, no citation graph)
scholar-corpus build \
  --source arxiv \
  --snapshot ~/data/arxiv-metadata-oai-snapshot.json \
  --categories cs.LG \
  --output ~/corpora/arxiv-ml-catalogue-only

# Summary report with coverage breakdown by strategy
scholar-corpus report ~/corpora/arxiv-ml

# Re-hash and verify against the manifest
scholar-corpus verify ~/corpora/arxiv-ml

# Tool version and default artifact directory
scholar-corpus info
  • Exit 0Success
  • Exit 1Error
  • Exit 2Usage error
  • Exit 3Coverage below the configured gate
  • Exit 4Verification failure (hash mismatch)

Quality

  • 100 tests. ruff and mypy strict clean. Enforced 85 percent coverage gate.
  • Library is presentation-agnostic: it never prints, reads argv, or calls sys.exit. Callers own all input and output.
  • Determinism treated as a correctness property and tested as one: same scope, same snapshots, same content SHA-256.