How Language Lineage is built

Every claim on this site is a typed record with a confidence score and an evidence URL. Research agents gather the raw facts, a person reviews every change, and a set of mechanical gates rebuilds and checks the whole site before anything ships. This page follows one fact through the machine.

One record, end to end

Here is a real record from the dataset, the one behind the answer to what Python is written in:

{
  "from_language": "lang:c",
  "to_language": "lang:python",
  "relationship": "runtime_written_in",
  "start_year": 1991,
  "end_year": null,
  "confidence": 0.98,
  "evidence_source": "https://github.com/python/cpython",
  "notes": "CPython reference implementation in C"
}

Every field is doing work:

• from_language and to_language: edges read from the implementation language to the thing it implements. lang:c pointing at lang:python means CPython, the Python reference implementation, is written in C.
• relationship: one of six vetted types, listed below. Nothing vague like "related to" is allowed in the schema.
• start_year: when the claim became true. An end_year of null means it still holds.
• confidence: a hand-assigned score between 0 and 1, printed wherever the claim appears, so a well-documented fact and a reported one never look the same.
• evidence_source: a URL anyone can follow. For this record, the CPython source tree itself.
• notes: the one-line human summary shown on relationship pages.

The six relationship types

The medians come from the live dataset and they tell the truth: implementation edges are backed by source trees and score 0.92, while influence claims are softer by nature and score 0.85. The gap is shown, not smoothed over.

01 Sources

The pipeline draws on three classes of sources, chosen because each one can be checked by a machine and cited to a reader.

• Wikidata structured claims. Designers, developers, licenses, influences, and implementation languages, as machine-readable statements rather than prose.
• Wikimedia Commons. Logo files, each with per-file license metadata that the pipeline records alongside the image URL.
• Primary repositories and documentation. The evidence URLs on edges point at source trees, release notes, and official docs, like the CPython repository in the record above.

02 Research agents

Two harvest agents run offline, and both write an audit report on every run. They are deliberately boring: they fetch, resolve, and record. They do not decide anything.

The fact harvester

scripts/harvestWikipediaContent.ts, run with npm run content:wikipedia. For each of the 152 nodes it:

• Resolves the node to a Wikipedia article and its Wikidata item, following redirects. Roughly 150 ambiguous names (Io, Hack, Raku, Nix) are pinned to exact article titles in a curated map, so the agent never lands on a disambiguation page.
• Pulls the structured claims below from the Wikidata item, plus the English description tagline.
• Resolves referenced entities to names in batches of 50. When an entity has no English label, which is common for organizations, it falls back to the English Wikipedia sitelink title with the disambiguation suffix stripped.
• Behaves politely: an identified User-Agent, spaced requests, and up to four retries with backoff.
• Runs incrementally and writes an audit report with per-node fact counts and a table of everything it could not resolve.

Current coverage: 148 of 152 nodes carry cited facts. The remaining 4 (mrustc, Odin, Wren, SWC) have no Wikipedia article with a Wikidata item to cite, so their pages fall back to dataset facts alone rather than invented prose.

The logo harvester

scripts/harvestWikimediaLogos.ts, run with npm run logos:wikimedia. Its policy is strict:

• Accepts only Wikidata P154 logo claims. P18 representative images are excluded on purpose: a screenshot or a photograph is not a logo.
• Resolves each accepted file on Wikimedia Commons and records its per-file license next to the URL, so attribution ships inside the dataset.
• Only fills gaps or upgrades stand-ins. It never overwrites a curated mark.
• Rejections are written down with reasons. Wikidata claims Icon's logo is an unrelated magazine's mark and FemtoLisp's is Julia branding; both are refused by name in the script.

96 of 152 nodes currently have a logo (51 from Devicon, 39 from Commons, 6 stand-ins). 56 have none, because the standing policy is that no logo beats a wrong logo.

03 Assembly

npm run dataset:v5 (scripts/generateV5Dataset.ts) deterministically merges the base dataset, the harvested logo metadata, and a set of curated corrections into one file: lineage_v5.json, 152 nodes and 443 edges. Run it twice and you get the same bytes.

Corrections travel with their reasons. Two real entries:

The output is the single source of truth. The interactive graph fetches this exact file at runtime, every static page is generated from it, and the downloadable dataset is the same file, not an export.

04 Human review

Agents propose; they never merge. Every change to the dataset or the generators lands as a git diff that a person reads before it reaches the main branch. This is the real chronology fix from the current dataset, exactly as the reviewer saw it:

This is the stage where judgment lives:

• Every one of the 443 edges carries a hand-assigned confidence score and an evidence URL; 443 of 443 have evidence attached, which is to say all of them. 259 score 0.9 or higher, and the floor in the dataset is 0.65.
• Contested history gets adjudicated, not averaged. Self-hosting compilers create chicken-and-egg loops (rustc is written in Rust) that no scraper untangles; the bootstrap_written_in edges recording how each loop actually started are curated by hand.
• 16 languages and tools get fully hand-written deep dives instead of generated prose: Python, JavaScript, Rust, Go, Java, C, C++, TypeScript, Ruby, Haskell, and more, plus engines like V8 and BEAM.
• Policy is decided here too: the licensing rules, the logo standard, and what qualifies for each of the six relationship types.

05 Validation gates

Before any change ships, three mechanical gates run. There is no override flag; a red gate stops the release until a person fixes the cause.

npm run type-check     # gate 1: strict TypeScript across app and pipeline
npm run seo:validate   # gate 2 and 3: dataset integrity + full site audit
npm run build          # regenerate everything, then compile

Gate 1: Types

The TypeScript compiler checks the application and every pipeline script against strict dataset interfaces, so a malformed record is a build error, not a surprise in production.

Gate 2: The dataset validator

validateDataset checks the graph itself, and it runs again in your browser every time the dataset loads:

• Every edge endpoint must resolve to a real node. No dangling references, ever.
• No duplicate ids.
• Required fields on every node and edge, including the confidence score and start year.
• Any edge scoring below 0.8 is flagged for review.

Gate 3: The site auditor

scripts/validateSeo.ts audits every generated page and exits nonzero on any error. The bar it enforces:

• Exactly one h1 per page, and no skipped heading levels.
• A unique title of at most 75 characters and a unique description of 75 to 180 characters, checked across all 300+ pages at once.
• A canonical URL on every page, and one canonical host everywhere: sitemap, robots.txt, redirects.
• JSON-LD that parses, and exactly one Dataset schema on the whole site, carrying the version, the license, and a direct download URL.
• llms.txt and llms-full.txt must list every page in the sitemap.
• An embed snippet on every language page, speakable markup on question pages.
• OG images within budget: 120 kB per image, 20 MB total.

The required result is 0 errors, 0 warnings. Not few. Zero.

06 Publication

npm run build regenerates the entire site from the dataset: 131 language pages, 21 tool pages, 120 question pages, 13 guides, 6 relationship pages, the timeline, the directory, and this page. Over 300 static HTML documents, plus the sitemap and the llms.txt index for AI crawlers, both derived from the rendered pages rather than maintained by hand.

• Every language page shows its sources: Wikipedia and Wikidata links for the facts, evidence URLs for the edges.
• The interactive graph at /explore reads the same lineage_v5.json the site offers for download.
• The dataset is licensed CC BY 4.0, with a ready-made citation block on the dataset page.

Corrections

Spot an error? Open an issue on GitHub. A correction is never a patch to one page: it becomes a dataset or generator change, re-enters the pipeline at the research or assembly stage, gets reviewed like any other diff, and has to clear every gate before it ships. The loop in the diagram is literal.

Agents scale, humans decide

The division of labor is strict, and it is the reason a two-person-scale project can maintain 443 sourced claims.

Known limits

• 4 nodes (mrustc, Odin, Wren, SWC) have no Wikipedia article with a Wikidata item, so their pages carry dataset facts only.
• 56 nodes have no logo. The bar is a P154 claim with a recorded Commons license; nothing else ships.
• Influence is soft by nature. The median influence edge scores 0.85 against 0.92 for implementation edges, and both numbers are printed rather than hidden.
• History is contested. Where sources disagree, as with Rust's 2010 public debut against its 2015 stable release, the dataset picks one value and records the disagreement in the notes field.

Explore the graph these gates protect →

Or go straight to the material: download the dataset, or read the pipeline source on GitHub.