Modeling Data Lineage in Neo4j: Make the Transformation a Node

The first lineage model almost everyone draws is (:Record)-[:DERIVED_FROM]->(:Record). It holds up for a while. If all you need to know is that B came from A, one relationship says so.

Edges run out of room

It stops holding up when someone asks which job produced B. Then they want to know when it ran, with what parameters and on which version of the code. Neo4j relationships can carry properties, so the obvious move is to put all of that on the edge. That breaks down quickly.

A real transformation usually has several inputs and several outputs. A run that reads three records and writes two becomes six edges, each holding its own copy of the same run metadata, which you now have to keep consistent. A relationship also can't have relationships of its own, so there's nowhere to attach the tool version, the person who triggered the run or the config file it read.

What I do instead is reify the event and give the run its own node: (:Record)-[:INPUT_TO]->(:TransformationRun {startedAt, tool, toolVersion, params})-[:PRODUCED]->(:Record). The run is stored once and connects to every input and output. You can query it directly too, so "show me every record produced by version 2.3 of the cleaning job" is a one-hop query rather than a scan over edge properties. The W3C PROV model uses the same shape (entities, activities, used, wasGeneratedBy). I take that as a decent sign this isn't just my taste.

Pick a direction

Neo4j stores every relationship with a direction. Cypher lets you ignore it in a query, but lineage is where you least want to. My rule is that lineage edges point the way data flows: input to run, run to output, source to derived. INPUT_TO and PRODUCED both point downstream, so any lineage path reads left to right as the history of the data.

W3C PROV points the other way with wasDerivedFrom, where the derived entity points back at its source. That's a valid choice too. Mixing the two is not. Once one pipeline writes (:Output)-[:FROM]->(:Input) and another writes (:Input)-[:FEEDS]->(:Output), every multi-hop query has to know which convention each segment used. The workaround that looks safe is matching undirected (-[*]-), which lets the traversal wander downstream into sibling outputs that have nothing to do with the record you asked about. The query still returns rows. They describe the wrong lineage.

So I pick one direction, enforce it at write time and name each relationship so it reads correctly in that direction.

The node everything descends from

It's natural to model a single (:Source {name: "raw-lake"}) that everything ultimately comes from. Then every record in the system sits a few hops from one node with millions of relationships. That's the supernode problem. Neo4j's index-free adjacency makes each individual hop cheap, but it doesn't make a million hops cheap. Any traversal that passes through that node fans out across its whole neighborhood. Click it in a graph browser and you get a hairball.

The fix I use is granularity. Model the ingestion batch, file or partition as its own node, something like (:IngestBatch {source, loadedAt, partition}), so no single node turns into the hub of the whole system. "The lake" becomes at most a property or a sparse parent of batches.

Bounding the path query

With that in place, the core lineage question is a bounded traversal backwards from the value you care about:

MATCH path = (src:Record)-[:INPUT_TO|PRODUCED*1..10]->(answer:Record {id: $recordId})
RETURN path

What comes back is a path someone can walk from the source batches through each run (with its parameters and start time) to the record they asked about.

Each derivation step is two relationships in the reified model (INPUT_TO then PRODUCED), so *1..10 means up to five transformations deep. Cypher won't reuse the same relationship twice within a single path, so a cycle can't loop forever. An unbounded * still enumerates every possible path, though. On a densely connected lineage graph that count grows combinatorially until the query eats your heap. I set the upper bound to match how deep the pipelines actually go. When they get deeper I raise it on purpose instead of dropping it.