Data Lineage

Data lineage answers the question: where did this number come from? A revenue figure on an executive dashboard may draw on orders from an e-commerce platform, adjustments from a finance system and exchange rates from an external feed, combined and transformed by several data pipelines before it reaches the screen. Lineage records each of those steps, so that anyone can follow the figure back to its sources and see what happened to it along the way.

Lineage has become essential as data flows have grown more complex. When a report shows an unexpected number, lineage shows where to look. When a source system changes, lineage shows which reports and models will be affected. Regulators in finance, healthcare and other sectors increasingly expect organizations to show how reported figures were produced. For enterprise architecture and IT finance, lineage reduces the time spent investigating data problems and the risk of making decisions on flawed figures.

Definition of Data Lineage

Data lineage is the documentation of data's origins, movements and transformations across an organization's systems. It typically shows sources, the processes that move and change data, and the destinations where data is stored or used.

Lineage is a key element of data governance and is often displayed in a data catalog, where users can see the upstream sources and downstream uses of any dataset.

Levels of Lineage

System-level lineage shows how data moves between applications and platforms, such as from a customer system to a data warehouse to a reporting tool. It is useful for architecture planning and impact analysis.

Dataset or table-level lineage shows which tables or files feed into others, which is the level most data teams use day to day.

Column or field-level lineage shows how individual fields are derived, including calculations and transformations. It is the most detailed and the most valuable for regulatory reporting and error investigation.

Business lineage presents the same information in business terms, showing how a reported measure is built from business concepts rather than technical tables.

How Lineage Is Captured

Automated lineage tools parse the code of data pipelines, SQL queries and integration jobs to identify how data moves and is transformed.

Some data platforms record lineage natively as data is processed, providing accurate and current information without separate parsing.

Manual documentation fills gaps where automation cannot reach, such as spreadsheets, manual transfers and older systems. Most organizations combine automated and manual methods.

Importance of Data Lineage in Enterprise Architecture

Lineage builds trust in data. When users can see where a figure comes from and how it was calculated, they are more likely to rely on it. It also helps resolve disputes when two reports show different figures for the same measure.

It speeds up problem solving. Tracing an error back to its source through lineage takes far less time than investigating each system by hand. Without lineage, data teams often spend days questioning colleagues and reading pipeline code to find where a figure went wrong.

It supports compliance. Regulations covering financial reporting, risk data and personal data often require organizations to demonstrate how data was produced and where it flows. Privacy laws, for example, require organizations to know every system that holds or receives personal data.

Impact Analysis and Change Planning

Before changing a source system, database structure or data pipeline, teams can use lineage to see every downstream report, model and application that will be affected.

This reduces the risk of breaking reports or analytics during upgrades, migrations and application rationalization.

Lineage also reveals redundant pipelines and unused datasets, which can be retired to reduce processing and storage costs.

For architects, system-level lineage complements application dependency mapping, showing how data flows alongside how applications connect.

Common Challenges

Lineage is hard to capture completely. Data passes through many tools, some of which do not expose their logic to automated tools. Spreadsheets and manual steps are the most common gaps, and often sit at critical points in reporting processes.

Lineage goes out of date when pipelines change, unless it is captured automatically or updated as part of change processes.

Very detailed lineage can be overwhelming. Presenting the right level of detail for each audience, such as system-level views for executives and column-level views for engineers, makes it usable.