International Journal of Science, Engineering and TechnologyAuthors: Srujana Parepalli
Abstract: As enterprise data ecosystems continue to expand in scale, velocity, and architectural diversity, ensuring end-to-end transparency, operational trust, and regulatory compliance has emerged as a critical and non-trivial challenge for organizations operating in data-intensive domains. Automated data lineage tracking, which systematically captures the origin, transformation logic, and propagation paths of data across heterogeneous systems, has therefore become a foundational capability for modern data governance, risk management, and advanced analytics platforms. This paper explores the evolution of automated lineage techniques, tracing their progression from early database provenance and data-warehouse dependency models to metadata-driven intelligence systems designed to operate in real-time, distributed, and continuously evolving environments. By synthesizing seminal research in warehouse lineage, standardized provenance frameworks such as W3C PROV, and distributed execution tracing mechanisms originally developed for large-scale systems observability, we present a unified architectural perspective on metadata-intelligent lineage systems. The study demonstrates how metadata abstraction, causal dependency modeling, and automated instrumentation collectively enable scalable, interoperable, and auditable lineage capabilities, supporting impact analysis, compliance verification, and operational diagnostics while laying the groundwork for self-describing and increasingly autonomous enterprise data pipelines.
DOI: https://doi.org/10.5281/zenodo.17986804
