Self-Healing Clinical Data Pipelines: An Agentic AI Framework for Autonomous Airflow Orchestration and Quality Anomaly Resolution

22 Mar

Authors: Naga RaviTeja Tadiboina, Sai Gowtham Tadiboina

Abstract: The growing adoption of Electronic Health Records (EHRs), Internet of Medical Things (IoMT) devices, and cloud-based platforms in the healthcare industry makes clinical data pipelines increasingly complex and requires them to be monitored continuously and reliably. Today's pipeline management systems, however, are based on rigid rules, rigid workflow scheduling, and manual troubleshooting, and do not rapidly detect the failure and recover from data quality issues. This paper presents a new solution based on Agentic AI called Self-Healing Clinical Data Pipeline Framework (SHCDP-AI) for clinical data pipelines monitoring, management and self-recovery. Combines Apache Airflow orchestration, reinforcement learning, and intelligent software agents to identify anomalies, investigate their origins, and automatically take corrective action. From the results of experiments carried out using simulated data pipeline for healthcare, it is clear that SHCDP-AI system has higher accuracy of anomaly detection (97.2%), lower average recovery time (24 s), high availability of the pipeline (99.1%), low false alarm rate (2.3%), and high efficiency of the pipeline (98.4%) than the traditional approaches to monitoring. The results indicate the potential effectiveness of the proposed framework under autonomous clinical data management for the reliability, resilience and continuity of clinical data systems in the future.

DOI: https://doi.org/10.5281/zenodo.24309147