International Journal of Science, Engineering and TechnologyAuthors: Mohana Ravi Teja Gutta
Abstract: Cloud-based Extract, Transform, and Load (ETL) processes have become important aspects of contemporary data management systems because of the growing amount of enterprise data being produced from various sources due to high volume, velocity, and variety. Nevertheless, typical ETL processes are subject to high latency, ineffective resource usage, high operational cost, and low scalability in managing large-scale tasks in cloud settings. To solve these problems, the Adaptive Performance Optimization Framework for Cloud-Based ETL Workflows (APOF-CETL) is presented in this paper that considers workload-dependent resource management, intelligent data partitioning, adaptive scheduling, and feedback-based optimization mechanisms. The experiment was performed on large-enterprise datasets, with the size of the workload varying between 100 GB and 500 GB. The proposed algorithm APOF-CETL was tested against two contemporary workflow scheduling algorithms – MOPWSDRL (Multi-Objective Workflow Scheduling using Deep Reinforcement Learning) and GRWS (Graph Attention Network-Based Reinforcement Workflow Scheduling). The findings show significant gains in various metrics. With the help of the proposed approach, processing latency was lowered to 298 ms, throughput was increased up to 421 GB/min, resource utilization efficiency became equal to 96.0%, and the success rate for large-scale workloads reached 99.8%. Additionally, the proposed method lowered the execution cost and energy expenditure, preserving its high scalability and successful task completion rate.
DOI: https://zenodo.org/records/22957463
