Journal of Frontiers in Multidisciplinary Research  |  ISSN: 3050-9726  |  Double-Blind Peer Review  |  Open Access  |  CC BY 4.0

Current Issues
     2026:7/1

Journal of Frontiers in Multidisciplinary Research

ISSN: 3050-9718 (Print) | 3050-9726 (Online) | Impact Factor: 8.10 | Open Access

A Conceptual Framework for Automating Data Pipelines Using ELT Tools in Cloud-Native Environments

Full Text (PDF)

Open Access - Free to Download

Download Full Article (PDF)

Abstract

The rapid growth of cloud-native architectures has transformed how organizations manage and operationalize data. In this context, automating data pipelines has become critical for ensuring agility, scalability, and efficiency in handling large-scale, dynamic datasets. This paper proposes a conceptual framework for automating data pipelines using Extract-Load-Transform (ELT) tools in cloud-native environments. Traditional ETL approaches often struggle with the volume and velocity of modern data workloads, whereas ELT strategies leverage the scalable compute power of cloud services to transform data post-loading, thereby enhancing performance and flexibility. The proposed framework integrates key cloud-native principles—such as containerization, microservices, serverless computing, and infrastructure as code—to orchestrate the data lifecycle from ingestion to insight. It emphasizes modular pipeline design, metadata-driven orchestration, dynamic schema evolution, and real-time monitoring to achieve seamless automation. Critical considerations such as data quality assurance, governance, security, and cost optimization are embedded throughout the pipeline's architecture to meet enterprise demands. By utilizing tools such as Apache Airflow, dbt, Fivetran, and cloud-specific services like AWS Glue and Google Cloud Dataflow, the framework enables automated, scalable, and resilient data operations. The paper also discusses the importance of decoupling compute and storage, employing containerized workloads for pipeline tasks, and leveraging managed orchestration services to reduce operational overhead. Case studies highlight the framework’s effectiveness in accelerating data delivery, improving reproducibility, and lowering time-to-insight across diverse industries. Furthermore, the framework addresses the challenges associated with multi-cloud strategies, hybrid architectures, and evolving data compliance requirements. Future research directions include integrating AI-driven optimizations for dynamic workload balancing, adaptive pipeline healing, and intelligent schema inference. Overall, this conceptual framework provides a robust foundation for enterprises seeking to modernize their data infrastructure and harness the full potential of cloud-native ecosystems. It lays the groundwork for continuous innovation in data engineering practices, positioning automated ELT pipelines as a cornerstone for next-generation analytics, machine learning, and digital transformation initiatives.

How to Cite This Article

Ayorinde Olayiwola Akindemowo, Eseoghene Daniel Erigha, Ehimah Obuse, Joshua Oluwagbenga Ajayi, Ayobami Adebayo, Afeez A Afuwape, Anthonette Adanyin (2021). A Conceptual Framework for Automating Data Pipelines Using ELT Tools in Cloud-Native Environments . Journal of Frontiers in Multidisciplinary Research (JFMR), 2(1), 440-452. DOI: https://doi.org/10.54660/.JFMR.2021.2.1.440-452

Share This Article: