Data Observability and Data Quality Automation: Building Self-Healing Data Pipelines

Author(s)	Ramesh Betha
Country	United States
Abstract	Modern data architectures have become increasingly complex, creating new challenges in ensuring data quality and reliability. This paper explores the emerging field of data observability and quality automation frameworks that enable organizations to build self-healing data pipelines. We present a comprehensive analysis of current challenges in data quality management, examine the evolution of observability practices from DevOps to DataOps, and propose a reference architecture for implementing intelligent data quality systems. Through case studies and empirical evidence, we demonstrate how organizations can significantly reduce data downtime, accelerate issue resolution, and build greater trust in their data assets through automated detection, diagnosis, and remediation capabilities. The paper concludes with a roadmap for future developments in self-healing data systems and guidelines for implementation across various organizational contexts.
Keywords	data observability, data quality, self-healing systems, DataOps, automation, machine learning, data reliability engineering
Field	Engineering
Published In	Volume 14, Issue 1, January-June 2023
Published On	2023-06-08
Cite This	Data Observability and Data Quality Automation: Building Self-Healing Data Pipelines - Ramesh Betha - IJAIDR Volume 14, Issue 1, January-June 2023. DOI 10.5281/zenodo.15104120
DOI	https://doi.org/10.5281/zenodo.15104120
Short DOI	https://doi.org/g897vs

About IJAIDR Fees & Payment Current Issue Publication Archive	Submit Research Paper Track Submission Status Publication Guidelines Publication Ethics Peer Review & Plagiarism	Join as a Reviewer Editors & Reviewers Reviewer Referral Program Get Reviewer Membership Certi.	Website/Journal Policies Usage Policy Content Policies Privacy Policy

Contact Us	Message on WhatsApp	+91-9687-182-185	editor@ijaidr.com

Journal of Advances in Developmental Research