Bad data can be responsible for the loss of around $15 million per year according to a study by Gartner. Poor quality data can cost your organisation in more than one way including data storage, loss of productivity and more. Implementing processes for data pipeline monitoring will ensure your data pipelines are running in line with expectations to deliver clean, good-quality data to your data storage solution.

In this article, we will explore data pipeline monitoring, the strategies you could implement, technologies on offer and the metrics you should be measuring.

What is data pipeline monitoring?

Most organisations will be dealing with a continuous stream of data, especially coming from a wide variety of sources such as company CRM, application data, social media data and more. The data being pulled from these sources will not be all relevant or of good quality, therefore during the data pipeline process of ETL (Extract, Transform and Load), this data will be extracted, cleansed, enriched and then loaded in its destination which can vary from a data warehouse to a data lake.

In order to ensure the reliability and accessibility of the data, organisations will invest in data pipeline monitoring. This is gaining or building data pipeline observability so you can spot and resolve data gaps, delays or dropout errors and put measures in place to mitigate these errors from recurring.

What data pipeline monitoring strategies are right for you?

Several strategies should be implemented for successful data pipeline monitoring and they include:

Test, test and test

Ensuring you have a robust testing strategy in place is essential. Testing does not have to be carried out manually, in fact in disciplines such as DevOps majority of the testing is automated, this ensures the continuity of systems without straining time and resources. A similar approach can be adopted for testing data to validate your data if your data and systems are running as they should. Some commonly used tests include:

• Scheme tests

• Custom fixed data tests

Regular audits

Regular audits are also essential for quality control. Carrying out regular, timely audits whether it is weekly, monthly or otherwise depending on your data, will help you spot any errors that may cause issues further down the line. Audits will uncover the reliability, quality and accuracy of your data flowing through the pipelines.

Make metadata a priority

Metadata can play an incredibly valuable role in data error resolution, however, it has been neglected in the past. Metadata can provide a connection point between complex technology stacks and can help data engineers identify how data assets are connected to identify and resolve any errors that arise.

What technologies to use for data pipeline monitoring?

There are three main categories of tools and technologies used for data pipeline monitoring and they are often referred to as three data observability pillars. The following are the categories:

• Metrics

• Logs

• Traces

Metrics for data pipeline monitoring

Metrics are the way to measure the performance of your data and they are essential to keep track of performance over time to ensure the goals and objectives are being met. Setting the right metrics helps you understand the performance of data pipelines and how they are functioning.

Technologies for metrics:

• Cloud Watch: Infrastructure monitoring/usage (CPU, DISK, performance, failures, scaling) and set alerts for defined threshold, for errors and logs

• Periscope: For data delay, latency, data pattern, data drop/spike

• Databricks Dashboard: For data delay, latency, data pattern, data drop/spike

• AWS Console: For traffic, load, latency, throughput, query logs, queue time, wait time, workload

• Prometheus (planned): Large-scale metric collection and analysis

• Grafana: Customizable dashboards and alerts based on metrics

Key metrics you measuring:

The metrics to measure will vary depending on the type of data pipelines being monitored, however, the following are some that should be used for the majority of data pipeline monitoring.

• Error/failure

• Traffic/Load

• Latency/performance

• Availability

• Reliability

Logs for data pipeline monitoring

Logs are the next step from metrics as they require and store a higher level of detail. They are a great way to measure and track the quality of data and with the right technologies for storing and managing logs, they can be invaluable.

Technologies for logs:

• Cloud Watch: Cloud watch logs for applications and services

• Python Logger: Customized log, log separations

• Elasticsearch: For indexing and querying log data

Traces for data pipeline monitoring

Traces are another pillar of data pipeline monitoring and they will trace the data that has been taken from a specific application. Standalone, they may not offer much value but used in collaboration with other tools such as logs and metrics, they can form a complete picture for anomaly detection.

Technologies for Traces:

• AWS Trusted Advisor: for recommendations and suggestion

• AWS Console: Performance

• Apache/MWAA Airflow: Orchestration, retry, logs, performance

Build, optimise and monitor your data pipelines with Ardent

With Ardent your data is handled by experts. We take a consultative approach to understand your unique challenges, goals and ambitions in order to deliver solutions that are right for you. If you are looking to build data pipelines from scratch, optimise and improve your existing data pipeline architecture for superior performance or monitor your data for data consistency, accuracy and reliability, we can help. We have helped clients hailing from a wide variety of industries from market research to media – discover their success stories:

• Making data science efficient with trusted data availability with expert operational monitoring and support

• Powerful insights driving growth for global brands with robust, scalable data pipelines with AWS infrastructure

• Improving data turnaround by 80% with Databricks for a Fortune 500 company

Explore our data pipeline development services or operational monitoring and support service.