Data Engineering

Data Engineering: From the Engineering Team to the Boardroom

Data Engineering: From the Engineering Team to the Boardroom

Data Engineering: From the Engineering Team to the Boardroom

Data engineering delivers its greatest value when technical improvements translate into measurable business outcomes. David Vallance explores how leaders can connect platform performance, trusted data and AI investment to productivity, efficiency, growth and profitability.

Data engineering delivers its greatest value when technical improvements translate into measurable business outcomes. David Vallance explores how leaders can connect platform performance, trusted data and AI investment to productivity, efficiency, growth and profitability.

David Vallance

5 minutes

KEY TAKEAWAYS

What you

need to know

Data engineering success should be measured by the business outcomes it enables, not technical performance alone.

  • Faster processing, improved reliability and better data quality only create value when they improve decisions, efficiency, customer outcomes or commercial performance.

  • The cost of maintaining legacy data environments can appear as lost engineering capacity, slower innovation and delayed commercial opportunities.

  • AI investment needs a measurable business objective and a trusted data foundation if organisations are to realise meaningful returns.

  • Technical, operational and business measures should be connected from the start so engineering teams and leadership share the same definition of success.

David Vallance | Director of Business Development & Sales, Ardent

David has over 25 years' experience in technology, business development and commercial leadership, spanning IT solutions, managed services, digital strategy and business transformation. He has worked with numerous global enterprises, helping harness technology to improve customer experience, operational efficiency and business performance.

Today, as Director of Business Development & Sales at Ardent, he brings that commercial perspective to data engineering, AI-ready data and modern data platforms. His focus is on connecting technology investments to measurable business outcomes, from greater productivity and smarter decision-making to business growth and profitability.

Why the real value of data engineering should be measured in business outcomes, not just technical achievements.

Much of my career has been spent helping organisations use technology to solve business problems, from large-scale digital transformation programmes to complex, integrated digital platforms.

The technology has evolved, and my focus today is data engineering, but one principle has remained remarkably consistent.

We're very good at explaining what technology does. We're not always quite as good at explaining what it means for the business.

And I wonder whether, in some cases, we're asking the wrong questions.

Impressive engineering. But what's the business outcome?

Working in data engineering, I'm regularly struck by the scale and sophistication of what's possible. We're talking about platforms processing hundreds of millions of records every hour, complex pipelines integrating multiple data sources, and migrations that can dramatically improve processing performance.

These are impressive technical achievements. They take considerable expertise to deliver, and I certainly wouldn't underestimate their significance.

But coming from a commercial background, my instinct is to ask a simple question.

So what?

I don’t mean that dismissively, quite the opposite. A platform processing 224 million records an hour is an extraordinary engineering achievement, but the number alone doesn’t tell me what changed for the business.

If a data platform delivers information 80% faster, what does that actually enable? Perhaps customers receive insights sooner, services become viable that weren’t before, or employees spend less time waiting for information and more time acting on it. Does that translate into lower infrastructure costs, accelerated revenue opportunities or a better customer experience?

Or have we simply made an existing process faster?

There’s an important distinction between improving technical performance and delivering measurable business value: the two are connected, but they’re not necessarily the same thing.

Technology is the means, not the end

In my previous work delivering large-scale digital platforms and transformation projects, the technology was rarely the ultimate objective.

We might have been building an enterprise website, an e-commerce platform or a member portal integrated with CRM, membership management, learning and event systems.

But these weren’t simply website projects; they were often business-critical platforms, bringing together multiple systems, processes and sources of information to improve how an organisation operated and served its customers.

Success might have meant increasing online sales, improving member engagement and retention, reducing administrative overhead or giving employees more time to focus on higher-value activities.

For example, a membership organisation might invest in a new digital platform that allows members to manage subscriptions, book events, access learning resources and update their information without contacting an administrator.

The technical delivery involves system integration, data synchronisation, workflow automation and creating a reliable user experience.

But the business case is about something else: reducing the cost of servicing members, improving satisfaction and retention, and enabling the organisation to grow without increasing administrative resources at the same rate.

That's a very different measure of success from simply launching a new website.

And I see a direct parallel with data engineering, where a business might need to migrate from Amazon Redshift to Databricks, modernise its data architecture or improve the reliability of its reporting. Those may be entirely valid technical requirements. But before discussing how to deliver them, shouldn't we understand why they matter?

The business impact of unreliable data

Consider something as apparently straightforward as management reporting.

Many organisations have invested heavily in business intelligence platforms, dashboards and analytics tools. Yet different departments can still produce different answers to what should be the same question.

Finance reports one figure for active customers, Marketing another and Operations a third. It’s tempting to blame the reporting tools, but the disagreement may start with what each department means by “active”.

One system might include trial users while another excludes them, with cancellation dates, renewal periods and customer status interpreted differently too. The pipelines could be working perfectly; the business simply hasn’t agreed what the numbers mean.

Solving this requires more than a new dashboard. It may involve establishing consistent business definitions, improving data governance, reconciling information across source systems, and introducing quality controls that identify discrepancies before they reach decision-makers.

These are data engineering and governance challenges, but the consequences are commercial.

If a subscription business cannot accurately measure churn, how confidently can it make decisions about customer acquisition spending? When departments disagree on the underlying figures, revenue forecasting becomes harder and time is spent reconciling information rather than using it.

The value of trusted reporting isn't simply having better dashboards. It's enabling better decisions, made with greater confidence and less wasted effort.

Faster data should enable a faster business

Another example is the modernisation of data platforms.

Technologies such as Databricks and Snowflake can help organisations process and analyse very large volumes of information, integrate different data sources and scale computing resources as demand changes.

Moving from legacy architectures to more modern platforms can improve processing performance, scalability, reliability and operational efficiency.

In some environments, separating computing resources for different workloads can also help organisations manage performance and infrastructure expenditure more effectively.

From an engineering perspective, these are significant improvements, but from a business perspective, the questions should go further.

If a market research company can process complex datasets faster, could it deliver more projects without increasing headcount at the same rate, improve profitability or win more business by getting insights to clients ahead of competitors?

If a media business can analyse audience behaviour more frequently, could it use those insights to increase viewer engagement and retention, strengthen its advertising proposition and ultimately grow revenue?

If a retailer can bring sales, inventory and customer information together more reliably, could it reduce lost sales from stockouts, lower the cost of excess inventory, improve margins and increase customer lifetime value?

Those are the commercial outcomes that can justify the investment.

Faster processing, better reporting and more reliable data are important achievements, but they’re ultimately valuable because of what they enable the business to do differently and the financial or commercial results that follow.

Of course, improved technical capability doesn’t automatically produce those results: business processes, people and decisions must change too. But that’s precisely why understanding the intended outcome at the outset matters.

Otherwise, we risk celebrating a successful technical implementation without knowing whether the organisation has realised its potential value.

What about the cost of doing nothing?

There's another dimension to the business case that I think deserves more attention. We naturally focus on the cost of implementing a new solution. But what is the cost of leaving the existing environment unchanged?

Consider an organisation with a capable internal data engineering team whose engineers might spend a significant proportion of their time monitoring pipelines, investigating failures, fixing integrations, reconciling inconsistent data and maintaining legacy processes.

All necessary work, but every hour spent keeping existing infrastructure running is an hour potentially unavailable for developing new capabilities.

The organisation might want to launch AI-powered services, introduce automation or create new data products, yet its most valuable technical people are consumed by maintaining what already exists.

The business cost isn't necessarily visible as a line item in the IT budget. It might appear as delayed product launches, slower innovation, missed commercial opportunities or an increasing reliance on a handful of individuals who understand critical systems.

This is where specialist engineering support and Managed Data Ops can contribute value.

By taking responsibility for agreed areas of platform monitoring, maintenance, reliability and optimisation, external specialists can work alongside internal teams, reducing operational pressure and creating more capacity for development.

The commercial case isn't simply whether external support is cheaper. It's whether the organisation can make better use of its existing people, reduce operational risk and deliver its priorities faster.

AI: An unprecedented opportunity, but where's the business case?

AI represents an extraordinary opportunity for organisations to unlock more value from their data, and the potential goes far beyond automating existing tasks. We’re talking about fundamentally changing how businesses operate, serve customers, develop products and create new revenue opportunities.

Organisations can analyse information at previously impractical speeds and scales, identify patterns that might otherwise go unnoticed, and make intelligence accessible to people who previously depended on specialist teams to retrieve and interpret it.

For many businesses, the opportunity is genuinely transformational, and recent developments are bringing that potential closer to everyday business operations.

At Microsoft Ignite 2026, Microsoft announced further integrations for Fabric IQ, extending access to governed business data and context through familiar tools such as Copilot, Excel and, in preview, ChatGPT and Claude.

The significance isn't simply that AI can access more information. It's that these tools can draw on consistent business definitions and trusted metrics, rather than leaving each application to interpret the underlying data independently.

Imagine a CFO being able to explore profitability across products and regions, or a COO identifying emerging operational inefficiencies, simply by asking questions of the organisation's governed data.

The potential to make information more accessible, accelerate decisions and identify new commercial opportunities is enormous.

But there's a crucial dependency.

As Clive Skinner, our Head of Architecture at Ardent, has observed, without the right foundations, AI risks simply scaling ambiguity.

If Finance and Sales already disagree about revenue definitions, or different systems calculate customer retention differently, making that information available through AI doesn't resolve the problem. It can simply make inconsistent answers easier to generate and distribute.

That's why a robust data strategy, clear business definitions, appropriate governance and reliable data engineering matter more than ever.

But even trusted data and sophisticated AI don't, by themselves, establish the commercial return.

If AI allows employees to analyse information in minutes rather than hours, what does the business gain? Can teams make better decisions, reduce costs, improve margins or identify revenue opportunities that would otherwise have been missed?

And if AI creates additional capacity, how is the organisation using it?

We've already seen examples of companies making workforce decisions based on anticipated AI efficiencies, only to reconsider when the expected capabilities or savings haven't fully materialised.

The opportunity is too significant to approach through technology adoption alone.

We need to connect three things from the outset: a measurable commercial objective, an AI strategy designed to achieve it, and a trusted data foundation capable of supporting it.

Because the greatest opportunity with AI isn’t simply to do what we already do faster, but to achieve things that weren’t previously possible and translate those possibilities into measurable business value.

Connecting the boardroom and the engineering team

Processing throughput, latency, system availability, data quality, infrastructure costs and pipeline reliability are essential measures of engineering success. They help technical leaders understand whether systems are performing as intended.

But they don't always tell the complete story.

The challenge is that the connection can become less visible as a programme moves from its original business case into technical delivery.

A transformation initiative might begin with an objective to reduce operational costs. That becomes a requirement to improve data integration, then a project to modernise pipelines. And ultimately, success gets reported as a 70% reduction in processing time.

That's a valuable engineering result. But have we established whether operational costs actually fell?

A CTO may need to demonstrate that a platform is more reliable and scalable. A CFO needs to understand the investment and financial implications. A COO may be more interested in operational efficiency, business continuity or delivery capacity.

They’re looking at the same investment through different lenses, and the opportunity is to connect those perspectives.

A useful approach is to establish three levels of measurement:

Level

What to measure

Examples

Technical improvement

What will change within the platform?

Faster processing, fewer failures, better data quality or lower infrastructure consumption

Operational benefit

What will this enable people and processes to do differently?

Reduce manual reconciliation, accelerate reporting, improve service reliability or release engineering capacity

Business outcome

What measurable difference should follow?

Lower operating costs, faster customer delivery, improved retention, reduced risk or additional revenue opportunities

These measures won’t always be immediately available, and not every benefit can be attributed directly to one engineering initiative. But agreeing what matters, establishing a baseline and identifying who owns the outcome gives everyone a clearer definition of success.

Perhaps we need to start with better questions

Some of the most important questions in a data engineering project aren't necessarily technical.

Before committing to a significant investment, I'd want to understand:

  • What business problem are we actually trying to solve?

  • What is that problem costing us today, financially or operationally?

  • What will the organisation be able to do that it cannot do now?

  • How will we measure success, and who will be accountable for realising the benefit?

  • What happens if we don't address the problem?

These questions shouldn't only shape the initial investment decision. They should be revisited throughout delivery and beyond, with commercial and technical leaders sharing responsibility for measuring whether the intended business outcomes are actually being achieved.

Engineering excellence needs commercial purpose

The challenge isn't simply whether we can build increasingly sophisticated data platforms and AI capabilities. It's whether we're consistently connecting that investment to something the business genuinely values.

A platform that processes hundreds of millions of records an hour is impressive. A platform that enables a business to serve customers faster, reduce costs, protect revenue or create new opportunities is commercially compelling.

Ideally, we should be able to demonstrate both.

Because ultimately, the measure of successful data engineering shouldn't just be how well the technology works. It should also be what the business can achieve because of it.

And if we can't answer that question, perhaps we need to ask ourselves whether we've really finished the job.

Data Engineering and Business Value FAQs

Frequently asked questions

The business value of data engineering comes from what reliable, well-designed data platforms enable an organisation to achieve. This can include faster decision-making, lower operating costs, improved customer experiences, greater productivity, reduced risk and new revenue opportunities.

Organisations should connect three levels of measurement: the technical improvement, the operational benefit and the resulting business outcome. This helps ensure that engineering performance remains connected to what the organisation is ultimately trying to achieve.

Faster processing is an important engineering achievement, but speed alone does not demonstrate business value. The more important question is what that improvement enables, such as faster customer delivery, greater capacity, reduced costs or better decision-making.

Unreliable or inconsistently defined data can create disagreement between teams, weaken forecasting and cause employees to spend time reconciling figures rather than using them. Trusted reporting enables decisions to be made with greater confidence and less wasted effort.

The cost can extend beyond the IT budget. Engineering teams may become consumed by monitoring pipelines, resolving failures and maintaining legacy processes, leaving less capacity for innovation, automation, AI and new commercial initiatives.

AI investment should connect a measurable commercial objective, an AI strategy designed to achieve it and a trusted data foundation capable of supporting it. Greater AI capability does not automatically produce business value without that connection.

NEWS

Latest Insights

Explore expert perspectives, practical advice and fresh thinking on data engineering, AI and technology. Drawing on real-world challenges and hands-on experience, our insights share lessons, emerging trends and ideas to help businesses make smarter technology decisions.

Data Engineering

Data Engineering: From the Engineering Team to the Boardroom

Data engineering delivers its greatest value when technical improvements translate into measurable business outcomes. David Vallance explores how leaders can connect platform performance, trusted data and AI investment to productivity, efficiency, growth and profitability.

5 minutes

Read insight

AI Readiness

Managed Data Services: What Should You Outsource and What Should Stay In-House?

Managed data services work best when operational responsibilities are handed over without losing strategic control. This guide explains what to keep in-house, what to outsource and when co-managed or managed data operations make sense.

6 minutes

Read insight

databricks and snowflake on red and blue background

AI Readiness

Databricks vs Snowflake: Whats right for you?

AI is changing the Databricks vs Snowflake debate. We look at custom model development, AI using trusted business data, analytics, day-to-day running and total cost to help you choose an approach that fits your organisation.

7 min read

Read insight

Data Engineering

Data Engineering: From the Engineering Team to the Boardroom

Data engineering delivers its greatest value when technical improvements translate into measurable business outcomes. David Vallance explores how leaders can connect platform performance, trusted data and AI investment to productivity, efficiency, growth and profitability.

5 minutes

Read insight

AI Readiness

Managed Data Services: What Should You Outsource and What Should Stay In-House?

Managed data services work best when operational responsibilities are handed over without losing strategic control. This guide explains what to keep in-house, what to outsource and when co-managed or managed data operations make sense.

6 minutes

Read insight

databricks and snowflake on red and blue background

AI Readiness

Databricks vs Snowflake: Whats right for you?

AI is changing the Databricks vs Snowflake debate. We look at custom model development, AI using trusted business data, analytics, day-to-day running and total cost to help you choose an approach that fits your organisation.

7 min read

Read insight

Data Engineering

Data Engineering: From the Engineering Team to the Boardroom

Data engineering delivers its greatest value when technical improvements translate into measurable business outcomes. David Vallance explores how leaders can connect platform performance, trusted data and AI investment to productivity, efficiency, growth and profitability.

5 minutes

Read insight

AI Readiness

Managed Data Services: What Should You Outsource and What Should Stay In-House?

Managed data services work best when operational responsibilities are handed over without losing strategic control. This guide explains what to keep in-house, what to outsource and when co-managed or managed data operations make sense.

6 minutes

Read insight

databricks and snowflake on red and blue background

AI Readiness

Databricks vs Snowflake: Whats right for you?

AI is changing the Databricks vs Snowflake debate. We look at custom model development, AI using trusted business data, analytics, day-to-day running and total cost to help you choose an approach that fits your organisation.

7 min read

Read insight

Ready to make your data work harder?

We build and manage the data foundations that remove the quality, consistency and reliability issues holding back AI, automation and better decisions.