The 5 Challenges CTO’s face when Considering “DataOps”

Data, coined the ‘new oil of the digital economy’, now permeates every function of an organisation. Data is validating businesses and teams, but this increase in value is not without its challenges.

The old model of having a central data team has shown its limits. With data becoming a key differentiator for innovation for product teams, each team is keen to own and manage this strategic asset themselves. This ensures teams know how to make sense of data and have a more insightful application of knowledge.

Legislation is also reflecting the expectation customers have around their data and more specifically how it is being used. Due to increased awareness around privacy, areas such as data lineage in AI have emerged.

In answer to this, Data Operations movements have emerged to prioritise the value of data management.

I was delighted to host this roundtable event to discuss the opportunities and challenges of this new approach. The event was facilitated by Jean-Marie Ferdegue, the Director of Platform Engineering at Babylon Health, who lead the discussion along with 15 CIOs and CTOs from leading organisations in healthcare, retail, market research and more.

The definition of DataOps

The new data ops transcends practical processes to consider the wider aspects of data success and value. And a new responsibility has arisen for platform teams to address the question of data, how it’s managed in a compliant way and made available to product teams.

As Ferdegue explained, in similar ways to the emergence of a DevOps movement years ago, the question of data ops is less one that a specific team can address and instead one of culture for the whole engineering organisation.

Naturally, this creates questions around what constitutes a team and how to build an effective culture.

Challenge 1: Building a DataOps culture

A common challenge faced was the need for a data-first culture in organisations. One participant offered the idea that, to build a strong culture, a platform team needed to pivot from being a cost center to have a place of its own.

This would involve transitioning to being a product team, with the accountability, public roadmap and own KPI that it implies.

Challenge 2: Communication across teams

Good communication is essential in any culture, though it’s often a battle to break down knowledge siloes between different areas and teams. One attendee cited the specific challenge of bringing teams together after a merger, where one company had a certain set of processes and another used operations and tools that clashed.

The experience taught them that to resolve these differences, they couldn’t stick to their product roadmap to the letter. They had to find the areas of interconnectivity and tweak the roadmap to suit.

Challenge 3: Balance between serving immediate tactical needs and building the future data platform

It isn’t uncommon for data teams to receive tickets and inbound requests that take up all of their time and stifle creativity. This creates a tension between servicing these very real immediate needs, and the long-term goal of building a future-proofed data platform often necessary to serve long-term company goals.

Our participants agreed that this tension required a common sense and fine-grained case-by-case judgement rather than a clear choice of one or another.

Challenge 4: Storing and securing an ever-evolving data set Many emphasised the need to store and secure data across many data stores, and with varying degrees of privacy. In the case of Babylon Health, such datasets include personal information but also private health data, governed by different regulations in the different territories it operates.

This creates inherent complexity, often compounded by the technical difficulties of emerging data store technologies that have not addressed legacy use cases (the case of backing up data streaming technologies was raised by a few participants).

Challenge 5: Navigating geographical nuances

In an increasingly globalised world, this is a challenge faced by any business looking to work internationally, particularly those operating in regulated spaces. To leverage economies of scale and platform speed, the organisation must develop a global solution to local regulation, which often drives an approach of “most secure drives the pack.” Here the tightest regulation (HIPAA in some areas of healthcare for example) drive the adoption of a solution for other markets.

Challenge 6: Securing and keeping the best talent

A senior engineering event wouldn’t be true to itself if it didn’t highlight the complexity of attracting and retaining key engineering talent, with a constant trend from the industry to look for unicorn engineers.

In a similar way to the DevOps movement, which went from experienced sysadmin to DevSecOps, the data side of engineering has a rigorous series of skill-sets that are essential. People with hands-on technical knowledge and an ability to architect complex data platforms while navigating technical legal implications are a must. The participants were all in agreement that searching for this type of profile was a difficult task.

End goals

Ultimately, there was agreement that despite the challenges faced, efforts to overcome them would pay-off in the long term and were the crucial steps to unlock the value of data, without compromising its security.

​