Data Discovery without governance: Descending into ‘spreadsheet hell’ all over again

“Through 2016, less than 10% of self-service business intelligence initiatives will be governed sufficiently to prevent inconsistencies that adversely affect the business.” Let’s think about how impactful this statement is for a moment.

This sentence might just be the biggest single forecast made in Gartner’s 2015 Magic Quadrant for Business Intelligence & Analytics Platforms report.

In other words, 90 percent of self-service Business Intelligence (BI) deployments lack the appropriate architecture, security and management functionality to prevent bad data being used, bad analysis being made and bad BI content being produced and acted on by your business.

The implications are enormous. In business, the only thing worse than inaction, is inappropriate action – and that’s exactly what you’ll get if acting on misguided ‘insights’ produced from an ungoverned BI environment.

Unfortunately, it seems that this situation is only worsening in the reporting and analytics industry. But, to understand why, perhaps a more historical view is required to lend the required perspective.

The IT – business divide

When BI tools first came about, they were (more or less) the exclusive domain of the IT department. Cumbersome to use, with unintuitive ‘interfaces’, an IT background and firm grasp of analytics / statistics were non-negotiable pre-requests.

The trouble with this necessarily IT-centric approach was that people of non-IT backgrounds were oftentimes unable access the data-based insights they desired to support timely decision-making processes. Those from business functions outside the IT department, who wanted to benefit from reporting and analytics, were unavoidably forced to plead for their requirements to be met. Those requirements would then have to be formerly outlined and submitted to the IT department. That request would then be placed in a queue. This process led to an inevitable bottleneck, where results were handed back to the business person several weeks or months later. This IT-ruled method resulted in three significant problems:

• The business requirements were left to be interpreted by IT folk who often did not appreciate the business needs driving the requests – a disconnect which often meant the resultant BI content did not meet the actual business requirements
• The time that elapsed between the request being lodged, and the results being delivered, often meant that the business had ‘moved on’ and the information was no longer relevant
• Even if the IT – business divide had been successfully breached – and relevant and actionable insights delivered, sharing that information with other stakeholders for decision-making purposes remained troublesome

In an attempt to combat this situation, and deliver more immediate insights and access to the business, some ‘bright sparks’ decided to try and remove IT from the BI equation…

Sacrificing trustworthiness for immediacy

This search for BI access and immediacy led to the development of desktop-based Data Discovery applications. The idea was to put data and data analysis in the hands of those outside the IT department. Though well intentioned, this may have just been one of the most detrimental movements to take hold of the BI industry.

While wrestling BI away from IT-only control might have ‘solved’ the immediacy part of the problem, it simultaneously created another, arguably far worse, situation:

• It led to a disintegration of governance and unrestrained increase in ungoverned BI environments
• Without appropriately skilled governance and oversight, the:
  • Quality, accuracy – and therefore trustworthiness – of content and ‘insights’ generated from BI tools became severely impaired
  • Security of important information became improbable, if not impossible

It’s this push for ‘self-service’ access to data and data analysis capabilities – blindly pursued by business and eagerly fueled by unabated, unquestioning market expansion – that has brought the BI industry full-circle.

At the beginning of the current iteration of the BI industry, specialist solutions and vendors sprung-up to answer the persistent problem of untrustworthy, stagnant data silos trapped inside myriad distant and disparate spreadsheets. Then, in the push to further ‘democratize’ data, and unshackle it from the tight bounds of IT’s inner-sanctum, the concept of self-service was born. This oft desktop-delivered development, designed to empower the individual, has further transformed over the past half-decade into the self-service Data Discovery we know today. But, like untrained and inexperienced adventurers without a compass, how can inexpert data explorers be expected to lead a triumphant expedition for knowledge?

The painful issue of data silos and unreliable analysis and outdated information has returned. Only, this time, it’s worse – for two reasons. Firstly, more people, with less analytical skill, have been encouraged to ‘discover’ fresh ‘insight’. Secondly, the IT department has been very deliberately shoved to one side during this transformative industry period. Oh, and enterprise-wide deployments? Forget it. Not with this ungovernable model. Aside from the practical barriers to enterprise BI, think about the cultural implications of today’s attempts at self-service Data Discovery: Pushing IT aside with one hand, then begging for assistance to somehow deliver company-wide insights with the other.

It’s this situation that led Gartner’s Research Vice President, Dough Laney, to remark: “As a result of the limited governance of self-service BI implementations, we see few examples of those that are materially successful — other than in satisfying end-user urges for data access.” Pretty blunt words from a usually reserved organization. And it’s this realization that has prompted Gartner and fellow industry experts to promote the importance of Governed Data Discovery.

Industry analyst Lindy Ryan summed-up the current quandary in her information-management.com piece Balancing Freedom and Control to Enable Governed Data Discovery: “Business-driven data discovery is becoming fundamental for organizations to explore, iterate, and extract meaningful and actionable insights from vast amounts of new and diverse internal and external data. To achieve the maximum benefits of discovery, analysts require the ability to move quickly and iteratively through the discovery process with as little friction – and as much IT-independence – as possible and with more self-service capabilities than ever before. However, with more data and more capabilities, the role of governance becomes increasingly critical to protect and secure both the data being used and the discoveries being generated.

“As constructs, data governance and data discovery seem inherently at odds. IT is held accountable for ensuring that data is accurate, complete, and secure per data owner policies, while business users want to freely explore all the data and earn insights without IT handcuffs. This is the familiar ‘freedom versus control’ paradox that seemingly puts the goals of IT and business analysts at a crossroads and brings governance to the forefront of the discovery conversation.”

So; what’s the solution to this predicament of the industry’s own making? How can we deliver on the need for Governed Data Discovery?

Welcoming expert governance to enable pervasive information sharing

The irony is, the ambition that began this detrimental cycle of self-service Data Discovery – the want for faster, broader access to fact-based insights across the business – is still the right ambition. The problem wasn’t, and isn’t, the intent. It’s the path that was taken to try and get there. By taking a desktop and / or self-service route, many technology vendors were attempting to, inadvertently perhaps, turn business people into data analysts. When was the last time you met a sales manager who desperately wanted to perform their own data mash-ups, or create their own reports from scratch?

So, how do you actually facilitate pervasive business-user-oriented BI deployments? The first step is to embrace the notion upon which most prolific social media platforms have grown their success and popularity. Studies analyzing the usage patterns of communities such as Facebook, Instagram, YouTube and LinkedIn et al show that only between one and ten percent of system users actually create and manipulate content. Around 90 percent are there to benefit from accessing, subscribing to, interacting with and sharing content with others. We should think of BI the same way. The bottom line is that the vast majority of potential BI users are business users who don’t want to be data analysts – and nor should they. BI should be delivered in a way that offers business users quick access to intuitive and reliable insights – via a single source of truth – that assists them to better perform the specific job function they were employed to undertake.

To make this ideal a reality, you need a single-integrated web-based BI platform for creating and sharing all BI content easily with other interested parties – to cater for the 90 percent of potential BI users; the BI consumer. What you don’t need, is countless user-specific tools hidden away in dark corners of dark rooms on individual devices.

Again, the irony is, to support the pervasive sharing of BI content, you need a BI environment and mindset that embraces the administrative role of data and IT experts – not one that pushes them away.

The centralized architecture of a single-integrated BI platform offers the governance features needed to use, integrate, deploy and manage BI at scale.

Giving platform administrators the required data, content, and function-based security mechanisms – to ensure that the right people have the right access to the best information possible – is critical. Specifically, a strong meta-data layer ensures that IT can deliver content creators universal access to the same information, uniform reporting definitions and failsafe approval processes.

This model of BI delivery offers:

• IT control
• Data analysts the framework and guidance required to discover great insights and create actionable BI content that can be rapidly published to consumers
• Business users the confidence to share and consume BI content pervasively as the ‘gospel truth’ on any device

As Ryan put it, achieving a governed BI environment and facilitating Governed Data Discovery is “intrinsically collaborative: A discovery layer fills the gap between business need to explore data and IT need to enforce governance. Subject experts are empowered in their business domain, and IT actively supports and takes a data management role in facilitating the needs of the business.”