I strongly believe that when people use facts and insights to guide their decisions, they and their organisations will be more successful. And yet, BI adoption rates (i.e. the numbers of users actually using a system they have access to) are still too low. In fact, according to the 2017 State of Analytics Adoption report, adoption actually dropped by 20% in 2016.
So why would a business user with access to data choose not to make use of it? This question has puzzled me for a long time, and now working in a BI company, I have the opportunity to help address this problem. In my professional life before joining Yellowfin, I saw Business Intelligence (BI) from both sides; as a developer of solutions and as an end-user. This has given me insight into the challenges faced by each side, both in building and using BI solutions.
Could the incorporation of machine learning (ML) into BI be the silver bullet we’ve been looking for?
The current limitations of BI
BI tools have proved enormously successful in putting data in the hands of users – providing the ability to track and manage Key Performance Indicators (KPIs), interact with and explore data visually, and collaborate and share with other users.
Frustratingly, however, large numbers of business users are simply choosing not to log in. I myself, as a former BI developer, am responsible for building solutions that had low utilisation rates. The reasons behind this have always intrigued me and I have made it one of my goals at Yellowfin to better understand this issue.
As developers of products and solutions, we would like to believe that business people love using our tools. In fact, many users don’t have either the time nor inclination to use a BI tool. What they are really looking for are the most appropriate insights to help them make better decisions, and they need to get these as efficiently as possible and at the right time.
One of the key limitations of BI tools has been that, whilst they are effective for viewing, exploring, and interacting with data, they are not very effective or efficient at uncovering hidden insight. The process of identifying hidden insights in a business intelligence solution is a manual one and requires both time and skill. In viewing their dashboards, users may notice something that they want to explore further. They can then interact in a number of ways. For example, changing chart types, pivoting, filtering and drilling down – with the hope of identifying the information they need. This form of data exploration or discovery is often done based on a hunch of what the answer might be; and as such, is subject to human bias. Finally, if the insight still eludes them, data is exported to Excel or the question is referred to a data specialist.
How Machine Learning will change Business Intelligence
Machine learning models are very effective at uncovering hidden patterns and insight in data. Data specialists have been using these techniques for many years to solve complex and specialised business problems. Now, advances in processing power have made the development and running of these complex mathematical models more accessible. Models that used to require expensive, high-end hardware are now able to be run on commodity platforms available to all.
Of late, we are starting to see a number of BI vendors incorporating machine learning capabilities into BI tools, bringing with it the promise of making BI much more effective at identifying hidden insights. BI platforms that can effectively combine these capabilities in a highly intuitive way, will soon become the norm. As users start to use this capability, they will start to expect it to always be there. Like GPS and other technologies that we now can’t imagine living without.
The combination of these capabilities effectively automates the process of uncovering insights that business users didn’t know were there. In a traditional dashboard, a business user looking at their top line sales might decide that the trend looks fine, and thus finds no need to look any further. However, hidden in the detail, in the underlying composition of the sales numbers, there may be a cause for concern. Some products may be performing well and others showing decline. This important insight is hidden from view.
In addition to finding these hidden insights, the automation of this process means that insights can be delivered much faster, thus allowing the business to act quickly and with better information. Automating these tasks should free up the time of the analyst’s role in organisations. Many analysts are performing routine tasks such as variance analysis, looking for anomalies, and authoring commentary for inclusion in reports. If these tasks are automated, the analyst can be freed up to work on higher-value tasks.
New challenges brought by Machine Learning
The automation of this insight requires additional platform capabilities to assist users in understanding what they are seeing. Even as someone who is quite data literate, there are many times I have stared for ages at a complex scatter plot or trellis chart wondering what it is trying to tell me. And those that are less data literate will find this challenge a much harder task.
The use of natural language narratives is a big step forward in helping all users to interpret and understand what the algorithms are telling us. Systems that can generate written interpretations of results add another mechanism to aid the users understanding.
Another key challenge to be addressed is trust. If a user is going to act on information provided by a machine, they need to be able to trust what they are being told. There are a number of aspects to this. First, the system needs to be transparent about what it is doing – including sharing what techniques or algorithms have been used. Second, is the ability to reduce noise. Nothing will erode trust faster than the machine throwing out dozens of meaningless correlations that confuse the user. The system needs to be able to select the most appropriate algorithm, and analyse the most appropriate data, in the context of the problem being solved. Further, analysis should be refined over time based on feedback from the user. Finally, these methods need to be used in the appropriate business context and be enabled in an intuitive way. For example, by selecting data points on a chart and enabling the analysis from a tool-tip, but also where necessary, asking the user to provide further context rather than simply assuming.
A data insights-driven approach
At Yellowfin, we believe that the combination of BI and machine learning is a critical industry development that will further empower users with insights, driving up adoption levels and thus improving the performance of our Customer’s organisations.
In our 7.4 release, we introduced our Assisted Insights capability. This is our approach to helping users uncover insights using machine learning. We have thought long and hard about the best and most intuitive way to include this in the BI workflow; how to ensure the context is maintained, noise is reduced, and trust is built. As always, we have attempted to meet the needs of different user types. We provide simple instant insights capability for business users browsing a dashboard, and more complex assisted discovery capabilities for skilled data analysts. And finally, for developers, we allow externally generated data science models to be imported into Yellowfin (using common XML standards) or called live from specialised data science platforms (such as R, H20.ai). Ahead of us, we have an exciting roadmap which will seamlessly build in greater sophistication to the platform and increasing degrees of automation and prediction.