In-memory Data Analysis and Agile BI
Agile is a style of working, not a methodology or process. This style includes a focus on collaboration, working software and responding to change…See this article for more info.
Recently I had the pleasure of working directly with a new client of ours – with a really interesting problem. The pleasure stemmed from seeing our in-memory analytical database in action and understanding the power of in-memory data analysis as an agile Business Intelligence tool.
It was an interesting case in that it highlighted the usefulness of in-memory analytics and the ability to rapidly proto-type reporting for the business. Basically the scenario was that this client had a data warehouse in place that had been designed and indexed for a purpose other than that intended for the reporting that was now required. The new purpose was to build an executive dashboard. The content of that dashboard was still being debated but the BI team wanted to get something into the hands of their users quickly so that they could get meaningful feedback for a more polished version.
The Problem
The BI team had all the data at their disposal – only it was incredibly granular and the queries that it supported were for branch level reporting – so every query was being filtered by branch and there were 1000’s of branches. Now they wanted to conduct aggregated analysis across all branches and business units on any daily time period – no weekly aggregations permitted. The problem this created was that the masses of data led to an average query run time was about 50 seconds and about 5 minutes to load the executive dashboard. Slow going if you want rapid builds for your agile BI project.
Now not a big problem I hear you say – why not just create a quick aggregate fact table? Easy, and these guys have an awesome tech team! Well it would have been if the data warehouse box wasn’t already 480 Gb out of 500 capacity. There was not enough space on disk to create a temp table for large queries and new disk was going to take weeks to procure (no kidding).
Step in Yellowfin’s in-memory database.
It took about 15 minutes to load the data - no disk space required. Not too bad considering the millions of records it was loading. However, the real benefit was that query times plummeted to about 5 seconds and the dashboard loaded in less than 15 seconds. What does this mean?
With vastly reduced queries times the business could now experiment with the reports they wanted on the dashboard and conduct iterative development which they could get into the hands of their end users. This was real agile business intelligence in action. Will the in-memory db be the final solution that they use? Who knows but it got them moving and understanding exactly what they needed data wise to deliver to their end user community.
Nice one Yellowifn!