What are the key components of Yellowfin Architecture

The following is a description of the key functional components of the Yellowfin architecture.

Configuration Database – stores all configuration data required by Yellowfin including information about users, security, data sources and content created in Yellowfin. Content created includes definitions of Reports, Charts, Dashboards, Presentations and Stories. Can be implemented in a variety of relational databases including SQLServer, MySQL and Oracle.

Reporting Data Sources – Yellowfin connects live to a wide variety of data sources including most popular relational and non-relational databases (with an appropriate jdbc driver), cubes (via MDX), flat files and API end-points (including out of the box connectors to popular cloud-applications). This framework is extensible, and Customers can create their own connections or use a third-party connection if one is available.

ETL Engine (optional) – this is an optional component that clients can utilise if they need to further transform their data to a form suitable for analytics. Data can be read from any source, combined, transformed and is then written back to any writeable DBMS that we support. A wide variety of transformation steps are available including merging data, splitting data, creating new columns, changing data types, replacing values and so on. In addition, Customers can connect an ETL work-flow to a third-party Data Science model to enhance data with the results of a predictive algorithm. We currently support models exported in pmml format, as well as live connections to R Servers, H20.ai and AWS SageMaker. The ETL step framework is extensible and Customers can build and plug-in their own custom-ETL steps.

Meta-data layer – this layer contains all of the necessary information about the data from which reports are to be built. This includes the table structures, join conditions, calculated fields, data formatting and so on. This data is then used by the Query Engine which generates DB specific queries in the form of SQL, MDX or other calls to dynamically query the live data connection. Data can also be sourced from a Stored Procedure or hand-coded SQL. Aspects of security are applied at this level including the application of access filters and/or data source substitution in order to support logical or physical segregation of data in a multi-tenant environment.

Yellowfin Report – this module is responsible for requesting data from the Query Engine, enhancing that data where necessary and generating the requested visualizations. Query Definitions contain the necessary detail of exactly which data is required to produce each report or visualization. There are various sub-engines which perform additional processing on the data once it is returned from the query engine:

• Post-processing Engine – this engine takes the raw data from a query and prepares it for presentation. This includes combining data from disparate data sources, performing cross-tabulation of data, converting data types, formatting data and also enriching data by performing complex post-processing. Post-processing can include the execution of advanced statistical calculations (such as percentage of total, StdDev, forecasts etc) and calling out to third-party applications to enrich data (such as invoking a predictive data-science model using the same framework as the ETL engine). Post processing functions are extensible, and clients can write and include their own functions to perform specific processing prior to data being displayed.
• Visualization Engine – this engine generates the visible content for users including Reports, Charts, Dashboards, Presentations and Stories. This framework is also extensible, for example Javascript charts can be included as well as other custom-built UI components.
• Raw Data Provider – the data sets returned from reports are accessible via API calls. Customers can perform custom processing on the underlying data sets prior to the data being presented in a Yellowfin visualization, or a custom UI element developed by the Customer.
• Data Cache – Yellowfin can be configured to cache data in order to speed up the end-user experience. A variety of data can be cached including actual report data, filter values, favourites and so on. Data refreshes can be scheduled via the Yellowfin scheduler.

Content Layer – Content can be consumed via the Yellowfin application from most popular browsers, from the native Yellowfin mobile App (IOS or Android) and individual components can be embedded into applications via API calls. Users will typically interact with Yellowfin Visualizations via a Dashboard, Presentation or Story – or a visualization embedded directly into a third-party application. The content layer can be reskinned and customized to meet the needs of the Customer. Other capabilities are available at the content layer including:-

• 3rd party content – commencing with the Story product, Customers can embed visualizations from third-party BI apps into Yellowfin. Content from Tableau, Qlik and PowerBI can be integrated and viewed alongside content from Yellowfin.
• Timeline Feed – the timeline feed serves personalised alerts and relevant content to users via the Yellowfin application, the mobile app or via a REST API.

Scheduling Engine – contains a flexible scheduling engine to schedule ETL jobs, Report distribution, Signals jobs and other system tasks. Some scheduled tasks can be triggered based for example on pre-defined thresholds being exceeded.

Signals Engine – the Signals engine automatically analyses data to identify data events of significance, and then generates personalized alerts that are delivered via the Timeline Feed, either in the browser client or mobile client. Signals jobs are configured in and leverage the meta-data layer and can be scheduled to run at any frequency using the Scheduling engine. The Signals engine is complex and contains a time-series generation engine, multiple event detection algorithms, a ranking and personalisation engine, and a notification engine.

Assisted Insights Engine – this is a set of algorithms that are invoked to automatically generate data insights on the fly. These insights are accessible through direct user interaction with a chart visualisation, through the report builder process and also to provide deeper explanations behind Signals.

What are the key components of the Yellowfin platform architecture?

Yellowfin consists of a Java Application Server (that runs on Apache Tomcat), and Repository Database. The Repository Database can exist on one of the following supported database systems: PostgreSQL, Microsoft SQL Server, MySQL, Oracle, DB2 or Ingres.

The application can be scaled for greater capacity and failover by clustering multiple Yellowfin Application Servers against the same repository database.