In March 2013, a multinational banking and financial services company was fined by the government of UK. Why? Due to their failure to provide updated information on customer risk profiles to the authorities in time. And how did that happen? The bank’s data processing batch systems failed to collect and process the necessary data from the banking applications on time. Welcome to the 21st century, where companies can lose customers, their brand and have serious business impact because of system failures.
Today, a business can lose between $84,000 and $108,000 (US) for every hour of IT system downtime. Application problems are the single largest source of downtime, causing 30% of annual downtime hours and 32% of downtime cost, on an average. The leading cause of application downtime is software failure followed by human error. Due to the exponential data growth, increase in real time processing and demand for greater availability and performance, the data processing environment shave become very complex.
Faced with these challenges, over the years multiple analytical solutions and operations management (OM) tools were developed to help data managers monitor and measure the data warehouse environment. These solutions are meant to help identify issues, isolate causes and resolve outages. However, these features are restricted and reactive in nature rather than being predictive. Also, they do not provide a business process aligned view of issues and failures which users require. Further the operational insights are inconsistent across multiple diverse tools deployed. All these lead us to understand that there is a need for improvement in the current tools.
Let’s understand certain inputs which could enable these improvements. The most important would be predictive monitoring capabilities which can actually predict and alert failures and delays in near real time before they actually occur and provide accurate insights to troubleshoot the issues. A comprehensive view aligned to business processes is required to identify the impact on the business deliverables. Lastly, consistent visualization and alert interfaces enable data managers have a uniform view of the otherwise heterogeneous environment.
The architecture to enable this process will include real time data acquisition and conversion of operational metadata into a standard format to enable seamless integration. The real time environment parameters and metrics need to be integrated and correlated with knowledge based predictive models built on historical data to derive predictions on possible failures and delays. The architecture will use predictive analytics to quantify expected data processing behavior in real time. Along with this the alert and visualization interfaces will provide alerts and set up communication channels with various stakeholders like support teams, operations management and business stakeholders.
If we do move to this proposed architecture, organizations will benefit from reduced risks, optimized costs and standardized operations. This proposed architecture catering to improving operational IT analytics is vendor agnostic and provides consistent experience across a range of data integration and business intelligence tools. Tomorrow customer centric organizations will have to become today’s operationally efficient ones.