Will you test a jet-ski engine and use the results of the test to extrapolate the engine’s performance on a cruiseship? I bet your answer is an emphatic “No”.Then, why is it that in enterprise IT, the Proof of Concept(POC) projects are miniscule compared to real life scenarios?Clearly, it’s time for enterprises torevisit the POC lifecycle, especially in the new digital paradigm powered by Artificial Intelligence (AI).
POC is probably the onlyeffective measureof evaluating the application of AI technologies in the enterprise. It helps validate the benefits and the business case for both the consumer and the solution provider. In the increasing clamour to implement AI for competitive advantage, the number of POCs executed by organizations is alsoincreasing. Reports from different customers, analysts and practitioners suggest a very high “hit” rate.However, when we look at the number of POCs that have successfully translated into production success, the numbers as well as opinions are mixed, suggesting the need for a deeper analysis.
While there is a trend wherein both the OEMs and carriers are collaborating to improve the reliability and accuracy of data sets, carriers are experimenting to unravel the complex co-relation between the data generated, interventions and improvement/deterioration in driving behavior – some going to the extent of onboarding professional psychologists to work with the analytics teams.
Business Objectives - One of the majorreasons for a POC not meeting expectations is the incorrect identification of abusiness problem and articulation of success criteria. For example, the organizationpicks a non-critical application for POC of an Artificial Intelligence forIT Operations (AIOps)project, in order to complete the whole process quickly. This, in turn,results in the AI model’s inability to learn adequately.Or take the case of an organization that selects a business process that requiresextensive human intervention across multiple stages, leading to process bottlenecks. Such instances result in a POC that does not instil confidence in stakeholders in terms of its ability to scale in the production environment.Hence,for POC success, it’s important to choose a process that is closely linked to the business objectives of the organization.
Data - Another factor that requiresclose attention is data. Is the sample set of data used inthe POC reflective of the enterprise problem? Here, not only is the size of data used inthe POC important but also the quality of data. Many times,both organizations and solution providers waste a significant amount of time on this step, but with little impact. It is important forbusinessto planin conjunction with internal risk and related functions and retrieve a data set that when applied on an AI model would make sense for production. My recommendation is to consider the available data set and use at least one-fifth of the data available in the relevant business process for the POC. Some experts suggest not putting a lot of effort into obtaining “perfect data”. This does not mean that businesses can pick up a data set that is not reflective of the business problem.
Infrastructure - POCs executed on sub-standard infrastructure and in an environment not aligned to production environments rarely scale up during actual project deployments. Many times, the security controls applied in POC environments are lenient as compared to the production environments.The success of such POCs falsely raises the hopes of stakeholders, leading to multiple issues in establishing a business case for the real problem. Ideally, the infrastructure and security controls used for POCs should be at least similar to the staging environments that exist in the enterprise.
Diligence - Finally,it’s vital to treat the POC with the same diligence that one would treat a production system. Doing so will lead to better communication, risk identification and transparent objective measurement of the POC exercise - in line with the expected business benefits visualized by stakeholders funding the whole exercise. It’s also critical to thoroughly understand the budgetary allocation for POC. While there will be a few roadblocks to getting approvals for a large budget typically required for an AI POC, stakeholders can devise a mechanism whereby the solution provider is accountable and rewarded for success - rather than the organization bearing the burden of investing upfront.
Understandably, POC programs will continue to dictate the introduction of AI programs in an enterprise. While not all POCs will succeed inreplicating a project in a live environment, collaboration across stakeholders - both enterprise and solution providers - canensure that POCs realize the desired Return on Investment (ROI).