In one of my recent papers, titled “Belling Schrödinger’s cat“, I had picked the analogy of Schrödinger’s thought experiment around how it was not possible to conceive if the cat was alive or not, and how in a similar fashion in business we have ambiguity in data points, that could potentially be used to extract superior decisions.
Examples of ambiguity arising out of a decision process can be seen in the following situations:
- Processing vendor payments – where the back-office associate suspects an invoice to be fraudulent or duplicate, but does not have hard evidence or the time to conduct the necessary checks
- Insurance claims – where the claim examiner has a gut feel that the claim is dubious but does not have the ability to conduct a full investigation, for reasons of bandwidth, data not available in time or not fully investigated as it was a low value claim
- Other instances where there is sensitivity around the individual concerned and the organization does not want to risk an adverse reaction.
These are scenarios that we commonly encounter, however today, advancements in technology is helping enterprises turn a different chapter. With today’s smart detection techniques, we have seen implementation moving from a rule-based regime to a probabilistic prediction of anomalies – as I spoke about earlier here.
In the above mentioned business scenarios, building an anomaly detection model that does not have a meaningful approach to handling the ‘unproven suspects‘ or ‘greys‘ could impact the predictive accuracy. We have seen that these ‘greys’ are typically handled in one of the following ways:
- Ignored greys from the analysis
- Assumed greys to be blacks (conservative stance – ‘Blacks’ denote the confirmed frauds)
- Assumed greys to be whites (assigned as the majority label – ‘Whites’ denote clean transactions)
All these options lead to loss of vital information and hence inferior decision making. To overcome this, we explored if there is a way to leverage the “two-state” nature of the grey set to build a better anomaly detection model, compared to the above-mentioned options. Essentially what we sought to do was to run a series of iterations and assume that the ‘grey’ data points could either be ‘blacks’ or ‘whites’ in each of these iterations and build this into our prediction model and then consolidate the results. The individual predictors could be any of the traditional classifiers.
The approach was implemented on three different datasets across three very different business domains. All three datasets contained a small number of known frauds and greys (under 1% each) and the remaining were normal / clean records.