Client Background
- Client: A multinational financial services corporation
- Industry: Financial services
- Products/Services: Credit cards, payment systems
- Areas of operation: Worldwide
Challenge
The client’s data processing system had been built over two decades across multiple lines of business with over seven thousand data pipelines. These pipelines/workflows carried out extract, transform, and load of data as well as various analytical calculations, which was then stored in the enterprise data warehouse. The legacy system had scalability issues, as the throughput was not adequate to meet the client’s growth plans. The tool also had low adaptability to new big data technologies. To add to that, the customer was paying millions of dollars as license cost for the legacy product.
The client was bracing for more sophisticated fraud detection, enhanced security protection, etc. and this required additional data processing for new data attributes being captured, real time data processing and quicker turnaround time.
Solution
Wipro built a new platform on Spark, the open source data processing framework, using Java/Scala due to its ability to scale up easily as well as process high and varied data volumes within a limited time.
The solution components included:
- A Spark based rule engine that was capable of using the existing business ruleset without any modification in order to reduce the development effort that would have been otherwise required to covert the ruleset to code
- A Java-based conversion platform to convert the existing legacy program to Spark/Scala code. This was used for most of the data-flows that had relatively lower level of complexity. The platform produced XML containing all the metadata and business rules as an intermediate step, which could be used for documentation of the business process
- The more complex data-flows containing a lot of complicated business rules underwent re-architecture and redesign. Data pipelines with similar patterns and functionality were assimilated, analyzed and then redesigned and implemented in Spark to ensure uniformity of the solution. One of the key aspects of the re-architecture was reusability of code across business processes
Business Impact
The transformation to Spark platform enabled the client with a comprehensive framework to manage big data processing and sophisticated analytics with speed and flexibility.
This led to a number of benefits, both short term, and long term:
- Implementation of the new platform across different lines of business brought down dependency on proprietary products thus saving millions of dollars in license cost
- The new rule engine built in Scala saved up to 35% effort in redesigning and development of the solution. It also provided scope for new development of rulesets thus enabling reusing of components
- The tool developed for conversion to Spark/Scala code, decreased up to 30% of the overall effort. It also brought in a uniformity in the dataflow implementation that could be easily followed and maintained
- The re-architecture and redesign resulted in streamlining of existing processes. Many redundant and obsolete pipelines were removed. It ensured better performance of jobs.
- The new solution brought in the opportunity to scale up the operation to incorporate sophisticated fraud detection, enhanced security protection, etc. Also, the solution could be easily integrated with new emerging Big Data technologies.
The new Big Data platform provided the customer the processing power required to minutely analyze the data and capture every aspect of the dynamics of the business. The company is geared for future business expansion as it is no more restricted by the computational power of the system.
Judhajit Senmazumdar, Director - Data Analytics & AI - Global Sales, Wipro
