Situation

A large international bank had a long-term strategy of reducing the company's data centre footprint. Using its Cloud-first approach, it sought to transform the banking customer experience. With more than 40,000 employees and operations in the US, UK, and Canada, this bank renders a variety of services in retail and commercial banking, credit cards, and auto loans.

To achieve this strategy, a key requirement was implementing a Cloudera EDH platform migration on cloud that included:

• Migration of PB Scale Cloudera Data Platform and associated workloads to AWS.
• Modernize Map Reduce, Hive and Pig Scripts on CDH to Spark based Workloads on Amazon EMR.
• Mitigate impact on Security, Compliance and Regulatory risks using AWS best practices.

Solution

Wipro’s Delivery Transformation group focused on agile transformation to reduce delivery times and drive best DataOps practices. The innovation council led the way by creating architectural design patterns to accelerate the migration of their Hadoop platform to cloud. Cloud-native modern design patterns were used to build the automated reusable frameworks for migration, data quality, EDW and ETL modernization accelerators, data lake and cloud data engineering tools, and big data applications, successfully deployed at scale using these frameworks. The following best practices were followed for EMR consumption:

• Mandatory usage of transient clusters for short running applications.
• Logical separation of jobs.
• Usage of unused EC2 capacity to run jobs and save costs.
• Auto scaling with spot instances and instance fleets.
• Complete AD integration with segregation of duties between developers and support.
• Self-service automated provisioning of infrastructure using CFT.

Business Outcome

The Hadoop platform migration journey, brought immediate benefits, including

• Increased IT and data center staff productivity gains.
• Reduced infrastructure costs enabled by cloud engineering best practices
• Improved business benefits by enabling real time insights and ease of innovation

In addition, the team added more than 20 global scrum teams across different LOBs, 1000+ Cloudera workloads were rationalized and re-engineered on AWS, and 500+ new big data applications built on AWS Hadoop platforms using cloud-native and open-source frameworks. The client’s long-term strategy of reducing the company's data center footprint was realized, going from eight in 2014 to three in 2018, and none by 2020.