Performance Engineering in Data Platform

The importance and significance of a data warehouse is a topic which is not discussed in the technology forum anymore as almost every organization - small or big - understands the business value that the data storage and analytics platform brings. However, the most common question that we hear when data volume grows and the platform matures is around the increased operational costs. In other words, as technology develops and processes streamline, the operational costs should ideally reduce. But that's not what customers experience with data storage and processing platforms. So why does the operational cost for data platform increase despite technological and process developments?

Few major reasons are improper design, lack of coding discipline, absence of platform engineers during solution and dearth of housekeeping. When the active user base increases, there is very little or no time to do any re-factoring or optimize the architecture if we fail to do it right at the first available opportunity. On one side, the volume of data being processed grows, and on the other, there arises strict SLA's to meet. Most often the support team spends more time in repetitive production issues, review cycles, platform maintenance and monitoring. To handle this pressure, the easiest way customers adopt is to add both computing power and human power. As a result, the operational cost keep raising.

What becomes an immediate need in such a context is Performance Engineering. Platform performance as an important solution parameter is not given due position. It is always looked upon at the end of the project cycle. Project timelines is another cause which forces people to skip or ignore it. What we don't understand is any effort which is put at later stage is always costlier with respect to data storage, process platforms. Data Platform Performance is something that starts with design, i.e., design right, coding to the design, deploying only the code which performs, continuous optimization and consistent refactoring. It is like any other engineering process. It is not possible to have a durable, sturdy end product without the exact inputs going in at the correct cycle. Building a framework which constitutes checklists, firewalls for each steps in the project cycle for performance, automated tools, and alerts would help to practice performance as an engineering process.

About the Author

Pon Prabakaran Shanmugam is a Principal Consultant with Wipro Analytics practice. He possesses exhaustive data architecture experience in the Financial Industry with strong data modeling, integration and analytical skills, and is an enthusiastic agile modeling proponent. He is also a strong believer of embracing open source technologies to make data architecture flexible and evolving.

Service Offerings

Client Themes

Service Offerings

Client Themes

Our Global Presence Global Site

Our Global Presence Global Site

Our Showcase Pages Locations

Performance Engineering

in Data Platform

About the Author

Service Offerings

Industries

Digital

HOLMES

Innovation

Contact Us

About Wipro

Location

Leadership

Investors

News and Events

Blogs

Partner Ecosystem

Careers

Sustainability

Contact Wipro