New technologies like 5G, plus constantly changing consumer demands, are forcing telecommunication service providers (TSPs) to create and rollout features faster and faster. For relatively recent applications developed with contemporary digital technologies, new features and functionalities can proceed relatively easily. But for legacy applications – typically commercial off-the-shelf (COTS) systems – rolling out new features can be a slow and tedious process which eventually results in slower rollout of new features to end customer.

Traditionally, TSPs have preferred COTS solutions for core business functions like billing, mediation, customer relationship management (CRM), order management, and others. COTS solutions, after all, are readily available, fulfill standard requirements, and, after their initial implementation, can be customized to serve an enterprise’s unique business requirements, thus reducing dependency on the solution vendor.

However, new feature releases will involve configuration changes. For example, a change in pricing plans or charge aggregation logics will necessitate a change in billing system; localized customizations could change the billing adaptor layer. This leads to problems in an environment where business demands are evolving rapidly, with many TSPs finding it difficult to keep their systems up to date. Moreover, creating multiple development environments to cater to multiple change cycles leads to additional maintenance headaches.

To work with customizations and configuration changes and still deliver production updates faster for a legacy COTS setup, Wipro recommends a comprehensive improvement strategy that is integrated with and centered around a fully automated continuous integration/continuous deployment (CI/CD) pipeline. 

Understand the Landscape

Here’s a good first step toward understanding the scale and size of the problem of rolling out changes to COTS solutions: Analyze the enterprise’s landscape and existing methodology. Wipro’s experience in building solutions for the telecommunications domain has illuminated some of the potential challenges and their impacts, detailed in the chart below. These problems negatively impact productivity and hinder the business goal of faster features rollout. 

Adopt a CI/CD Approach

Once the landscape and potential bottlenecks are understood, TSPs with a legacy COTS environment should consider adopting a fully automated CI/CD approach to manage customizations and configuration changes and still achieve faster production delivery. 

In broad terms, this CI/CD approach solution involves three stages: 

Stage 1: Process optimization
Start by evaluating processes such as release management, environment data refresh, and test management to identify and remove redundancies. Next, optimize time-consuming steps in these processes such as wait time for manual interventions or a complex process which can be simplified into a smaller number of steps

Stage 2: Tools selection and enhancement
A reliable, user-friendly tool that supports integration to pipeline is essential to a CI/CD approach. Examples include: 

• An application lifecycle management (ALM) tool that enables all development tasks to be tracked and reported throughout the project lifecycle (e.g., Jira, CLM, or TFS).
• A source code management (SCM) tool with clear merging and branching strategy defined and peer code reviews enabled. Open-source SCM tools like Git can reduce costs substantially; other new-age tools can be easily integrated into a development pipeline.
• A test tool that can be integrated with a pipeline through a Command Line Interface (CLI) or APIs and have test cases that can be easily invoked by the pipeline orchestrator such as Jenkins. Tools such as Wipro AssureNxt framework or Selenium can be evaluated based on specific use cases.
• A code security scan can be integrated with a pipeline through plugins, CLI, or APIs. This tool can help identify security vulnerabilities during the development stage so they can be addressed before moving to production.
• A code quality scan can be integrated with code quality check tools using plug-ins to perform predefined code quality checks and verify whether the release package meets quality standards. Custom rules can also be defined for the application to review the code format, giving more control to the application owner.

Stage 3: Build a fully automated CI/CD pipeline for all environments
The CI/CD pipeline integrates all the tools and processes to enable a single-click, end-to-end delivery of changes to production. 

• Part 1: Automation  
  Identify manual, repeated tasks and automate them via scripts, playbooks, and open-source or free tools integrated with the pipeline through CLI or APIs. Some of the most obvious areas for automation include:

1. Unit test and regression test cases suites
2. Build and deployment steps
3. Smoke test execution after deployment
4. Simple FTP tasks can be scheduled and automated using jobs
5. E-mail/Microsoft Teams notification automation on triggers, alerts

• Part 2: Build the pipeline
  Build the pipeline in line with the delivery plan to streamline the entire build-test-release-deploy process into one single CI/CD loop. Some key features the pipeline could offer include:
  
  a)  One environment, one pipeline: A pipeline with clear start and end points helps establish owners, split responsibilities, and ensures proper access management. For example, for the development and agile pipeline, access and ownership will reside with developers, while for preproduction and production, access and ownership will stay with the release-management team.
  b)  Interactive dashboards: Having a single interactive dashboard helps everyone on the team visualize status, issues, and the owners responsible.
  c)  Single-touch deployments: Integration of the pipeline with SCM tools is the first step toward a single-touch pipeline execution. For example, tools like Jenkins can apply various methods such as polling the SCM on commits, attaching webhooks to get an automatic trigger on a defined action (like commit or merge), tagging or commit message.
  d)  Parallel deployments: The pipeline orchestrator handles parallel job execution through a single pipeline to shorten execution time and allow multiple developers to deploy simultaneously.
  e)  Custom e-mail notifications to stakeholders: Using HTML-based e-mail templates integrated with Jenkins, all stakeholders can be informed on pipeline status automatically.
  f)   All the development environments in sync with each other, enabled by Jenkins
  g)  Security and quality code tools integrated with the pipeline.

Go to Market Faster

An integrated CI/CD pipeline, along with process improvements, represents a holistic solution that can enable faster and smoother rollouts of changes to production. Providing an additional layer of quality and security checks is a step to saving a lot of reactive effort when encountering future issues.

The steps outlined above address concerns posed by most of the visible layers of a COTS set-up. The exception is the core code of COTS products, which remains with the product vendor and is expected to evolve according to CI/CD demands. But the rest of the layers, which are under the purview of TSP, can be managed as follows:

• Configuration and customizations build around the core can be fully automated and integrated with the pipeline to accelerate faster delivery of products to production.
•  ITSM processes and team collaboration processes can be made leaner and more efficient by simplification of processes and using appropriate toolsets respectively.  

Fig 1: Layers to legacy COTS application with potential accelerators for enabling faster rollout

Next Steps

Enterprises looking to accelerate deployment of COTS revisions or enhancements should consider the CI/CD methodology. But before embarking on such an endeavor, consider these prerequisites for success:

• Understand team maturity and skillsets before handing over more responsibility and tools. Team mindset and culture plays a big role in adoption/rejection of any change.
• Changes being introduced in processes and/or tools should be walked through well with DevOps teams.
• Allow time for onboarding end-users and giving them time to get comfortable with new toolsets. Proper on-boarding includes user guides and Standard Operating Procedure documents to ensure the new process is understood and accepted.
• Tools operation, housekeeping, upgrades and maintenance must be done regularly.
• Tools infrastructure (things like CPU utilization, memory, and space) must be monitored, and alerts must be configured on utilizations that reach a threshold to ensure capacity and performance issues are addressed proactively.

The CI/CD approach can be a powerful way to roll out better software and do it faster. For example, Wipro recently worked with T-Mobile Netherlands to bring about one such transformation for the company’s BSCS Billing application, which has generated significant improvements.

Implementing a CI/CD pipeline for a legacy application such as BSCS (20+ years old) was only possible with the skills and know-how of Wipro, being a big IT company
T-Mobile Billing DevOps team