- On which devices, platforms and OS versions should my application be tested?
- How quickly can I get to the market?
- Even if I do a comprehensive test now, how do I prepare for future device and OS upgrades?
- What kind of tests should I run?
- Can the test scripts be reused across platforms? Will the scripts run on all the platforms?
- Can I use automation to run the scripts? Which automation tool should I opt for, given that they are as confusing as the mobile device landscape?
- How do I ensure end-user satisfaction on the application after the test?
- How quickly can the tests be done?
- How do I guarantee security of the application?
- How do I evaluate application performance?
- How can I also test the performance of the device environment (server side)?
- What will the test cost?
It shouldn’t come as a surprise to discover that for adequate coverage with respect to mass markets, an application might have to get tested on 15 to 20 device combinations [Platform-Version-Device Type]. It is evident that testing applications for mobile environments calls for a completely different approach, infrastructure and expertise.
The Complexities of Testing
There are several areas that need attention when creating a strategy for mobile application testing.
Functionality Testing: Functionality testing is essential across all valid use cases and boundary / special conditions because it ensures a good user experience.
Some of the critical factors to evaluate include ease of navigation and screen transitions, response speed of the device and application, ability to customise and personalise the application and the intuitiveness of the touch interface. Such testing must be done among various user groups against benchmarks to ensure adaption and quality feedback about the application’s functionality.
Performance and Usability Testing: While application stability and performance are of prime importance, they are dependent on a variety of other factors ranging from the mobile device to load and application response, server utilisation and network conditions. A number of test tools such as Shunra, Keynote, Neotys, Gomez, SDK Tools and LoadRunner are available for performance testing. Selecting the right tool based on the scope of the test and costs is critical.
Automation: Test solutions should be platform agnostic. In order to ensure that testing time is reduced, the solutions should be reusable and scalable. Automation leads to shorter test cycles and productivity improvements of around 25 to 30 per cent.
Security Testing: The mobile and personal nature of smartphones makes it a very vulnerable device to security threats. Security testing for mobile applications comes in three tiers – physical end-point device security, enterprise level security for devices (Mobile Device Management - MDM) and application security. Security check lists include password locked keypads, policy enforcements via MDM solution, remote data wipe in the event of non-compliance / theft / accidental loss of the device, application security checks via tools and vulnerability exploitation exercises. Security testing requirements of the mobile application includes evaluating OS specific vulnerabilities, application specific vulnerabilities, network vulnerabilities, storage vulnerabilities, data vulnerability and physical security of devices.
Device Proliferation: With such severe fragmentation in the device space, application testing has to be done across a variety of devices. This means having a very large inventory of handsets available for tests.
Upgrades: Rapid platform upgrades means keeping pace with them and validating the impact of the upgrades on the applications as soon as possible. Though platform owners don’t wish to impact any public application interfaces leveraged by 3rd party developers, at times there could be impact on the application due to an upgrade. It is a good practice to check the release notes of the platform upgrade to look for any potential impact / enhancement opportunity for the application. For businesses this means identifying the core regression suite to validate the impact of the change. This must be executed every time there is a platform / application upgrade. Tests should be based on upgrade release notes that identify the impact areas. Finally, this calls for automation of core regression suite for quick execution with increasing coverage to include more features.
Infrastructure and Connectivity: Mobile testing requires dedicated infrastructure. To function, most mobile applications require connecting to the application servers. This creates the need for having Wi-Fi connectivity to the enterprise backbone (limited to application servers). There are several other ways to gain connectivity to application servers and it is important from a scaling perspective to ensure that test environments can be leveraged by geographically distributed testing team. This, in turn, may impact compliance to IT security policies and require intervention from the IT department.
Addressing the Complexities of Testing
Mobile application testing requires several sophisticated tools to ensure comprehensive and reliable test management. The most common tools required are for test automation, performance testing and security testing.
Test automation is a must for mobility testing. This is evident from the fact that the same use case has to be run and validated on a growing number of devices and platforms. A reusable and scalable automation solution can help optimise the test effort. This significantly reduces the time required for testing and also the cost of testing.
Automation solutions can be broadly classified in two categories – cloud based and desktop based. Tools must be selected for reusability across platforms and devices while keeping in mind the entire application portfolio. For a majority of the enterprises selecting and maintaining test automation tools can be a confusing task prone to expensive errors. This is one reason why it is best to partner with a technology provider who has a deep understanding of mobile test automation and who also has an understanding of your specific test needs and priorities.
Another way to manage mobile application testing is to set up an in-house mobile test automation lab. This means equipping the lab with devices (various platforms and form factors), mobile infrastructure (secure Wi-Fi, internet, backend, automation stations, PI servers), tools (test, performance, security) and data (test data, benchmarks etc.). This is an expensive exercise. Most enterprises also find it difficult to retain the skills and expertise required to manage and maintain such facilities that are not core to their respective business.
The Solution and the Benefits
In today’s highly fragmented, fast growing and dynamic mobile market it is tough for enterprises to own and operate mobile application testing infrastructure with high utilisation levels to justify the investment. It is best for businesses to work with a partner who can deliver testing as a service. Such a partner would ideally have the required skills and experience and also the infrastructure to deliver a reliable testing service. As an example a partner must offer the following to its customers:
- Creating a comprehensive test strategy (device coverage, functional testing, tools etc).
- Solutions and tools for testing.
- Evaluation of onboard tools in enterprise environment.
- Automation capabilities to ensure best quality in shortest possible time.
- Scalable test infrastructure.
- Mobility test experts.
- Ready-to-customize business processes and use cases.
- On demand services.
- Total mobility validation.
- A global presence.
The benefits of such a partnership mean that an enterprise can remain platform agnostic and focus on the application rather than on the device. Rapid automated testing of applications across platforms and versions will ensure faster time to market, with a high degree of functional and security assurance in addition to deep test coverage - all this without the headache of managing the test infrastructure or making CAPEX investments with uncertain ROI.
It is inevitable that your business will adopt mobile technologies to improve efficiency and become an ‘always available’ business. However, in a fast-changing mobile environment, application testing must be done in days instead of weeks, to ensure that your business remains agile, productivity is not lost and customer satisfaction is not impacted. How your business overcomes these challenges will determine how mobile it is and how competitive it remains. One of the best approaches to mobile testing, therefore, is to engage with a technology partner that offers mobile testing as a service.