Introduction
Mobile phones are becoming the equivalent of the Swiss Army knife. They are being designed for the needs of urban warriors who want to do everything with a single device. The ‘always-on' mobile ecosystem has gone beyond the boundaries of telephony. It records activities, provides event alerts, keeps users in touch with their personal lives and assists in business processes like closing transactions and completing payments. These sophisticated smartphone capabilities are giving rise to novel ways in which to address the needs of customers, employers and partners. As a result, mobile adoption is on a steep adoption curve. A Cisco study suggests that by the end of 2013, the number of mobile-connected devices will exceed the number of people on earth. By 2017 there will be almost 1.4 mobile devices per person.
The most obvious inference that can be made from the growth of mobility is that access to alerts, data, intelligence and applications on the move will filter down to the lowest levels in industry and enterprise. The corollary to this is obvious: businesses will begin to mobile-enable their web presence. Mobile applications are already becoming a natural extension of web applications. While this sounds simple, it presents a major engineering challenge to mobile developers. This is because what is not immediately obvious is the fact that the mobile landscape is vastly different from that of the desktop / laptop landscape.
Mobile devices are built on a variety of fragmented platforms that include multiple versions of iOS, Android, Windows, BlackBerry, Symbian, Bada, Brew and dozens of others (as of today, there are 17 versions of Android alone published since its beta release at the end of 2007). Mobile devices themselves come in a vast variety of form factors, resolutions, processing capabilities and with a diverse range of sensors and interface options. Within this environment, bristling with differing hardware, software and connectivity capabilities are applications in three distinct flavours:
Mobile Web Apps: These are server side apps that bring together the vast capabilities of the web and the functionality of touch-enabled smartphones. These apps are quick to deploy since they don’t need to go through approvals as they don’t depend on a mobile app store or marketplace for distribution.
Native Mobile Apps: These are platform-specific applications designed for a device’s OS and firmware. They leverage the device’s sensors and capabilities. Typically, a native app developed for one mobile device will have to be re-worked and adapted for another device. Native apps may enjoy a higher level of security than web apps.
Hybrid Mobile Apps: These apps are designed using web technologies (HTML5 being amongst the most popular) but run inside the device’s native container. A hybrid app is powerful because it exposes the device’s capabilities (say a camera or an accelerometer using an abstraction layer) to what is essentially a web app. This is not good news from a business point of view as usability, capability and security of hybrid mobile apps can be different across devices.
For businesses this means an exceptionally large burden on testing mobile applications across devices and device versions. Let’s consider some of the key questions (see figure above 1.0) thrown up when testing mobile applications to understand the complexity of mobile testing:
Mobile Application Testing
It shouldn’t come as a surprise to discover that for adequate coverage with respect to mass markets, an application might have to get tested on 15 to 20 device combinations [Platform-Version-Device Type]. It is evident that testing applications for mobile environments calls for a completely different approach, infrastructure and expertise.
The Complexities of Testing
There are several areas that need attention when creating a strategy for mobile application testing.
Functionality Testing: Functionality testing is essential across all valid use cases and boundary / special conditions because it ensures a good user experience.
Some of the critical factors to evaluate include ease of navigation and screen transitions, response speed of the device and application, ability to customise and personalise the application and the intuitiveness of the touch interface. Such testing must be done among various user groups against benchmarks to ensure adaption and quality feedback about the application’s functionality.
Performance and Usability Testing: While application stability and performance are of prime importance, they are dependent on a variety of other factors ranging from the mobile device to load and application response, server utilisation and network conditions. A number of test tools such as Shunra, Keynote, Neotys, Gomez, SDK Tools and LoadRunner are available for performance testing. Selecting the right tool based on the scope of the test and costs is critical.
Automation: Test solutions should be platform agnostic. In order to ensure that testing time is reduced, the solutions should be reusable and scalable. Automation leads to shorter test cycles and productivity improvements of around 25 to 30 per cent.
Security Testing: The mobile and personal nature of smartphones makes it a very vulnerable device to security threats. Security testing for mobile applications comes in three tiers – physical end-point device security, enterprise level security for devices (Mobile Device Management - MDM) and application security. Security check lists include password locked keypads, policy enforcements via MDM solution, remote data wipe in the event of non-compliance / theft / accidental loss of the device, application security checks via tools and vulnerability exploitation exercises. Security testing requirements of the mobile application includes evaluating OS specific vulnerabilities, application specific vulnerabilities, network vulnerabilities, storage vulnerabilities, data vulnerability and physical security of devices.
Device Proliferation: With such severe fragmentation in the device space, application testing has to be done across a variety of devices. This means having a very large inventory of handsets available for tests.
Upgrades: Rapid platform upgrades means keeping pace with them and validating the impact of the upgrades on the applications as soon as possible. Though platform owners don’t wish to impact any public application interfaces leveraged by 3rd party developers, at times there could be impact on the application due to an upgrade. It is a good practice to check the release notes of the platform upgrade to look for any potential impact / enhancement opportunity for the application. For businesses this means identifying the core regression suite to validate the impact of the change. This must be executed every time there is a platform / application upgrade. Tests should be based on upgrade release notes that identify the impact areas. Finally, this calls for automation of core regression suite for quick execution with increasing coverage to include more features.
Infrastructure and Connectivity: Mobile testing requires dedicated infrastructure. To function, most mobile applications require connecting to the application servers. This creates the need for having Wi-Fi connectivity to the enterprise backbone (limited to application servers). There are several other ways to gain connectivity to application servers and it is important from a scaling perspective to ensure that test environments can be leveraged by geographically distributed testing team. This, in turn, may impact compliance to IT security policies and require intervention from the IT department.
Addressing the Complexities of Testing
Mobile application testing requires several sophisticated tools to ensure comprehensive and reliable test management. The most common tools required are for test automation, performance testing and security testing.
Test automation is a must for mobility testing. This is evident from the fact that the same use case has to be run and validated on a growing number of devices and platforms. A reusable and scalable automation solution can help optimise the test effort. This significantly reduces the time required for testing and also the cost of testing.
Automation solutions can be broadly classified in two categories – cloud based and desktop based. Tools must be selected for reusability across platforms and devices while keeping in mind the entire application portfolio. For a majority of the enterprises selecting and maintaining test automation tools can be a confusing task prone to expensive errors. This is one reason why it is best to partner with a technology provider who has a deep understanding of mobile test automation and who also has an understanding of your specific test needs and priorities.
Another way to manage mobile application testing is to set up an in-house mobile test automation lab. This means equipping the lab with devices (various platforms and form factors), mobile infrastructure (secure Wi-Fi, internet, backend, automation stations, PI servers), tools (test, performance, security) and data (test data, benchmarks etc.). This is an expensive exercise. Most enterprises also find it difficult to retain the skills and expertise required to manage and maintain such facilities that are not core to their respective business.
The Solution and the Benefits
In today’s highly fragmented, fast growing and dynamic mobile market it is tough for enterprises to own and operate mobile application testing infrastructure with high utilisation levels to justify the investment. It is best for businesses to work with a partner who can deliver testing as a service. Such a partner would ideally have the required skills and experience and also the infrastructure to deliver a reliable testing service. As an example a partner must offer the following to its customers:
The benefits of such a partnership mean that an enterprise can remain platform agnostic and focus on the application rather than on the device. Rapid automated testing of applications across platforms and versions will ensure faster time to market, with a high degree of functional and security assurance in addition to deep test coverage - all this without the headache of managing the test infrastructure or making CAPEX investments with uncertain ROI.
Summary
It is inevitable that your business will adopt mobile technologies to improve efficiency and become an ‘always available’ business. However, in a fast-changing mobile environment, application testing must be done in days instead of weeks, to ensure that your business remains agile, productivity is not lost and customer satisfaction is not impacted. How your business overcomes these challenges will determine how mobile it is and how competitive it remains. One of the best approaches to mobile testing, therefore, is to engage with a technology partner that offers mobile testing as a service.
Sudheer M, Practice Head – Mobility Testing and Automation
Sudheer has 12 years of experience in Wireless Technologies & Mobility and is currently working as the Practice Head for Wipro Mobility Testing. He handles Enterprise Mobility Testing and is also responsible for Mobility Test solutions. He has developed automation solutions for Smart Devices, Android Platform and Chipset Validation
This white paper focuses on an alternative approach to Risk-Based Testing, based on dynamic reassessment of risk and its benefits.
Find out how Mobility Assurance is playing a vital role in Organization's IT strategy today, and how it differs from traditional Testing processes
Energy and Utilities have traditionally been conservative in their approach to innovation and technology, especially when it comes to Business Solution Platforms
Locations
