Mobile Testing Strategy

Introduction

Talk about a difficult area of mobile development. Testing mobile applications and Web Sites.

First we need to decide which platforms we wish to test:

• What is the lowest iOS device we’re testing on? 3GS, 4, 4S?
• (Worse) What are the Android devices and versions we’re testing on?
• Hmmm, Windows Phone market share in Europe just rose above the magic 10% threshold, should we start testing on that as well?

It’s all about the target audience, you stupid!

I hear you, and I agree, it is all about the target audience and the platforms, devices and OS versions the people therein use. However, in many cases you still need to check a large number of possible combinations. Certainly in case of a Web Application or Web Site.

Happy and few are those people that can just say I make something for the latest version of iOS and iPhone device only because my target audience always upgrades immediately.

Targeting highest value customers

Another interesting approach some take is to test only on devices of the highest-value customers. Customers that make a difference to the company’s bottom line. Typically these customers tend to have more recent devices, narrowing the set of devices considerably. But not every company can do this.

Customer specific situations you need to take into account

And as if that wasn’t difficult enough in and by itself, some organizations impose additional limitations that will make it even harder, like: “our test environment cannot be made accessible on the internet due to our stringent security policies.”

Important points to describe

To have a clear vision on the costs and effort you’ll need to foresee you need to think about -at least- the following subjects and describe them in your test strategy:

• Platforms and (in case of web) browsers you’ll support
• Devices and (test)services you’ll use.
• How often you’ll be renewing devices
• Which tests you plan to automate and how
• Which test infrastructure you need to test your application (web services? push notifications service?) and -if not yet available online- how you’ll make them available
• How will you try to mimic live conditions (e.g. low connection speeds, restricted memory etc)
• How you’ll be measuring “performance” (both perceived and real)
• More generally: when will you consider your app a success, and how will you measure that your app is matching these expectations?

In all cases

Take your time to investigate how you’ll tackle this. Things that are trivial in “normal” development settings, sometimes are very hard or down-right impossible in “mobile” development.

Luckily, better testing tools for mobile development are becoming available.