The Android world is becoming more fragmented as time goes on. There are now 4000 different unique devices with endless (not really, but it seems that way) permutations of Android1. And then you have the Amazon tablet, Kindle Fire going its own way. Based on Android 2.3, and with 54% of the tablet market2, App developers can’t ignore it. Additionally, some of the new features of Android version 4.1 are not backwards compatible, so when tasked with Android app testing, many of our clients find themselves in a quandary of what devices to test? Which highlights the larger problem, how do you write Apps in the Android market to cover as many devices as possible without testing on every device?
Android App Testing – How to Choose Devices to Test
When our clients ask us to do android app testing and what devices to test, we discuss the concept of ‘a family of devices’ that provides coverage for as many devices as possible without breaking the bank. Android devices have varying characteristics, but the following factors are the main differentiators.
1) Manufacturer market share: This is not necessarily a ‘characteristic’, but it is main factor. Market share rules and you have to make sure your App works on the market leaders.
2) Device Market share: (see 1 above).
3) Version of Android: With Android versions 2.3 and earlier having over 80% of the Android market, using new 4.1 features, like Google Now which predicts what you want, become problematic. Using the newer, but not backward compatible, features means not having access to over 80% of the Android market.
4) Screen Size: From tablets, 800×1280, to the small screens, 240×320; the difference is quite large. You can write an App to run on both the small mobile phone screen and the tablet size screen or you can write different versions of your App. Either way, there are compromises to be made.
5) Input type: Keyboard, touch or both.
It is possible to test with a small number of phones to get a fairly good representative idea of how your App’s quality will be perceived in the market place. Depending on the month and day (new devices are being released and gaining/losing market share), 15-20 devices can achieve up to 70-85 percent market coverage. Increasing to 25-30 devices can give you more coverage and more confidence that your App will work on a large percentage of the devices out there. After that, the law of diminishing returns takes over.
Some Apps, by the very nature of their solution must make choices that limit the size of the Android market that they can go after. Others have more flexibility to make compromises so that their App can work on as many devices as possible. Whichever your direction, the choices are not easy.
1 The Essential Guide to Mobile App Testing, uTest eBook, p. 21.