5 Things You Need To Know About Before Jumping into Mobile A/B Testing
As the Android vs. iOS saga continues, most major apps establish their presence on both platforms. Many opt for iOS first, since there are only a few device models (as opposed to the ever-expanding portfolio of Android devices and manufacturers).
Nevertheless, some have been swayed by the advantages of Android. Our friends at Stack Overflow cited iteration speed and first-party support for alpha and beta testing as the main reasons for choosing Android as the first platform for which they developed their native mobile presence.
Even if the platform differences are not immediately apparent to you, taking them into account is key to a successful cross-platform app. It may be easy to assume that what works for one platform will translate into success on the other, but this type of logic will get you in trouble down the road.
In case you’re new to A/B testing, here is the concept in a nutshell:
Mobile A/B testing has a big part to play in identifying the differences needed to be successful on both iOS and Android. Here are some of the things we’ve seen that have the greatest impact when testing on the platforms:
According to this comScore report, iOS users tend to be younger and wealthier: 19% of iPhone owners are between the ages of 18-24 years old (compared with just 16% of Android owners), and 41% of iOS users are in the $100,000+ income bracket (compared with just 24% of Android users).
Android has been shown to be popular with professional and business users. Hacker types also have been drawn to Android because of the possibilities that an open platform offers.
Takeaway: Think critically about the differences in general audiences across the two platforms, and how you can play to those differences through UI, UX, price points, and features.
For example, iPhone users may respond better to a promotion or feature that makes a cultural reference to something that young, affluent users will recognize. Android users, on the other hand, may be motivated more by a feature that lets them customize their experience.
Despite Android’s impressive growth in market share of devices sold and active users, iOS has been shown to generate more money for developers year after year. iPhone users are more likely to purchase something from their mobile device, and 23% have purchased something on mobile previously (as opposed to 17% on Android).
Flurry took the opportunity at GDC this week to open their datasets on Android games, showing that not only is the Android population skewed toward young males, but some mix of in-app purchases and ad-based revenue is optimal for many mobile games.
Takeaway: A/B test different monetization strategies on iOS and Android in order to capture the most overall value from both platforms. iOS users generally are more likely to download paid apps and make in-app purchases, whereas Android users may be monetized more easily through advertising and lead generation. You also can test different price points, and you may find that one platform’s users have a greater tolerance for higher price points.
Engagement and native UX perhaps are where iOS and Android differ the most. For example, Android’s “intents” allows users to share content using any installed app from any other app, which doesn’t exist on iOS.
iOS has a tendency to attract “power users” who are more likely to engage in all major content categories (social media, news, e-commerce, and games) for longer average session times.
Takeaway: Consider testing user flows and experiences that complement the behavior typical of users on the platforms. You may want to test more streamlined and straightforward activation flows for Android, whereas you may be able to rely on iOS users to engage unilaterally for longer periods of time before you offer them an in-app purchase or ask them to share some content with friends.
Additionally, there are distinct UX conventions in iOS and Android – such as navigation, or how actions are displayed – that may work for one platform and not the other.
4. Device Type and OS Versions
A key challenge in Android development is the relatively large range of different devices that use the platform. From Samsung to HTC and Google’s own phones, Android represents a mosaic of different price points, screen sizes and resolutions, and hardware that can make it an unpredictable platform at times.
iOS runs on significantly fewer device types, but, nevertheless, can be complicated by different hardware and OS versions that may not have the capability to run some apps correctly or at all.
Takeaway: When A/B testing new features on Android or iOS, segment by device type and OS to account for how changes affect users on each combination of hardware and software. You may be able to drive more desirable user behavior by pushing only new features on certain devices while keeping the old feature set for other devices. The same may apply to OS versions.
5. Speed of Iteration
At a recent Android meetup at Google’s NYC offices, we heard from Kinsa Health’s CEO Inder Singh about the benefits of developing for Android. One of his key perceived advantages of Android: the speed at which Kinsa can act upon user data, and iterate faster, through methods like A/B testing, without having to go through lengthy app approval processes.
Although new technology in A/B testing mobile apps has cut down the iteration time for both iOS and Android, there still is a strong case to be made that Android is a more flexible platform for which to implement iterative feedback loops and respond quickly to user data.
The time saved through quicker iteration should not be confused with quicker tests, though. Stopping tests early leads to the risk of a false positive (also known as a Type I error). In other words, a statistically underpowered test could indicate an effect (like a 20% boost in conversions) is present, when, in fact, the results are skewed because of a narrow sample population.
Takeaway: If you’re not already leveraging new technologies to quickly run A/B tests and make app feature or UI decisions for your mobile app based on data (and not opinions), there’s no reason not to get started.
And, when you’re ready to commit changes to your app’s binary that have been validated through rigorous testing and analysis, it will be faster to push those changes through the Google Play Store than the Apple App Store.
Source: Kiss Metrics