3rd Research in the large workshop, this time at MobileHCI2012

Having visited already the previous serious of Research in the Large (2010, 2011) I was very much looking forward to meeting the large researchers again investing approaches, tools, research questions for conducting research by deploying apps.
This year I was talking about three topics. In the first paper [1] we investigated the compliance of users of updating their mobile apps. We analyzed the lifecycle of various versions of a released app and found that after one week still half of the users run an old version. Thus, we recommend that developers actively notify their users inside the app to conduct an update.
Throughout the workshop we received interesting comments regarding whether could differentiate within the non-updaters between active and no-longer active users. Furthermore, it could be interesting to cluster the users by age, categories of apps etc. in order to find out whether updaters and non-updaters share certain characteristics. Finally, it would be interesting to track whether the newly introduced 'update all' feature in Android increases the compliance. We might also collect updates histories of apps from appbrain.com and compare our results with other apps.
Then I talked about our analysis of apps [2]  (based on appaware.com stats) available in the Android market. We analyzed the volumes of apps being downloaded corresponding to the overall sales. In contrast to amazon which largely live on the longtail of products, the Android market clearly is a hit market, where the most popular products contribute the largest parts of the sales. Our conclusion was that niche apps are still hard to find on the Android market (in contrast to the recommender systems revealing niches in amazon), thus research should focus on developing recommender systems. Furthermore, we found there is a significant amount apps generating large revenues with rather low download numbers by premium prices. Thus, developers should consider to increase the price of their apps significantly.
Finally, I presented NFC Heroes [3] we motivates users by a card trading fight game to upload information about deployed NFC tags.
The feedback we received from the workshop to give more hints to users where NFC tags are, e.g. information about local deployment activities as NFC tags at bus stops in San Francisco. Later, we might also building a service listing places and services of NFC tags in order to educate and inform users where they could use the NFC feature. Finally, we should consider to also listen to NFC intends  from other apps on the phone in order to capture any NFC interaction of the user's phone.

 In the workshop we had 7 talks in total which all latest for only 7min followed by 7min of discussion. This gave us enough time in the afternoon for some more in depth discussion about how to recruit users and what to learn form app stores.
To start off the discussion Frank Bentley first motivated why to do field studies at all (compared to lab studies): investigate real usage, impact of new systems, usability, inspiration, qualitative understanding, or quantitative confirmation of a previous small study.
Then Frank listed various option for recruiting users: in addition to the practice of small studies via mechanical turk, craigslist, posters in stores, snowball sampling, classmates and lab-mates, in the large we can populate the what's new section app stores, app store ads, targeted Facebook ads, re-tweets, personal tweets, presentation, and advertising. Another idea mentioned was to treat user's privacy as the user currency, the user owns and can decide to release for receiving specific services/benefits in return. Finally, our discussions revealed that either way user are recruited it is essential that papers clearly document how this was done. This is the only way that readers and reviewers can judge on the validity of the results.
The second part of the discussion focused on what kind of data and which research questions could be specifically tackled by doing research using app stores. On the one hand we get stats about downloads and locations of apps, logs of button presses, and partly demographics. However, what we don't get is the context of use and insights into why people did or didn't an activity. Overall, we are limited to tasks users in the wild want to do which may limit to tasks which are fun or particularly useful. Finally, a remaining question is as the number of apps in the markets develops and matures, whether the effort of releasing a decent app would be still feasible in near future.
David Aymann Shamma reported about his experience collecting and analyzing data from a chatting app for watching videos remotely together. David focused a lot on responsibility and ethics of treating the data being gathered throughout the project. Handling this massive amounts of data and also highly sensitive data would be a new dimension of what we as researchers would be usually used to. David also recommended to be generous and consider deleting all the session which appear to be odd or incomplete.

Overall, I really enjoyed meeting all these people sharing an interest for doing research in the large. I'm looking forward to help running this next year again. It's probably also time to come with a manifesto or guidelines describing how to do research in the large. Collective experience of researchers in the domain (e.g. [4]) should motivate the community to consider this new research method going far beyond the established modes of user studies.

[1] Update Behavior in App Markets and Security Implications: A Case Study in Google Play, Andreas Möller, Florian Michahelles, Stefan Diewald, Luis Roalter, Matthias Kranz, 3rd workshop on Research in the large (at MobileHCI 2012), San Francisco, September 21, 2012. [pdf]
[2] Examining the Long Tail of Android Market, Nan Zhong, Florian Michahelles, 3rd workshop on Research in the large (at MobileHCI 2012), San Francisco, September 21, 2012. [pdf]
[3] NFC Heroes – Observing NFC Adoption through a Mobile Trading Card Game, Lukas Murmann, Florian Michahelles, Matthias Kranz, 3rd workshop on Research in the large (at MobileHCI 2012), San Francisco, September 21, 2012. [pdf]

[4] My App is My Experiment: Experience Studiesin Mobile App Stores, Henze, N., Pielot, M., Poppinga, B., Schinke, T., & Boll, S. (2011). International Journal of Mobile Human Computer Interaction , 3 (4), 21.