Development and Demonstrational Instantiation of a Method for the Structured Content Analysis of Smartphone Apps

. The publication discusses the adaptation of dedicated known methods for structured content analysis for the usage of smartphone apps, taking into account the specifics of these dynamic media types and resulting consequences for the procedure and content rating. The background is the increasing popularity of apps and the growing demand for precise analysis of the status quo regarding the content and functions of apps on the market, especially within design-oriented business informatics. Within the literature, no uniform, comparable and systematic approach to this could be identified so far. Therefore, an existing standard for the evaluation of apps from the medical context has been adapted for general application to apps. This provides a helpful procedure for researchers, practitioners, and students and ensures consistency across studies. The feasibility of the proposed method and assessment standard were confirmed by a successful demonstrational instantiation based on an app review of energy information systems for private customers.


Introduction
Smartphones and related apps are constantly increasing in popularity. For years the distribution within the population has been growing steadily across all age groups, and there is still potential in developing nations. We can observe the currently highest smartphone penetration rate with nearly 82% in the US, followed by European countries, for instance, the UK with 79% and Germany with 78% [1]. The rate is even higher in specific age groups, such as over 95% of the 14 to 49 age group in Germany [2]. As of Q1/2021, approximately 3.5 million (Google Play Store) and 2.2 million (Apple App Store) apps were available for download in the leading App Stores, by Google and Apple [3]. The Windows and Amazon stores are, with over 1 million apps, increasingly relevant as well. This means that there is a strong basis in terms of smartphone distribution in all relevant markets and their applicability for economic and scientific purposes, in which significant investments are made [4]. Apps are thus to be preferred for many applications that are intended to reach end-users, also due to their technical features (e.g., mobile availability, enabling various forms of communication; push suitability; and a broad multimedia spectrum). This is also in line with the ongoing research for mobile-only adoption, also in households with age 55+ [5]. Even in very sensitive areas, such as finance and banking, which are sometimes characterized by user restraint, the literature observes a rapid increase in user acceptance [6]. In general, smartphones and apps are seen as a potentially very useful way of delivering information to interested parties -regardless of time and place [4], [7].
Need for research. In terms of the rapid spread of smartphone apps and their applications in the economy, there is a growing need for their analysis. Analogous to other types of media (e.g. online content or TV artifacts), which have been systematically analyzed within the literature, a researcher needs to get an overview of existing implementations in advance [8], [9]. For instance, this overview of extracted functions and features of apps is highly interesting for the development of a new app or an app prototype by researchers. Especially, this is relevant for a successful application of the design science research methodology for modern information systems research, as stated within the common framework descriptions. As it is essential to start with a complete picture of the environment and ideally build on existing research, the method of content analysis is one option to achieve this (within the context of systematic reviews) [10]- [13].
Generic methods are available for this purpose, but they do not yet address the specifics of a digital, fast-moving environment and the media type "smartphone apps" in a proper scope. Thus, a researcher is currently required to carry out comprehensive research and transfer those methods within several iterations. We could not identify any generalized and holistic approach to analyze smartphone app content. This gap is also confirmed by the results of a systematic literature review, as presented in Section 2. In addition to this, statements within the literature explicitly describe the difficulty for users and researchers to identify and evaluate apps: • "There is a large and ever-increasing number of mobile phone health, wellness, and medical applications on the market. However, there is little guidance or quality assurance available for consumers." [14]; • "(…) there is no consensus on the use of any specific methodology for evaluating apps (…)" [15]; • "Given the rapid proliferation of smart phone apps, it is increasingly difficult for users, health professionals, and researchers to readily identify and assess high quality apps (…)", "Little information on the quality of apps is available, beyond the star ratings published on retailers' Web pages, and app reviews are subjective by nature and may come from suspicious Sources (…)", "Previous attempts to develop mobile app evaluation criteria have been too technical or specific (…)" [16].
This shows that the procedures described in the common literature for content analysis cannot be fully adapted to smartphone apps without limitations. Little useful documentation is accessible, especially regarding the assessment of the user-perceived quality of apps. In many cases, app store ratings (in text form and aggregated as star ratings) are the only form of user perception which is available for further analysis. Furthermore, these ratings are sometimes alienated by subjectivity [16]- [18]. To sum it up no applicable recommendation for a systematic content analysis of apps could be identified for the field of business informatics. This circumstance leads to the research question: How must an existing method for content analysis be adapted to the specifics of smartphone apps so that a comprehensible, transparent and systematic content analysis can be performed by a researcher?
Research goal. Thus, we aim to provide a method description for performing content analysis based on the specific media type "smartphone app" to fill the identified need. This should offer: a) An applicable, in particular to the design-science-oriented business informatics research, procedure for content analysis of smartphone apps. b) A standardized set of criteria for the time-efficient, initial evaluation of features of the apps ensuring their comparability.
The development of the method itself is guided by established procedures as stated by the literature providing a generic framework as a guidance instrument, checklist, and tool for quality control [19], [20]. Our contributions can, therefore, be formulated as follows: c) Overview of literature on methods of content analysis regarding smartphone apps; d) Derivation of a systematic approach for the content analysis of smartphone apps; e) Transformation of domain-specific evaluation categories for general application to apps.
Overview. The article is structured as follows: Within Section 2, the basis for the development of the method is compiled from the literature. In Section 3, the design of the analysis procedure is presented as well as the suggested set of criteria for the evaluation of apps. This is mainly derived from literature and integrates different features of the highly dynamic media type "smartphone app". Section 4 describes the evaluation of the method as demonstrational instantiation based on a real use case (identification of functional components of apps in the field of energy feedback systems). The whole research is based mainly on method creation guidelines, e.g., [19] and underlying literature for content analysis.

2
Related Work

Content Analysis in General
The procedures for performing a content analysis are well documented in the established international literature. In many cases, those guidelines follow similar premises and steps. To conduct a content analysis based on very dynamic digital media types (smartphone apps), we identified the limitation of the procedures to classical media, e.g., textual artifacts, and a lack of guidance for researchers. Although content analysis should be "applicable to all contexts" [9] there is a more active and participatory role of the user compared to traditional media. As a result, smartphone apps are not time-stable and, therefore, unsuitable for multilevel, comprehensive coding procedures. The volatility of these new media types results in several difficulties: during the feature extraction process, during coding, and documentation. One example: When using search engines to identify the units of the content analysis, the influence of algorithms in the weighting of the display or sorting can make reproducibility difficult, and many elements of the object can be left undetected [9], [21]. However, the basis for the analysis lies in the established methods, serving as an estimation of the distribution of certain characteristics/features. This can be conducted via a simple counting procedure of the relevant characteristics. Either based on all available analysis objects (complete survey) or for an extract (sample). The literature differentiates between qualitative and quantitative content analysis, as well as "mixed methods" such as the integrative content analysis. The common approach for content analysis described within the literature, [8], [9], [21]- [25], is further used as a basis for the following research process in this article.
In the following, we will examine whether there is any specific documentation in scientific publications on content analysis of smartphone apps.

Status Quo within the Content Analysis of Smartphone Apps
The goal of this initial literature search is to find a possible solution for the initially formulated need for research: a usable method to systematically analyze smartphone apps and document the features. The search terms (content analysis OR Content Analysis) AND (Apps OR Smartphone-Apps) were used in scientific search engines "Elsevier ScienceDirect", "ResearchGate", "SpringerLink" and "Google Scholar". The overview of relevant works is documented in Table 1. NO [31] The publications were analyzed based on their title and abstract in the first place and were included in the following deep-dive-evaluation if the abstract referred to the method of content analysis as well as smartphone apps as the objects of analysis.
Concerning identified secondary sources, 30 relevant publications overall were identified. Most of these publications are from the medical sector and contain frameworks from the healthcare sector and analyze medically relevant smartphone apps (e.g., for smoking prevention or fitness). 14 publications do not document any specific method used for the described content analysis and therefore seem to proceed "intuitively" but conclusively, for instance, [26], [27]. 12 publications use or refer to procedures that are similar or at least oriented on classical content analysis procedures. Some of these publications also refer to established standards in the medical field, e.g., the PRISMA Statements for systematic literature reviews or the PPM Precede-Proceed Model, e.g. [28], [29]. Five of these publications use the standard for coding the identified apps, called MARS (Mobile App Rating Scale), developed by [16]; for instance, MARS is used by [30], [31]. MARS contains 6 main categories and 23 subcategories, which focus on the quality of the apps under the premise of providing an easy-to-use and adaptable measurement framework [16]. Four publications did not meet the search criteria. Finally, the strong domain reference to publications in the medical field is particularly noticeable.
Despite the use of referenced procedures, no dedicated method could be identified that represents a re-usable procedure for the content analysis of smartphone apps or provides recommendations for authors. This results in various ambiguities and problems, especially regarding platform specifics, which are expressed by the authors. What could be identified, however, is a standard used for the subsequent categorization of apps (inclusion of general Information, without reference to the researcher's individual research needs). It also originates from the medical field (MARS) and pays special attention to qualitative aspects.

Development of the Method and Assessment Standard
Since no usable method (without adjustment or optimization for the media type "apps") could be identified during the systematic literature review, an adaptation was conducted based on the available scientific literature. In particular, the special features of smartphone apps were in focus, as highly volatile digital media with a high degree of user interaction. The proposed procedure also integrates the identified standardization approach and is intended to serve as a helpful method guideline for researchers, practitioner, and students.

Adaption of a Procedure for Content Analysis
Several approaches for content analysis were compared, combined, and adapted to the specifics of the media type "smartphone apps" in order to create a new method for supporting the execution of content analysis by eliminating the previously necessary adaptation and transfer tasks. Core components of the proposal method refer to proven procedures of content analysis of traditional media types. In this section, we will focus on special features of smartphone apps. The whole research process is visualized in Figure 1.

Phase 1: Theoretical considerations.
Preparations are necessary before implementation. The trigger is usually a concrete research interest of the scientist (in the literature also called "knowledge interest", i.e. the question "What exactly should be investigated?" [21]). This must be concretized by dedicated motivation and precise research questions (scope is here the application of the proposed method by the reader) and sets the basis for the crucial coding frame of the research (dimensions or main categories) [24]. These form a framework for the subsequent research activities and deciding on the concrete procedure. To obtain a broad view of the already existing content of the research, it is necessary to determine the theoretical foundation as well as the current state of research (structured literature review). Depending on the content of the research, it is useful to form hypotheses and define terms. Finally, it is advisable to reflect on the current state of the literature to adapt the further procedure if necessary or, if the required information has already been identified by other studies, to terminate the process at this point.

Phase 2.1: Planning the content analysis.
Before starting with the actual implementation, the problem definition must be further explained. If necessary, the individual research questions and assumptions made must be reflected upon. This is relevant due to the broadened knowledge base of the researcher after the execution of phase 1 and, especially, after an examination of the literature. Usually, different aspects are strengthened by the research needs of other authors; some aspects may already have been answered. This phase ends with the selection of the platforms or app stores to be used and the creation of the search phrases to be used. Regarding the platforms, some special features should be considered first: • Leading platforms are the Apple App Store and the Google Play Store, according to the current statistics [3]. However, the method itself is generalizable and can be applied to other platforms (with appropriate adjustments in terms of data extraction). For simplicity, we rely on the two leading platforms by Google and Apple. • The accessibility of the search on the platforms is partly restricted. While the Google Play Store at https://play.google.com/ can be searched using the search phrases, the Apple App Store is only accessible from the end devices (iPhone / iPad) and only reveals apps for the respective device. Via the browser (common access path for corresponding research purposes), help applications such as https://fnd.io/ or https://theappstore.org/ must be accessed, which use of the Apple API. In addition, a country-specific App Store may have to be selected and the same search repeated in several country stores.

Phase 2.2: Preparation of the content analysis.
In the preparation phase, two main steps are performed: the search for concrete analysis objects (apps) and the essay of the category system. During the search for apps, it is also advisable to systematically record the test material as a basis for the subsequent refinement through coding. Regarding implementation: • The displayed items vary depending on the platform (e.g., the Google Play Store does not show the total number of hits). There are differences in the release date and update date, as well as the number of downloads. • If necessary, the displayed apps are sorted according to a non-publicly accessible algorithm, which prioritizes the items and, if applicable, includes user-specific, individual features. Searches should therefore be carried out and documented as comprehensively as possible.
Concerning the category system, we recommend differentiating: 1. Individual elements of the researcher executing the proposed method, to reflect the research questions. These aspects vary from investigation to investigation. The category system (dimensions, subcategories) needs to be developed for the individual research object. For instance, as in the demonstrational instantiation in Section 4. 2. In addition, a suggested set for app analysis can be adapted. This contains a category system for formal and evaluative rating categories and creates comparability of investigations among themselves. The category system can be used app-agnostic and independent from the individual research topic. This increases the quality of coding and reduces the preparation effort for the researcher.

Phase 2.3 Test of content analysis.
Before coding identified apps, so-called test or pilot coding is recommended. This may lead to iterative adjustments to the category system. Together with the test of the quality of the coding (e.g., intercoder reliability), the need for additional training may become apparent.

Phase 2.4 Execution of the content analysis.
All preparatory steps are now complete, and the entire set of identified analysis objects can be added to the holistic coding according to the category system. The collected information is recorded by the coders, in the simplest case, supported by an Excel spreadsheet. Due to the challenges in dealing with dynamic online content (apps are subject to similar dynamics), as mentioned by [9], [21], we recommend documentation of the app (e.g., text and images based on the App Store) with the date of access, to be able to reference the presentation available at the corresponding time if necessary. A specific challenge is given by the possibly hidden elements, which are not visible on the platforms based on the description and screenshots [21]. The installation of the app can be a remedy for this, but it requires much more time and personnel investment. Furthermore, the use of special apps (e.g., for analyzing the power consumption of households) is sometimes not possible without hardware or access information. Our experience with content analysis based on the demonstrational instantiation of the method (see Section 4) shows that a sufficiently accurate survey of the functions is possible based on screenshots and descriptions of the platforms. If necessary, further information from the developers on separate websites can be integrated. This procedure allows for sufficient analysis accuracy while saving time and bypassing possible installation limitations (for instance, smartphone apps for the use case of the demonstrational instantiation can be used with specific hardware installed in the house or apartment onlywhich limits the analysis to the usage of screenshots and descriptions). But this may vary from case to case, based on the specifics of the desired app categories as well as on the needed level of granularity of the analysis.

Phase 3: Result analysis & presentation of the results.
Subsequent cleaning and preparation of the results are carried out in a scientific environment, e.g., using software tools for statistical analysis, e.g., SPSS. The following interpretative process creates, in addition to the quantitative presentation of the results, the added value for the final research reports and publications.

Derivation of the Generalized Set of Criteria for Evaluation
In addition, a standard for evaluating apps in the context of content analysis is to be introduced and tested. The whole category set is visualized as Figure 2. The literature describes the following category system classes [21]: • Formal categories: Concrete facts that can be captured in the process of coding without the coder's room for interpretation. Examples: Name, medium, date, scope, the ranking of the app, genre, or category, etc. • Content categories: They are used to represent the content focus of the analysis and usually relate directly to the individual, underlying research questions of the researcher executing the proposed method. Examples include topic, existence, and specification of searched functional elements of an app, aspects of the reviews, etc. • Rating categories: The elements of this category generate the highest differences between the coders in terms of subjectivity. Concerning apps, we speak in particular of user reviews and content-evaluating statements. In addition, overall ratings of the app's suitability for fitting in with the "feel" of the coder are also conceivable (so-called "valence" of a contribution).
Interaction of individual and comparable elements: Only those category system classes that have no direct reference to the individual research content are suitable for adoption. This is confirmed in the context of the limitation of a publication, which states that "for one's research work, category systems from other projects can only be directly adopted if the research question is also identical" [22]. For all other categories, a set of standardized and re-usable elements is helpful to increase the comparability of app analysis studies and to facilitate processing for the researcher.
MARS as a starting point. Within the literature (see Section 2.3), a standard for coding apps has been identified. The so-called MARS (Mobile App Rating Scale) contains six main categories and 23 subcategories and provides a uniform measurement framework. An overview of the elements can be seen in [16], as well as in detail in the corresponding classification template [55]. It is primarily used in medically oriented publications and will be adapted for the evaluation of smartphone apps of any kind in the following. Ideally, the content should be collected without installation, just by using the descriptions and screenshots as well as additional information on developer websites. Adapted categories for app analysis: Elements taken over from MARS were marked with a "*" in Figure 2. As an additional supplement and support for the content analysis, corresponding test questions were derived to facilitate a subjective answer by the coder. For these elements, the authors suggest a five-level scale. Separate scores are formed for each main category in the evaluation [16], [55]. All five elements were taken from the main category, Commitment. No elements were taken over from the main category Functionality, because installation and use of the app is required. This is shifted to a downstream in-depth analysis of the prioritized apps. From the main category, Aesthetics, all three elements were inherited. Most elements from the main category, Information, were also adopted. By removing the mentioned category Functionality, we have added a new item within this category, complementary to MARS and motivated by the Android Developer Guidelines [56]: Functions, to be able to cover the overall expectation to functional elements and behaviors supplementary to the very specific research questions of the Content Categories. A reduction was made concerning Accuracy of app description, since installation and use are also required here. Furthermore, Evidence base was not adopted since the evidence base is usually not obtained during initial examination, but also requires extensive research as part of the in-depth app analysis. All four elements of the main category App subjective quality were adopted. All elements of the main categories, according to MARS were assigned to the evaluating category since they require a certain degree of subjectivity by the coder. Content categories that directly relate to the individual core of the research are to be created separately and recorded within the framework of the coding.

Use Case: Smart Metering in Germany
For the evaluation of the adapted method for the content analysis of smartphone apps as well as the adapted rating standard, a demonstrational instantiation based on a real use case is performed. This is especially important for the evaluation of rating categories of an app (see Section 4.2). Its foundation is based on scientific method creation guidelines [19] and was successfully used (after this presented evaluation and with iterative adjustments) in real-life context for the analysis of smartphone apps for smart metering as described in the final report document [58]. The background is the introduction of smart meters (digital, remotely readable electricity meters) or so-called intelligent measuring devices. This rollout is required by law in Germany as part of the energy turnaround, with a gradual implementation and equipping of all households by 2023 [59]. The resulting potentials lie, for instance, in the display of consumption data in realtime, the implementation of new comfort features (notifications of activity in the household, individual device consumption, benchmarking with comparable households), and possible energy savings through timely feedback [60]- [62]. This drastically increases the transparency of private energy consumption and provides customers with a means of controlling costs. Customers are increasingly actively demanding this transparency, partly due to growing ecological awareness [63].
The visualization and analysis of data from smart metering is an essential prerequisite for generating added value. While various possibilities for data acquisition are described in the literature [64], the focus of the following instantiation is on the presentation level of information for the target group of private households. Previous studies have shown (see Section 1) that smartphone apps are a suitable channel for this target group, also, in the application context of energy data evaluation. There are already implementations on the market which are to be examined with the described method for the structured collection of the contained functional components. Therefore, this instantiation has the research goal of identifying functional components of apps in the field of energy data analysis in private households with the aim to validate the designed procedure and rating scheme of Section 3.

Instantiation of the Content Analysis
The content analysis is carried out using the procedure described in the previous sections to evaluate its suitability: Phase 1: Theoretical considerations. The concrete research interest lies in the identification of functional components of smartphone apps available on the market to analyze the energy consumption of private households. For the use case explained in Section 4.1 the creation of an initial overview of the state of the art in research and practice is important before building a possible new smartphone app to avoid double work and unnecessary investments. For the described use case it is also interesting to identify possible correlations between features of the apps and the user ratings or perceived satisfaction of the users. Therefore, the creation of a new app can be focused on the most important and helpful features and optimizing customer-centricity, thanks to the initial extraction of the status quo.

Phase 2.2: Preparation.
The search terms were used in the Google Play Store one after the other. All displayed apps were checked, and a total of 1,392 hits were reviewed. For iOS, https://fnd.io/ requires the selection of the country store. All search terms were applied to the following stores: US, UK, NL, and DE. The collection of results took place with direct comparison to the list enriched by the Google Play Store and provided 55 new results with a total of 1,079 hits. After performing the search and preliminary check as well as matching between the app stores and cleaning up duplicates, 164 apps remain. Concerning the category system, the proposed subdivision into formal, content, and rating categories was applied. Suggested elements of the formal and rating categories were adopted 1:1. In terms of the content category, five elements were be established for the subcategory Data Acquisition, 24 elements for the subcategory Basic Features, 11 elements for the subcategory Special Features, and five elements for the subcategory "Configuration Options". Each subcategory has received a score (sum of the contained features per app) for evaluation. Phase 2.3: Test. The apps selected for pre-testing were used to iteratively extend the features of the category system. The sample coding was performed on 40 randomly selected apps of the identified total. Two coders were independently involved in the sample coding. The intercoder reliability was calculated according to Holsti [21] for the formal category is r=0.99. It is close to the value of 1 required for formal categories. For the content category, a solid Holsti coefficient of r=0.87 on average was determined. Furthermore, Cohen's Kappa was calculated for each element. Of 45 elements, 28 are above the threshold of 0.60, which is considered moderate [21]. For the evaluating category, a Holsti coefficient of r=0.69 could be determined, which is low due to the high degree of subjectivity in evaluating categories [21]. The underlying elements are based on the MARS evaluation framework, which has an excellent internal consistency of alpha=0.9 [16]. The coding was comprehensively reviewed, with a focus on content elements with low corresponding reliability values.

Phase 2.4: Execution.
After performing the search, pre-test, and duplicate cleanup, 164 apps are remaining. Apps that are still being updated and maintained only will be subject to content analysis. The last update (concerning Android versions, most apps are represented in both stores) may therefore be carried out at most one year ago (deadline 01.07.2019). Apps without this Information were kept and further investigated. Overall, there were 133 apps left. Furthermore, only apps that have received a rating above 3 (of 5 stars) from the users (either iOS or Android) were the subject to content analysis. Apps that could be too new to receive a rating were still included in the analysis. Apps that have been given a poor rating by users should not be subjected to the analysis to focus on a positive user acceptance and correspondingly involved functional components. There remained 112 apps. Apps identified by secondary literature were added (+3). This resulted in 115 apps for an in-depth analysis according to the set category system.

Phase 3: Processing of results.
After successful completion of the content analysis, initial evaluations showed that of 115 apps, only 33 apps had more than 100 ratings (measured by the Google Play Store or Apple App Store) and were therefore subject to active user participation. Seventeen apps had more than 100,000 downloads (measured by Google Play Store) and thus attracted the attention of a wide range of users. The app "EDF & MOI" from the French energy provider Groupe EDF achieves top values in terms of popularity and user satisfaction. The most functional components (basic features, special features, and configuration options) are realized by the two apps of German origin, "Energy Buddy" and "Powerfox". The application of the MARS evaluation procedure described in Section 3.2 reveals that apps with a correspondingly high rating have an above-average number of functional components (e.g., "Powerfox"), are often new on the market, and are among the outstanding applications. In many cases, these apps are supported by additional media coverage or reviews. The average MARS Total Score is 2.71. Apps with a MARS Total Score above 3.0 (rating: "acceptable" [16]) have an average rating of approx. 4/5 stars. Within this article, a more detailed interpretation of the results and transfer into a dedicated report is not intended and is postponed to a follow-up publication.

Conclusion
In this article, we describe an adapted method for performing a content analysis for smartphone apps with an associated standardized category set for the evaluation of the identified analysis objects. This provides a suggestion for the stated research question and the need for further research. Subsequently, the method is evaluated by a demonstrational instantiation with a real existing research interest. The developed procedure is based on instructions of the literature on content analysis in classical media, e.g., newspaper articles. It addresses the lack of a procedure for smartphone apps, which have grown in popularity. The analysis of content and functions of smartphone apps also has an increasing relevance within design-oriented research projects. A systematic approach has not yet been documented in the literature. The documentation of the process saves researchers a considerable amount of transfer effort when adopting a method of traditional content analysis to the context of highly dynamic media such as apps. In addition, the proposed set of criteria provides a basis for comparability across multiple analyses. In particular, the special features of App Stores as necessary platforms for the search process are discussed, which, when carried out for the first time, generate high research and testing effort for the researcher and, also, cause different problems and resulting limitations mentioned in the identified literature. Previous publications about content analysis with apps either proceed without an explicit method or use approaches from the medical field. Furthermore, different procedures have been used for the evaluation of the identified objectsagain, many in the context of medical publications only.
The evaluation of the method has demonstrated the feasibility and led to small, iterative adjustments in the descriptions. Thus, the procedure could be guided, run through quickly and efficiently, and its focus was placed on the content of the coding. The first results of the MARS Score, which have shown the identified apps were suitable based on the assessment of the coders, also prove themselves in practice. This is evident from the above-average user ratings. In addition, the apps with high scores show an above-average number of features and are also, on average, newer apps. The MARS framework was assigned exclusively to the evaluating categories and revised in advance. Therefore, a time-saving extraction is possible using the documentation provided by the app developer (description text and screenshots, without the need to install the app, which is not always possible due to access restrictions). The proposed method offers a useful standard for the rating of the identified apps, which facilitates comparison with other app analysis and publications. During the instantiation process, the questions could be answered easily by the coders.
Based on the performed instantiation, the described method could be successfully tested and iteratively improved. As a result, it can be used for further investigations. The identified category set for the evaluation of apps was adapted, and also used, outside the domain of medical publications. Based on the first evaluation the method serves to increase the comparability of app analysis. In the following investigations, an intensive examination of the content components of the research object of the evaluation will take place. This will contain an adequate analysis, interpretation, and transfer of the results in a separate research report with the goal to identify which subcategory (a feature of the app) causes the highest degree of usage and satisfaction among users and how this interacts with the previous assessments of the coders.