Evaluating Company ’ s Readiness for Adopting Product Line Engineering : a Second Evaluation Round

Product lines have emerged in the software industry as an attractive approach to perform planned reuse of code. Nevertheless, a product line solution is not appropriate in all cases and also requires some conditions to be implemented successfully. The literature offers several contributions regarding the adoption of product lines. However, only a few of them are able to support decision-makers in making informed decisions in favor of or against following this approach. We proposed APPLIES, a framework for evaluating the organization’s motivation and preparation for adopting product lines. This article presents the second version of the APPLIES framework as well as the second iteration of the evaluation of this approach. This evaluation consisted of (i) a workshop with a practitioner who had experience in adopting the product line production approach and; (ii) a review of the content by five product line experts. The results obtained from the evaluation resulted in modifications to the framework content, mainly to simplify the statements and eliminate redundant elements. Also, we detected new functionalities and modifications that we expect to be resolved in the following evaluation iterations. Further evaluations and improvements are needed to mature the framework. Moreover, we expect to incorporate APPLIES into a process that covers the aspects that a company must consider before deciding to adopt this production paradigm.


Introduction
A product line is a set of similar products that share common characteristics, meet the requirements of a particular market segment and are implemented from a common set of core assets in a prescribed way [1].Product Line Engineering (PLE) is a systematic and comprehensive process aimed to develop and maintain product lines.
Product line engineering helps companies to reduce their efforts through two key aspects: systematic variability management and planned reuse.However, like any initiative that involves organizational change, adopting a product line engineering approach entails challenges that not all organizations are prepared to face.For instance, Brockers [2] mentioned that the way in which the insurance industry develops software results in a significant obstacle for introducing a product line engineering approach.One reason of these difficulties is that the knowledge needed to define a product line is distributed among insurance companies and software vendors, while another reason is the speed of innovation required by the insurance sector.Also, despite the advances in validation and certification of products derived from a product line [3], Habli and Kelly [4] determined that these concerns are barriers to implement product lines in safety-critical industries such as avionics.
Adopting a product line production approach is a change initiative that aims to benefit the business by improving its competitiveness and profitability.However, it is also well-known that change initiatives are time-consuming and costly [5], which explains why different authors encourage evaluating the change readiness before introducing any change in the organization [6]- [9].In that sense, we hypothesize that assessing a company's change readiness is important in helping decision-makers to decide whether or not to adopt a product line.
Previous studies have reported that both motivation and preparation are factors that affect the effectiveness of organizations in implementing change.Particularly, Armenakis et al. [8] proposed two key components to achieve the change readiness: (i) the belief that the change is needed, which means there is motivation to perform the change; and (ii) the belief that the individuals and the organization have the capacity to undertake the change [8], which means they are prepared to do it.However, while there has been significant effort spent in the product line community to plan and monitor the adoption of a product line [10]- [29], less attention has been given to help companies to evaluate their motivation and preparation for adopting a product line solution.
An alternative to address this need is to hire an external consulting team who provides expert opinion.In this case, the Product Line Technical Probe (PLTP) or the Product Line Quick Look (PLQL) approaches are ideal solutions [30].However, none of those alternatives are available in the public domain, and anyone interested in assessing his organization must request the Software Engineering Institute (SEI) for a consulting regarding PLTP or a PLQL and pay the corresponding consulting services.Another alternative is to use free-access approaches.For instance, Schmid and John [31] evaluate whether a specific domain has sufficient potential for the reuse.Fritsch and Hahn [32] propose a method to evaluate whether a systematic product line development would be helpful for an organization according to its target market and the set of potential products.Tüzün et al. [33] propose a decision support system to help companies to evaluate whether adopting a product line is feasible or not in their organization and to select a transition strategy to migrate towards a product line approach.Finally, Mazo et al. [34] propose a process and a collection of tables to evaluate the technical and economic feasibility of a product line production strategy in the software industry.However, none of these approaches evaluate why a product line approach might be useful for a company in the first place, and they only consider a reduced set of assessment criteria to evaluate company preparation for adopting this paradigm.
Given the above observations, the general goal of our research is guided by the following research question: how to help companies to evaluate their motivation and preparation for adopting a product line approach?
As an alternative we created APPLIES [35], a framework for evaluAting organization's motivation and Preparation for adoPting product LInES.This framework allows stakeholders to assess and prioritize factors that motivate the organization to adopt a SPL.In addition, APPLIES evaluates success factors that denote preparation for adopting a product line engineering approach.APPLIES was developed and improved iteratively.We follow a design-science methodology where the framework itself and all the related materials and tools are designed, evaluated and improved as they are evaluated in different scenarios.The design science methodology includes six activities: problem identification and motivation, definition of the objectives for a solution, design and development, demonstration, evaluation, and communication [36].The first two activities were previously mentioned in this introduction, and the other activities have been carried out iteratively.Two iterations were carried out in order to release APPLIES version 2.0.a1.
Iteration 1: A literature review was conducted (i) to identify the key factors that drive companies into a product line approach, and (ii) to identify the key factors to evaluate the company's preparation to adopt a product line.Based on this review, a first version of the framework (version 1.0.a1) was developed and reviewed internally by the research team.Then, some adjustments were made to this version and a next version (1.0.a2) was delivered and evaluated with a survey and an interview with two researchers in the PLE field.This preliminary evaluation led to the identification of minor issues such as unclear definitions that produced misunderstandings.Details of this iteration were published in [35].
Iteration 2: Version 1.0.a2 was evaluated by a practitioner and five SPL academics with industry experience.Feedback received in both evaluations activities were the basis to design a new version of the framework (i.e., APPLIES v-2.0.a1).This article reports the details of Iteration 2 that had two evaluations: a workshop with a practitioner with experience in adopting product line engineering, and an evaluation based on the judgment of five experts in PLE.As a result of these evaluations, the research team collected suggestions and observations to improve the APPLIES framework and release a new version named APPLIES v-2.0.a1.
The results of the retrospective evaluation were encouraging because they indicate that APPLIES is perceived as useful and could be valuable to software companies considering PL adoption.Regarding the evaluation with experts, their suggestions led to improvements to unify redundant content, eliminate unnecessary items and simplify the text of some others that were incorporated in the version presented here.However, some findings, described in the evaluation section (Section 3), have not yet been incorporated into the framework because we want to collect further evidence before deciding how to manage them.
The remainder of this paper is structured as follows: The design and development of the framework are explained in Section 2. Section 3 presents the empirical evaluation.Section 4 highlights major-related work and shows the novelty and complimentary issues of the APPLIES framework regarding related work.Finally, Section 5 presents the conclusions and future research.

Presentation of the APPLIES Framework
APPLIES stands for the framework for evaluAting organization's motivation and Preparation for adoPting product LInES.This framework aims to provide quality information to companies that do not know whether a product line approach is feasible to their case.For instance, this framework can be applied by consultants, project managers, product managers or anyone who needs information to convince management levels to adopt a product line initiative.APPLIES is not intended to replace decision-makers.Instead, it provides a systematic approach to support their judgments before making any decision either for or against the adoption of a product line.
APPLIES is divided into three levels: Conceptualization, Operationalization and Implementation explained in Figure 1 and presented in detail in the rest of this section.
The Conceptualization level provides the theoretical basis of APPLIES.The Operationalization level makes the evaluation tangible through a process, guidelines, and charts that articulate the theoretical basis.Finally, the Implementation level provides the support users require to provide inputs to the APPLIES framework and obtain the results.In addition to the above separation, APPLIES consists of two articulated components to evaluate a company's motivation and preparation for adopting a product line engineering approach.The first component is called APPLIES-motivation and the second component is called APPLIESpreparation.APPLIES-motivation: helps companies to identify factors that motivate the adoption of a software product line approach in a company.APPLIES-motivation aims to answer the question: which specific needs of a company might be addressed by following a product line production approach?APPLIES-preparation: assesses to what extent a company is prepared to transit towards a product line approach and helps users to identify points where particular attention is necessary.Hence, APPLIES-preparation aims to answer the question: to what extent a company is prepared for adopting a product line production approach?

Conceptualization Level
The conceptualization level defines the theoretical basis that APPLIES uses to evaluate the company's motivation and preparation for adopting a product line approach.For this purpose, there are two assessment models at this level, one for each component, that contain and organize the constructs under evaluation: the motivation assessment model and the preparation assessment model.Both assessment models were the result of a preliminary review of diverse published works on product lines, as presented in [35].Subsequently, the pertinence and understandability of both assessment models were empirically evaluated as presented in Section 3.

Motivation Assessment Model
The motivation assessment model includes 20 concrete "signals," henceforth referred to as adoption drivers, which indicate that a product line approach could be useful for a company.
Most adoption drivers are signals reported by companies with experience in adopting product lines.Adoption drivers were classified according to a SWOT structure of strength-based drivers, weakness-based drivers, opportunity-based drivers and threat-based drivers.The category in which each driver was placed depends on whether the driver was considered a strength, weakness, opportunity or threat to companies.For instance, if a company retains its customers as a result of the variability between its products, then the company has a strength that could be exploited by following a product line approach.On the other hand, if a company must repeat the maintenance work for each customer, then the company has a weakness that could affect its productivity.In turn, this weakness could be addressed with a product-line production approach.

Preparation Assessment Model
The preparation assessment model organizes the factors in a hierarchical structure composed of 3 dimensions, 17 criteria, and 58 sub-criteria.Notice that previous version of APPLIES (v.1.0.a2) has three dimensions, 17 assessment criteria and 67 sub-criteria, but after the evaluations reported in this article, nine of those sub-criteria were removed because they were redundant or unnecessary, as is explained in Section 3.
At the top level, the dimensions are the perspectives of analysis from which APPLIES assesses the company preparation i.e. operational, economic, and technical.
The operational dimension includes criteria for evaluating to what extent the product line approach fits in the organizations' culture, processes or practices.The technical dimension includes criteria for evaluating to what extent the organization has the technical capability for transitioning towards a product line approach.Finally, the economic dimension includes criteria for evaluating to what extent the associated cost and benefits would materialize if a product line approach was adopted.
At the middle-level, each assessment criterion is a signal that we use to determine how well prepared the company is for adopting a product line approach.For instance, the domain knowledge, the commitment of the managers or the potential products of the product line are some of those assessment criteria.
Finally, at the lower level of the preparation assessment model, the sub-criteria are the information that users rate to evaluate the company's preparation.
In the preparation assessment model each assessment criterion has an importance that indicates how indispensable the criterion is for adopting a product line.The possible values are "very important (3)", "important (2)" and "desirable (1)".They were assigned, as explained in previous work [35], according to the number of references found for each criterion in the literature."Very important" criteria are those for which we find the most entries in the literature, while "desirable" criteria are those for which we find the least number of entries.

Operationalization Level
This level makes the evaluation tangible through the processes, guidelines and charts that articulate the theoretical basis defined at the conceptual level.At this level APPLIES has a process for evaluating company's motivation and another process for evaluating company's preparation.Both processes are explained in this section along with the charts for summarizing the results and the response format to capture users' responses.

Process to Identify the Company's Motivation
APPLIES-motivation follows a three-step process to identify drivers that motivate a company for adopting a product line approach.These steps are: (i) select the adoption drivers, (ii) calculate the alignment score, and (iii) draw the results into the alignment chart and the adoption share chart.Each step is explained below.

(i) Select adoption drivers
Users of APPLIES-motivation select from the collection of adoption drivers providing those that are aligned with the organization's case.Figure 2 shows an extract of the motivation instrument in which users of APPLIES-motivation complete their selection.Drivers could be selected or not selected, therefore here APPLIES uses a binary response format with two binary categories "yes" and "no".
(ii) Calculate the alignment score The tool that implements APPLIES calculates automatically the percentage rate of adoption drivers selected by the user.This percentage represents to what extent the organization feels aligned with the adoption drivers that have been reported in the literature.Also, the tool that implements APPLIES calculates the share of the selected drivers among the four types of drivers.This information shows whether the company's motivation for adopting a product line comes from organizational strengths, opportunities, weaknesses or opportunities.
(iii)Draw the result into charts Two charts present the alignment score calculated in the previous step.Figure 2 shows an example of both charts.Part (B) of this example shows that seven over 20 of the adoption drivers were selected in a dummy example case.From this selection the 57% of the drivers were motivated by organizational weaknesses (Figure 2 part C).

Process to Assess the Company's Preparation
APPLIES-preparation follows a five-step process to assess the preparation of a company.Those steps are: (i) customize the importance value of the assessment criteria (optional), (ii) calculate the relative importance of each assessment criterion, (iii) go through each assessment criterion and rate their sub-criteria one by one, (iv) calculate the grade of each assessment criterion, and (v) synthesize and plot the results.Steps (i) and (iii) require input from APPLIES users, while the remaining steps are automatically executed in the assessment instrument that supports APPLIES-preparation (see Section 2.3).Each step is explained below.
(i) Customize the importance value of the assessment criteria (optional) Companies might have different and independent prioritization criteria.APPLIESpreparation allows users to customize the importance of any criteria at any time during the evaluation.This customization is done with a number between zero and three.Zero means the criterion is irrelevant to the organization, one means the criterion is desirable, two means the criterion is important and three means it is very important.From now on, this number will be called customized importance value.

(ii) Calculate the importance of each assessment criterion
The relative importance represents how relevant each criterion is in comparison with the preparation of a company for adopting a product line approach.There are three categories that APPLIES-preparation includes to represent this importance: very important, important and desirable.APPLIES-preparation calculates the relative importance of each assessment criterion combining information from two different strategies: evidence from the literature and stakeholder opinions.
On the one hand, each assessment criterion has a suggested importance value that we assigned from the literature review as explained in Subsection 2.1.2.On the other hand, from getting information from stakeholders, APPLIES-preparation considers the customized importance value assigned by the user in step (i) of this process.In this third step, APPLIES-preparation averages both values, to calculate the importance of each criterion.For instance, the suggested importance for the criterion commitment of key actors in the assessment model is very important, which has a numeric value of 3. If a user of APPLIES assigns a value of 1, which means desirable, to this criterion, the resulting relative importance will be "important.",which corresponds to the number 2 that is an average of 3 and 1.

(iii) Go through each assessment criterion and rate their sub-criteria one by one
In this step, users rate the sub-criteria of each assessment criterion one by one.Assessment criteria whose relative importance is equal to "Not applicable" could be omitted because this relative importance means that these criteria are not important for the evaluation.Users should select a category from a five-points Likert-response format to grade the subcriteria.This response format contains the same number of positive and negative positions and provides the "don't know" option as a neutral value to give respondents a 'get out' if they cannot answer the question or when they are unsure.The accepted values are: (5) strongly agree, (4) agree, (3) don't know, (2) disagree, and (1) strongly disagree.
Figure 3 shows a screenshot of the assessment tool that implements the assessment in APPLIES-preparation.The tool that implements APPLIES automatically calculates the score for each assessment criterion.This score is the average of the scores of each sub-criterion.For instance, the criterion commitment of key actors has two sub-criteria.If the evaluator rates the first subcriterion with "strongly agree" and the second sub-criterion with "disagree," the resulting score is 3.5 which is the average between 5 for "strongly agree" and 2 for "disagree".

(v) Synthesize the results
Inspired by a SWOT analysis [37], in this step the assessment criteria are classified into opportunities, threats, weaknesses, and strengths according to their score.The current version of APPLIES-preparation sets in three the cut-off score that separates good results from bad results.Internal criteria with a score below of the cut-off value are weaknesses and external criteria in the same case are threats.On the other hand, internal criteria with scores equal to or upper than three are classified as strengths, and external criteria with the same scores are classified as opportunities.A SWOT matrix named "preparation grid" graphically presents the resulting SWOT classification.Figure 4 shows a screenshot of this preparation grid with dummy data.Each assessment criterion has a position in this matrix depending on its resulting score, its relative importance and its type (internal, external).Internal criteria are placed on the right side of the chart, while external criteria are placed on the left side.Moreover, the Y-axis depicts the score from zero to five, while the X-axis has three values that correspond to the relative importance of each criterion.For instance, in Figure 4 the assessment criteria reuse potential, and commitment of key actors are among the weaknesses because both have a score lower than three.
The preparation grid also has a color progression in the background that goes from green to red.This background pretends to call attention to crucial points: we assume that very important criteria with high scores are more relevant than desirable criteria with high scores.Similarly, very important criteria with low scores are more critical than desirable criteria with low scores.
This summary chart gives information to detect the weakest and strongest points of the company about how prepared it is for adopting a product line approach.Furthermore, this result helps companies to plan their short-term and long-term strategies in case they are interested in adopting a product line approach.
A novelty of this version of APPLIES, compared to the previous version published in [35] is the calculation of the overall score regarding company's preparation for adopting product lines and the score for each dimension.This score is calculated as a weighted average between the assessment criteria (see Figure 4 part b).The weight of each criterion depends on its relative importance.Based on the results, companies with a preparation score lower than three are considered "low prepared".Companies with a score between 3 and 4 (not included) are considered partially prepared, and companies with a preparation score equal or upper than 4 are considered well prepared (see Figure 4 part c).For now these values were arbitrarily defined considering that good performance is equal or superior to 80%, and low performance is less than 60% over 100%.It is expected that these values will be refined as APPLIES is used in real environments where we can assess the influence of the framework on decision making and adjust the values accordingly.

Implementation Level
Two evaluation instruments were defined from the evaluation models presented in Subsection 2.1.Both instruments are self-administered questionnaires made up of 20 and 58 items respectively.The items that make up the instruments are close-ended statements that users grade.
A tool implemented as an Excel book is the part of APPLIES visible to users.This tool implements both the evaluation instruments and the evaluation processes presented in Subsection 2.2 and it is in charge of capturing user responses, processing them automatically and plotting the results.

Empirical Evaluation
APPLIES was evaluated through a workshop with the CEO of a software company, who had practical experience in adopting product lines, and five PLE experts.This section presents both empirical evaluations.These evaluations were an early attempt to explore how useful, understandable and easy to use the framework was from the point of view of an industry practitioner as well as to improve the content of the assessment models that provides the conceptual basis for the assessments.

Evaluation Purpose
The research questions that guided the workshop were oriented to evaluate the (i) perceived usefulness, (ii) intention to use, and (iii) content understandability of the APPLIES framework from the perspective of a practitioner with previous experience in adopting a product line approach.The research questions addressed were:  RQ1: How useful do the practitioner perceive the APPLIES framework and why?  RQ2: Would the practitioners intend to use the APPLIES framework, why/ why not?  RQ3: How understandable do the practitioner perceive the content of the APPLIES framework?Why?  RQ4: What additions or modifications could improve the APPLIES framework?

Evaluation Settings
Acople Tecnológico is a small Colombian company with more than seven years of experience in the software sector.Since its foundation, Acople has created software solutions that use biometric devices to address needs such as access control or time recording.The company has a consolidated customer base and several renowned Colombian companies have successfully used its applications.Acople Tecnológico began the incremental transition to a product line approach in 2014.Since then and to date the company has transformed its previous applications into a product line of biometric applications.
The evaluation of APPLIES was carried out as a workshop with the CEO of Acople Tecnológico.The selection of the company was based on the suitability of the organization given its experience adopting a product line approach and its willingness to provide access to the information needed during the evaluation.Furthermore, the CEO was suitable to perform the evaluation because: (i) he had more than 15 years of experience in the IT business; (ii) he actively participated during the transition from a standard software development approach to a product line approach in his company, and (iii) he knows in detail the conditions of the company before and after adopting a product line approach.

Design
We designed this workshop in four stages in which the CEO used the APPLIES framework from a retrospective position given that the company had already implemented a product line production approach.The four stages planned were: (i) pre-interview, (ii) framework presentation, (iii) interaction with the tool that supports APPLIES, and (iv) post-interview.
First, we conducted a pre-interview followed by a 30 minutes training session in which the CEO of Acople Tecnológico learned the main concepts of APPLIES and the objectives and procedures of the evaluation.After the training session, interaction with the tool that supports APPLIES started.In this part, we asked the CEO and obtained his consent to record the screen while he performed tasks we assigned to him.During this part, the main researcher was available to answer any questions, but she could not interfere with the CEO's activities.Finally, once the CEO completed all the tasks, we asked him to complete a questionnaire that evaluates his intention to use APPLIES in the future and his perception of the usefulness and ease of use of APPLIES.
Handouts for the activities: We prepared as handouts: (i) slides presenting the APPLIES framework.(ii) a questionnaire to know the context of Acople Tecnológico before carrying out the adoption, these questions were formulated during the pre-interview; (iii) a questionnaire to know the CEO's opinion after using APPLIES; and (iv) the tasks to be carried out in APPLIES, These tasks consisted of following the steps explained in Section 2.2 to assess the motivation and preparation to adopt a product line approach in the case of Acople Tecnológico.
Data collection methods: The data collection methods of this evaluation were interviews, observation and the strategy of think aloud.Semi-structured interviews were conducted during the pre and post interviews.The observation took place while the CEO completed the tasks assigned during the evaluation.The researcher did not interfere with the activities performed by the interviewer, but the researcher answered questions and made clarifications when it was necessary.Furthermore, we instructed the CEO to speak out loud and explain his actions while he was doing the assigned tasks.The observation was registered in real time by taking notes and recording the CEO's voice.Data were collected only in the presence of the main researcher in a remote way using Skype.However, to reduce bias in the analysis, we used two different methods to collect information (interviews and observation) and more than one researcher participated in the analysis of the data.
Analysis procedure: audio recordings were transcribed with an "intelligent transcription" strategy, i.e. a transcription that intends to improve the text for readability, e.g. by adding personal pronouns and articles omitted in hurried speech.Later, the transcripts were coded following a descriptive coding strategy, i.e. a strategy where codes are short phrases that summarize the basic topic of a passage of qualitative data [9].This strategy was useful to find differences and similarities among the comments.The interview was conducted in Spanish and the answers were transcribed using the original language.

Results
Acople Tecnológico started its transition towards a product line approach with only basic knowledge on this topic: "We had heard about the topic and we were starting to get involved in this subject".Now, after four years of experience in product line engineering, the CEO considers that they "have practical experience and we had done complex tasks to conduct the migration … however, there is still much to be learned about product lines".
In this section, the findings of the evaluation are presented through the answers to each research question.

RQ1/RQ2: Perceived usefulness and intention to use
The results on perceived usefulness and intended use are positive.In fact, according to our findings, from a practitioner with experience in adopting product lines, APPLIES is useful and could be recommended.Also, the CEO would potentially use APPLIES if he had a new project related to the adoption of product lines in the company.
The following are the comments that support the above-mentioned statements."I would recommend and would have used APPLIES if it had existed when the company took the decision; …, in general I do not see any reason for not recommending it".The CEO clarifies that the only factor he believes could discourage the use of APPLIES in the decision-making process would be that the use of the framework would require paying a cost that the company could not afford.On this point, APPLIES is a free initiative and therefore we do not believe that an economic factor can become an obstacle to its use.Other positive comments regarding the perceived usefulness are: "I think APPLIES is a very good tool, not only because of the calculation of the scores in each evaluation criterion but also because of the last chart (referring to the preparation grid).This chart summarizes the information very nicely….,I consider APPLIES is something good that can be recommended".
"The tool shows the reality of how we were at that time.For instance, the criterion with the worst score was the process discipline, and it was the criterion where there were many things to improve, in fact, we had to change this part completely".At that time, the decision was inspired more by what the company could gain instead of a systematic analysis of the motives and the conditions to perform this change.In this regard, the CEO also said that APPLIES could help people to think on "things that can be important, but that you just discovered when the APPLIES framework asked about".
Finally, in the satisfaction assessment, the CEO assigned the highest score to statements indicating that: (i) APPLIES offered him a repeatable approach to evaluate his company's preparation for adopting a product line approach; (ii) using APPLIES he could evaluate to what extent his company is prepared for adopting a product line approach; (iii) using APPLIES, he would feel more confident to decide for or against adopting a product line approach in his company; (iv) APPLIES results would help him to understand which factors he needed to put attention at when a company adopts a product line approach; and (vi) APPLIES results would help him to detect criteria in which company's members would be willing to change and to detect those in which some resistance could be expected.

RQ3: Understandability
The CEO completed all assigned tasks and had no critical errors using the tool.He considers he could provide the inputs that APPLIES needs easily and that APPLIES was easy to follow.However, in our observations we noticed that the CEO made some non-critical mistakes while he performed the assigned tasks.For instance, he modified or erased unintentionally some texts as well as some formulas for calculating the resulting scores.
Regarding the content, during the workshop different clarifications were required to explain better the meaning of some sub-criteria.For instance, the sentence "product line champion" of one of the assessment sub-criteria was unclear to CEO.Moreover, regarding the language clarity the CEO comments "I have a technical background, so in general I understand the concepts included in APPLIES, but I am not sure the language is clear enough for other types of stakeholders".Additionally, he said: "I think the questions are fine but there are some that need more context and some others that can be separated into parts".
Previous comments are also supported by the satisfaction questionnaire in which he gave a low score to the questions related to the understandability of the adoption drivers and the subcriteria.
Concerning the resulting charts, the CEO indicated that the percentages in the chart "share of selected drivers" were difficult to read because the results overlap each other.In addition, regarding the time required to interpret the graphs, we noticed that the interpretation took to the CEO a few minutes, while what we want to get to is that people could interpret the resulting charts in seconds.
In summary, regarding the understandability, the results showed us that the content of the framework and the charts that summarize the results could be still improved.Some content improvements were already incorporated as the results of the evaluation cycle especially with the experts, as explained in Subsection 3.2.3.
RQ4: Suggested improvements that will be considered for the next iteration Tool usability: the CEO expressed that he would prefer a web-tool for APPLIES.He believes a web-based presentation will make the tool more intuitive, which, in turn, will remove the need for preliminary training and will avoid unintentional text modifications as actually happened during the workshop.
Motivational charts: charts that summarize the motivation do not consider the operational, economic and technical dimensions evaluated in the preparation assessment.The CEO indicates that it would be interesting to relate these dimensions of analysis in the motivational part to make more evident, for instance, to what extent the strengths that motivate adopting the product line approach influence the company's preparation.
Calculate the potential products: according to the CEO it would be useful to incorporate the calculation of the potential products because this information also affects the decision.The item 31 of the preparation assessment model (in Table 2 of Appendix) asks about the number of potential products, but it does not provide any support to calculate this value.We will take into account this suggestion in order to evaluate whether we find feasible to incorporate it in a future version of APPLIES or, perhaps, in a next stage of the evaluation.
Add the option "don't APPLY" to the response formats: when APPLIES was used to evaluate in retrospective the case of "Acople Tecnológico", the CEO found that some subcriteria were not relevant to the company.However, the response format for rating the subcriteria did not include any option similar to "don't APPLY or N/A".
The CEO suggested including this additional option in the response scale to offer more flexibility to users when there are items that do not affect the company under evaluation.

Expert Opinions
Expert opinions are a series of scientific efforts, used for interpreting data, predicting system behavior, and evaluating uncertainties [38].The expert review has been used in the literature to ensure that the content of the artifact under development is correct and relevant for the domain under study.For instance, in [39] experts were consulted to contrast and compare findings from a literature review.In [40] experts were consulted during the development of an assessment instrument for measuring key factors of success in software process improvement, while in [41] they were consulted to evaluate the content of an instrument to measure the resilience safety culture.

Evaluation Settings
The research questions that guided this evaluation were oriented to explore to what extent the assessment models of APPLIES were pertinent and complete.The research question addressed was: What modifications do require the assessment models to have a more understandable and pertinent content?Why?

Design
We performed the expert review based on the guidelines proposed by Li and Smidts [38].Initially, four experts were invited to participate in the expert review of APPLIES.Once they accepted the invitation, we sent an email with a guideline that presents APPLIES and a questionnaire to evaluate the content of the assessment models.As explained in Subsection 2.1, the assessment model constituted the conceptual basis of APPLIES.Two experts sent back the completed questionnaire while the other two did not respond within 30 days.Due to the above, a fifth expert was contacted.This expert agreed to participate in the interview and the experts three and four also accepted to participate but only in the interview.In total, five experts participated in the interview, but only two of them answered the questionnaire.Two experts reviewed the content in detail during the interview, and one expert reviewed the structure in general but did not go deeper in the details of the content.The first contact with experts took place in January 2018, while the last interview took place in May 2018.The interviews were conducted individually, three of them face-to-face and two of them virtually.

Interview design
Interviews with experts were divided into four stages: setup, framework training, questionnaire, and closing.
Setup: The researcher explained to the expert the purpose and conditions of the interview and provided him the assessment tool.This tool already had data from a fictitious company to have information to show in the resulting charts.When possible, the interview was recorded.Finally, the researcher asked questions about its context and previous experience of the interviewee.
Framework training: The researcher presented and explained the components of APPLIES (APPLIES-motivation and APPLIES-preparation) while asking the expert to interact with the assessment tool at the same time that the explanation was done.
Questionnaire: Once the expert became familiar with the tool, he was asked questions to know his perception of the usefulness, completeness and potential limitations of APPLIES, as well as to know other works that could be of interest to consider.
Closing: The interviewer asked the post-questionnaire questions to evaluate the quality of the interview itself and thanked the expert for his time and feedback.
Handouts for the activities: We prepared the following handouts for the review of the experts: (i) the questionnaire in which experts could explain which items they considered confusing, redundant or unnecessary; (ii) a document named "expert guidelines" that presents the fundamentals of the version 1.0.a2;(iii) a questionnaire oriented to know the context of the expert (these questions were formulated during the pre-interview); and (iv) a questionnaire oriented to know the expert's opinion after interacting with APPLIES.
Expert selection: This evaluation was performed with five experts that satisfied the following conditions: they (i) have proved experience by publications or have participated in product-line related projects; (ii) have experience in consulting or managing the adoption of product lines in real settings such as enterprises or research projects; (iii) have diverse backgrounds and affiliations to achieve a wider perspective and independence in the feedback received as Wohlin et al. [42] recommend; and: were willing to participate in the evaluation.Above conditions were adapted from Li and Smitdts [38] who argue that several criteria are necessary when choosing an expert reviewer.
Four of the experts held a Ph.D. in computer science; all of them have at least ten years of professional experience and a minimum of three years in the product line engineering field.All of them participated as researchers in product-line related projects, one of them has been a consultant and one of them also worked as a software architect in the implementation of product lines.Finally, two of them got involved in the decision of adopting a product line approach in real contexts, one of them worked in a project to migrate legacy systems towards a product line approach, while the other participated in a project in which product lines have been used to support the customization of video-games for supporting the speech therapy.
Expert opinion aggregation: The results extracted were not aggregated because we wanted to analyze the suggestions proposed by each expert separately.It should be noted that we did not seek consensus among the experts, but looked for potential points for improvement and, therefore, once the different opinions had been collected, we processed them separately.We then compared the diverse opinions to identify commonalities and differences.

Results -Content of the Assessment Models
This expert review resulted in changes classified in the following seven types: C1: Modification suggested by experts; C2: Wording improvement; C3: Delete; C4: Delete due to merging with other items; C5: Adjust to incorporate other items; C6: No change; C7: Open issue.Figure 5 shows how many changes per type were made in both assessment models.Each one of them is explained in the rest of this subsection.Table 1 and 2 in Appendix contain the assessment models resulting from the changes here described.
C1. Modification suggested by experts and C2.Wording improvement Six adoption drivers and 25 sub-criteria had vague definitions, required style improvements, were improved by the experts during their revision or were redefined or renamed using simple language for practitioners.For instance, the assessment criterion "The product line project will have a "product line champion" or "angel" at a high level of the hierarchy" was changed to "The product line project would have a project leader with authority to make decisions and support the idea of change."The item "The company is pledged to the industry standards (e.g. for protocols, interfaces, components)" was rewritten as "The company follows industry standards."

C3. Remove
In the preparation assessment model the sub-criteria "The organization is a pioneer in the development of products" and "The organization has a strong software engineering capability" have been removed as some experts suggested.The first sub-criterion was eliminated because it represents a situation that could be ideal for many projects, but it is not an exclusive requirement to adopt a product line production approach.The second criterion was eliminated because it is an abstract concept that is evaluated concretely with other items that belong to the same assessment criterion.

C4. Delete due to merge with other items and C5. Adjust to incorporate other items
Seven sub-criteria were combined because they overlap with other sub-criteria.For instance, in the preparation assessment model the sub-criteria: "Current customer needs are known", "Future customer needs could be predicted" and "The organization could predict the evolution of the products" were merged into the sub-criteria "It is foreseeable a future market for the products that will be developed under the scope of the product line".
C6.No change Four items in the motivation assessment model and 11 items in the preparation assessment models are the same reported in the version 1.0.a2 of APPLIES.
C7. Open issue Items annotated with code C7 indicate changes, suggestions or problems that we want to review in more detail before making any change.In total, eight items in the motivation assessment model and 14 items in the preparation assessment models were annotated with this code.
Furthermore, two items in the motivation assessment model and three in the preparation assessment model were modified by recommendation of the experts or in order to improve the style, but still they might need extra changes in future versions of the framework.For instance, one of the experts suggested reviewing the adoption assessment model in detail because he considers this model has overlapping elements.One reason that according to the expert, explains this overlapping is that different levels of abstraction are mixed in the same motivation assessment model.For instance, some weakness-oriented adoption drivers are related to technical artifacts, while others are related to business processes.Problems in technical artifacts can be the result of problems in business processes which would lead to the overlapping.Other drivers where overlaps were reported were: Gain customers and New Market; Similar market and New market; Similar market, similar incoming plans and technology advances; Customize products in short time and New market; Scattered source code, uncontrolled product parts, and Similar products implemented differently.
Furthermore, two of the experts mentioned that the criteria "customer connection" and their sub-criteria are not exclusive to evaluate the preparation for adopting a product line approach and could therefore be useless for evaluating the preparation to adopt a product line approach.

Threats to Validity
Before the execution of each evaluation we designed a protocol to guide the evaluation.However, there are threats to the validity of the study, which are briefly described and grouped according to the research method.

Workshop Generalization of the results:
We interacted only with one member of a software company not randomly chosen and therefore we cannot make statistical generalizations about our results or the validity of the framework.However, as mentioned at the beginning of this section, the purpose of the evaluation was to find areas for improvement in APPLIES and not to validate it.The process of evaluation and improvement of APPLIES must continue in order to achieve that the proposal reaches the state of sufficient maturity to be validated.

Effects of one interpretation of one single data source:
The transcript of the interview results were discussed in the research team.Also, we kept the interview extracts that justify the reasoning behind the resulting analysis to keep traceability of the data source and the corresponding interpretation.Moreover, two different data collection methods were used to data triangulation.Although it can be said the evaluation could be limited in the sense that data came from one single informant (the CEO of Acople Tecnológico), in total the evaluation had three hours of duration which allowed us to capture valuable information.Moreover, we felt that the atmosphere in the workshop was open and trustworthy, while the responder did not hesitate to answer our questions honestly.
False or incomplete information: We cross-checked interview recordings, notes of observations and video-recordings for contradictions to reduce the amount of false or incomplete information.However, our findings may be incomplete and missing important factors due to the lack of another type of source of information.
Misconceptions and misunderstandings between the collected data and the reality: we sent a preliminary version of this article back to the CEO, to ask for his revision in order to avoid misunderstandings among our interpretations and the reality.

Experts Reviews
Overconfidence bias: Expert opinion is a qualitative method that requires a degree of subjectivity and implies the problem of relying on informal evidence that may be influenced by personal opinion [43].Tversky and Kahneman [44] explain that experts' revision is subject to overconfidence bias.Overconfidence bias is the tendency in which a person's subjective confidence in his or her judgments is reliably greater than the objective accuracy of those judgments, especially when confidence is relatively high.To mitigate this threat experts were trained about the purpose and components of the framework before answering the interview questions.This strategy allowed encoding the knowledge of the expert with the knowledge needed to answer the questions proposed during the interview.Moreover, we asked the experts for the justifications of their answers as Chibbet et al. cited by [38] suggest.
A limited number of experts: According to Li and Smidts [38], if an expert is "perfect" (i.e.internationally recognized expert with experience in academy and industry), the number of experts required for an expert elicitation process is one.However, there is a possibility that the expert will make a mistake or that his knowledge is inadequate, and therefore it is safer to use more than one expert.This study included five experts in total.
Construct validity: This threat refers to ensuring the correctness of the measures involved in the investigation, the relevance of the used concepts, and the proper chain of evidence.Construct validity was mitigated by conducting a pilot test with members of the Centre de Recherche in Informatique (CRI) before running the interviews with the experts.Also, the transcripts of the interviews and the summary of the improvements suggested in the content questionnaire were consolidated and discussed in the research team.We kept the interview extracts that justify the reasoning behind the resulting analysis.In this way the traceability of the data source and the corresponding interpretation is assured.

Related Work
Many researchers have been interested in the adoption of a product line approach from various points of view.For instance, some works present strategies for planning and assessment of an eventual adoption of the PLE paradigm [22], [23], [34], [45], [46], while others propose strategies for transitioning towards a product line approach once the planning step is done [22], [24]- [29].Many proposals present alternatives to evaluate the maturity of a product line initiative once this initiative is being implemented in a company [12]- [21], and others report experiences in adopting product line initiatives [27], [29], [47]- [58] or estimating the quantity of material needed to guaranty a production plan of products [59].However, even if those approaches are interested in the adoption of a product line, their focus is not on assessing a company's motivation or preparation for adopting a product line approach.
Extensive preparation is a crucial factor for transitioning successfully when any change is conducted and therefore, being prepared for adopting a product line approach is fundamental to avoid wasting company's time and money.However, to best of our knowledge, there is no unified framework of reference that gathers criteria for assessing to what extent current and future company practices encourage or impede a product line adoption.On the contrary, those factors are scattered over different sources in the literature.For instance, Bandinelli and Sagardui [60] propose an overview of the benefits and risks that a product line adoption might imply.Schmid and John [31] define a method to evaluate whether a domain has sufficient potential for reuse based on structured interviews.Fritsch and Hahn [32] propose a method that analyzes the target market and the potential products of a company to evaluate whether a systematic product line development would be helpful for an organization or not.The Software Engineering Institute proposes the "Product Line Technical Probe" (PLTP) [30].This method is a diagnostic that examines the preparation of a company to succeed with a software product line approach based on the 29 practice areas specified in the SEI Framework.Finally, Tüzün et al. [33] propose a decision support system to help companies to select what transition strategies will help them to migrate towards a product line approach.
APPLIES is complementary to these existing approaches.On the one hand, APPLIESpreparation includes the criteria proposed by the previous approaches but also considers other factors retrieved from the literature.Furthermore, APPLIES-preparation provides a defined process and a tool to operationalize the assessment.On the other hand, unlike the previous approaches, APPLIES-motivation also analyzes the motivation of a company for adopting a product line approach.This motivation is analyzed with a collection of "symptoms" that indicate to what extent a company might be interested in adopting a product line approach.As far as we know, none other approach has considered analyzing the company's motivation for adopting a product line as APPLIES-motivation does.

Conclusions and Future Work
This article presents the version 2.0.a1 of the APPLIES framework as well as the second iteration of the evaluation that has been carried out in the process of developing and consolidating this framework.APPLIES is an alternative that seeks to help companies to evaluate their motivation and preparation for adopting a product line approach.The ultimate goal of APPLIES is to provide quality information to companies that want to know whether a product line is viable to their case or not.The second cycle of evaluation reported in this article consisted of (i) a workshop with an industrial practitioner who had experience adopting the product line production approach and; (ii) a content review by five experts.
The results obtained from the evaluation with the practitioner allowed us to understand that APPLIES is a practical tool, perceived as useful to be used in real cases.Both the experts and the practitioner suggested functionalities and modifications that we expect to resolve in further iterations of evaluation and improvement.On the other hand, the suggestions of the experts led to modifications in the content of the assessment models mainly to simplify sentences and eliminate redundancies.Also, we incorporated in this version of the framework the calculation of the level of preparation (well prepared, partially prepared and not prepared).Therefore, the content of the evaluation models, together with the calculation of the level of preparation are modifications that differentiate this version of the framework (2.0.a1) from our previous work [35].
Regarding the future work, APPLIES is still under development and therefore new iterations of evaluation and improvement have been carried out.One of these was an evaluation conducted with 14 practitioners of a postgraduate course in software engineering.In this experiment we evaluated human factors such as perceived usefulness and understandability over the version of the framework presented in this article.This evaluation was reported in [61].
Other evaluations are also planned.For instance, regarding the quality of the content we plan to perform a second round of expert revisions.Regarding the perceived usefulness and understandability of the framework, we plan to conduct usability tests with potential users.
Finally, regarding the intention of using APPLIES in real contexts, we are implementing the APPLIES framework as the online service of a product line engineering tool suite called VariaMos [62] and we are looking for companies that want to use APPLIES to evaluate APPLIES in real settings.

APPENDIX
This appendix has two tables presenting the motivation assessment model and the preparation assessment model of APPLIES v.2.0.a1.To facilitate traceability between versions, each table contains the previous version of the text and points out the wording changes between versions.Also, the tables show a code to identify and keep track of the change reasons.In cases where there is no the text of the version 1.0.a2this means that the text was not modified between previous and current versions of APPLIES.The code of each change is the same as explained in Section 3: C1: Modification suggested by experts; C2: Wording improvement; C3: Delete; C4: Delete due to merging with other items; C5: Adjust to incorporate other items; C6: No change; C7: Open issue.The reference numbers in tables comply with the list of references of the article.The organization retains its current customers due to the variability among their products [13] C6 2

Gain customers
The organization gains new customers due to the variability of their products [13]  APPLIES v1.0.a2

C1
The organization gains new customers due to the variability among their products [13] 3 Legacy code The enterprise has reusable legacy code that it would like to translate into a more maintainable form [63]  APPLIES v1.0a2

C1
The enterprise has reusable legacy code that it would like to refactor into a more maintainable form [63] Opportunity-based drivers 4 Customize products in short time The market requires an increasing number of specialized products in a short time.These products might be developed by the organization [48], [54], [56] C7 5 New market There is a new niche market with shared characteristics.The company expects to enter to this market [48] C7 6 Similar market The market is "similar enough" to set features that can be shared across product variants.However, there are still differentiating characteristics [64]  APPLIES v1.0.a2

C7
The market is "similar enough" to set features that can be shared across product variants and at the same time there are still differentiating characteristics [ The company has identical or highly similar software code segments due to poor reuse practice such as cut and paste programming practices [50], [54], [55], [65]  APPLIES v1.0.a2C1 91 The company has identical or highly similar software code segments due to poor reuse practice such as clone-and-own [50], [54], [55]

7
The organization can control how the product is developed, e.g.there are no design constraints such as specific components use or support of customerspecific interfaces [64] C7 The organization can balance customer and productcentered approaches in product development [12], [71] C7 92 9 The organization has the potential of offering new products that market has not anticipated [16], [64] C7 Domain knowledge (very important) 10 Engineers have enough expertise to decide what is common and variable in the area where the product line might be implemented [52], [63] Engineers have enough expertise to decide what is common in the domain [52], [63] C5 11 The organization has the know-how to capture the requirements that will cover the product line [13], [16] C6 12 The organization has successful products in the area where they want to adopt a product line approach [63], [64] The organization has successful products in the targeted market [63], [64] C2 The organization could predict the evolution of the products [48], [73], [74] C4 22 It is foreseeable a future market for the products that will be developed under the scope of the product line [16], [29], [31], [32], [48], [54], [60], [64], [72]- [74] A future market for the products under the scope of the product line is foreseeable [16], [29], [31], [32], [54], [60], [64] C5 Openness (very important) 23 Individuals in the organization are open to change how they interact with colleagues and other departments [15], [26], [67] Individuals in the organization are open to change how to interact the different business areas [15], [26], [67] C1

C4
Organizational willingness (very important) 26 If necessary the organization is willing to do internal changes such as modify the organizational structure [29], [54], [70], [75] The organization might be will to perform internal changes required to adopt a product line approach [29], [54], [70], [75] C2 27 Different organizational units are able to work collaboratively [15], [29], [48], [55], [76] C5 28 Merged in the item 28 The cross-functional activities around departments can be synchronized [15], [55] C4 29 Individuals would be willing to try new techniques, tools or ways to do things [63] The individuals would be committed to try product line engineering methods [63] C1 Potential products (very important) 30 The organization develops more than one product that shares functionalities and could be integrated into the product line [32] The organization develops more than one product that could be integrated into a product line [32] C1 93 31 There is potential to create at least three different products that share common functionalities [32], [77] The potential product line could create at least three different products [32], [77] C1 32 Products of the product line will be developed with a stable technology (refresh rate >5 years) [32], [48], [77], [78] C7 Process discipline (very important) 33 Requirements are documented in a structured and disciplined way [13], [29], [49] C7 34 The company defined, specified and, optionally, optimized its processes using any systematic approach.e.g.CMMI [18] The organization has experience applying standardized engineering procedures for developing its products.i.e.CMMI [18] C1 35 There is explicit documentation which is up-to-date and available for being reviewed [13], [39], [64], [71] C2, C7 36 The organization has formal plans to maintain its assets [16] C6 Product line awareness (very important) 37 The organization staff has knowledge about product line engineering.Otherwise, they are willing to learn [22] C6 38 If needed, the organization would provide the infrastructure, training, and materials that could require a software product line [63], [64], [69], [79] If needed, the organization would provide capacity building about product line engineering [63], [64], [69], [79] C2 39 The company will allocate employees' time for capacity building about product line engineering [64] Technical dimension

Commonality and variability (very important)
40 Potential products have common features [13], [27], [32], [58], [63] C6 41 Potential products have common nonfunctional requirements [32] Potential products have common qualities [32] C1 42 Potential products have differentiating features [64] Potential products have differentiating characteristics [64] C1 Reuse aptitude (desirable) 43 The company has established reuse processes for providing or consuming external assets [29] The company has established reuse processes for providing or consuming external components [29] C1 44 Current organizational structure allows developing reusable assets [29] Current organizational structure allows developing reusable components [29] C1 45 Staff members are considering possible future use when they develop software artifacts [15], [29] Management and developers considering possible future use when they develop software artifacts [15], [29] C2 46 Technical staff is willing to develop new software based on reusable components rather than build it from scratch.[69] Technical staff agree developing new software based on reusable components rather than build it from the scratch [69] C1 47 There is low coupling between existing software artifacts [31] C7 48 There is high cohesion between existing software artifacts [31] C7 49 There is an explicit organization vocabulary shared by the staff about terms related to the area where a product line approach is interesting [29] There are unified and shared understanding of domain-specific terms among the staff [29] C1, C2 Reuse potential (very important) 50 The legacy code could be incorporated into the product line approach with minor or no changes [29] C6 94 51 Assets which might be incorporated into the product line, are reliable to use [69] Components, which might be incorporated into the product line, are reliable to use.[69] C1 52 Current products are common enough for reusing key design elements [49] C6 53 There are technical or business artifacts that might be reused [69], [76], [78] There are technical and business artifacts such as field service technician documentation, customer engineering, or interaction that might be reused [69], [76], [78] C2 54 Architectures of different products are modeled and aligned [29], [80] Architectures of different products are aligned [29], [80] C1, C7 55 The company follows industry standards [29], [60], [63] The company is pledged to the industry standards (e.g., for protocols, interfaces, components) [29], [60], [63] C1 56 There is technology compatibility among products to incorporate into the product line [29] C6 Software engineering capability (very important) 57 The organization has a strategy to control the change management in their artifacts [15], [16], [18], [29], [76], [78] C6 58 Removed: It is eliminated because it is an abstract concept that is evaluated concretely with the other items that belong to the same assessment criterion The organization has a strong software engineering capability [56], [64], [76] C3 59 There are established developer roles, such as software architects, testers, developers [29] There are established developer roles, such as software architects and reuse coordinators [29] C2 60 Staff have technical competence in the technology that might be used for implementing the product line [22], [64] C7 61 Proprietary tools used to develop the current products are not a barrier to adopt a product line approach [64] The organization is independent on proprietary tools built and used to support a single system approach [64] C1, C7

Economic dimension Economic indicators (desirable)
62 The potential return on investment (ROI) of the software product line meet the organizational financial target [16], [60], [63], [78] C7 63 The maintenance cost of the former products without planned reuse is higher than migrating the products towards a product line approach [16] C7 Readiness to invest (important) 64 There is a budget for investments [27], [63], [75] C6 65 The organization is willing to do regular investments to keep code healthy and keep the number of variations manageable [69] C6

Readiness to pay migration costs (desirable)
66 The enterprise is willing to pay costs for training people in SPL engineering [69] The enterprise is willing to pay the costs for training people in SPL engineering [69] C2 67 The enterprise is willing to pay costs for building the reusable assets that will support the product line initiative [29] The enterprise is willing to pay costs for building the core asset base [29] C1

Figure 2 .
Figure 2. Screenshot of the evaluation instrument in the part that supports: (A) data collection for APPLIES-motivation, (B) chart that summarizes the alignment of the company with the proposed adoption drivers, and (C) chart that summarizes the share among the four SWOT categories of the selected drivers.

Figure 3 .
Figure 3. Screenshot of the evaluation instrument in the part that supports the data collection for APPLIES-preparation

Figure 4 .
Figure 4. (a) Example of the preparation grid chart with dummy data.(b) and (c) overall preparation score and guidelines to interpret the resulting score

Figure 5 .
Figure 5. Number of changes in the motivation assessment model (left) and preparation assessment model (right) by type of change If needed, individuals are open to change their current practices[15],[26],[67] If needed, individuals are open to change their development practices[15],[26],[67] C5 25 Merged in the item24 If needed, individuals are open to change their task assignments[15],[26],[67]

Table 1 .
Adoption drivers that make up the motivation assessment model of APPLIES v 2.0.a1

Table 2 .
Dimensions, assessment criteria and sub-criteria that belong to the preparation assessment model