Evaluating the Persuasive and Motivational Design Method: Supporting Design for Sustainable Behavior Change

Introducing new technology within an organization requires a change in its employees' ways of working and thinking. Such a change is necessary to increase the adoption of the technology. The main issue is that employees do not sufficiently grow with innovation being promoted. When a change program is initiated, often it focuses on a short-term change, whereas a more sustainable change in behavior is necessary to fully adopt the technology. This article concerns the validation and evaluation of a design method called Persuasive and Motivational Design (PMD). PMD is a design tool aiming to bring about a sustainable behavior changes during the development of Information Systems. It is constructed using existing psychological methods and models. We used a survey amongst first users and the case of a significant application to get both a qualitative and quantitative evaluation of our method. Our conclusion is that PMD is a promising technique. Furthermore, the results of this evaluation indicate some ways to further refine and improve our method.


Introduction
The introduction of technology within organizations requires a change in its employees' ways of working and thinking. This is especially the case when the new technology is a major improvement in the way of working. For the successful introduction of the new technology, a smooth transition from the old to the new situation is an absolute condition. Many investigations such as [1], [2], [3], have shown that a purely top-down introduction of new technologies is not the most successful method. Other studies, [4], [5], [6], show that the early involvement of employees pays off. In this paper we restrict ourselves to evaluating the smooth acceptance of new ways of working by employees. There are several behavior change programs which focus only on awareness and not on the needed sustainability [4] and design methods and which focus only on a methodologye.g. gamification [7], [8], instructional design [9], [10], inclusive design [11], [12] and do not fully support psychological models which would enable the target audience to achieve the needed behavior change. We conclude that the main issue is that, using theses methods, employees do not sufficiently grow with the innovation being promoted. Another point is that a change program often focuses on a short-term change whereas a more sustainable change in behavior is necessary in order to fully adopt the new technology [5].
This article focuses on the evaluation of a design method, called Persuasive and Motivational Design (PMD), introduced in [13], which is briefly summarized in this paper. This method uses (game) psychological models in its design for sustainable behavior change within organizations. The structue of the method is analogous to a modern game. The reason for this is that experiences with game psychological models show that they allow for raising awareness [14], mapping behavior [15], and facilitating a sustainable behavior change [16]. The PMD-method has some similarities to the A 3 -framework, as proposed in [17], however our method focusses more on organizational behavior and how this can be sustainable in the long-term.
The goal of this article is to evaluate the PMD method for its efficacy and adoption and to validate its usefulness and effectiveness in practice. In order to do this, we used a questionnaire in which we measured the actual and perceived efficacy, and applied PMD in a special development project to measure its adoption in practice. When applying PMD to a particular case, we have used a few assumptions. First, the current situation and organizational goals have been described. Second, the project team is a mixture of domain experts and system analysts. Lastly we assume that motivational elements (such as game elements) do not always work on the target audience and, therefore, a behavioral analysis is required.
This article is part of a larger project regarding the development of an effective and professional design method to be used in practical situations. A first paper [18] outlined the possibilities of using motivational elements such as game elements to enhance tasks of knowledge workers in an organization by focusing on some desired behavior. In a second paper [13] we introduced PMD as a new method to be used for the design of sustainable behavior changes within organizations, based on the findings discussed in the first paper.
The article is structured as follows. In Section 2 we discuss the research methodology used. In particular, we explain what motivated our choice for the combination of evaluation techniques we used. In Section 3 we present an overview of the PMD-method. Section 4 concerns the online survey and Section 5 concerns the application of PMD to a case. In Section 6 we discuss the results of the online survey and case study and then we combine the results into an overall evaluation. In Section 7 we discuss the overall evaluation of the PMD-method, draw some conclusions, and discuss future steps to refine and improve the method.

Methodologies Used
The need for a framework, to address the sustainability part of behavior changes, originated from existing findings that most behavior change programs focus only on the first step of change. They do so by creating awareness [4] that "changes regarding behavior are usually necessary when introducing a […] system within an organization" [13]. The evaluation of the suggested PMD-method serves to better understand and asses this method [19]. Verschuren [20] mentions that, in the evaluation stage of design-oriented research, "it should also be evaluated whether they (parts of a model) form a coherent and balanced whole". For information on evaluation techniques for systems analysis and design modeling methods, see [21].
In work psychology, from which the PMD-method originates [13], the two most used evaluation techniques are surveys and case studies [5]. A survey is often used to get a better insight into a new method and is described as "a good evaluation technique for systems analysis and design methods" [20]. A case study enables more "generalizable settings [...] which could not be easily replicated in a laboratory setting" [21]. We chose to combine both techniques to evaluate the usage of the PMD-method, because this enables us to provide more convincing and reliable findings [22].
We used a mix of research paradigms. The first one is Design Science as described by Hevner [23]. In his approach Hevner distinguishes seven guidelines for an iterative design process. These guidelines address: design as an artifact, problem relevance, design evaluation, research contributions, research rigor, design as a search process, and research communication. This article concerns, as well as the 'research communication' aspect, the 'design evaluation' aspect. Hevner describes this as follows: "the utility, quality, and efficacy of a design artifact must be rigorously demonstrated via well-executed evaluation methods".
Two other research paradigms we used come from applied psychology; more specifically, work psychology [5]. Work psychology distinguishes between two research approaches: 'Scientific enquiry' and 'Problem solving' [5], [24]. The former is about getting all the details for answering the research questions using literature study, and then publishing the findings in a scientific article, and the latter is about working together in research with stakeholders in order to solve some problem [5]. To evaluate the PMD-method by working together with stakeholders, we chose to conduct a case study in practice, and a survey among students of a PMD-method course (for more information about the course see [13] and Section 4: 'Setting up a survey among users and students').
In Design-Oriented research, the evaluation of a new artifact is very important and our evaluation is based on 'The Designing Cycle' by Verschuren [20]. The designing cycle emphasizes six stages: first hunch (small set of goals to be realized with the artifact to be designed), requirements and assumptions (requirements to be fulfilled with the frame defined by the goals), structural specifications (deriving the design from the requirements), prototype (realization of the artifact), implementation (putting it into practice) and evaluation (does the prototype fit the design goals?). Our previous paper [13] covers the first four stages, where the PMD-method is the prototype; the current article covers the implementation and evaluation parts.
To evaluate the PMD-method using a survey and a case study, we used the theoretical 'Method Evaluation Model' by Moody [25]. This model closely relates to the use of innovation and technology within organizations [25], which our PMD-method focusses on, by supplementing the widely used Technology Acceptance Model (TAM) [26].
The Method Evaluation Model consists of six parts:  Actual Efficiency: the effort required to apply a method.  Actual Effectiveness: the degree to which a method achieves its objectives.  Perceived Ease of Use: the degree to which a person believes that using a particular method would be free of effort.  Perceived Usefulness: the degree to which a person believes that a particular method will be effective in achieving its intended objectives.  Intention to Use: the extent to which a person intends to use a particular method.  Actual Usage: the extent to which a method is used in practice. The actual efficacy (measured by combining the actual efficiency and the actual effectiveness) of the PMD-method was obtained through the results of the case study, in which the PMDmethod was put into practice. The perceived efficacy (measured by the perceived ease of use combined with the perceived usefulness) and the adoption in practice (measured by the intentions combined with the actual behavior) of the PMD-method were obtained through the results of the survey amongst distributed students and users of the method. See Figure 1 for an overview.

Persuasive and Motivational Design
The Persuasive and Motivational Design (PMD) method was developed in order to help to design for a sustainable behavior change that would be necessary when introducing some innovation within organizations. The method consists of a four-layered framework, consisting of seven steps, in an iterative approach to design, for a sustainable behavior change, supporting the PMD-method, which is depicted in Figure 2. The four layers concern: goal definition, mapping behavior, motivational change and testing. These layers are the result of the three main concepts of the PMD-method -Goals, Behavior and Motivation [13] and a need for testing to facilitate an iterative approach. The PMD-method further suggests the use of several models and methods, in each step, to complete the framework.  [13] showing the steps of the framework with preconditions (P i ) and postconditions (Q i ) with 1 i The seven steps of the framework are: 1. Conceptual framework definition (by identifying Problems and Objectives) 2. Requested Behavior 3. Current Behavior 4. Changing Current Behavior to Requested Behavior 5. Intrinsic motivation check (allow for sustainability) 6. Elements selection 7. Playtest The first step in the framework is to define the problem and objectives: Which objectives, or goals, do we ultimately want to reach; or which problem do we want to solve? Motivational design used in, e.g. games, can make it possible to achieve these goals in an engaging manner [16], [27], [28] and [29]. Input for this step is the need for change when introducing a new way of working (via some innovation) within an organization. The outputs of this step are the identified problems and goals to be addressed by the solution.
The second step in the framework is mapping the requested behavior and distribution of this requested behavior within an organization. This can be done by the use of a behavior or personality model. To map behavior, we suggested the Bartle Player Types [15], [13]. Bartle distinguishes four kinds of 'player types' by plotting them onto two dimensions. These dimensions are: 'Players vs. the World' and 'Acting vs. Interacting'.
The four player types are described as [15]:  Killers: acting on other players  Achievers: acting on the world  Socializers: interacting with other players  Explorers: interacting with the world To formalize and map this behavior using the Bartle Player Types, we suggest registering the number of characteristics for each type, and the degree to which the various BPT can be associated with that behavior. This characterization and role can be seen as a vector in a fourdimensional BPT-space. The similarity between the player (or behavior) vector and the association with it , can be obtained by taking the inner vector product [13]: (1) Inputs for this step are the identified problems and objectives in the first step of the framework. The output of this step is the (distributed) required behavior to address these problems and reach the set objectives.
The third step in the framework describes the current behavior of the target audience and the distribution within the organization. Again, we suggest the use of the Bartle Player Types and the use of vectors to map this current behavior. Inputs for this step are the objectives from step two. The output for this step is the current (distribution of) behavior in the organization, based on the situation defined in the first step.
Step four is about how we can change the current behavior into the required behavior. To facilitate such a behavior change, we need to identify the associated trigger. For this we suggested the 'Fogg Behavior Model' (FBM) [30] as described in [13]. Fogg states that three elements are important in order to reach a certain target behavior: Motivation, Ability and a corresponding Trigger. To identify what level of motivation the target audience has reached regarding the set objectives, we suggested the Organismic Integration Theory by Ryan and Deci [31]. "The Organismic Integration Theory distinguishes four levels in extrinsic motivation [...]: External regulation (doing something because we must do it, sometimes get punished if we will not do it), Introjection (doing something for status, self-esteem or social acceptance), Identification (doing something because we can identify with the meaning but still need some trigger) and Integration (doing something because it is related to our own goals, no real need for a trigger but not yet fully intrinsic)" [13]. Inputs for this step are the required and current behavior identified by steps two and three. The output of this step is identification of the needed behavior change.
The next step in the framework is an intrinsic motivation check to allow for a sustainable behavior change. To allow for the target audience to be intrinsically motivated, we suggested the use of the 'Keys to Fun' by Nicole Lazzaro [32] in this step. "Nicole Lazzaro distinguishes four 'keys' to fun: Hard fun (Emotions from Meaningful Challenges, Strategies, and Puzzles), Easy fun (Grab Attention with Ambiguity, Incompleteness, and Detail), Serious fun (Generate Emotion with Perception, Thought, Behavior, and Other People, getting better at something that matters) and People fun (Create Opportunities for Player Competition, Cooperation, Performance and Spectacle; Fun out of playing with others)" [13]. To keep the target audience intrinsically motivated we suggested focusing on the three major items which, according to Daniel Pink, are needed to accomplish this [33]: Mastery, Autonomy, and Purpose. Input for this step is the identification of the needed change in step four. The output of this step is a way to trigger the required behavior and get people intrinsically motivated to perform this behavior, leading to a sustainable behavior change.
In step five, based on the results of all previous steps, the right elements to change the behavior within the target audience can be selected, which is the output for this step. These elements are the foundation of the design of the solution. Finally, in step seven, the design which has been created can be tested with a selection of the target audience to find out if the right behavior is triggered and maintained.
Each step in the framework provides the input for the next step, moreover step seven can be used as input for step one, if at the first attempt, the output of step seven does not cover the set objectives in step one. This allows for an iterative approach to Persuasive and Motivational design. For a more detailed description of the framework and the suggested models to be used within each step see [13]. Section 5 will show how the PMD-method, including our framework, was used in practice, therefore showing to what extent the method helps in processing the input needed to generate the desired output for each step.

Setting up a Survey among Users and Students
To measure the perceived efficacy and the adoption, in the practice part of the 'Method Evaluation Model', we needed as much information as possible from people who knew about the PMD-method and might have used it. The PMD-method has been taught at Atos, a leader in digital services with a focus on business technology, which has circa 100,000 employees; in the form of two different courses (Foundation and Advanced) involving 55 students (both internal Atos professionals and external people) [13]. These students (and users) of the PMD-method were the target audience of the online survey. Since the size of the target audience was small and all could be reached via e-mail, there was no need for sampling and the whole target audience was asked to fill out the survey anonymously. They had one month to fill out this online survey.
The questions in the online survey were created using the TAP (Topic, Applicability and Perspective) paradigm [34] and were based on the methods suggested in the PMD-method and the suggested survey items from the TAM [26].
The survey was based on the last three phases of the Method evaluation model [25] and consisted of questions regarding the PMD-method: its concepts, the framework, and suggested models and suggested theories. The questions were related the Method evaluation model as follows:  Experience: to obtain results on the Perceived Usefulness  Ease of use: to obtain results on the Perceived Ease of Use  Impressions and grades: to obtain results on the Intention to Use  Usage of the method or parts of it: to obtain results on the Actual Usage A total of 37 respondents filled out the survey, which is about 67.27 % of the entire target population. Because of our qualitative approach, we believe that the 37 respondents gave us a good sense of the perceived efficacy and adoption in practice regarding the PMD-method. Because the entire target audience was quite a small group, we asked an extended number of questions to gather as much information as reasonably possible.

Applying the Method to a Case
The next part of evaluating the PMD-method was to put the method into practice, enabling a case study. The results of this case study in a practical scenario were used to determine the actual efficacy and performance part of the 'Method Evaluation Model'. We conducted the case study at two separate units of the Netherlands Tax and Customs Administration (NTCA), "Uitgeverij: Online Innovatie" (Publisher: Online Innovation) and "Academie: Leren en Ontwikkelen" (Academy: Education and Development). The total population of the combined unit teams was about 30 people. In our study, we used a multiple case study approach, as described by [35]. This implies that we did two individual case studies; one for each of the teams of the NTCA. The findings have been integrated into one study for this research.
At the end of 2011 the NTCA started using a Social Enterprise Platform (SEP) to support online collaboration between employees of different departments, such as those at the Publisher and Academy departments. The research question raised by the NTCA for this case study was: "To what extent can Persuasive and Motivational Design be used to stimulate more online collaboration using the SEP, so that employees will reach their work-related goals more efficiently?".
At the start of the case study we created a baseline, to install an idea of how employees initially experience offline and online collaboration and how they put this into practice. During the creation of the baseline, we used two different kinds of data gathering methods: Interviews and Online Survey. Because of time aspects we needed some selective sampling in the target population for the interviews. We used Stratified Sampling and selected a sample from each stratum (Manager, Senior Expert, Junior Expert) using Simple Random Sampling. The sampling procedures are based on the work of Segers [36]. For the survey, we selected all the people in the target population with the exception of those whom we selected for the interviews.
The questions for the interviews and the online survey (of this case study) were the same, and they were, like the online survey amongst the students, created using the TAP (Topic, Applicability and Perspective) paradigm [34]. The questions were based on a conceptual model, which consisted of the research elements and research variables of this research. The conceptual model we used is depicted in Figure 3. After the interviews were conducted, we transcribed and coded the results using the procedures described by Chi [37]. 70 % of the respondents who were asked to fill out the survey did so, and 100 % of the respondents who were asked to participate in the interviews did participate.
The results of the baseline study were a good motivation to continue the case study. The attitudes of the respondents regarding the SEP were good, although the overall experience of the SEP was improvable. Werbach and Hunter [16] discuss the idea that motivational design (like gamification) can only work if the system or asset is of sufficient quality, i.e. PMD will not make the system or asset itself any better. On the other hand, if the final result of the overall experience of the SEP were much higher, the conclusion would have been that PMD could not add much value and possibly could even lead to the demotivation of employees [27], [31] and [38].
Further, the application of the PMD-method is described step by step as was shown in Section 3.

Step 1
The following objective was defined for the motivational design: Motivating employees of the NTCA to collaborate using an SEP to reach work-related goals more efficiently. The results of the baseline study showed that the 'Experience of the SEP' was the main reason why this goal has not yet been achieved. So, the main focus for designing the new solution using the PMDmethod was improving the experience.

Step 2
We wanted the target audience to collaborate more online and for this we wanted them to use the possibilities of the SEP more efficiently. To stimulate collaboration, we wanted the players to be more focused on their team than on themselves as individuals.
In this step of the framework, we focused on the situation of online collaboration using an SEP. Even in such a situation, people could not be categorized as just one player type [5]. To map the full required behavior (regarding all the player types), we listed (per player type) the desirable characteristics needed to reach the objectives.
 Killers: (Killer) enthusiasm to work with the SEP, enforce online collaboration (with colleagues), giving frank opinions and seeing how the player (or the player's 'team') is doing regarding others.  Achievers: Achieving work-related goals and getting better in their daily jobs.  Socializers: Interacting with colleagues (also from other divisions and teams), helping each other, interest in what is going on, and focused on (human) networking.  Explorers: Explore the SEP and share those experiences and find better ways (in the SEP) to collaborate online. The desirable characteristics resulted in player vector: . For each player type we registered the degree (between 0 and 1) in which the various Bartle player types (BPT) can be associated with that player. To create this role vector, we assigned a weight of 0.25 to the behavior of the 'killer,' because this behavior does help, in a limited way, to reach the set objectives as long as this player type is not quite dominant within the organization. To the behavior of the 'achiever' and the 'socializer' we assigned a weight of 1, because these behaviors relate the most to the defined objectives. We assigned a weight of 0.5 to the behavior of the 'explorer,' because, whilst we wanted our players to start exploring the SEP, this behavior should not take over.
The assigned degree to each of the types resulted in role vector: . The weights were assigned together with the stakeholders within the NTCA. The final desired behavior is depicted in Figure 4.

Step 3
For each player type, we again listed the behavior which belonged to our target audience. The behavior we listed was a result of the questions we asked in the survey, during the baseline study, and of observations. The current behavior is listed below per player type:  Killers: A limited imposition of ideas, giving new ideas no chance and pushing an opinion without listening to others.  Achievers: Achieving work related goals, getting better in their daily jobs, using the knowledge of others and sharing knowledge when they get something in return.  Socializers: Interaction with direct colleagues, helping direct colleagues, and interest in what is going on (direct colleagues and own department).  Explorers: Some exploration of the system (or world). The current characteristics result in player vector: . To create the role vector for the current behavior, we compared the results from the baseline with observations of daily practice, together with the stakeholders of the NTCA. In this way we assigned a weight of 0.5 to the behavior of the 'killer' and the 'explorer,' because we see this behavior, more or less, in daily practice. To the behavior of the 'achiever' we assigned a weight of 1, because we saw that the focus in the daily jobs of our players was mostly on this kind of behavior. We assigned a weight of 0.25 to the behavior of the 'socializer,' because we observed this kind of behavior only in a limited way in daily practice.
The assigned degree to each of the types resulted in role vector: . The final current behavior of our players is depicted in Figure 5.

Step 4
In our case we saw that a change in behavior was necessary; from the behavior belonging mostly to the 'achiever' to that of the 'socializer', resulting in a desired shift from acting to interacting, and from world (in our case the SEP) to players (in our case colleagues), thus achieving a change regarding both dimensions of the Bartle player types.
To facilitate the behavior change, we used the answers in the survey and interviews, regarding motivation and ability, combined with observations, to conclude the current motivation and ability of the target audience. Data derived from our baseline showed that the majority of the respondents were on the 'identification' level of motivation. In order to explain why people are not very much intrinsically motivated, we also asked several questions regarding the elements of intrinsic motivation as discussed by Daniel Pink [33]. The results of the baseline study showed the following results regarding the intrinsic motivation of the respondents (on a scale of 1-10): Autonomy: 6.27, Mastery: 5.31 and Purpose: 5.42. These results confirmed that the respondents were not yet fully intrinsically motivated.
As described above, it was necessary to have some trigger to effectuate the desired behavior in our players. The Fogg Behavior Model (FBM) [30] distinguishes three kinds of trigger: Spark (focused on motivation, when ability is high), Facilitator (focuses on ability, when motivation is high) and Signal (more a reminder, when ability and motivation are high). The baseline showed that motivation and ability were not very high; and measured on a scale from 1-10: Motivation: 6.1 and Ability: 6.38. Figure 6 depicted where our target audience were currently positioned in the FBM. As we see, the dot in Figure 6 is as close to the 'facilitator' as to the 'spark' trigger. Using Persuasive and Motivational Design we wanted to increase the motivation of the players and select a trigger which focuses more on ability. During our baseline study we also concluded that respondents did see some value in the SEP but did not fully know how to use it optimally. The trigger which focuses on ability is the facilitator. The goal of this trigger is to show how easy it is to collaborate online and therefore to reach work-related goals more efficiently.

Step 5
To allow for the change in behavior to be sustainable, we focused on the people fun 'key'. This key is mostly related to the Bartle player type 'socializer.' Just as with the player types, we did not focus solely on one kind of fun, so parts of the other 'keys' were also included in our design. The main focus was on people fun. To keep the required behavior triggered in the target audience, next to some element of fun, we introduced the concept of flow. Flow is described by Csikszentmihalyi [39] as "effortless action many people feel in moments that stand out the best in their lives". In our Persuasive and Motivational Design, it is therefore necessary to create the right kind of flow by keeping the focus on mastery and progression.

Steps 6 and 7
The next step was to select the right kind of elements based on the data derived from the baseline and the filled-out framework. The following game elements were selected:  Dynamics: -Rules and Restrictions: There were some rules which described the 'game arena' (often referred to as the Magic Circle within games, as described by Huizinga [40]) and the game rules describe what was allowed within this magic circle. -Emotions and Relations: We helped provide some feeling of meaningfulness to the employees regarding online collaboration. It helped them reach their work-related goals more efficiently by creating a network-oriented organization instead of the current hierarchical organizational form.  Mechanics: -Collaboration: The mechanical collaboration was the game element most related to our objectives. -Feedback: Feedback was used to show how the online collaboration proceded and where this might be improved. Feedback was shown visually by displaying some type of score (quantitative; based on the kinds of feedback as described by [41]). -Rewards: Online collaboration was stimulated by the use of rewards. Since the target audience was still extrinsically motivated, the use of rewards helped in the short term.
In the long term the focus would be more on intrinsic rewards (such as meaningfulness).  Components: -Social graphs: Employees could compare themselves with other colleagues (in an overall competitive manner and a social manner) to see how well they were collaborating. -Avatars: Every colleague would have a personal profile, on which they could place an avatar. Rewards could be given for completing a profile and keeping it up-to-date.

Results
In this section we discuss the survey (see Section 4) results and PMD-method application (see Section 5) results and combine these results for final evaluation of the PMD-method.

Survey Results
The final results of the survey were analyzed by taking the mean average of the answers given by the respondents, which was possible because most answers were numerical values and the distribution is approximately symmetric [42]. In this way a good overview could be generated for each part of the PMD-method (concepts, framework, models and theories), e.g. for each model or theory proposed in the PMD-method, questions were asked about the perceptions, intentions and actual usage. Questions about the 'experience' rendered answer categories from one (very bad) to five (very good); questions about 'ease of use' rendered answer categories from one (very bad) to five (very good); questions about impressions rendered answer categories from one (very bad) to five (very good); questions about overall grades rendered answers from one (very bad) to ten (very good) and questions regarding actual usage rendered answer categories from 'never' to 'more than 10 times a year'. The outcomes are listed in Table 1 and are a result of combining all the related questions. Results regarding which steps (of the framework), models and theories contributed or did not contribute to the design are listed in Table 2.  The results of the online survey among the students of the Persuasive and Motivational Design method gave an overview of the perceptions, intentions and behavior. The overall perceived ease of use scored a five (5) out of five (5) and the perceived usefulness scored a four (4) out of five (5). The overall perceived efficacy was graded as very good among the respondents of the online survey. Their intention to use the PMD-method scored a four (4) out of five, showing a high intention to use the PMD-method in their daily jobs. The actual usage of the PMD-method showed that on average the method was used one (1) to four (4) times a year, with some major outliers using the method more than ten (10) times a year. This shows that the adoption in practice scored very well.
Some interesting information emerged when questions regarding parts of the PMD-methods were asked related to identifying which parts contributed most or least to the final design. Looking at the framework, the respondents answered that steps one to four contributed most to the final design and steps five to seven needed some more work. The models which contributed most to the final design were, according the respondents, the 'Bartle Player Types' and the 'Fogg Behavior Model'. The models 'Activity Loop' and 'Keys to Fun' were considered to contribute least to the final design. When it came to the theories suggested in the PMD-method, the 'Meaningfulness', 'Self-determination theory' and 'Intrinsic Motivation' were considered to contribute most to the final design, whereas 'Classic Conditioning' was considered to contribute least. Figure 7 depicts the results of the survey in the 'Method Evaluation Model'.
The students stated that they could apply the theory as a whole and that they also used the theory outside gaming. Some examples of the additional applications where PMD was used are: (e-)Learning, change management, satisfaction survey, process optimization, consulting, system development, innovation, and coaching. The respondents also stated that time, budget and commitment from stakeholders was often a problem while using the PMD-method. Some stakeholder commitment was needed before an actual design could be produced, and consequently the results were better and more sustainable. This was later acknowledged by the stakeholders.

Case Study Results
Based on the results of the case study, we evaluated the actual efficacy of the PMD-method together with the stakeholders at the NTCA. Looking at the performance of the PMD-method, only a small amount of effort was needed to apply the method, therefore a score of three (3) out of five (5) was given to the actual efficiency. Some steps were felt to be somewhat redundant and some models and theories we considered not to contribute a lot to the final design. However, it was felt that the degree to which the PMD-method achieved its objective was quite high. It was considered that the PMD-method provided enough tools to create a design which was able to achieve the goal for the case study. A score of four (4) out of five (5) was given to the actual effectiveness.
The case study showed that almost all steps in the framework were necessary to design a solution using the PMD-method, with the exception of step five, regarding Fun. However, we feel it is important to have some 'Intrinsic Motivation Check' based on fun but now suggest this step should be a part of the subsequent step 'Game Element Selection'. We used all the models as suggested in each step of the framework. We believe that the Bartle Player Types and the Fogg Behavioral Model contributed most to the design. There are a number of other representations of mapping behavior [13] as alternatives to the Bartle Player Types. In this case the BPT-model was sufficient, but when a target audience should be analyzed more in depth we, argue that the BPT might not suffice. In that case we suggest the use of the Big Five approach [43]. The model which contributed least to the design was the 'Keys to Fun'. Elements of fun in our design were selected by the analysis of the required and current behavior of the target audience and the 'Keys to Fun' model did not really complement this.
In conjunction with the models we also used all of the theories as suggested in the method as discussed in [13]. The Organismic Integration theory and the elements regarding Intrinsic Motivation contributed the most to our design. They provided us with the right tools to analyze the motivational part of the target audience and helped us pick the right elements to increase motivation and design for intrinsic motivation. Another instrument, which was really helpful, was the use of a conceptual model to describe the objectives and problems associated with the design. We feel that, of all the techniques applied, none proved unnecessary for designing the solution.

Combining the Results
With the results of the case study we could finalize our evaluation by putting the results into the 'Method Evaluation Model'. The final results are depicted in Figure 8 showing both the survey (see Section 6.1) and case study (Section 6.2): results one next to the other.

Discussion and Conclusion
There are several similarities between the results of the online survey and the case study.
Step five (Intrinsic Motivation Check), together with the suggested 'Keys to Fun' model, is considered to contribute the least to the final design. Our suggestion would be to remove this step from the framework. However, intrinsic motivation is regarded as most important, as shown by both the results from the survey and the case study; so this step and theory should be maintained in the PMD-method. This could be integrated in either step four (to check whether the target audience is intrinsically motivated or identify why this could be lacking) or step six (to select the elements for the design, focusing on missing elements of intrinsic motivation). There are also alternative techniques, which would be of better use in different scenarios; for instance the Bartle Player Types. However, it was argued that this might not be sufficient to map complex behavior within organizations. The use of the Big Five was suggested instead. Overall, we see that where PMD started out as a means to design for a solution, the method proves itself is valuable in other contexts as well.
Further research will include refinement of the PMD-method and an evaluation of the results of deploying the PMD-method. Our current evaluation has provided valuable material to refine the PMD-method. Where this evaluation covered only the efficiency of the PMD-method itself, as a supporting system, the next steps are to evaluate additionally the end-product of this method. Does the design really effectuate a behavior change within an organization?
Overall it can be concluded that, in general, whilst the PMD-method, as a supporting system works, it could use some refinement. However, we can also conclude that, with the use of the right game elements and motivation, underpinned by the application of a dedicated game psychology framework, we can deliberately and purposefully design a solution to stimulate a necessary sustainable behavior change.