Interventions to improve organizations are undertaken quite often in both practice and research. Sometimes the outcomes (or the effect) of the interventions are evaluated, but the implementation (process) of the interventions is seldom the subject of evaluation. The intervention presented in the current paper includes one of the first quantitative process evaluations of an organizational intervention. Biron and Karanika-Murray (2014) reviewed many studies of organizational health intervention programs and found that the two issues most commonly studied were the effects of an intervention program and the factors influencing a specific outcome. Thus, they posited that organizational health intervention programs focus solely on evaluating results or outcomes rather than processes. This may be problematic because results-oriented models focus on the explanation of variations in a given outcome rather than on how organizational health intervention programs work (Biron, Burke, & Cooper, 2014). In organizational interventions and changes, both the type of the change (e.g., downsizing) and the method or mechanism of implementing the change (i.e., how the intervention is implemented; the intervention process) are important to evaluate (Biron, Karanika-Murray, Daniels, Hasson, Nielsen, Randall, Saksvik, & von Thiele Schwarz, submitted); Nielsen & Randall, 2012; Saksvik, Nytrø, Dahl-Jørgensen, & Mikkelsen, 2002.

We selected a salutogenic intervention for our process evaluation in the present project mainly because it involved the whole organization and was based on participation and involvement from all employees. We believe this is the fundamental principle behind success of all kinds of interventions at the organizational level, including more traditional interventions where the intention is to remove stressors. Rooted in positive psychology, Bauer and Jenny (2013) have argued for a shift in focus from the removal of negative psychosocial features in the workplace, to the development and promotion of the positive aspects of work. In the salutogenic model for health the main research objective is health promotion.

Kelloway, Hurrell, and Day (2008) originally proposed the concept of «countervailing interventions». According to proponents of countervailing intervention, turning the focal point of intervention programs to the development and promotion of positive salutogenic elements of the work environment may have counteractive effects on the negative features of the same work environment (Bauer & Jenny, 2013). In the present project we aimed to evaluate an intervention labelled «the employeeship program». The employeeship program aimed primarily at strengthening the psychosocial work environment by raising the employees’ awareness of and competence in interpersonal relationships, thereby increasing employees’ ability to take responsibility for their everyday work and their working environment. Prevailing opinion states that humans possess strengths such as courage, insight, perseverance, and hope, and that these strengths act as buffers against negative experiences like mental illness (Snyder & Lopez, 2005). Bakker and Derks (2010) point out that the negativity bias has become the focal point of mainstream psychology. Milch, Giæver, Vaag, and Saksvik (2013) maintain that a similar pattern can be found in occupational psychology. It has also become common for researchers to focus on negative psychosocial experiences (e.g. stress and absenteeism) in the work environment. The authors argue that more attention should be centred on developing and increasing positive psychosocial experiences, while simultaneously indirectly reducing the negative. This has now become a new trend in the traditional approach as well; to reduce stress and improve the mental and physical health of employees by introducing positive change initiatives (Biron et al., 2014).

According to Kelloway and colleagues (2008), there is a wide range of countervailing intervention program types. Programs promoting psychosocial work environments have existed for a long time (Milch et al., 2013). Although countervailing intervention programs appear to be popular in organizations, research on their initiation, implementation, and effectiveness is very scarce (Kelloway et al., 2008). Since typical countervailing intervention programs are not designed to reduce negative psychosocial experiences at work, they tend to have a different objective than traditional intervention programs. For this reason models used to evaluate traditional intervention programs cannot be used to evaluate countervailing intervention programs.

Best methods to evaluate workplace changes or intervention processes have not yet been established despite some initial studies, especially from the organizational changes literature (Armenakis & Bedeian, 1999; Burke, 2002; Tvedt & Saksvik, 2012). A few of these studies used a relatively open qualitative or observational study design (e.g., Dahl-Jørgensen & Saksvik, 2005), and more recently, a couple of other studies have used cross-sectional surveys (Randall, Nielsen, & Tvedt, 2009; Tvedt, Saksvik, & Nytrø, 2009). In the present study, we examined whether survey questions from previous studies were associated with employees’ and managers’ perceived success of an employeeship intervention.

Previous evaluations of the implementation process of organizational interventions used single process or single item measurements only (Murta, Sanderson, & Oldenburg, 2007). The most common measure has been the degree of participants’ exposure to the intervention. Findings from studies comparing the degree of participants’ exposure and intervention outcomes have been inconsistent, with some reporting more favourable health outcomes with higher exposure (Elo, Ervasti, Kuosma, & Mattila, 2008; Murta et al., 2007; Randall, Griffiths, & Cox, 2005) and others not finding such differences between participants in different exposure groups (Arnetz, 1996; Hancock & Craig, 1996; Wilson et al., 2010). Studies where intervention participants have been compared with non-participants are rare, but one study showed gender, personality and some work environment differences between the two groups in the same organization (Vaag, Saksvik, Theorell, Skillingstad, & Bjerkeset, 2012). Two studies have investigated employees’ perceptions of intervention implementation in relation to their health outcomes (Nielsen, Randall, & Albertsen, 2007; Randall et al., 2009). In a study by Randall and colleagues (2009), employees’ perceptions of their line managers’ attitudes and actions were more important for their future health status than their exposure to the content of the organizational intervention, which was team training. Employees’ perceptions of information and communication concerning an intervention have also been related to the intervention effects (Jimmieson, Terry, & Callan, 2004). Employees with higher levels of participation in the change process showed less resistance to change and more achievement of goals and commitment to their organization (Lines, 2004). Similarly, high levels of participation in the change process were associated with low levels of behavioural stress symptoms and higher job satisfaction in post-intervention measurements (Nielsen et al., 2007), and decreased self-reported work demands, increased social support and decreased stress levels (Eklöf, Ingelgård, & Hagberg, 2004). These results provide further support for the possibility that factors concerning program implementation have a major impact on program effects (Randall et al., 2009). In the present study our ambition was to include a broader scope of implementation items in order to identify factors that provide a better understanding of the process part of interventions.

In 2012, the economy and real-estate unit (ERU) at the Norwegian University of Science and Technology initiated an intervention called the «employeeship program» (EP). The intervention was implemented because an employeeship survey showed that several sections of the unit had psychosocial work environments in need of improvement. The focus of the intervention was on how to improve unit performance and employee job satisfaction. Participation in the program was mandatory for all 409 ERU employees. A second goal of the intervention was to improve the ERU’s quality of customer service by increasing employees’ customer communication skills. In this study, we had two main aims:

1. We wanted to evaluate the items constructed for process evaluation by a research group in Denmark, building on the works of Randall et al., 2009 and Tvedt et al., 2009. A model has been developed that gives an overview of the different categories that these items comprise (Tvedt & Saksvik, 2012), but the research group in Denmark developed their own categories based on the Randall et al. validation study (see the homepage of Arbejdsmiljoforskning, 2014). Our evaluation was conducted both by using a quantitative approach (factor and regression analysis) and a qualitative approach (personal interviews).
2. We also wanted to learn how the final process items extracted were linked to the participants’ own perceptions of the success of the intervention (the employeeship program). When performing this analysis we controlled for other variables (gender, personality, engagement, and organizational commitment) that can be associated with the success of interventions (Vaag et al., 2012). The research question that guided the qualitative part of the study was: «Does process evaluation by interviewing add important aspects to information gained by surveying employees?» Interviews may for example add to surveys by examining specific issues raised in the survey in more detail, by adding to the evaluation of the factors found in the survey, or by examining additional issues not covered in the surveys. The interview guide, thus, included both open questions about the process and specific questions based on the survey.

The core objective of our study was, thus, to evaluate a mandatory salutogenic intervention to learn more about the contributions of these kinds of interventions to health related outcomes and to study a quantitative evaluation approach of the implementation of the intervention.

Methods

The employeeship program.
Setting, participants and study design. Invitations to participate in this study were initially sent out by e-mail to ERU employees. Different sections of the ERU were also informed about the program at section meetings by their section leaders a few weeks before the program started. It took about one year to implement the intervention for all employees, starting February 15, 2012 and ending March 20, 2013. The EP was conducted by Kibu AS, an external consultant company that works to develop internal communication and collaboration processes of organizations. The program consisted of three full-day workshops, spread over a period of six to nine months for each section. Each workshop had a group of about 30–50 participants. Employees in different sections participated in the EP together as one group as much as possible. The workshops took place at a course centre outside the workplace during working hours.

The purposes of the three workshop days were as follows:

1. Day one involved an employee psychological test called the Diversity Icebreaker1. This test measures preferences for communication, interaction, and different problem-solving styles (Diversity Icebreaker Homepage, 2013). Individual employees’ strengths and weaknesses and organizational strategies to overcome these weaknesses were identified.
2. Day two consisted of practical exercises in collaboration and communication. The exercises were somewhat different between groups, but all exercises were built around the same model. The model involved five stages: Planning, Implementation, Reflection, Identification of improvement, and Identifying actions for transferring new insights to daily work.
3. The third day was a brief course in customer communication skills. Some groups learned about communication methods and created a communication plan for their section, while other sections underwent training in verbal and non-verbal communication with an actor.

The survey (quantitative evaluation).
In total, the survey had 34 questions which covered employee demographic information, personality, engagement, health and other work-environment scales. The questionnaire used in this study was based on well-established and previously validated instruments used to assess occupational health and well-being. It included measures of engagement (Schaufeli, Salanova, Gonzalez-Roma, & Bakker, 2002), personality (Eysenck Personality Questionnaire [EPQ-12]; Eysenck & Tambs, 1990), organizational commitment (Mowday, Steers, & Porter, 1979) and overall satisfaction with the intervention (as measured on a Likert scale from 1 to 6 of «How satisfied are you with the employeeship program?»). Questions about the process evaluation were collected from Randall et al. (2009) and Tvedt et al. (2011). A model has been developed that gives an overview of the different categories that these items comprise (Tvedt & Saksvik, 2012), but the research group in Denmark developed their own categories based on the Randall et al. (2009) validation study (see the homepage of Arbejdsmiljoforskning, 2014). The final 20 questions can all be found in the Randall et al. study (2009) and/or in the Tvedt et al. study (2011). These 20 questions were placed at the beginning of the questionnaire and the overall satisfaction question at the end. Questions about the intervention process were structured in three scales measuring how the implementation was initiated and carried out, and what improvements it resulted in. The implementation of the intervention (20 items) was rated on a 5-point scale ranging from «totally agree”» to «totally disagree».

The questionnaire was sent to all 409 intervention participants by email, using an online survey system called Select Survey (www.selectsurvey.net). Reminders were sent by email about three and six weeks later. If employees completed less than half of the questions, their survey responses were excluded from our analyses.

Evaluation of EP: Data acquisition, participants and analyses. On the issue of sample size, there are many opinions. There are some that recommend a ratio of 10:1 from participants to items. Tabachnick and Fidell (2013) maintain that the researcher ought to have at least 300 participants in order to conduct a factor analysis. They, however, add that a sample of around 150 should be sufficient, especially if there are numerous marker variables with high loadings above .80. The number of factors was decided upon based on the criteria of a scree test and the Kaiser’s eigenvalue over 1. The screeplot was thereafter inspected, and it showed an obvious break after the second component, which suggested retaining only the first two components. This was further supported by the results from the Parallel Analysis that was conducted from random data of a similar size (20 variables x 190 respondents). Parallel analysis allows researchers to determine the number of factors to retain after running a factor/principal component analysis (PCA) (Pallant, 2013). In this paper, we have made use of the Monte Carlo PCA for Parallel Analysis. As a second step, descriptive statistics and intercorrelations were calculated for all variables identified in the study.

The question about satisfaction with the employeeship program was used in a regression model as the dependent variable with gender, personality, engagement, commitment, and the intervention process variables identified in the PCA as predictors (Enter procedure in two blocks, with the process variables in the second). Gender and personality were treated as control variables, and engagement and commitment were expected to be associated with the employeeship program, either as a prerequisite for success or as a consequence of the program, but the direction of association could not be investigated.

The interviews (qualitative evaluation).
Sample. Seven respondents were identified and recruited for the interviews, with help from the section leaders. Since the interviews were considered as supplemental to the survey, we deemed this a suitable size for the sample. The respondents recruited represented independent sections and different levels in the organization. There were several reasons why help from the section leaders was necessary in the recruiting process. The interviews were conducted during working hours, and therefore it was necessary for section leaders to arrange for employees to take time off to participate. Section leaders sent out invitations by e-mail. Some of the participants volunteered, and some were asked to participate based on their section, as well as for practical reasons (as mentioned above). Based on the various working environments in the different sections it was interesting to have some diversity in the sample (for example leader experience, seniority, or gender), as this potentially could lead to different perspectives on the findings from the survey.

The interview procedure. The interviews were retrospective; they took place about three months after the intervention had ended. The semi-structured interview administration guide included open-ended questions for the interviewees about their typical working conditions (e.g., their type of work, psychological and social conditions in the workplace, and management), how the EP was received (i.e. either positively or negatively), and potential future changes (in terms of the interviewees’ individual working conditions and their workplace). The mean interview time was approximately 40 minutes, with a range from 32 to 54 minutes. The interviews were tape recorded, transcribed and analysed.

Analytical approach. Thematic analysis was adopted to analyse the interview material. This is a method «for identifying, analysing, and reporting patterns (themes) within data» (Braun & Clarke, 2006). In contrast to other types of qualitative analysis (such as Interpretative Phenomenological Analysis, Grounded theory or discourse analysis) thematic analysis is not bound to a theoretical or epistemological framework (Braun & Clarke, 2006). Thematic analysis is performed through several steps, and can be summarized as data that is taken through the process of coding to establish meaningful themes. The actual analysis is not a linear process in the sense that the analyst goes back and forth between the data and the codes, as well as between the themes and the codes.

Our analytical approach was driven by our theoretical interest in our research question, and can therefore be classified as a deductive thematic analysis or a «top down» approach (Braun & Clarke, 2006). The goal was to get a nuanced account of elements in the implementation process, and not to give a rich thematic description of the entire dataset. Both the coding and the theme development were therefore driven by the research question presented above. The coding was carried out line-by-line by using a simple three-column form where the first column was blank and allowed room to explore themes later in the analysis, the second column was blank and open for writing codes, and the third column consisted of the transcribed interview. Quotes or text blocks that the researcher found relevant according to the research question were marked and given a code to capture the essence of the chosen segment in the data. The initial codes were as close to the original statement of the informants as possible. A theme was defined according to Braun and Clarke (2006) as «something important that relates to the research interest, and represents some level of patterned response or meaning within the data set» (p. 82). In this regard, constructs that embraced a number of initial codes were identified as a theme. During the process of theme development, themes were continuously revised; some themes would be subdivided, and others would be combined with the purpose of fitting the data. Therefore, this step of the analysis involved more interpretation. A digital mind map was applied to assist the process of theme development.

Results

Survey results.
A total of 171 employees completed the questionnaire (42% of those contacted). However, 44 of these employees completed less than half of their questionnaire, and so were excluded. Thus, the final sample comprised 127 employees.

The PCA produced a significant result in the Bartlett’s test of sphericity (Chi² = 1275.431, df = 91, p < .001) in the correlation matrix, indicating that the analyzed matrix was not an identity-matrix. The Kaiser-Meyer-Olkin measure of sampling adequacy was KMO = .910, which, according to Field (2009), is an acceptable value. The Kaiser criterion suggested a four-component solution explaining 53.0% of variance in the items. However, one of the components had only one item, and, the Cronbach’s alpha was .64 for the items in the third component, which is lower than recommended by Field (2009), who has a limit of > .70, thus leaving us with two final factors: factor 1, which was labelled Communication, and factor 2, which was labelled Leadership (see Table 1).

The regression analysis showed in step 1 that gender and engagement were significant predictors, but in step 2 only the two process variables contributed significantly to the success of the intervention (See Table 2, 3, and 4). The model explained 59% of the variance in satisfaction with the intervention (EP).

Interview results.
Three themes were identified in thematic analysis. These were: (1) Management Communication (information provided prior to the intervention and reported relevance to daily work), (2) Content, and (3) Structural Factors. To provide a fuller presentation of the data material, each theme was divided into subcategories. In the following sections, the themes and their subcategories are presented through excerpts from the interviews, to demonstrate the presence of each theme within each subcategory.

Management Communication. There were two categories identified under the theme of Management Communication. The first category, «Articulated information prior to the intervention», captures the fact that many of the interviewees reportedly found the information given to them by their managers about the EP prior to the measure vague or insufficient. The following quote illustrates this category:

According to the interviewees, in some cases vague information led to assumption-making about the upcoming intervention based on previous failed interventions, and thus encouraged them to have negative attitudes towards the upcoming intervention.

The second category under the theme Management Communication is called «Articulated relevance to daily work». This category also involves management communication, but differs from the above category in that it mainly relates to the actual intervention workshops. The data material implies that a clearer explanation of how each exercise or part of the workshop were related to the participants’ daily work responsibilities would have been beneficial. The following statement illustrates this category:

The data material further reflects that for those who did not understand the relevance of the intervention to their daily work, the workshops or the exercises could be experienced as meaningless or as a waste of resources and thereby create negative attitudes towards the employeeship program.

Content. Another major theme identified in the data qualitative analysis, is the Content of the intervention. Based on analysis, this theme was divided into two categories. The first category, «Diversity Icebreaker test», reflects the fact that many of the interviewees emphasized the importance of the Diversity Icebreaker questionnaire and the process around the test. New insights in relation to this category were often associated with changes in the psychosocial work environment. The following statement was uttered in the context of the Diversity Icebreaker and illustrates the presence of this category:

The second category under the same theme is called «Practical exercises», and reflects that several of the interviewees perceived the practical exercises in collaboration and communication of the second workshop day, as rewarding. The following quote illustrates the presence of this category:

Both the psychological test and the practical exercises can, based on the data, also be understood as catalysts for conversation and discussion about the work environment amongst employees.

Structural Factors. As a theme, Structural Factors reflect that the structure of the intervention was referred to as important several times by participants. The first subcategory under this theme has been named «Time interval between workshops» and simply reflects that several of the participants thought they would have gotten more out of the intervention if there had been a shorter time span between each workshop. Interviewees indicated that important lessons from previous workshops were perhaps forgotten, so they felt they had to start over again in this workshop. The following statement illustrates this category:

The second subcategory under the theme Structural Factors is called «Gathering under different circumstances». This category was placed under this theme to reflect that the framework for the intervention facilitated that the employees got to know each other better. The data indicates that the reasons for this could be: that they met at a different place than usual, they were placed in an unfamiliar social setting, and they talked with people they did not usually talk to. All this was spoken of positively—without exception— in interviews, and may be seen as an important factor for any changes in the psychosocial work environment associated with the intervention. The following statement illustrates this category:

Discussion

Our survey showed that the two main factors involved in this intervention process were communication and leadership. These factors were also the most important ones in understanding the success of the outcome of the intervention, even when the regression model contained scales that measured personality and engagement. According to our survey, participants generally responded positively to the intervention, and to some extent this was reinforced in the interview responses. The employees felt it was important to come together outside of their daily work to learn how to communicate better with their colleagues, and to get to know each other better. The interviews identified some areas for improvement in the intervention, such as tighter meeting plans and breaking down negative attitudes that employees held of employeeship programs due to past experiences (as these negative attitudes took away from their engagement in the current program).

The survey findings and interview findings complemented each other but did not agree completely. The interview findings highlighted important aspects of the insufficiency of the factors «leadership» and «communication». Most importantly, the interviews’ theme of «management communication» suggests that both leadership and communication are essential factors in the intervention implementation process. Leadership through communication creates attitudes that are crucial for the outcome of the intervention.

According to our test of the 20 items in the questionnaire (developed to evaluate the intervention process), many were relevant. Fifteen items came out in the two factors; the remaining five items did not collectively make a distinct factor, but the content of the questions suggested that factors like «resistance to change» and «involvement» can be developed as more robust factors. The interviews also revealed that the intervention process should also address negative participant attitudes about the program before the start of it, as well as the structure or timing of the workshop activities. One can therefore assume that clearly articulated information prior to the intervention is of importance for the outcome of the intervention. It is also reasonable to assume that this is related to the participants’ «willingness to change» or resistance to change. This means that the intervention process is about both content (what) and dynamics (how). This will be further elaborated below.

This study shows the significance of leadership in both the process and employees’ perception of the intervention program. Since several studies (Nielsen & Randall, 2012; Randall et al., 2009; Tvedt et al., 2009) have found employee perception to be highly important in the success of any intervention program, other influencing factors (such as leadership) should be taken seriously in any evaluation. While it might be tempting to quickly assume that the program will yield positive results just because the leadership initiated it, this is not always the case. In other words, leadership initiating an intervention program does not automatically translate into positive results. There are a host of other factors that have been found to be important (Dahl-Jørgensen & Saksvik, 2005). In their study of the impact of two organizational interventions on the health of service-sector workers, Dahl-Jørgensen and Saksvik (2005) identified the following as influential to the results of their study: employees’ readiness, managers’ restricted time used on an intervention, employees’ identification with and involvement in the program, and high turnover. According to Dahl-Jørgensen and Saksvik (2005), one of the problems often found with intervention programs is the hiring of an external expert who oftentimes fails to involve the stakeholders in the organization. In the current study, an external consultant (Kibu) designed a program that involved the leadership at an early stage. The leadership then got their employees involved. Our study results showed that the intervention was met with employee interest and enthusiasm.

Moreover, previous studies carried out on leadership behaviour have shown that the role of leaders, especially immediate supervisors/line managers, is very important for their subordinates. Psychosocial factors (e.g. staff turnover, job engagement, burnout, stress, and job satisfaction) have all been found to be directly or indirectly related to leadership behaviours. Thus it is not surprising that the role of the immediate leader/supervisor was found to be crucial to the intervention process in our study.

When leaders of an organization initiate an intervention, several questions may quickly arise. Will the employees be able to see things along the same lines as their leaders? Are the employees receptive to such programs? Results from the current study show that the leaders, although being the original initiators of the intervention program, did a good job in guiding the employees through the intervention process. The role of leaders in transformational leadership is relevant in this respect (Bass, 1978). Biron et al. (2014) emphasize the importance of management support in implementing interventions, in terms of financial support, freeing up employees for the time needed to plan, develop and implement the intervention, and getting the workers’ engagement, respect and involvement in the process of the change (Biron et al., 2014). However, few studies have focused on how managers can be supported in the intervention. The role of line managers can be important for the prevention of stress for both leaders and their employees.

The other process aspects of communication and provision of information in the organization during the intervention program were found to have an impact on the employees’ appraisal of the program. The communication component included 11 items from two different sub-scales of the process evaluation scales. This suggests that communication is especially important during the planning and implementation of an intervention program. One reason for this is that communication makes it possible for all stakeholders to be «on the same page» regarding activities and tasks concerning the intervention at hand. If everyone is aware of what to do, at what time, and with what intensity, implementation becomes easier and foreseeable. Additionally, employees and all those concerned are able to master the necessary efforts to make the intervention successful. Employees might feel less motivated if they are not properly informed about the need for the intervention or its purpose and goal.

Other studies have found that employees’ readiness and feelings of ownership of the program are essential. «Readiness» addresses issues concerning the level of preparedness of those that will be exposed to the content of an intervention. Nielsen and Randall (2012) and Nielsen and colleagues (2010) found that when employees are not prepared or ready for an intervention program, exposure to such programs may have unintended adverse effects on productivity. Furthermore, some employees might engage in sabotaging activities simply because they are not ready for the particular intervention program. Similarly, it is critical that employees also feel ownership of a program. In situations where the program is being initiated by leadership with the help from a consultant (as was the case in the present study), creating a tangible sense of employee ownership of the program can encourage employees to have positive attitudes toward participation. In their study of the evaluation of process and contextual issues in an organizational-level work stress intervention, Biron, Gatrell, and Cooper (2010) found that it is highly important that stakeholders feel ownership of a program in order for them to be motivated and committed to it. They proposed the employment of different effective strategic tools to bring about the feeling of ownership and commitment to the program. This is where the use of sufficient information and open communication becomes relevant. When employees and stakeholders are provided with information regarding the particular intervention program, and are also able to contribute to the planning and improvement of such programs (as was, to some extent, the case in this study), it is plausible to argue that these employees will be more prepared and have a stronger sense of ownership of the program. Although the provision of information might be vital to achieve a positive intervention outcome, a host of other factors (e.g. participants’ mental states and the context of the intervention) also play an important role in a desirable, successful intervention.

An organization is a complex system. When evaluating interventions within such a system, it is important for evaluators to be aware of the various problems they may encounter. These problems include both practical and methodological challenges, and the significant amount of time and resources required in an intervention (Craig et al., 2008). Even though we found that the intervention was positive, it may be difficult to apply our results to a different context. Weak findings or results in intervention trials may be attributed to poorly designed or theorized interventions, or inadequate methods for evaluation. The pre-existing context will also influence the effects observed (Hawe, Shiell, Riley, & Gold, 2004). Nielsen and Abildgaard (2013) proposed a framework for the evaluation of organizational interventions. It includes the context of the intervention, as well as organizational actors, the mental models of those actors, and intervention design and process. These four categories are said to be crucial to intervention evaluation, and are based on recent research on intervention and process-oriented organization theory (Nielsen & Abilgaard, 2013).

Methodological considerations.
During data collection, a voluntary questionnaire was sent to employees in the estate management section at NTNU. Since employees were expected to answer the questionnaire while at work, factors like a hectic and busy working day as well as sicknesses and absenteeism might have played a role in the relatively low number of employee responses. Social desirability bias may have also influenced their responses (Donaldson & Grant-Vallone, 2002). Meltzoff (2007) maintains that insufficient self-report surveys tend to be less reliable due to participant distortions of self-perception and the presence of self-serving biases. Self-reporting may be more useful for measuring subjective feelings about experiences such as joy, sadness, or physical pain. If participants feel that the employer or leadership would have access to both results from the study as well as the respondents’ individual answers, this may also influence their responses.

In this study an explanatory factor analysis (EFA) was chosen to address the main aim of assessing whether a factor structure could be found from the 20 process questionnaire items. A confirmatory factor analysis (CFA) was excluded because the aim of the study was not to build a model or theory of process evaluation, but to just extract factors that had sufficient Cronbach’s alpha to be included in the regression analyses.

A strength of this study is the use of complementary survey and interview methods, which improved our understanding of how we should best evaluate the process part of interventions. Biron and Karanika-Murray (2014) have required process models to find out more about what makes an intervention work. The interview responses included negative perceptions of the outcome of the intervention, but since participants were asked to give their honest, critical opinions, this may have contributed to what we found. Although the study was conducted in one organization only, the different departments that participated may have made the interviews more context-specific. Since context is important to understanding the contribution of the intervention process, interviews may have an advantage over surveys that most often are based on standard validated scales. Regarding the retrospective format of the interviews, it should be taken into consideration that recall bias represents a possible threat to the validity of the findings.

Conclusion

The present study found two aspects of the intervention processes which provided relevant and valuable information for the success of the intervention: Leadership and communication. In the past, researchers have listed several factors (e.g. leadership action and attitude, employees’ involvement and readiness) as influential to the effectiveness of an intervention program. In this study, we found that good implementation of the intervention involving leadership role and communication also predicted levels of participants’ satisfaction with this kind of salutogenic intervention program. Since participants’ satisfaction with the content and implementation of an intervention program previously has been linked to its success, there is a need for researchers as well as employers to pay more attention to it.

References

Armenakis, A. A., & Bedeian, A. G. (1999). Organizational change: A review of theory and research in the 1990s. Journal of Management, 25, 293–315. doi: 10.1177/014920639902500303.

Arbejdsmiljøforskning (2012). Piop-prosjektet. http://www.arbejdsmiljoforskning.dk/da/projekter/participatorisk–intervention-fra-et-organisatorisk-perspektiv—piop.

Arnetz, B. B. (1996). Techno-stress: A prospective psychophysiological study of the impact of a controlled stress-reduction program in advanced telecommunication systems design work. Journal of Occupational and Environmental Medicine, 38(1), 53–65. doi: 10.1097/00043764-199601000-00017.

Bakker, A. B., & Derks, D. (2010). Positive Occupational Health Psychology. In S. Leka, & J. Loudmoth (Eds.), Occupational Health Psychology (pp. 194–224). Oxford, England: Blackwell Publishing.

Bass, B. M. (1998). Transformational leadership – Industrial leadership: Industry, military, and educational impact. Mahwah, NJ: Lawrence Erlbaum.

Biron, C., Karanika-Murray, M., Daniels, K., Hasson, H., Nielsen, K., Randall, R., Saksvik, P. Ø., & von Thiele Schwarz, U. (Submitted). Hunting the trolls of psychosocial risks: A research agenda for intervention process evaluation.

Biron, C., Burke, B., & Cooper, C. L. (Eds.) (2014). Creating healthy workplaces: Stress reduction, improved well-being, and organizational effectiveness. Farnham, UK: Gower Publishing.

Biron, C., & Karanika-Murray, M. (2014). Process evaluation for organizational stress and well-being interventions: Implications for theory, method, and practice. International Journal of Stress Management, 21(1), 85–111. doi: 10.1037/a0033227.

Biron, C., Gatrell, C., & Cooper, C. L. (2010). Autopsy of a failure: Evaluating process and contextual issues in an organizational-level work stress intervention. International Journal of Stress Management, 17(2), 135–158. doi: 10.1037/a0018772.

Bauer, G., & Jenny, G. (2013). Concepts of salutogenic organizations and change: the logics behind organizational health intervention research. New York: Springer Education.

Braun, V. & Clarke, V. (2006). Using thematic analysis in psychology. Qualitative Research in Psychology, 3(2), 77–101. doi: 10.1191/1478088706qp063oa.

Burke, W. W. (2002). Organizational change. Theory and practice. Thousand Oaks, CA: Sage.

Craig, P., Dieppe, P., Mcintyre, S., Mitchie, S., et al. (2008). Developing and evaluating complex interventions: the new medical research council guidance. BMJ, 337, 979–983. doi: 10.1136/bmj.a1655.

Dahl-Jørgensen, C., & Saksvik, P. Ø. (2005). The impact of two organizational interventions on the health of service sector workers. International Journal of Health Service, 35(3), 529–549. doi: 10.2190/P67F-3U5Y-3DDW-MGT1.

Diversity Icebreaker Homepage (2013, November 28). Retrieved from http://www.diversityicebreaker.com/home.aspx.

Donaldson, S. I., & Grant-Vallone, E. J. (2002). Understanding self-report bias in organizational behavior research. Journal of Business and Psychology, 17, 245–260. doi: 10.1023/A:1019637632584.

Eklöf, M., Ingelgård, A., & Hagberg, M. (2004). Is participative ergonomics associated with better working environment and health? A study among Swedish white-collar VDU users. International Journal of Industrial Ergonomics, 34(5), 355–366. doi: 10.1016/j.ergon.2004.04.013.

Elo, A. L., Ervasti, J., Kuosma, E., & Mattila, P. (2008). Evaluation of an organizational stress management program in a municipal public works organization. Journal of Occupational Health Psychology, 13(1), 10–23. doi: 10.1037/1076-8998.13.1.10.

Eysenck, S. B. G., & Tambs, K. (1990). Cross-cultural comparison of personality: Norway and England. Scandinavian Journal of Psychology, 31(3), 191–197. doi: 10.1111/j.1467-9450.1990.tb00830.x.

Field, A. (2009). Discovering statistics using SPSS (3rd Ed.). London: SAGE Publications.

Hancock, K., & Craig, A., (1996). The influence of a healthy lifestyle program in a work environment: A controlled long-term study. Journal of Occupational Health and safety Australia and New Zealand, 12, 193–206.

Hawe, P., Shiell, A., Riley, T., & Gold, L. (2004). Methods for exploring implementation variation and local context within a cluster randomized community intervention trial. Journal of Epidemiology & Community Health, 58, 788–793. doi: 10.1136/jech.2003.014415.

Jimmieson, N. L., Terry, D. J., & Callan, V. J. (2004). A longitudinal study of employee adaptation to organizational change: the role of change-related information and change-related self-efficacy. Journal of Occupational Health Psychology, 9(1), 11–27. doi: 10.1037/1076-8998.9.1.11.

Kelloway, E. K., Hurrell Jr, J. J., & Day, A. (2008). Workplace interventions for occupational stress. In K. Näswall, J. Hellgren, & M. Sverke (Eds.), The individual in the changing working life (pp. 419–441). Cambridge: Cambridge University Press.

Lines, R. (2004). Influence of participation in strategic change: resistance, organizational commitment and change goal achievement. Journal of Change Management, 4(3), 193–215. doi: 10.1080/1469701042000221696.

Meltzoff, J. (1998). Critical thinking about research: psychology and related fields. Washington, D.C.: American Psychological Association.

Milch, V., Vaag, J., Giæver, F., & Saksvik, P. Ø. (2013). Building healthy organizations through music and culture interventions. In I. G. Bauer & G. Jenny (Eds.), Concepts of salutogenic organizations and change: the logics behind organizational health intervention research. New York: Springer Education.

Mowday, R. T., Steers, R. M., & Porter, L. W. (1979). The measurement of organizational commitment. Journal of Vocational Behavior, 14(2), 224–247. doi: 10.1016/0001-8791(79)90072-1.

Murta, S. G., Sanderson, K., & Oldenburg, B. (2007). Process evaluation in occupational stress management programs: a systematic review. American Journal of Health Promotion, 21(4), 248–254. doi: 10.4278/0890-1171-21.4.248.

Nielsen, K., & Abildgaard, S. J. (2013). Organizational interventions: A research based framework for the evaluation of both process and effects. Work & Stress: An International Journal of Work, Health & Organisations, 27, 278–297, doi: 10.1080/02678373.2013.812358.

Nielsen, K., Randall, R., & Albertsen, K. (2007). Participants’ appraisals of process issues and the effects of stress management interventions. Journal of Organizational Behavior, 28(6), 793–810. doi: 10.1002/job.450.

Nielsen, K., Randall, R., Holten, A., & E.R. González. (2010). Conducting organizational-level occupational health interventions: What works? Work and Sress, 24(3), 234–259. doi: 10.1080/02678373.2010.515393.

Nielsen, K., & Randall, R. (2012). Opening the black box: Presenting a model for evaluating organizational-level interventions. European Journal of Work and Organizational Psychology, 22 (5), 601–617 doi: 10.1080/1359432X.2012.690556.

Pallant, J. (2013). SPSS survival manual: a step by step guide to data analysis using IBM SPSS. Maidenhead: McGraw-Hill.

Randall, R., Nielsen, K.M., & Tvedt, S. D. (2009). The development of five scales to measure employees’ appraisals of organizational-level stress management interventions. Work & Stress, 23, 1–23. doi: 10.1080/02678370902815277.

Randall, R., Griffiths, A., & Cox, T. (2005). Evaluating organizational stress-management interventions using adapted study designs. European Journal of Work and Organizational Psychology, 14(1), 23–41. doi: 10.1080/13594320444000209.

Saksvik, P. Ø., Nytrø, K., Dahl-Jørgensen, C., & Mikkelsen, A. (2002). A process evaluation of individual and organizational occupational stress and health interventions. Work & Stress, 16(1), 37-57. doi: 10.1080/02678370110118744.

Schaufeli, W., Salanova, M., González-Romá, V., & Bakker, A. (2002). The Measurement of Engagement and Burnout: A Two Sample Confirmatory Factor Analytic Approach. Journal of Happiness Studies, 3(1), 71–92. doi: 10.1023/A:1015630930326.

Snyder, C. R., & Lopez, S. J. (2005). Handbook of positive psychology. New York: Oxford University Press.

Tabachnick, B. G., & Fidell, L. S. (2013). Using multivariate statistics. Boston: Pearson.

Tvedt, S. D., & Saksvik, P. Ø. (2012). Finding the content of the process part of interventions. In C. Biron, M. Karanika-Murray, & C. L. Cooper (Eds.), Managing psychosocial risks in the workplace: The role of process issues. Routledge/Psychology Press.

Tvedt, S. D., Saksvik, P. Ø., & Nytrø, K. (2009). Does change process healthiness reduce the negative effects of organizational change on the psychosocial work environment? Work & Stress, 23, 80–98. doi: 10.1080/02678370902857113.

Vaag J., Saksvik, P. Ø., Theorell, T., Skillingstad, T., & Bjerkeset, O. (2012). Sound of well-being – choir singing as an intervention to improve well-being among employees in two Norwegian county hospitals. Arts & Health: An International Journal for Research, Policy and Practice, iFirst article, 1–10. doi: 10.1080/17533015.2012.727838.

Wilson, M. G., Basta, T. B., Bynum, B. H., DeJoy, D. M., Vandenberg, R. J., & Dishman, R. K. (2010). Do intervention fidelity and dose influence outcomes? Results from the move to improve worksite physical activity program. Health Education Research, 25(2), 294–305. doi: 10.1093/her/cyn065.

1. Diversity Icebreaker has been developed by Human Factors AS and is a questionnaire used in seminars where the aim is to improve communication and interaction in the group or between different departments or subsidiaries in a bigger company or organization. Based on the results from the questionnaire, the participants are divided into three categories (Red, Blue, and Green). Each color represents a certain set of preferences. It is suggested that the facilitator follows a certain seminar structure to bring out the effect of helping the participants develop a shared understanding about effective ways to communicate and work together with people with different preferences. [↩]