• Nyheter
    • Pluss-innhold
    • Arbeidsliv
    • Psykologi-folk
    • Nye bøker
    • Podkaster og videoer
      • Pia og psyken
      • Psykologlunsj
      • Psykologisk salong
      • Videoer
  • Ideer
    • Ytringer
    • Bokutdrag
    • Spalter
      • Forebygg depresjon med Arne Holte
      • Fra terapirommet med Kirsti Jareg
      • Hverdagspsykologi med Eirik Hørthe
      • Kjærleik & liv med Anne Marie Fosse Teigen
      • Kritisk tenkning med Torstein Låg
      • Menneskets natur med Leif Edward Ottesen Kennair
      • Gutta fra Psykologlunsj
      • Månedens klassiker
  • Aktiviteter
  • Stillinger
  • Bli abonnent
  • Kontakt oss
    • Vil du annonsere?
    • Send innlegg
    • Ansatte
  • Nyheter
    • Pluss-innhold
    • Arbeidsliv
    • Psykologi-folk
    • Nye bøker
    • Podkaster og videoer
      • Pia og psyken
      • Psykologlunsj
      • Psykologisk salong
      • Videoer
  • Ideer
    • Ytringer
    • Bokutdrag
    • Spalter
      • Forebygg depresjon med Arne Holte
      • Fra terapirommet med Kirsti Jareg
      • Hverdagspsykologi med Eirik Hørthe
      • Kjærleik & liv med Anne Marie Fosse Teigen
      • Kritisk tenkning med Torstein Låg
      • Menneskets natur med Leif Edward Ottesen Kennair
      • Gutta fra Psykologlunsj
      • Månedens klassiker
  • Aktiviteter
  • Stillinger
  • Bli abonnent
  • Kontakt oss
    • Vil du annonsere?
    • Send innlegg
    • Ansatte

Scandinavian Psychologist.

  • Volumes
    • 6 – 2019
    • 5 – 2018
    • 4 – 2017
    • 3 – 2016
    • 2 – 2015
    • 1 – 2014
  • Aims and scope
  • Author guidelines
  • Advisory board
  • Volumes
    • 6 – 2019
    • 5 – 2018
    • 4 – 2017
    • 3 – 2016
    • 2 – 2015
    • 1 – 2014
  • Aims and scope
  • Author guidelines
  • Advisory board
Op-Ed

A new approach to psychological measures in leadership research

The semantic theory of survey responses casts doubt on the prevalent use of surveys in leadership research, write Jan Ketil Arnulf and colleagues.

SIMILAR MEANINGS: The authors of this opinion piece suspect that survey data do not reflect attitudes to workplace phenomena. Instead, they may just be assessments of the similarity of the language in the applied items. Photo: Flazingo Photos / Flickr.

Jan Ketil Arnulf, Kai Rune Larsen, Øyvind Lund Martinsen & Chih How Bong

Last updated: 16.11.19  Published: 07.03.15

Citation

Arnulf, J. K., Larsen, K. R., Martinsen, Ø. L., & Bong, C. H. (2015). A new approach to psychological measures in leadership research. Scandinavian Psychologist, 2, e4. https://doi.org/10.15714/scandpsychol.2.e4

Abstract

A new approach to psychological measures in leadership research

Is survey data a source of new information, or could surveys just be begging their questions? The authors of this opinion piece suspect that survey data in leadership research do not reflect attitudes to workplace phenomena. Instead, they may just be assessments of the similarity of the language in the applied items. In a recent article in the journal PLOS ONE, this possibility was tested in a new theory called the semantic theory of survey responses (STSR). In a follow-up study, language links leadership to other phenomena such as heroism, in ways that are difficult to entangle by traditional survey research.

Keywords: charisma, heroism, latent semantic analysis, leadership, semantics, survey research.

Author(s)

Author affiliations: Jan Ketil Arnulf – BI Norwegian Business School, Oslo, Norway; Kai Rune Larsen – Leeds Business School, University of Colorado at Boulder, USA; Øyvind Lund Martinsen – BI Norwegian Business School, Oslo, Norway; Chih How Bong – Faculty of Computer Science & Information Technology, University of Malaysia at Sarawak, Malaysia. Contact information: Jan Ketil Arnulf, BI Norwegian Business School, Nydalen, N‑0442 Oslo, Norway. Email: jan.k.arnulf@bi.no. Language: English. This article is an opinion piece.

News article

(In Norwegian:) The first author Jan Ketil Arnulf is interviewed about this opinion piece in Psykologisk.no (in Norwegian). See also the research article on the overlapping semantics of leadership and heroism.

Forfatterinfo

Jan Ketil Arnulf

Jan Ketil Arnulf er professor ved Handelshøyskolen BI. Han er spesialist i klinisk psykologi og arbeids- og organisasjons­psykologi. De siste årene har han konsentrert seg om hvordan ledelse utøves og skapes. Han er forfatter av boken Hva er ledelse.

Kai Rune Larsen

Kai Rune Larsen er førsteamanuensis i ledelse og entreprenørskap ved Leeds Business School, University of Colorado i Boulder.

Øyvind Lund Martinsen

Øyvind Lund Martinsen er professor ved Institutt for ledelse og organisasjon ved Handelshøyskolen BI i Oslo.

Chih How Bong

Chih How Bong er forsker i informatikk ved University of Malaysia i Sarawak, Malaysia.

Is survey data a source of new information, or could surveys just be begging their questions? Organizational behavior is a field in psychology that relies heavily on surveys as research method. Theories about motivation, leadership and general workplace behaviors are frequently tested by asking managers and employees to fill out questionnaires. The data are then analyzed in complex statistical models, and usually interpreted as expressions of the underlying constructs. We have been suspecting that such data do not reflect attitudes to workplace phenomena. Instead, they may just be assessments of the similarity of the language in the applied items.

In a recent article in the journal PLOS ONE, we tested this possibility in a new theory called the semantic theory of survey responses, or STSR (Arnulf, Larsen, Martinsen, & Bong, 2014). We used text algorithms to assess the overlap of meaning among the items of scales used to measure common leadership surveys such as the Multifactor Leadership Questionnaire (MLQ), Leader–Member Exchange (LMX), and the Ohio State University Leadership theory inventory, the LBDQ. We also included a five-factor personality inventory, the NEO-FFI.

Related variables such as economic exchange, intrinsic motivation, organizational citizenship behavior (OCB), work effort, turnover intention, and job satisfaction were also included. The semantic values were compared to the observed statistics from human respondents in four large samples.

Determined by semantics

The main finding was that the observed statistics were predictable a priori from the semantic properties of the items. By the most conservative estimate, semantics explained 79% of the variance in the inter-item correlation matrix of the MLQ. Depending on assumptions, semantics could explain as much as 86% of the variance. Similarly, in a large sample surveyed for transformational leadership, LMX, the Ohio State 2-factor theory, as well as motivational variables, semantics explained between 47 and 87% of the variation.

Moreover, it seems that mediational relationships may also be determined by semantics. In a commonly applied regression model, intrinsic motivation was found to mediate between transformational leadership and outcome variables such as work effort and OCB. These relationships were also mutually explainable through semantics. Only in the case of the NEO Personality Inventory did semantic values fail to explain a significant part of the variation.

Our findings cast doubt on the prevalent use of surveys as research methods in this field because semantics (similarity in item content) is only supposed to play a role in the within-scale variation. Relationships between different variables should ideally be caused by patterns in the attitude strength of respondents. To ensure this, methodologists recommend various types of factor analysis to demonstrate that the scales are relatively independent and not inherently correlated (Nunnally & Bernstein, 2010).

As expected, we found that intra-scale relationships (measured by Cronbach’s alpha) were semantically determined. However, we could also show that the same semantic relationships pervaded into all meaningful relationships among the leadership and organizational behavior variables. Confirmatory factor analysis could neither detect nor prevent this, as better fit indices were associated with more obvious semantic determination of data patterns.

Four consequences

These findings may have some alarming consequences for prevalent methodologies in leadership research. First, the semantic patterns are properties of items involved a priori in obtaining responses. If the item correlation matrices are largely caused by the language recognition of respondents, the emerging statistical patterns really don’t tell us anything we didn’t know already (Smedslund, 1994).

Second, as theoretically argued (Van Knippenberg & Sitkin, 2013), it seems that many variables in leadership research have tautological relationships. As we could show in all three samples, the values of outcome variables were already given by their semantic relationships to the independent variables, suggesting some theoretical confounding in the basic concepts used. As two of the authors show in another empirical study, language links leadership to other phenomena also, such as heroism, in ways that are difficult to entangle by survey research (Arnulf & Larsen, 2015).

Third, the prevalent statistical modeling techniques used to demonstrate variable independence seem inadequate to this purpose, a possible explanation for the frequently observed levels of common method variance in leadership research (Podsakoff, MacKenzie, & Podsakoff, 2012).

A fourth and illustrating consequence concerns the cross-cultural validity of survey data. The crucial point in semantic relations is that they only reflect what is proposed by the linguistic structure of the sentence. If similarities in survey correlation matrices are observed across languages, it merely means that the survey was correctly translated. In our case, the items were fed into the text analytical algorithms in American English, but the respondents were all Norwegians responding to Norwegian versions of the surveys. This issue casts doubt about inferences from cross-cultural survey data as these merely show how the same statements are expressible across languages. Such data provide no information about how actual behavioral interactions are comparable, such as leadership or employee behavior across cultural divides. This possibility has also been proposed by psychological theories exploring the discrepancy between attitudes, language, thought, and action (Gollwitzer & Sheeran, 2006; Parks-Stamm, Oettingen, & Gollwitzer, 2010; Prinz, Aschersleben, & Koch, 2009).

A new approach to psychological measurements

Finally, we believe that our study offers a new approach to understand why some psychological measurements carry more predictive value for future behaviors or effects than others. Mere compliance with semantic structures in surveys could be a reason why some instruments show appropriate psychometric properties but little predictive validity (Bing, LeBreton, Davison, Migetz, & James, 2007). In our study, the five-factor model did not seem strongly influenced by semantics, but is known to be predictive of a number of life events ranging from health to career success (McCrae & Costa, 2004).

The observed statistics were predictable from the semantic properties of the items.

We understand that our findings and claims may be counterintuitive and problematic to some, and welcome debate about these topics. Concomitantly, our findings are a product of technologies in digital text analysis that have emerged in recent decades and are continually being refined. With the advent of such technologies in information search and knowledge storing, our study is probably just the beginning of new methodological approaches that will complement the standard statistical methods in use. We believe this is a promising path for the revitalization of the empirical foundations of leadership research.
References

Arnulf, J. K., & Larsen, K. R. (2015). Overlapping semantics of leadership and heroism: Expectations of omnipotence, identification with ideal leaders and disappointment in real managers. Scandinavian Psychologist, 2, e3. doi: 10.15714/scandpsychol.2.e3.

Arnulf, J. K., Larsen, K. R., Martinsen, Ø. L., & Bong, C. H. (2014). Predicting survey responses: How and why semantics shape survey statistics in organizational behavior. PLOS ONE, 9(9), 1–13. doi: 10.1371/journal.pone.0106361.

Bing, M. N., LeBreton, J. M., Davison, H. K., Migetz, D. Z., & James, L. R. (2007). Integrating implicit and explicit social cognitions for enhanced personality assessment: A general framework for choosing measurement and statistical methods. Organizational Research Methods, 10(1), 136–179. doi: 10.1177/1094428107301148.

Gollwitzer, P. M., & Sheeran, P. (2006). Implementation intentions and goal achievement: A meta-analysis of effects and processes. Advances in Experimental Social Psychology, 38, 69–119. doi: 10.1016/S0065-2601(06)38002-1.

McCrae, R. R., & Costa, P. T. (2004). A contemplated revision of the NEO Five-Factor Inventory. Personality and Individual Differences, 36(3), 587–596. doi: 10.1016/S0191-8869(03)00118-1.

Nunnally, J. C., & Bernstein, I. H. (2010). Psychometric theory (3rd Ed.). New York, NY: McGraw-Hill.

Parks-Stamm, E. J., Oettingen, G., & Gollwitzer, P. M. (2010). Making sense of one’s actions in an explanatory vacuum: The interpretation of nonconscious goal striving. Journal of Experimental Social Psychology, 46, 531–542. doi: 10.1016/j.jesp.2010.02.004.

Podsakoff, P. M., MacKenzie, S. B., & Podsakoff, N. P. (2012). Sources of method bias in social science research and recommendations on how to control it. In S. T. Fiske, D. L. Schacter & S. E. Taylor (Eds.), Annual Review of Psychology. Vol. 63 (pp. 539–569). Palo Alto: Annual Reviews.

Prinz, W., Aschersleben, G., & Koch, I. (2009). Cognition and action. In E. Morsella, J. A. Bargh & P. M. Gollwitzer (Eds.), Oxford Handbook of Human Action. Boston: Oxford University Press.

Smedslund, J. (1994). Nonempirical and empirical components in the hypotheses of 5 social-psychological experiments. Scandinavian Journal of Psychology, 35(1), 1–15. doi: 10.1111/j.1467-9450.1994.tb00928.x.

Van Knippenberg, D., & Sitkin, S. B. (2013). A critical assessment of charismatic-transformational leadership research: Back to the drawing board? The Academy of Management Annals, 7(1), 1–60. doi: 10.1080/19416520.2013.759433.

Redaksjonen anbefaler

Emosjonelt ustabil personlighets­forstyrrelse: Pårørende kan falle i en av to grøfter

  • Nyheter, Pluss

Engstelig tilknytning: Når partnerens usikkerhet styrer forholdet

  • Nyheter, Pluss

Maren ville ikke dø alene. Men telefonen hun ringte til, reddet i stedet livet hennes

  • Nyheter, Pluss

Hypomani: En langvarig lykke med mørke skyggesider

  • Nyheter, Pluss

Frykten for å stamme fikk han til å besvime på scenen

  • Nye bøker, Pluss

Vi har en tendens til å ignorere kroppen når vi snakker om psykologi

  • Hverdagspsykologi med Eirik Hørthe, Pluss

Det finnes veier ut av håpløsheten

  • Nyheter, Pluss

Mangler du glede, motivasjon og livslyst? Da lider du kanskje av anhedoni

  • Nyheter, Pluss

Økning i ADHD-diagnoser, mener FHI: Norsk spesialist reagerer på analysen

  • Nyheter, Pluss

Anne B. Ragde drar heller på hytta enn til psykolog

  • Nyheter, Pluss

Mishandling i barndommen gjør det vanskeligere å gjenkjenne egne følelser

  • Nyheter, Pluss

Gode mennesker har et personlighetstrekk til felles

  • Nyheter, Pluss

Hvorfor blir noen med ADHD først diagnostisert i voksen alder?

  • Nyheter, Pluss

Peder Kjøs gir livet terningkast fire

  • Nyheter, Pluss

God kommunikasjon redder ekteskap som lider av «phubbing»

  • Nyheter, Pluss

Highasakite-Ingrid: – Jeg har vært god på å lage noe fint ut av noe vondt

  • Nyheter, Pluss

Sykelig narsissisme: – Jeg tenker at det er en selvfølelse på speed

  • Nyheter, Pluss

ME-forsker mistenker at sykdommen skyldes immunsvikt

  • Nyheter, Pluss

Fastlegen mener vi bør ignorere flere helseråd og bli mer fornøyde med det vi allerede gjør

  • Nye bøker, Nyheter, Pluss

Ønsker mer fokus på det psykologiske aspektet i møte med en pasient

  • Nyheter, Pluss

Så du har fått diagnosen ADHD. Hva nå?

  • Nyheter, Pluss

Dette er de vanligste barndoms­traumene

  • Nyheter, Pluss

I årevis har han drevet psykedelisk terapi i det skjulte

  • Nyheter, Pluss

Omfattende studie avdekker hvordan traumer i barndommen endrer hjernens utvikling

  • Nyheter, Pluss

Et hjerte må bæres i et annet hjerte for å vokse seg sterkere

  • Hverdagspsykologi med Eirik Hørthe, Pluss

Sanna Sarromaa var fanget i et psykisk voldelig forhold: – Det kan skje den sterkeste

  • Nye bøker, Nyheter, Pluss

Bivirkninger av ADHD-medisin: – Jeg visnet bort og ble et skall av meg selv

  • Nyheter, Pluss

Derfor kan forsvarsmekanismer også fungere til din fordel

  • Nyheter, Pluss

– Mangel på selvrespekt er et sentralt element i depresjon

  • Nyheter, Pluss

Desorganisert tilknytning: Når forholdet blir kaotisk og forvirrende

  • Nyheter, Pluss

Siste saker

Depresjon gjennom livet øker faren for demens

  • Nyheter, Pluss

Mange menn lider i stillhet bak kule fasader. I verste fall tar en av dem livet sitt

  • Nyheter, Pluss

Lavkarbo kobles til depressive symptomer

  • Nyheter, Pluss

De to mest unyttige tingene å bekymre seg for, ifølge psykiater

  • Nyheter, Pluss

Derfor er det så vanskelig å behandle stress

  • Nyheter, Pluss

Psykologen som forlot Meta: – Ekspertgruppen var bare vinduspynt

  • Nyheter, Pluss

Vi ser – og det forplikter

  • Ytringer

Systematisk bruker­medvirkning forutsetter at ulike perspektiver høres

  • Ytringer

Den sosiale masken: Når du mister deg selv i rollen du spiller for andre

  • Nyheter, Pluss

Når det blir viktigere å få rett enn å forstå

  • Ytringer

Frp-politiker langer ut mot ekspert: – Burde ta en titt i speilet

  • Nyheter, Pluss

Mange føler seg ekstra alene i ferien – dette kan du gjøre

  • Nyheter, Pluss

Fortsatt ubesvart

  • Ytringer

Tech som terapi? Derfor stoler stadig flere på klokken og ringen for å unngå utbrenthet

  • Nyheter, Pluss

Nervesystemet: Refleksen som ble et liv

  • Ytringer

Er det vanskelig å få barna til å sove på sommerferie? Her er søvnekspertenes råd

  • Nyheter, Pluss

Når hjernen bremser: Hvorfor depresjon ikke bare handler om tristhet

  • Ytringer

Sammenbruddet hennes avslørte både ADHD, autisme og systemets blindsoner

  • Nyheter, Pluss

Det er særlig ett råd psykologen sjelden klarer å følge selv

  • Nyheter, Pluss

Skole, skjerm og stress – kan yoga være løsningen?

  • Nyheter, Pluss

«Psyk» og de brysomme blant oss

  • Ytringer

Ny møteplass for fagfeltet inviterer til dialog: – Ofte det som mangler i feltet

  • Nyheter, Pluss

Stortingspolitikere reagerer: – Varselet må tas på alvor

  • Nyheter, Pluss

Tverrfaglig blikk og subjektiv erfaring – det helsevitenskapen mangler

  • Ytringer

Mest lest

– Den vanligste personlighets­forstyrrelsen er lettest å overse

    Visse livsstiler øker faren for demens betraktelig

      – Psykisk vold dreper kjærlighet

        Gaslighting: – En ondskapsfull teknikk for å ta kontroll over et annet menneske

          Slik utnytter narsissisten din emosjonelle intelligens

            Sinte voksne barn

              Oppdaget mulig årsak til emosjonelt ustabil personlighets­forstyrrelse

                Nevroforsker om ADHD-diagnosen: – Det er ikke en enhetlig tilstand

                  Dette skjer med oss når vi opplever det mystiske fenomenet dissosiasjon

                    En bestemt oppførsel hos barn kan være tegn på senere angstlidelse

                      Hva skal til for å komme over et traume?

                        Dette er den skjulte formen for narsissisme

                          Slik er kjærlighetslivet med en narsissist

                            De tre søylene for god psykisk helse

                              Pia la om kostholdet og ble kvitt angsten

                                Tegnene på at du sliter med kronisk stress

                                  Med én enkel påstand kan du nå avsløre om noen lyver

                                    Tre faktorer kan svært presist forutsi psykiske lidelser

                                      Nye følelsesfunn i dypet av høysensitive hjerner

                                        Narsissisme – kan du holde ut?

                                          Hvorfor er det så vanskelig å gjøre det slutt?

                                            Noen personlighetstrekk beskytter mot demens – andre øker faren

                                              Emosjonelt intelligente foreldre blir oftere utbrent, antyder ny studie. Det kan skade foreldreevnen deres, mener forskerne

                                                Er du et A- eller B-menneske? Det henger sammen med personligheten din

                                                  Meld deg på nyhetsbrev fra Psykologisk.no

                                                  • Psykologisk.no AS​
                                                    C. J. Hambros plass 5
                                                    0164 Oslo
                                                    912 389 782 MVA
                                                  • Tips oss
                                                  • Kundeservice
                                                  • Skriv innlegg
                                                  • Bli annonsør
                                                  • Redaksjon
                                                  • Personvern
                                                  • Ansvarlig redaktør
                                                    Pål Johan Karlsen
                                                  • Nyhetsredaktør
                                                    Jonas Hartford Sundquist
                                                  • Administrasjons­sjef
                                                    Vera Thorvarsdottir
                                                  Facebook-f Twitter Linkedin

                                                  Psykologisk.no er medlem av Mediebedriftenes Landsforening og Fagpressen, og arbeider etter Vær Varsom-plakatens regler for god presseskikk.

                                                  Kopibeskyttet © 2025