Home Uncategorized SLR Paper - Analyzing & Visualizing Data

Uncategorized

SLR Paper – Analyzing & Visualizing Data

Prepare an SLR (systematic literature review) paper on the topic Analyze & Visualize Data. Topic/Idea should be new (shouldn’t copy someone else) and should follow SLR format, attached is the sample SLR papers.

DeadLine 8/9/2020

Data Analyticsdata visualization

SAMPLE_SLRs/10.1.1.441.9182

Information and Software Technology 51 (2009) 7–15
Contents lists available at ScienceDirect

Information and Software Technology

journal homepage: www.elsevier .com/ locate/ infsof
Systematic literature reviews in software engineering – A systematic
literature review

Barbara Kitchenham a,*, O. Pearl Brereton a, David Budgen b, Mark Turner a, John Bailey b, Stephen Linkman a

a Software Engineering Group, School of Computer Science and Mathematics, Keele University, Keele Village, Keele, Staffs, ST5 5BG, UK
b Department of Computer Science, Durham University, Durham, UK

a r t i c l e i n f o
Available online 12 November 2008

Keywords:
Systematic literature review
Evidence-based software engineering
Tertiary study
Systematic review quality
Cost estimation
0950-5849/$ – see front matter � 2008 Elsevier B.V. A
doi:10.1016/j.infsof.2008.09.009

* Corresponding author. Tel.: +44 1622 820484; fax
E-mail address: barbara@kitchenham.me.uk (B. Ki
a b s t r a c t

Background: In 2004 the concept of evidence-based software engineering (EBSE) was introduced at the
ICSE04 conference.
Aims: This study assesses the impact of systematic literature reviews (SLRs) which are the recommended
EBSE method for aggregating evidence.
Method: We used the standard systematic literature review method employing a manual search of 10
journals and 4 conference proceedings.
Results: Of 20 relevant studies, eight addressed research trends rather than technique evaluation. Seven
SLRs addressed cost estimation. The quality of SLRs was fair with only three scoring less than 2 out of 4.
Conclusions: Currently, the topic areas covered by SLRs are limited. European researchers, particularly
those at the Simula Laboratory appear to be the leading exponents of systematic literature reviews.
The series of cost estimation SLRs demonstrate the potential value of EBSE for synthesising evidence
and making it available to practitioners.

� 2008 Elsevier B.V. All rights reserved.
Contents
1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8
2. Method . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8
2.1. Research questions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8
2.2. Search process . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8
2.3. Inclusion and exclusion criteria . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9
2.4. Quality assessment . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9
2.5. Data collection. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9
2.6. Data analysis . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9
2.7. Deviations from protocol . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9
3. Results. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10

3.1. Search results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10
3.2. Quality evaluation of SLRs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10
3.3. Quality factors . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10
4. Discussion. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11

4.1. How much EBSE Activity has there been since 2004? . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11
4.2. What research topics are being addressed? . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11
4.3. Who is leading EBSE research?. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12
4.4. What are the limitations of current research? . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12
4.5. Limitations of this study . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13
5. Conclusions. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13
Acknowledgements . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15
References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15
ll rights reserved.

: +44 1622 820176.
tchenham).

mailto:barbara@kitchenham.me.uk

http://www.sciencedirect.com/science/journal/09505849

http://www.elsevier.com/locate/infsof

8 B. Kitchenham et al. / Information and Software Technology 51 (2009) 7–15
1. Introduction

At ICSE04, Kitchenham et al. [23] suggested software engineer-
ing researchers should adopt ‘‘Evidence-based Software Engineer-
ing” (EBSE). EBSE aims to apply an evidence-based approach to
software engineering research and practice. The ICSE paper was
followed-up by an article in IEEE Software [5] and a paper at Met-
rics05 [17].

Evidence-based research and practice was developed initially in
medicine because research indicated that expert opinion based
medical advice was not as reliable as advice based on the accumu-
lation of results from scientific experiments. Since then many do-
mains have adopted this approach, e.g. Criminology, Social policy,
Economics, Nursing etc. Based on Evidence-based medicine, the
goal of Evidence-based Software Engineering is:

‘‘To provide the means by which current best evidence from
research can be integrated with practical experience and
human values in the decision making process regarding the
development and maintenance of software” [5].

In this context, evidence is defined as a synthesis of best quality
scientific studies on a specific topic or research question. The main
method of synthesis is a systematic literature review (SLR). In con-
trast to an expert review using ad hoc literature selection, an SLR
is a methodologically rigorous review of research results. The
aim of an SLR is not just to aggregate all existing evidence on a re-
search question; it is also intended to support the development of
evidence-based guidelines for practitioners. The end point of EBSE
is for practitioners to use the guidelines to provide appropriate
software engineering solutions in a specific context.

The purpose of this study is to review the current status of EBSE
since 2004 using a tertiary study to review articles related to EBSE
and, in particular, we concentrate on articles describing systematic
literature reviews (SLRs). Although SLRs are not synonymous with
EBSE, the aggregation of research results is an important part of the
EBSE process and, furthermore, is the part of the EBSE process that
can be readily observed in the scientific literature. We describe our
methodology in Section 2 and present our results in Section 3. In
Section 4 we answer our 4 major research questions. We present
our conclusions in Section 5.
Table 1
Selected journals and conference proceedings.

Source Acronym

Information and Software Technology IST
Journal of Systems and Software JSS
IEEE Transactions on Software Engineering TSE
IEEE Software IEEE SW
Communications of the ACM CACM
ACM Computer Surveys ACM Sur
ACM Transactions on Software Engineering Methodologies TOSEM
Software Practice and Experience SPE
Empirical Software Engineering Journal EMSE
IEE Proceedings Software (now IET Software) IET SW
Proceedings International Conference on Software Engineering ICSE
Proceedings International Symposium of Software Metrics Metrics
Proceedings International Symposium on Empirical Software

Engineering
ISESE
2. Method

This study has been undertaken as a systematic literature re-
view based on the original guidelines as proposed by Kitchenham
[22]. In this case the goal of the review is to assess systematic lit-
erature reviews (which are referred to as secondary studies), so
this study is categorised as a tertiary literature review. The steps
in the systematic literature review method are documented below.

2.1. Research questions

The research questions addressed by this study are:

RQ1. How much SLR activity has there been since 2004?
RQ2. What research topics are being addressed?
RQ3. Who is leading SLR research?
RQ4. What are the limitations of current research?

With respect to RQ1, it may be a concern that we started our
search at the start of 2004. We recognise that the term ‘‘systematic
literature review” was not in common usage in the time period
during which literature reviews published in 2004 were
conducted. However, there were examples both of rigours litera-
ture reviews and of meta-analysis studies prior to 2004
[37,41,42,10,33,29,30,13]. Furthermore, the concepts of evidence-
based software engineering had been discussed by research groups
in Europe for some time before 2004 as part of some (unsuccessful)
European Commission Research proposals. Thus, although we
would not expect papers published in 2004 to have been directly
influenced by the EBSE papers [23,5] or the guidelines for system-
atic reviews [22], we thought it was important to have some idea
of the extent of systematic approaches to literature reviews before
the guidelines were made generally available.

To address RQ1, we identified the number of SLRs published per
year, the journal/conferences that published them and whether or
not they referenced the EBSE papers [23,5] or Guidelines paper [22].

With respect to RQ2, we considered the scope of the study (i.e.
whether it looked at research trends, or whether it addressed a
technology-centred research question) and the software engineer-
ing topic area. With respect to RQ3, we considered individual
researchers, the organisation to which researchers were affiliated
and the country in which the organisation is situated.

With respect to limitations of SLRs (RQ4) we considered a num-
ber of issues:

RQ4.1. Were the research topics limited?
RQ4.2. Is there evidence that the use of SLRs is limited due to lack

of primary studies?
RQ4.3. Is the quality of SLRs appropriate, if not, is it improving?
RQ4.4. Are SLRs contributing to practice by defining practice

guidelines?

2.2. Search process

The search process was a manual search of specific conference
proceedings and journal papers since 2004. The selected journals
and conferences are shown in Table 1. The journals were selected
because they were known to include either empirical studies or lit-
erature surveys, and to have been used as sources for other system-
atic literature reviews related to software engineering (e.g. [10 and
36]).

Each journal and conference proceedings was reviewed by one
of four different researchers (i.e. Kitchenham, Brereton, Budgen
and Linkman) and the papers that addressed literature surveys of
any type were identified as potentially relevant. Kitchenham coor-
dinated the allocation of researchers to tasks based on the avail-
ability of each researcher and their ability to access the specific
journals and conference proceedings. The researcher responsible
for searching the specific journal or conference applied the detailed
inclusion and exclusion criteria to the relevant papers (see Section
2.3). Another researcher checked any papers included and ex-
cluded at this stage.

B. Kitchenham et al. / Information and Software Technology 51 (2009) 7–15 9
In addition, we contacted Professor Guilherme Travassos di-
rectly and Professor Magne Jørgensen indirectly by reviewing the
references in his web page. We did this because Professor Travas-
sos had reported to one of us that his research group was attempt-
ing to adopt the SLR process and because Professor Jørgensen was
known to be the author of a substantial number of SLRs.

2.3. Inclusion and exclusion criteria

Peer-reviewed articles on the following topics, published be-
tween Jan 1st 2004 and June 30th 2007, were included:

� Systematic Literature Reviews (SLRs) i.e. literature surveys with
defined research questions, search process, data extraction and
data presentation, whether or not the researchers referred to
their study as a systematic literature review.

� Meta-analyses (MA).

Note, we included articles where the literature review was only
one element of the articles as well as articles for which the litera-
ture review was the main purpose of the article.

Articles on the following topics were excluded

� Informal literature surveys (no defined research questions; no
defined search process; no defined data extraction process).

� Papers discussing the procedures used for EBSE or SLRs.
� Duplicate reports of the same study (when several reports of a

study exist in different journals the most complete version of
the study was included in the review).

2.4. Quality assessment

Each SLR was evaluated using the York University, Centre for
Reviews and Dissemination (CDR) Database of Abstracts of Re-
views of Effects (DARE) criteria [3]. The criteria are based on four
quality assessment (QA) questions:

QA1. Are the review’s inclusion and exclusion criteria described
and appropriate?

QA2. Is the literature search likely to have covered all relevant
studies?

QA3. Did the reviewers assess the quality/validity of the included
studies?

QA4. Were the basic data/studies adequately described?

The questions were scored as follows:

� QA1: Y (yes), the inclusion criteria are explicitly defined in the
study, P (Partly), the inclusion criteria are implicit; N (no), the
inclusion criteria are not defined and cannot be readily inferred.

� QA2: Y, the authors have either searched 4 or more digital
libraries and included additional search strategies or identified
and referenced all journals addressing the topic of interest; P,
the authors have searched 3 or 4 digital libraries with no extra
search strategies, or searched a defined but restricted set of jour-
nals and conference proceedings; N, the authors have search up
to 2 digital libraries or an extremely restricted set of journals.

� QA3: Y, the authors have explicitly defined quality criteria and
extracted them from each primary study; P, the research ques-
tion involves quality issues that are addressed by the study; N
no explicit quality assessment of individual primary studies
has been attempted.

� QA4: Y Information is presented about each study; P only sum-
mary information about primary studies is presented; N the
results of the individual primary studies are not specified.
The scoring procedure was Y = 1, P = 0.5, N = 0, or Unknown (i.e.
the information is not specified). Kitchenham coordinated the
quality evaluation extraction process. Kitchenham assessed every
paper, and allocated 4 papers to each of the other authors of this
study to assess independently. When there was a disagreement,
we discussed the issues until we reached agreement. When a ques-
tion was scored as unknown we e-mailed the authors of the paper
and asked them to provide the relevant information and the ques-
tion re-scored appropriately.

2.5. Data collection

The data extracted from each study were:

� The source (journal or conference) and full reference.
� Classification of the study Type (SLR, Meta-Analysis MA); Scope

(Research trends or specific technology evaluation question).
� Main topic area.
� The author(s) and their institution and the country where it is

situated.
� Summary of the study including the main research questions

and the answers.
� Research question/issue.
� Quality evaluation.
� Whether the study referenced the EBSE papers [23,5] or the SLR

Guidelines [22].
� Whether the study proposed practitioner-based guidelines.
� How many primary studies were used in the SLR.

One researcher extracted the data and another checked the
extraction. The procedure of having one extractor and one checker
is not consistent with the medical standards summarized in Kitch-
enham’s guidelines [22], but is a procedure we had found useful in
practice [2]. Kitchenham coordinated the data extraction and
checking tasks, which involved all of the authors of this paper.
Allocation was not randomized, it was based on the time availabil-
ity of the individual researchers. When there was a disagreement,
we discussed the issues until we reached agreement.

2.6. Data analysis

The data was tabulated to show:

� The number of SLRs published per year and their source
(addressing RQ1).

� Whether the SLR referenced the EBSE papers or the SLR guide-
lines (addressing RQ1).

� The number of studies in each major category i.e. research
trends or technology questions (addressing RQ2 and RQ4.1).

� The topics studied by the SLRs and their scope (addressing RQ2
and RQ4.1).

� The affiliations of the authors and their institutions (addressing
RQ3).

� The number of primary studies in each SLR (addressing RQ4.2).
� The quality score for each SLR (addressing RQ4.3).
� Whether the SLR proposed practitioner-oriented guidelines

(addressing RQ4.4).
2.7. Deviations from protocol

As a result of an anonymous review of an earlier version of this
paper, we made some changes to our original experimental proto-
col (see [24] Appendix 1):

� We explained our concentration on SLRs as part of EBSE.

10 B. Kitchenham et al. / Information and Software Technology 51 (2009) 7–15
� We extended the description of our research questions.
� We asked the authors of studies for which the answers to certain

quality questions were unknown to provide the information.
� We clarified the link between the research questions and the

data collection and analysis procedures

3. Results

This section summarizes the results of the study.

3.1. Search results

Table A1 (in Appendix 1) shows the results of the search proce-
dure. Although we identified 19 articles by this search process, one
of the articles [19] is a short version of another article [18]. Thus
we identified 18 unique studies. In addition, we found another
two other studies that had been subject to peer review: one by ask-
ing researchers about their current work [1] and the other by
searching the Simula Research Laboratory website [14]. Other
potentially relevant studies that were excluded as a result of apply-
ing the detailed inclusion and exclusion criteria are listed in Table
A2 in Appendix 1. One of the excluded papers positioned itself as
an EBSE paper but did not specify how it applied the EBSE princi-
ples [26].

Two studies were published in conference proceedings as well
as in journals: Galin and Avrahami [7] is a conference version of
Table 2
Systematic review studies.

ID Author Date Topic type Topic area

S1 Barcelos and Travassos [1] 2006 Technology
evaluation

Software arch
evaluation m

S2 Dyba et al. [4] 2006 Research trends Power in SE e

S3 Galin and Avrahami [7,8] 2005 &
2006

Technology
evaluation

CMM

S4 Glass et al. [9] 2004 Research trends Comparative
CS, IS and SE

S5 Grimstad et al. [11] 2006 Technology
evaluation

Cost estimati

S6 Hannay et al. [12] 2007 Research trends Theory in SE

S7 Jørgensen [15] 2004 Technology
evaluation

Cost estimati

S8 Jørgensen [14] 2007 Technology
evaluation

Cost estimati

S9 Jørgensen and Shepperd
[16]

2007 Research trends Cost estimati

S10 Juristo et al. [18,19] 2004 &
2006

Technology
evaluation

Unit testing

S11 Kitchenham et al. [20,21] 2006 &
2007

Technology
evaluation

Cost estimati

S12 Mair and Shepperd [27] 2005 Technology
evaluation

Cost estimati

S13 Mendes [28] 2005 Research trends Web research

S14 Moløkken-Østvold et al.
[31]

2005 Technology
evaluation

Cost estimati

S15 Petersson et al. [32] 2004 Technology
evaluation

Capture–reca
inspections

S16 Ramesh et al. [34] 2004 Research trends Computer sci
S17 Runeson et al.[35] 2006 Technology

evaluation
Testing meth

S18 Torchiano and Morisio [38] 2004 Technology
evaluation

COTS develop

S19 Sjøberg et al. [36] 2005 Research trends SE experimen

S20 Zannier et al. [40] 2006 Research trends Empirical stu

a Runeson et al. suggest how practitioners can use their results but do not explicitly
Galin and Avrahami [8] and Kitchenham et al. [20] is a conference
version of Kitchenham et al. [21].

The data extracted from each study are shown in Tables A2 and
A3 (in Appendix 1). Summaries of the studies can be found in [24],
Appendix 3.

3.2. Quality evaluation of SLRs

We assessed the studies for quality using the DARE criteria (see
Section 2.4). The score for each study is shown in Table 3. The fields
marked with an asterisk in Table 3 were originally marked as un-
known and were re-assigned after communicating with the study
authors.

The last column in Table 5 shows the number of questions
where the researchers were in agreement. All disagreements were
discussed and resolved.

The results of the quality analysis show that all studies scored 1
or more on the DARE scale and only three studies scored less than
2. Two studies scored 4 ([15 and 21]) and two studies scored 3.5
([14 and 40]).

3.3. Quality factors

We investigated the relationship between the quality score for
an SLR and both the date when the article was published, and
the use or not of the guidelines for SLRs [22]. The average quality
scores for studies each year is shown in Table 4. Note, for this anal-
Article
type

Refs. Include
practitioner
guidelines

Num.
primary
studies

itecture
ethods

SLR Guideline
TR

No 54

xperiments SLR Guideline
TR

No 103

MA No No 19

trends in SLR No No 1485

on SLR Guideline
TR

Yes 32

experiments SLR Guideline
TR

No 103

on SLR No Yes 15

on SLR No Yes 16

on SLR GuidelineTR No 304

SLR EBSE paper No 24

on SLR Guideline
TR

Yes 10

on SLR No No 20

SLR Guideline
TR

No 173

on SLR No No 6

pture in SLR No No 29

ence research SLR No No 628
ods SLR EBSE paper No a 12

ment SLR No No 21

ts SLR Guideline
TR

No 103

dies in ICSE SLR No No 63

define guidelines.

Table 3
Quality evaluation of SLRs.

Study Article type QA1 QA2 QA3 QA4 Total
score

Initial rater
agreement

S1 SLR Y P N Y 2.5 4
S2 SLR Y P P P 2.5 4
S3 MA Y P* P P 2.5 4
S4 SLR Y P N P 2 4
S5 SLR Y Y N Y 3 4
S6 SLR Y P N Y 2.5 4
S7 SLR Y Y* Y Y 4 4
S8 SLR Y Y P Y 3.5 4
S9 SLR Y Y N Y 3 4
S10 SLR P N P P 1.5 4
S11 SLR Y Y Y Y 4 4
S12 SLR Y P* N Y 2.5 4
S13 SLR Y N P P 2 4
S14 SLR Y Y* N Y 3 4
S15 SLR P Y N Y 2.5 3
S16 SLR P P N P 1.5 3
S17 SLR Y N N Y 2 2
S18 SLR Y N N N 1 4
S19 SLR Y P N P 2 3
S20 SLR Y Y Y P 3.5 3

Table 4
Average quality scores for studies by publication date.

Year

2004 2005 2006 2007

Number of studies 6 5 6 3
Mean quality score 2.08 2.4 2.92 3
Standard deviation of quality score 1.068 0.418 0.736 0.50

B. Kitchenham et al. / Information and Software Technology 51 (2009) 7–15 11
ysis we used the first publication date for any duplicated study. Ta-
ble 4 indicates that the number of studies published per year has
been quite stable. The average quality score appears to be increas-
ing, the Spearman correlation between year and score was 0.51
(p < 0.023) The average quality scores for studies that did or did not refer- ence the SLR guidelines are shown in Table 5. A one way analysis of variance showed that the mean quality score of studies that refer- enced the SLR guidelines [22] compared with those that did not, was not significant (F = 0.37, p = 0.55). Thus, it appears that the quality of SLRs is improving but the improvement cannot be attrib- uted to the guidelines. 4. Discussion In this section, we discuss the answers to our research questions. 4.1. How much EBSE Activity has there been since 2004? Overall, we identified 20 relevant studies in the sources that we searched, as shown in Table 2. 19 studies were classified as SLRs and one study was classified as a meta-analysis [8]. Twelve studies addressed technology evaluation issues and 8 addressed research trends. We found that 8 studies referenced Kitchenham’s guide- Table 5 Average quality score for studies according to use of guidelines. Referenced SLR guidelines Did not reference SLR guidelines Number of studies 8 12 Mean quality score 2.69 2.46 lines [22] and two referenced the EBSE paper [5]. Thus, half the studies directly positioned themselves as related to Evidence- based Software Engineering. With respect to where SLRs are published, IEEE Software and IEEE TSE each published 4 studies, JSS published 3 and IST pub- lished 2. Thus, it appeared that IST’s attempts to encourage the publication of SLRs, was unsuccessful [6]. However, a further check of IST publications (on September 17th 2008 using the search string systematic AND review) found seven more SLRs, whereas sim- ilar searches of TSE and JSS found no new SLRs. Initially, we were surprised that ACM Computer Surveys did not include any relevant software engineering studies, although the journal published a systematic literature review on the topic of education [25]. An automated search of ACM Computer Surveys using the ACM digital library on September 20th 2008, found no software-related surveys that used the systematic review method- ology. However, the apparent lack of software SLRs in ACM Com- puter Surveys may be because, with a maximum of four issues per year, the journal is likely to have a significant publication lag. 4.2. What research topics are being addressed? With respect to the topic of the articles, eight were related to re- search trends rather than specific research questions. In terms of the software engineering topic area addressed by the SLRs: � 7 related to software cost estimation (one of those covered research trends), in addition, the four studies that included evi- dence-based guidelines all related to cost estimation. � 3 articles related to software engineering experiments (all inves- tigated research trends). � 3 articles related to test methods. In the area of cost estimation, researchers are addressing spe- cific research questions including: � Are mathematical estimating models more accurate than expert opinion based estimates? � No. [15]. � What is the level of overrun of software projects and is it chang- ing over time? � 30% and unchanging [31]. � Are regression-based estimation models more accurate than analogy-based models? � No. [27]. � Should you use a benchmarking data base to construct an esti- mating model for a particular company if you have no data of your own? � Not if you work for a small company doing niche applications [21]. � Do researchers use cost estimation terms consistently and appropriately? � No they confuse prices, estimates, and budgets [11]. � When should you use expert opinion estimates? � When you don’t have a calibrated model, or important con- textual information is not incorporated into your model [14]. The testing studies have investigated: � Whether testing is better than inspections. 12 B. Kitchenham et al. / Information and Software Technology 51 (2009) 7–15 � Yes for design documents, No for code.[35]. � Different capture–recapture methods used to predict the defects remaining after inspections. � Most studies recommend the Mh-JK model. Only one of 29 studies was an application study [32]. � Empirical studies in unit testing. � Empirical studies in unit testing are mapped to a framework and summarized [18]. Table A1 Sources searched for years 2004–2007 (including articles up to June 30 2007). Year 2004 2005 2006 2007 Total IST (Total) 85 95 72 47 299 IST (Relevant) 0 2 2 0 4 IST (Selected) 0 0 2 0 2 JSS (Total) 139 122 124 43 428 JSS (Relevant) 4 0 0 0 4 JSS (Selected) 3 0 0 0 3 IEEE SW (Total) 51 52 48 24 175 IEEE SW (Relevant) 1 0 5 2 9 IEEE SW (Selected) 1 0 3 0 4 TSE (Total) 69 66 56 25 216 TSE (Relevant) 2 1 0 3 7 TSE (Selected) 0 1 0 3 4 CACM (Total) 148 141 158 64 511 CACM (Relevant) 1 0 0 0 1 CACM (Selected) 1 0 0 0 1 ACM Sur (Total) 12 11 13 3 39 ACM Sur (Relevant) 0 0 1 0 1 ACM Sur (Selected) 0 0 0 0 0 TOSEM (Total) 10 12 12 6 40 TOSEM (Relevant) 0 2 0 0 2 TOSEM (Selected) 0 0 0 0 0 SPE (Total) 64 59 68 29 220 SPE (Relevant) 0 0 0 0 0 SPE (Selected) 0 0 0 0 0 ICSE (Total) 58 58 36 64 216 ICSE (Relevant) 0 0 1 0 1 ICSE (Selected) 0 0 1 0 1 ISESE (Total) 26 50 56 n/a 132 ISESE (Relevant) 0 2 1 n/a 3 ISESE (Selected) 0 2 0 n/a 2 IET SW (Total) 22 28 22 9 81 IET SW (Relevant) 0 0 0 1 1 IET SW (Selected) 0 0 0 0 0 EMSE (Total) 14 19 20 12 61 EMSE (Relevant) 1 0 0 0 1 EMSE (Selected) 1 0 0 0 1 Metrics (Total) 36 48 n/a n/a Metrics (Relevant) 1 0 n/a n/a 1 Metrics (Selected) 1 0 n/a n/a 1 Total 734 761 685 326 2506 Total relevant 10 7 10 6 33 Total selected 7 3 6 3 19 4.3. Who is leading EBSE research? Overall, the set of studies are dominated by European research- ers who have been involved in 14 of the studies, in particular the Simula Research Laboratory in Norway which has been involved in 8 of the studies. The two researchers who contributed to more than two SLRs, Jørgensen (5) and Sjøberg (3), are both affiliated to the Simula Research Laboratory. Only four studies had North American authors. The success of the Simula Research Laboratory in applying the principles of EBSE and performing high quality SLRs is supported by the strategy of constructing databases of primary studies re- lated to specific topic areas and using those databases to address specific research questions. A database of cost estimation papers from over 70 journals [16] has been the basis of many of the de- tailed cost estimation studies authored or co-authored by Jørgen- sen and the database of 103 software experiments [36] has allowed researchers to assess a number of specific research trends in software experimentation. 4.4. What are the limitations of current research? With respect to whether research topics addressed by SLRs are somewhat limited (RQ4.1), a relatively large number of studies re- late to research practice rather than questions concerning specific software engineering practices and techniques. This is disappoint- ing since this type of study benefits researchers rather than practi- tioners, and evidence-based software engineering is meant to be of benefit to practitioners. However, three of the research trend stud- ies addressed the quality of current experimental studies and iden- tified areas for improvement, and improved empirical methods might be expected to benefit practitioners in the longer term. Fur- thermore, the Jørgensen and Shepperd study [16], although classi- fied as a research trends study, is also an example of a mapping study (i.e. a study that aims to identify and categorise the research in a fairly broad topic area). The availability of high quality map- ping studies has the potential to radically change the nature of software engineering research. Mapping studies can highlight areas where there is a large amount of research that would benefit from more detailed SLRs and areas where there is little research that require more theoretical and empirical research. Thus, instead of every researcher undertaking their own research from scratch, a broad mapping study provides a common starting point for many researchers and many research initiatives. On September 17, 2008, the SCOPUS search engine found already 23 citations of this paper of which only four were self-citations. This suggests that the research community has already recognised the value of a good mapping study. For studies that investigated technology questions, the majority have been in the cost estimation field. Of the conventional software engineering lifecycle, only testing, with three studies, has been addressed. Juristo et al. [18,19] found only 24 studies comparing unit test- ing techniques. This is extremely surprising given that unit testing is a software activity that is relatively easily studied using experi- ments since tasks are relatively small and can be treated in isola- tion. We found this particularly curious in the light of 29 experiments that compared test–retest methods of predicting remaining defects after inspections [32] which is a far less central element of software engineering practice than unit testing. Juristo et al.’s study was based on a search of only the ACM and IEEE elec- tronic databases, so this may be an example of area where a broad- er search strategy would be useful. Looking at the number of primary studies in each SLR (RQ4.2), unsurprisingly, the research trends studies were based on a larger number of primary studies (i.e. 63–1485) than the technology eval- uation studies (i.e. 6–54). However, the results confirm that some topics have attracted sufficient primary studies to permit SLRs to address detailed research questions, although, as yet, only a lim- ited number of topics are addressed. With respect to the quality of SLRs (RQ4.3), the results of the quality analysis show that all studies scored 1 or more on the DARE scale and only three studies scored less than 2. However, relatively few SLRs have assessed the quality of the primary studies included in the review. This is acceptable in the context of studies of re- search trends but is more problematic for reviews that attempt to evaluate technologies. With respect to the contribution of SLRs to software engineer- ing practice (RQ4.4), of the 12 SLRs that addressed research ques- B. Kitchenham et al. / Information and Software Technology 51 (2009) 7–15 13 tions only four offered advice to practitioners. This is an issue where there needs to be improvement, since Evidence-based Soft- ware Engineering is meant to impact practice not just academia. 4.5. Limitations of this study The procedures used in this study have deviated from the ad- vice presented in Kitchenham’s 2004 guidelines [22] in several ways: � The search was organised as a manual search process of a spe- cific set of journals and conference proceedings not an auto- mated search process. This was consistent with the practices of other researchers looking at research trends as opposed to software technology evaluation. � A single researcher selected the candidate studies, although the studies included and excluded were checked by another researcher. � A single researcher extracted the data and another researcher checked the data extraction, as suggested by Brereton et al. [2]. The first point above implies that we may have missed some relevant studies, and thus underestimate the extent of EBSE-re- lated research. In particular, we will have missed articles published in national journals and conferences. We will also have missed articles in conferences aimed at specific software engineering top- ics which are more likely to have addressed research questions rather than research trends. Thus, our results must be qualified as applying only to systematic literature reviews published in the major international software engineering journals, and the major general and empirical software engineering conferences. With respect to the second point, given our interest in system- atic literature reviews, we are likely to have erred on the side of Table A2 Candidate articles not selected. Source Authors Reference Yea TSE T. Mens and T. Tourwé 30(2), pp 126–139 200 TSE S. Balsamo, A. Di Marco, P. Inverardi 30(5), pp. 295–309 200 IET Software S. Mahmood, R. Lai and Y.S. Kim 1(2), pp 57–66 200 IEEE Software D.C. Gumm 23(5) pp. 45–51 200 IEEE Software M. Shaw and P Clements 23(2) pp. 31–39 200 IEEE Software M. Aberdour 24(1), pp. 58–64 200 IEEE Software D. Damian 24(2), pp. 21–27 200 JSS E. Folmer and J. Bosch 70, pp. 61–78 200 IST Hochstein and Lindvall 47, pp. 643–656 200 IST S. Mahmood, R. Lai, Y.S. Kim, J.H. Kim, S.C. Park, H.S. h 47, pp. 693–707 200 TOSEM J. Estublier, D. Leblang, A. van der Hoek, R. Conradi, G. Clemm, W. Tichy, D. Wiborg-Weber pp. 383–430 200 TOSEM Barbara G. Ryder, Mary Lou Soffa, Margaret Burnett pp. 431–477 200 ACM Surv J. Ma and J. V. Nickerson 38(3), pp. 1–24 200 ISESE S. Wagner 200 including studies that were not very systematic, rather than omit- ting any relevant studies. For example, the literature review in the primary study, that was assigned the lowest quality score [38], was only a minor part of the article. The third point means that some of the data we collected may be erroneous. A detailed review of one of our own systematic liter- ature reviews has suggested that the extractor/checker mode of working can lead to data extraction and aggregation problems when there are a large number of primary studies or the data is complex [39]. However, in this tertiary study, there were relatively few primary studies and the data extracted from the selected arti- cles were relatively objective, so we do not expect many data extraction errors. The quality assessment criteria proved the most difficult data to extract because the DARE criteria are somewhat subjective. However quality criteria were evaluated independently by two researchers, hopefully reducing the likelihood of erroneous results. 5. Conclusions Although 10 of the SLR studies in this review cited one of the EBSE papers [5] or the SLR Guidelines [22], the number of SLRs has remained extremely stable in the 3.5 years included in this study. Furthermore, Table A2 (see Appendix 1) also makes it clear that many researchers still prefer to undertake informal literature surveys. However, we have found that the quality of SLRs is improv- ing, suggesting that researchers who are interested in the EBSE ap- proach are becoming more competent in the SLR methodology. The spread of topics covered by current SLRs is fairly limited. Furthermore main stream software engineering topics are not well represented. However, even if these areas are unsuitable for SLRs aimed at empirical assessments of software technology, we believe r Title Reason for rejection 4 A survey of software refactoring Informal literature survey 4 Model-based performance prediction in software development Informal literature survey 7 Survey of component-based software development Informal literature survey 6 Distribution dimensions in software development Literature survey referenced but not described in article 6 The golden age of software Architecture Informal literature survey 7 Achieving quality in open source software Informal literature survey 7 Stakeholders in global requirements engineering: lessons learnt from practice Informal literature survey 4 Architecting for usability: a survey Informal literature survey 5 Combating architectural degeneration: a survey Informal literature survey 5 A survey of component-based system quality assurance and assessment Informal literature survey 5 Impact of software engineering research on the practice of software configuration management Informal literature survey 5 The impact of software engineering research on modern programming languages Informal literature survey. No clear search criteria, no data extraction process. 6 Hands-on, simulated and remote laboratories: a comparative literature review Not a software engineering topic 6 A literature survey of the quality economics of defect-detection techniques Informal literature survey although quantitative data tabulated for different testing techniques. 14 B. Kitchenham et al. / Information and Software Technology 51 (2009) 7–15 it would be possible, and extremely valuable, for leading software engineering researchers to undertake mapping studies of their do- main similar to that provided by Jørgensen and Shepperd study [16] for cost estimation research. Table A3 Author affiliation details. ID Authors Institution S1 Barcelos Federal Unive Travassos Federal Unive S2 Dybå SINTEF & Sim Kampenes Simula Labora Sjøberg Simula Labora S3 Gavin Ruppin Acade Avrahami Lipman Electr S4 Glass Computing Tr
Ramesh Kelley Busine
Vessey Kelley Busine

S5 Grimstad Simula Resea
Jørgensen Simula Resea
Moløkken-Østvold Simula Resea

S6 Hannay Simula Resea
Sjøberg Simula Resea
Dybå SINTEF & Sim

S7 Jørgensen Simula Resea

S8 Jørgensen Simula Resea

S9 Jørgensen Simula Resea
Shepperd Brunel Univer

S10 Juristo Univsidad Pol
Moreno Univsidad Pol
Vegas Univsidad Pol

S11 Kitchenham Keele Univers
Mendes University of
Travassos Federal Unive

S12 Mair Brunel Univer
Shepperd Brunel Univer

S13 Mendes University of

S14 Moløkken-Østvold Simula Resea
Jørgensen Simula Resea
Tanilkan OSLO Univers
Gallis Simula Resea
Lien Simula Resea
Hove Simula Resea

S15 Petersson Lund Univers
Thelin Lund Univers
Runeson Lund Univers
Wohlin Bleking Instit

S16 Ramesh Kelley School
Glass Computing Tr
Vessey Kelley School

S17 Runeson Lund Univers
Andersson Lund Univers
Thelin Lund Univers
Andrews University of
Berling Lund Univers

S18 Sjøberg Simula Resea
Hannay Simula Resea
Hansen Simula Resea
Kampenes Simula Resea
Karahasanović Simula Resea
Liborg BNP Paribas
Rakdal Unified Consu

S19 Torchiano Norwegian Un
Morisio Politecnico de

S20 Zannier University of
Melnik University of
Maurer University of
In the area of cost estimation there have been a series of
systematic literature reviews. This accumulation of evidence in a
specific topic area is starting to demonstrate the value of evi-
dence-based software engineering. For example, the evidence
Country of institution

rsity of Rio de Janeiro Brazil
rsity of Rio de Janeiro Brazil

ula Laboratory Norway
tory Norway
tory Norway

mic Center Israel
onic Engineering Israel

ends USA
ss School, Indiana University USA
ss School, Indiana University USA

rch Laboratory Norway
rch Laboratory Norway
rch Laboratory Norway

rch Laboratory Norway
rch Laboratory Norway
ula Research Laboratory Norway

rch Laboratory Norway

rch Laboratory Norway
sity UK

iténcia de Madrid Spain
iténcia de Madrid Spain
iténcia de Madrid Spain

ity & NICTA UK & Australia
Auckland New Zealand
rsity of Rio de Janeiro Brazil

sity UK
sity UK

Auckland New Zealand

rch Laboratory & OSLO University Norway
rch Laboratory Norway
ity Norway
rch Laboratory & OSLO University Norway
rch Laboratory Norway
rch Laboratory Norway

ity Sweden
ity Sweden
ity Sweden
ute of Technology Sweden

of Business, Indiana University USA
ends USA
of Business, Indiana University USA

ity Sweden
ity Sweden
ity Sweden
Denver USA
ity Sweden

rch Laboratory Norway
rch Laboratory Norway
rch Laboratory Norway
rch Laboratory Norway
rch Laboratory Norway

Norway
lting Norway

iversity of Science and technology Norway
Torino Italy

Calgary Canada
Calgary Canada
Calgary Canada

B. Kitchenham et al. / Information and Software Technology 51 (2009) 7–15 15
gathered by means of the SLRs has overturned existing ‘‘common
knowledge” about the efficacy of models compared with expert
opinion and the size of project overruns. Furthermore in this area
we are beginning to see the publication of evidence-based guide-
lines aimed at practitioners, which is a specific goal of evidence-
based software engineering.

This review suggests that the Simula Research Laboratory, Nor-
way is currently the leading software engineering institution in
terms of undertaking SLRs. The research group has benefited from
developing extremely effective research procedures to support
their secondary studies. We recommend other research groups
adopt similar research procedures, allowing the results of their
own literature reviews to build up into a data base of categorised
research papers that is available to initiate research programmes
and provide the references needed for research articles.

The results in this study suggest that the current output of EBSE
articles is strongly supported by European researchers. However, if
EBSE is to have a serious impact on software engineering research
and practice, it is important that researchers in other areas of the
world take an increased interest in a formal approach to literature
reviews, particularly, the US, because of its leadership in software
engineering research.

This study suffers from a number of limitations; in particular,
we have restricted ourselves to a manual search of international
journals and conferences. We plan to extend this study by under-
taking a broader automated search for other SLRs over the same
time period. This has the joint aim of extending the generality of
this study and investigating a number of issues associated with
systematic literature reviews in software engineering i.e. whether
we should use manual or automated searchers, and whether re-
stricted searches provide reliable results. We also plan to repeat
this study at the end of 2009 to track the progress of SLRs and evi-
dence-based software engineering.

Acknowledgements

This research was funded by The Engineering and Physical Sci-
ences Research Council (EPSRC) EBSE Project (EP/C51839X/1).
Short, preliminary versions of this study were presented at the RE-
BSE2 workshop at ICSE07 and the EASE07 Conference at Keele
University.
Appendix 1. Tables of the systematic review results.

See Tables A1–A3.

References

[1] R.F. Barcelos, G.H. Travassos, Evaluation approaches for software architectural
documents: a systematic review, in: Ibero-American Workshop on
Requirements Engineering and Software Environments (IDEAS), La Plata,
Argentina, 2006.

[2] O.P. Brereton, B.A. Kitchenham, D. Turner Budgen, M. Khalil, Lessons from
applying the systematic literature review process within the software
engineering domain, Journal of Systems and Software 80 (4) (2007) 571–583.

[3] Centre for Reviews and Dissemination, What are the criteria for the inclusion
of reviews on DARE? 2007. Available at ).

[4] T. Dyba, V.B. Kampenes, D.I.K. Sjøberg, A systematic review of statistical power
in software engineering experiments, Information and Software Technology 48
(8) (2006) 745–755.

[5] T. Dybå, B.A. Kitchenham, M. Jørgensen, Evidence-based software engineering
for practitioners, IEEE Software 22 (1) (2005) 58–65.

[6] M. Dyer, M. Shepperd, C. Wohlin, Systematic Reviews in Evidence-Based
Software Technology and Software Engineering 47 (1) (2005) 1.

[7] D. Galin, M. Avrahami, Do SQA programs work – CMM works. A meta analysis,
IEEE International Conference on Software – Science, Technology and
Engineering (2005).

[8] D. Galin, M. Avrahami, Are CMM program investments beneficial? Analyzing
past studies, IEEE Software 23 (6) (2006) 81–87.
[9] R.L. Glass, V. Ramesh, I. Vessey, An analysis of research in computing
disciplines, CACM 47 (6) (2004) 89–94.

[10] R.L. Glass, I. Vessey, V. Ramesh, Research in software engineering: an analysis
of the literature, Information and Software technology 44 (8) (2002) 491–506.

[11] S. Grimstad, M. Jorgensen, K. Molokken-Ostvold, Software effort estimation
terminology: the tower of Babel, Information and Software Technology 48 (4)
(2006) 302–310.

[12] J.E. Hannay, D.I.K. Sjøberg, T. Dybå, A systematic review of theory use in
software engineering experiments, IEEE Transactions on SE 33 (2) (2007) 87–
107.

[13] W. Hayes, Research synthesis in software engineering: the case for meta-
analysis, Proceedings 6th International Software Metrics Symposium, IEEE
Computer Press, 1999. pp. 143–151.

[14] M. Jørgensen, Estimation of software development work effort: evidence on
expert judgement and formal models, International Journal of Forecasting 3 (3)
(2007) 449–462.

[15] M. Jørgensen, A review of studies on expert estimation of software
development effort, Journal of Systems and Software 70 (1–2) (2004) 37–60.

[16] M. Jørgensen, M. Shepperd, A systematic review of software development cost
estimation studies, IEEE Transactions on SE 33 (1) (2007) 33–53.

[17] M. Jørgensen, T. Dybå, B.A. Kitchenham, Teaching evidence-based software
engineering to university students, in: 11th IEEE International Software
Metrics Symposium (METRICS’05), 2005, p. 24.

[18] N. Juristo, A.M. Moreno, S. Vegas, Reviewing 25 years of testing technique
experiments, Empirical Software Engineering Journal (1–2) (2004) 7–44.

[19] N. Juristo, A.M. Moreno, S. Vegas, M. Solari, In search of what we
experimentally know about unit testing, IEEE Software 23 (6) (2006) 72–80.

[20] B. Kitchenham, E. Mendes, G.H. Travassos, A systematic review of cross-
company vs. within-company cost estimation studies, Proceedings of EASE06,
BSC (2006) 89–98.

[21] B. Kitchenham, E. Mendes, G.H. Travassos, A systematic review of cross- vs.
within-company cost estimation studies, IEEE Transactions on SE 33 (5) (2007)
316–329.

[22] B.A. Kitchenham, Procedures for Undertaking Systematic Reviews, Joint
Technical Report, Computer Science Department, Keele University (TR/SE-
0401) and National ICT Australia Ltd. ( 0400011T.1), 2004.

[23] B.A. Kitchenham, T. Dybå, M. Jørgensen, Evidence-based software engineering,
in: Proceedings of the 26th International Conference on Software Engineering,
(ICSE’04), IEEE Computer Society, Washington DC, USA, 2004, pp. 273–281.

[24] B. Kitchenham, O.P. Brereton, D. Budgen, M. Turner, J. Bailey, S. Linkman, A
Systematic Literature Review of Evidence-based Software Engineering, EBSE
Technical Report, EBSE-2007-03, 2007.

[25] J. Ma, J.V. Nickerson, Hands-on, simulated and remote laboratories: a
comparative literature review, ACM Surveys 38 (3) (2006) 1–24.

[26] S. Mahmood, R. La, Y.S. Kim, A survey of component-based system quality
assurance and assessment, IET Software 1 (2) (2005) 57–66.

[27] C. Mair, M. Shepperd, The consistency of empirical comparisons of regression
and analogy-based software project cost prediction, International Symposium
on Empirical Software Engineering (2005) 509–518.

[28] E. Mendes, A systematic review of Web engineering research, International
Symposium on Empirical Software Engineering (2005) 498–507.

[29] J. Miller, Can results from software engineering experiments be safely
combined?, in: Proceedings 6th International Software Metrics Symposium,
IEEE Computer Press, 1999, pp 152–158.

[30] J. Miller, Applying meta-analytical procedures to software engineering
experiments, JSS 54 (1) (2000) 29–39.

[31] K.J. Moløkken-Østvold, M. Jørgensen, S.S. Tanilkan, H. Gallis, A.C. Lien, S.E.
Hove, A survey on software estimation in the Norwegian industry, Proceedings
Software Metrics Symposium (2004) 208–219.

[32] H. Petersson, T. Thelin, P. Runeson, C. Wohlin, Capture–recapture in software
inspections after 10 years research – theory, evaluation and application,
Journal of Systems and Software 72 (2004) 249–264.

[33] L.M. Pickard, B.A. Kitchenham, P. Jones, Combining empirical results in
software engineering, Information and Software Technology 40 (14) (1998)
811–821.

[34] V. Ramesh, R.L. Glass, I. Vessey, Research in computer science: an empirical
study, Journal of Systems and Software 70 (1–2) (2004) 165–176.

[35] P. Runeson, C. Andersson, T. Thelin, A. Andrews, T. Berling, What do we know
about defect detection methods?, IEEE Software 23 (3) (2006) 82–86.

[36] D.I.K. Sjøberg, J.E. Hannay, O. Hansen, V.B. Kampenes, A. Karahasanovic, N.K.
Liborg, A.C. Rekdal, A survey of controlled experiments in software
engineering, IEEE Transactions on SE 31 (9) (2005) 733–753.

[37] W.F. Tichy, P. Lukowicz, L. Prechelt, E.A. Heinz, Experimental evaluation in
computer science: a quantitative study, Journal of Systems and Software 28 (1)
(1995) 9–18.

[38] M. Torchiano, M. Morisio, Overlooked aspects of COTS-based development,
IEEE Software 21 (2) (2004) 88–93.

[39] M. Turner, B. Kitchenham, D. Budgen, O.P. Brereton, Lessons learnt undertaking
a large-scale systematic literature review, in: Proceedings of EASE’08, British
Computer Society, 2008.

[40] C. Zannier, G. Melnik, F. Maurer, On the success of empirical studies in the
international conference on software engineering, ICSE06 (2006) 341–350.

[41] M. Zelkowitz, D. Wallace, Experimental validation in software engineering,
Information and Software Technology 39 (1997) 735–743.

[42] M. Zelkowitz, D. Wallace, Experimental models for validating computer
technology, IEEE Computer 31 (5) (1998) 23–31.

http://www.york.ac.uk/inst/crd/faq4.htm

Systematic literature reviews in software engineering – A systematic literature review
Introduction
Method
Research Questionsquestions
Search Processprocess
Inclusion and Exclusion exclusion criteria
Quality Assessmentassessment
Data Collectioncollection
Data Analysisanalysis
Deviations from Protocolprotocol

Results
Search Resultsresults
Quality evaluation of SLRs
Quality Factorsfactors

Discussion
How much EBSE Activity has there been since 2004?
What research topics are being addressed?
Who is leading EBSE Research?research?
What are the limitations of current research?
Limitations of this study

Conclusions
AcknowledgementAcknowledgements
Tables of the Systematic Review Resultssystematic review results.
References

SAMPLE_SLRs/2_pdfsam_INFSOF-S-07-00181

Open Research Online
The Open University’s repository of research publications
and other research outputs

Motivation in software engineering: a systematic
literature review
Journal Item
How to cite:

Beecham, Sarah; Baddoo, Nathan; Hall, Tracy; Robinson, Hugh and Sharp, Helen (2008). Motivation in
software engineering: a systematic literature review. Information and Software Technology, 50(9-10) pp. 860–878.

For guidance on citations see FAQs.

Version: Accepted Manuscript

Link(s) to article on publisher’s website:
http://dx.doi.org/doi:10.1016/j.infsof.2007.09.004

Copyright and Moral Rights for the articles on this site are retained by the individual authors and/or other copyright
owners. For more information on Open Research Online’s data policy on reuse of materials please consult the policies
page.

oro.open.ac.uk

http://oro.open.ac.uk/help/helpfaq.html

http://dx.doi.org/doi:10.1016/j.infsof.2007.09.004

http://oro.open.ac.uk/policies.html

Motivation in Software Engineering: A Systematic Literature
Review

SARAH BEECHAM, NATHAN BADDOO AND TRACY HALL
University of Hertfordshire
and
HUGH ROBINSON AND HELEN SHARP
The Open University

__________________________________________________________________________________________

OBJECTIVE – In this paper we present a systematic literature review of motivation in Software Engineering.
The objective of this review is to plot the landscape of current reported knowledge in terms of what motivates
developers, what de-motivates them and how existing models address motivation.

METHOD – We perform a systematic literature review of peer reviewed published studies that focus on
motivation in Software Engineering. Systematic reviews are well established in medical research and are used to
systematically analyse the literature addressing specific research questions.

RESULTS – We found 92 papers related to motivation in Software Engineering. 56% of the studies reported that
Software Engineers are distinguishable from other occupational groups. Our findings suggest that Software
Engineers are likely to be motivated according to three related factors: their ‘characteristics’ (for example, their
need for variety); internal ‘controls’ (for example, their personality) and external ‘moderators’ (for example, their
career stage). The literature indicates that de-motivated engineers may leave the organisation or take more sick-
leave, while motivated engineers will increase their productivity and remain longer in the organisation. Aspects
of the job that motivate Software Engineers include problem solving, working to benefit others and technical
challenge. Our key finding is that the published models of motivation in Software Engineering are disparate and
do not reflect the complex needs of Software Engineers in their career stages, cultural and environmental
settings.

CONCLUSIONS – The literature on motivation in Software Engineering presents a conflicting and partial
picture of the area. It is clear that motivation is context-dependent and varies from one engineer to another. The
most commonly cited motivator is the job itself, yet we found very little work on what it is about that job that
Software Engineers find motivating. Furthermore, surveys are often aimed at how Software Engineers feel
about ‘the organisation’, rather than ‘the profession’. Although models of motivation in Software Engineering
are reported in the literature, they do not account for the changing roles and environment in which Software
Engineers operate. Overall, our findings indicate that there is no clear understanding of the Software Engineers’
job, what motivates Software Engineers, how they are motivated, or the outcome and benefits of motivating
Software Engineers.

__________________________________________________________________________________________

1. INTRODUCTION

In this paper we present findings from our systematic literature review on motivation in Software Engineering

(SE). Since Bartol and Martin’s literature review in (1982) no comprehensive body of research has been

published to provide a complete picture of the available material on motivation in Software Engineering. By

updating work in this area we help SE managers, Software Engineers and interested researchers to determine the

current state of research in Software Engineering motivation. Our systematic approach to analyzing published

studies enables us to identify reliably where the literature has recurring themes, where it presents conflicting

findings, and where are there gaps in the existing body of work.

Manuscript

Motivation in Software Engineering is reported to have the single largest impact on practitioner1 productivity

(Boehm 1981) and software quality management (McConnell 1998), and continues to be ‘undermined’ and

problematic to manage (Procaccino et al. 2005). While there is increasing recognition amongst practitioners and

the academic community that motivation is an important issue, no systematic literature review has been

undertaken to bring together the published work of motivation in a Software Engineering setting

(MoMSE(cfs) 2005).

Motivation is increasingly cited as a particularly pernicious people problem in Software Engineering. In

DeMarco and Lister’s (1999) survey, motivation was found to be one of the most frequently cited causes of

software development project failure. The Standish report (1995) amplifies this finding by reporting that having

access to competent, hard working and focused staff is one of ten success criteria for software projects. As

McConnell (1998) points out,

“Motivation is a soft factor: It is difficult to quantify, and it often takes a back seat to other factors
that might be less important but are easier to measure. Every organisation knows that motivation is
important, but only a few organizations do anything about it. Many common management
practices are pennywise and pound-foolish, trading huge losses in motivation and morale for
minor methodology improvements or dubious budget savings.”

Some studies in this area suggest that conventional approaches to motivation within the industry might be

outdated. They have concentrated on rewards and recognition, e.g. (ProjectLink 2006), whereas some experts

have identified Software Engineers as having a distinctive personality profile (Capretz 2003) that are instead

motivated by the nature of the job, e.g. technical success and challenging technical problems (Tanner 2003;

Ramachandran and Rao 2006).

Given the importance of motivating Software Engineers, we conduct a systematic literature review of what

motivates Software Engineers and whether Software Engineers are indeed a homogeneous group with similar

needs. A systematic literature review evaluates and interprets all available research relevant to a particular

research question or topic area. It aims to present an evaluation of the literature relative to a research topic by

using a rigorous and auditable methodology. We have followed guidelines derived from those used by medical

researchers, adapted and applied by Kitchenham et al (2004; 2006) to reflect the specific problems of Software

Engineering research. We summarise evidence that establishes what motivates Software Engineers and how

existing theoretical frameworks represent motivation in Software Engineering. We look to the literature to

answer five research questions:

RQ1: What are the characteristics of Software Engineers?

RQ2: What (de)motivates Software Engineers to be more (less) productive?

RQ3: What are the external signs or outcomes of (de)motivated Software Engineers?

RQ4: What aspects of Software Engineering (de)motivate Software Engineers?

1 The way Software Engineer (as a practitioner) and Software Engineering (as a field) have been referred to over the 26 years covered in this
study has evolved significantly. IT, IS, SE, analysts, developers, programmers are examples of some of the terms used for the practitioner
role/field. In this survey we use the term ‘Software Engineer’ (SE) to refer to any of these roles and Software Engineering to refer to the
field. However, when quoting or referring to a particular paper, we use the term used in the study.

RQ5: What models of motivation exist in Software Engineering?

In this review the literature often characterises Software Engineers (SE’s) as a homogeneous group of high

achievers (Couger and Zawacki 1980; Capretz 2003). These studies suggest that Software Engineers are

somehow different to non-Software Engineers, a view reinforced by Wynekoop and Walz (1998) who found

“important differences in personalities exist between IS employees and the general population”. On the other

hand, Ferratt and Short (1986) question the existence of differences between IT and non-IT employees. They

found that IT employees (including IT managers) within the technical-professional sub-occupations were not

more motivated by achievement needs than corresponding subgroups of non-IT employees. Although they did

find that meaningful work was the highest motivator for these IT subgroups.

Couger and Zawacki’s (1980) seminal work on motivation in software engineering was conducted over 20 years

ago. Yet, this work has been used throughout the period of this review as the central model of Motivation in

Software Engineering. The environment for software engineering has changed considerably since that time, e.g.

with the increase in outsourcing, open source development, new technical concepts and languages, new

lightweight development methods and so on. So, in this review we consider whether this work is still as valid as

it was.

This paper is organised as follows. In Section Two we describe the method we used for our systematic literature

review, this involves producing and following rules in a protocol that is independently validated. We also report

on the quality of the included papers in this section. Section Three presents results of our synthesis of the

literature, including geographical spread, temporal aspects and publication details. Section Four reports the

results of our synthesis of identified themes based on our five research questions. In Section Five we discuss our

key findings. Section Six presents some limitations of this study, and finally in Section Seven we present our

conclusions.

2. METHOD

2.1 Introduction

In accordance with systematic review guidelines (Kitchenham 2004) we take the following steps:

1. Identify the need for a systematic literature review (MoMSE(cfs) 2005)

2. Formulate review research question(s)

3. Carry out a comprehensive, exhaustive search for primary studies

4. Assess and record the quality of included studies

5. Classify data needed to answer the research question(s)

6. Extract data from each included study

7. Summarise and synthesise study results (meta-analysis)

8. Interpret results to determine their applicability

9. Write-up study as a report

These steps are detailed in our protocol (See (Beecham et al. 2006) or

http://homepages.feis.herts.ac.uk/~ssrg/MOMSEProto.htm). We developed our protocol by running three

separate pilot studies involving four researchers who performed searches based on rules given in the protocol.

After several refinements the protocol was peer reviewed by an independent expert in systematic review

development in Software Engineering.

The remainder of this methodology section summarises the process presented in our protocol. Where more

information is required please refer to (Beecham et al. 2006).

2.2 Resources Searched

Key words and synonyms were drawn up for each research question. Then the following databases were

searched using these key words:

 ACM Digital library

 EI Compendex

 Google scholar

 IEEE Explore

 Inspec

 ISI Web of Science

 ScienceDirect

 UH University’s electronic library (voyager.herts.ac.uk)

To ensure we did not overlook any important material, additional searches were performed directly on key

conference proceedings, journals and authors.

2.3 Document Retrieval

Our searches elicited over 2,000 references. Evaluating the title and abstract enabled us to reject approximately

1,500 of these. We then looked at 519 papers in full to establish a final list of 92 papers.

2.4 Procedures for Including and Excluding Studies

Any published work that directly answers our research questions and was published between 1980 – date is

considered for inclusion in our review. To be included the study must also be published in a journal paper,

conference proceedings, or empirical experience report based on theoretical or previous rigorous research.

Studies are excluded that are opinion pieces or viewpoints that do not reference any other study or are not based

on empirical work. We also excluded studies external to Software Engineering, which focus on cognitive

behaviour, general group/team motivation and dynamics, manager motivation, user/end user motivation and

acceptance of technology, gender differences and education (e.g. motivating IT students to learn).

We included studies that focus on motivation and satisfaction in Software Engineering. We included satisfaction

as it is often used to measure Software Engineer motivation. For example, satisfaction is considered in great

detail in the Job Diagnostics Survey for Data Processing Personnel (JDS/DP) tool (Couger and Zawacki 1980)

that is used extensively to measure Software Engineer motivation.

https://oufe.open.ac.uk/exchweb/bin/redir.asp?URL=http://homepages.feis.herts.ac.uk/~ssrg/MOMSEProto.htm

2.5 Study Quality Assessment Checklists

Each accepted study is assessed against a quality checklist. Scores are given according to whether the study

presents clear, unambiguous findings based on evidence and argument. Quality scores for the 92 papers are

given in Table 1:

Table 1: Quality scores of Accepted Papers

QUALITY (scores) Total

Poor

(<26%) Fair (26%-45%) Good (46%-65%) Very Good (66%-85%) Excellent ( >86%)

Number of Studies 6 10 32 32 12 92

Percentage of papers ~6.5% ~10.5% ~35% ~35% ~13% 100%

Over 82% of papers included in our literature review have quality scores that are good to excellent.

2.6 Data extraction and synthesis

We used Endnote version 9 (www.endnote.com) to record reference details for each study. How each study

answers the research question(s) was recorded on a separate results form. We synthesised the data by identifying

themes emanating from the findings reported in each accepted paper. These identified themes gave us the

categories reported in our results section. In our results section we present frequencies of the number of times

each theme is identified in different studies. We give each occurrence the same weight. The frequencies merely

reflect how many times a given characteristic or motivator is identified in different papers, not how important it

may be.

A sensitivity analysis was performed on these studies based on population, location, year and type of study. The

sensitivity analyses gave us information on where the data might be biased. They are also reported in our results

section. Our protocol provides full details of this process.

2.7 Validation

We performed two validation exercises:

A. Inter-rater reliability: We ran an inter-rater reliability test on the 519 paper references we found in our

initial search. A group of primary researchers looked at each of these papers in greater detail (9 papers proved

unobtainable). 95 papers were accepted by the primary researchers. An independent researcher looked at 58

randomly selected rejected and accepted papers (approx every 10th paper from an alphabetical list of the 519

papers). A 99.4% agreement was recorded with the original assessments. This high level of agreement gives us

considerable confidence in our acceptance/rejection decisions.

B. Independent assessment: We performed a final validation exercise on the 95 ‘accepted papers’.

An independent expert in motivation in Software Engineering recorded how each paper addressed our

research questions. Again there was a high level of agreement between the primary researchers and

http://www.endnote.com/

the independent expert (99.8%), and any disagreements were discussed. There were only three papers

that could not be agreed on, and these went to arbitration with a third, independent researcher who

determined whether the papers should be included and how each study addressed our research

question(s). This process resulted in 100% agreement. Three of the accepted papers were rejected as a

result of this exercise, leaving 92 papers for inclusion.

3. RESULTS – BACKGROUND

3.1 Type of study

Figure 1 shows that out of the 92 studies, 86% are empirical, i.e. findings are based on direct evidence or

experiment. The 11% theoretical or conceptual studies are based on an understanding of the field from

experience and reference to other related work. There are a small number of studies (3%) that are either reviews

of the literature or secondary studies, where empirical work is re-examined.

empirical
86%

theoretical
11%

literature review
3%

Figure 1: Types of studies in our accepted papers

Data collection methods used in the empirical studies include: surveys and questionnaires, field studies,

structured and semi-structured interviews (face-to-face and by telephone), analysing results of programming

tests, data reviews, case studies, focus groups and controlled experiments.

Out of the 79 studies that are empirically based, only 5 studies do not include questionnaires. Figure 2 shows

how these data collection methods are divided with 94% (78% + 16%) of the empirical studies employing

questionnaire survey instruments:

78%

16%

5% questionnaire/survey

multiple data collection
methods with
questionnaire

multiple data collection
methods without
questionnaire

other

Figure 2: Data collection methods used in the empirical studies

3.3 Temporal view of publications

Figure 3 shows that over the last 26 years there is a recent increase in published papers covering Software

Engineer characteristics and motivation in Software Engineering.

Figure 3: Number of papers included in the review by five year intervals

This recent increase may be a reflection of a growing awareness of the importance of motivation in Software

Engineering. Alternatively, this increase may just match a general rise in published papers in Software

Engineering.

3.4 Data Sources

5
10
15

20
25
30

35
40
45

1980-84 1985-1989 1990-1994 1995-1999 2000-2005/6

N
u

m
b

er
o

f
p

ap
er

s
(T

o
ta

l
92

)

Figure 4: Publication sources of our included studies

Figure 4 gives a breakdown of where our 92 papers are published. The majority are published by the special

interest group on computer personnel research with fewer papers reaching the more widely known journal

publications.

3.5 Geographical distribution of papers

A high percentage of the empirical studies in our review are concentrated on work carried out in the USA

(56%), as shown in Figure 5:

Gl
ob

Au
str

al
ia

Au
str

Ca
na

Eg
yp

Gr
ee

Ho
ng

K
on

g
Isr

ae
l

Ja
pa

M
ex

ico

Ni
ge

ria

No
rw

Ch
ina

Si
ng

ap
or

So
ut

h
Af

ric
a

Ta
iw

an UK US
A

N
u

m
b

e
r

o
f

st
u

d
ie

Figure 5: Countries represented in the empirical studies

N
U

M
B

E
R

O
F

P
A

P
E

R
S

KEY JOURNALS AND CONFERENCE PROCEEDINGS

0
5

10
15
20
25
30
35
40
45
50

SIG
CPR

IEEE
PROCS

INF &
MAN

COMM OF
ACM

MIS JSS OTHERS

Key: SIGCPR = ACM Special Interest Group for Computing Personnel Research/ IEEE Procs = IEEE
Proceedings/ INF & MAN = Information and Management; JSS = Journal of Systems and Software

Seventeen countries are represented, and although the nine global studies involve all continents, countries such

as India are not well represented in the literature. It is clear that when we synthesise findings from all the studies

we are giving a predominantly Western view of motivation in Software Engineering.

4. RESULTS – MOTIVATION IN SOFTWARE ENGINEERING

This section reports on how the literature represents motivation in Software Engineering. Figure 6 gives an

overview of how our research questions work together to give a comprehensive view of our topic. Citations for

the 92 papers included in this section are given in numeric form with a bibliography in Appendix 1.

By investigating the five research questions in Figure 6, we aim to gain a broad picture of what the literature is

reporting on motivation in Software Engineering. We collected information on Software Engineer

characteristics (RQ1) to broaden our understanding of the underlying constructs relating to what (de)motivates

Software Engineers to be more/less productive (RQ2). We then took a more in depth view of Software Engineer

(de)motivators to uncover areas specific to the Software Engineering task itself (RQ4). To see how motivation is

measured and the potential benefits or otherwise of motivating Software Engineers, we researched the external

signs of (de)motivated Software Engineers. Finally, we looked at how all aspects of motivation are modelled in

the Software Engineering literature (RQ5).

Figure 6: The relationship between our five Research Questions

4.1 Do Software Engineers form a distinct occupational group?

Figure 7 shows that papers fall into three categories when considering whether Software Engineers form a

distinct occupational group.

YES
54%

NO
24%

YES&NO (depending
on context)

22%

Figure 7: Do Software Engineers form a Distinct Group?

Figure 7 shows that 76% (54% plus 22%) of papers find that Software Engineers do form a distinct occupational

group, albeit that in 22% of the cases this is context dependent. However, 24% of papers report that Software

Engineers do not form a distinct occupational group and in some contexts this rises to 46% (24% + 22%). Six

papers included in our software characteristics group of papers do not cover this question and are therefore

excluded from this analysis.

The following sub-sections look at each of our research questions in more detail.

4.2 RQ1 – Software Engineer Characteristics

43 papers were identified as answering Research Question 1 (RQ1), “What are the characteristics of Software

Engineers?”

These 43 papers identify 24 attributes which relate to ‘characteristics’ of SEs. However a closer inspection

shows that these attributes can be structured into three linked categories. The first category contains the ‘raw’

characteristics of Software Engineers. The second contains factors that control whether or not a particular

individual will have those characteristics. The third contains moderators which determine the strength of a

characteristic within an individual. Figure 8 shows how Characteristics, Control Factors and Moderators seem

to relate to one another: an individual will have a given Characteristic (e.g. need for stability) depending on

their Control Factors (e.g. their Myers Briggs Type Index (MBTI) score), and the strength of this characteristic

is Moderated by contextual factors, such as the country they live and work in. This structure implies a different

profile of characteristics for every individual Software Engineer, and that over time, an individual’s motivation

changes. Both of these implications are consistent with findings in the literature.

Figure 8: Determinants of software engineer characteristics

Using this structure, we have identified 16 ‘raw’ Software Engineer characteristics of which growth oriented,

introverted and need for independence are the most cited. The literature often uses the term ‘Career Anchor’ to

describe a person’s characteristics. A person’s career anchor is (1) his or her self concept of talents and abilities;

(2) his or her self concept of basic values; and (3) the individual’s evolved sense of motives and needs (Schein

1996).

Table 2 presents our literature review results for characteristics.

TABLE 2: Software Engineer Characteristics
Ch: Software Engineer Characteristics Paper references Frequency

(# of studies)
Ch.1 Need for stability (organisational stability) [1-5]; 5
Ch.2 Technically competent [1, 3, 6-8] 5
Ch.3 Achievement orientated (e.g. seeks promotion) [2, 9-11] 4
Ch.4 Growth orientated (e.g. challenge, learn new skills) [4, 7, 9, 12-17] 9
Ch.5 Need for competent supervising (e.g. needs respect

and appreciation, given a clear job to do and goals)
[2, 3, 8, 18] 4

Ch.6 Introverted (low need for social interaction) [12-14, 19-22] 7
Ch.7 Need for involvement in personal goal setting [14] 1
Ch.8 Need for feedback (needs recognition) [14, 15] 2
Ch.9 Need for Geographic stability [3] 1
Ch.10 Need to make a contribution (needs worthwhile/

meaningful job)
[3, 4, 8] 3

Ch.11 Autonomous (need for independence) [3-5, 11, 13, 17, 22] 7
Ch.12 Need for variety [3, 5, 17, 23] 4
Ch.13 Marketable [5, 17] 2
Ch.14 Need for challenge [5, 11, 17, 20] 4
Ch.15 Creative [17, 24] 2
Ch.16 Need to be sociable/identify with

group/organisation/supportive relationships
[4, 8, 9, 17, 25] 5

Control Factors relate to an individual’s personality, their internal make-up and their strengths and weaknesses.

These control factors seem to determine the existence of various ‘raw’ characteristics.

Table 3 presents our literature review results for control factors.

TABLE 3: Software Engineer Controllers
Con: Controllers Paper references Frequency

(# of studies)
Con.1 Personality Traits ( e.g. introverted, thinking) [6, 10, 19, 24, 26-28] 7
Con.2 Career Paths (Managerial/Technical) [3, 4, 29] 3
Con.3 Competencies (how good they are at their job) [6, 11, 30, 31] 4

Table 3 shows there are many studies that report personality traits of Software Engineers. Our findings suggest

that an engineer’s personality, career path preference and competencies will control whether each of the 16

characteristics listed in Table 2 form part of his or her make-up.

Moderators are external factors that influence characteristics, for example, environmental conditions, type of

work and role are moderators. Our findings suggest that moderators change the strength of a particular Software

Engineer characteristic. Table 4 presents our results for moderators.

TABLE 4: Software Engineer Moderators
Mod: Moderators (context) Paper references Frequency (#

of studies)
Mod.1 Career stage (age & experience, e.g. Apprentice,

colleague, mentor, sponsor)
[6, 26, 32-37] 8

Mod.2 Culture (relating to different countries) [2, 5, 9, 13, 15, 20, 25,
38]

Mod.3 Job type/role/ occupational level [16, 20, 39, 40] 4
Mod.4 State of IT profession (a snap shot of an evolutionary

process)
[22, 37, 39, 41, 42] 5

Mod.5 Type of organisation (e.g. promotion
opportunities/rules) – relating to lifestyle

[4] 1

Finally, Table 4 shows that career stage and culture are often cited in the literature as moderating an engineer’s

characteristics. To a lesser extent the literature considers that the type of job and the type of organisation will

also moderate an engineer’s characteristics. This means that these factors are likely to moderate the strength of

each characteristic an individual software engineer has.

4.2 Research Question 2

62 papers answered Research Question 2, “What (de)motivates Software Engineers to be more (less)

productive?”

This section is divided into papers that identify Motivators; De-motivators; and Implementation Factors.

Implementation factors are issues that need to be considered when applying motivators and influence the

effectiveness of motivators. Table 5 summarises the frequencies of papers relating to motivators:

TABLE 5: RQ2 – Motivators in Software Engineering
Motivators References Frequency

(# of studies)
M.1 Rewards and incentives (e.g. scope for increased pay
and benefits linked to performance)

[7, 23, 36, 43-53] 14

M.2 Development needs addressed (e.g. training
opportunities to widen skills; opportunity to specialise)

[3, 7, 22, 25, 43, 44, 48, 49, 54-
56]

M.3 Variety of Work (e.g. making good use of skills, being
stretched)

[9-11, 25, 29, 37, 43, 44, 48, 50-
52, 55, 57]

M.4 Career Path (opportunity for advancement, promotion
prospect, career planning)

[3, 9, 11, 25, 29, 37, 43, 44, 47,
48, 50-52, 55, 57]

M.5 Empowerment/responsibility (where responsibility is
assigned to the person not the task)

[7, 11, 44, 54, 57, 58] 6

M.6 Good management (senior management support, team-
building, good communication)

[7, 10, 18, 22, 25, 37, 44, 46, 48,
49, 51, 53, 54, 56, 59, 60]

M.7 Sense of belonging/supportive relationships [8, 10, 21, 22, 25, 43-45, 49, 56,
61-64]

M.8 Work/life balance (flexibility in work times, caring
manager/employer, work location)

[4, 25, 43-45, 64, 65] 7

M.9 Working in successful company (e.g. financially stable) [4, 44] 2
M.10 Employee participation/involvement/working with
others

[23, 33, 43, 44, 49, 52, 54, 58, 60,
66, 67] [10, 25, 49, 63, 68]

M.11 Feedback [9, 10, 20, 23, 33, 37, 45, 56, 67,
69]

M.12 Recognition (for a high quality, good job done based
on objective criteria -different to M1 which is about making
sure that there are rewards available).

[7, 8, 10, 22, 23, 25, 46, 48, 49,
51, 54, 68]

M.13 Equity [52, 67, 70] 3
M.14 Trust/respect [8, 33, 58, 70] 4
M.15 Technically challenging work [11, 22, 42, 46, 48, 54, 59, 64, 65,

67, 68]
11

M.16 Job security/stable environment [23, 25, 43, 46-48, 50, 56, 59, 71] 10
M.17 Identify with the task (clear goals, personal interest,
know purpose of task, how it fits in with whole, job
satisfaction; producing identifiable piece of quality work)

[7-9, 11, 18, 20, 22, 23, 33, 47-51,
53, 54, 56, 67, 68, 72]

M.18 Autonomy (e.g. freedom to carry out tasks, allowing
roles to evolve)

[7, 9-11, 33, 56, 67-69] 9

M.19 Appropriate working conditions/environment/good
equipment/tools/physical space/quiet

[4, 7, 47, 64, 67, 73] 6

M.21 Making a contribution/task significance (degree to
which the job has a substantial impact on the lives or work of
other people)

[8, 9, 11, 33, 48, 61] 6

M.22 Sufficient resources [25, 48] 2

Table 5 shows that the most frequently cited motivators in the literature are, ‘the need to identify with the task’

such as having clear goals, a personal interest, understanding the purpose of task, how the task fits in with the

whole, having job satisfaction; and working on an identifiable piece of quality work. Having a clear career path

and a variety of tasks is also found motivating in several papers.

Table 6 lists the de-motivators found in the literature.

TABLE 6: RQ2 – De-Motivators in Software Engineering
De-Motivators References Frequency

(# of
studies)

D.1 Risk [1] 1
D.2 Stress [43, 66, 69, 74, 75] 5
D.3 Inequity (e.g. recognition based on management
intuition or personal preference)

[7, 43, 56, 66] 4

D.4 Interesting work going to other parties (e.g.
outsourcing)

[45] 1

D.5 Unfair reward system (e.g. Management rewarded
for organisational performance; company benefits
based on company rank not merit)

[7, 70] 2

D.6 Lack of promotion opportunities/stagnation/career
plateau/boring work/poor job fit

[37, 56, 61, 76, 77] 5

D.7 Poor communication (Feedback deficiency/loss of
direct contact with all levels of management)

[7, 13, 20, 39, 56] 5

D.8 Uncompetitive pay/poor pay/unpaid overtime [7, 13, 20, 56, 77, 78] 6
D.9 Unrealistic goals/ phoney deadlines [7, 13, 23, 42, 56, 77] 4
D.10 Bad relationship with users and colleagues [42, 51, 56, 74] 4
D.11 Poor working environment (e.g., wrong staffing
levels/unstable/insecure/lacking in investment and
resources; being physically separated from team)

[4, 7, 10, 22, 23, 56, 73, 74, 79] 9

D.12 Poor management (e.g. poorly conducted
meetings that are a waste of time)

[7, 20, 22, 23, 42, 47, 56, 79] 7

D.13 Producing poor quality software (no sense of
accomplishment)

[7, 23, 56] 3

D.14 Poor cultural fit/stereotyping/role ambiguity [42, 51, 63] 3
D.15 Lack of influence/not involved in decision
making/no voice

[23, 56] 2

Table 6 shows that poor working conditions and lack of resources are reported as de-motivating in 9 separate

studies.

TABLE 7: RQ2 – IMPLEMENTATION FACTORS
IMPLEMENTATION Factors References Frequency

(# of
studies)

IMP 1: Job Fit [12, 13, 15, 22, 37, 38, 63, 75, 80] 9
IMP 2: Tailoring practices [11, 43, 59, 62, 71, 75] 6
IMP 3: Long term/ short term strategies [43] 1
IMP 4: Temporal effects [1, 42, 56, 72, 78, 81] 6
IMP 5: Individual differences [5, 29, 55, 56, 67] 5

The literature reports that to implement the motivators noted in Table 5, factors listed in Table 7 need to be

considered. How the job fits with an individual’s needs is considered important in 9 separate studies. Here,

motivation is viewed as a function of the ‘fit’ between the individual and the organisational job setting. The

concept of ‘job-fit’ is detailed in the work of social scientist McClelland in 1975.

4.3 Research Question 3

18 papers were identified as answering Research Question 3, “RQ3: What are the external signs or outcomes of

(de)motivated Software Engineers?”

Table 8 lists the external signs associated with motivated or de-motivated software engineers, as identified in

these 18 papers.

TABLE 8: External signs of motivated and de-motivated software engineers

External signs References Frequency
(# of studies)

Ext1: Retention [21, 25, 32, 38, 43, 45, 50, 52, 57, 62, 66, 81] 12
Ext2: Project delivery time [16, 82] 2
Ext3: Productivity [6, 21, 58, 68, 73] 5
Ext4: Budgets [81] 1
Ext5: Absenteeism [50] 1
Ext6: Project Success [68] 1

The majority of the studies cited retention as the major outcome of motivated or de-motivated software

engineers. Twelve studies showed that motivated engineers tend to stay in their jobs longer than de-motivated

engineers. Five studies reported that productivity is affected by motivated/de-motivated engineers.

4.4 Research Question 4

Eighteen papers answered research question 4, “What aspects of Software Engineering (de)motivate Software

Engineers?”

Table 9 identifies themes based on (de)motivators relating to the software engineering activity itself. Factors

related to salary or other motivators extraneous to software engineering itself have not been included in this

analysis. This question is an offshoot of our research question 2 that takes a more general view of all motivators

found in software engineering.

TABLE 9: Aspects of Software Engineering that motivate Software Engineers

Motivating Aspects of software engineering field References # of studies
Asp1: Problem Solving (the process of understanding and solving a problem in
programming terms)

[10, 22, 83] 3

Asp2: Team Working [61, 84] 2
Asp3: Change [2, 11, 42, 85] 4
Asp4: Challenge (Software Engineering is a challenging profession and that in
itself is motivating)

[22, 42, 61,
65]

Asp5: Benefit (creating something to benefit others or enhances well-being) [10, 61, 83] 3
Asp6: Science (making observations, identifying, describing, engineering,
investigating and theorising, explaining a phenomena)

[77, 83] 2

Asp7: Experiment (trying something new, experimentation to gain experience) [55, 83] 2
Asp8: Development practices (Object Oriented, XP and prototyping practices) [85, 86] 2
Asp9: Software process/lifecycle – Software development, project initiation
and feasibility studies, *maintenance (*also found a de-motivating activity)

[77] 1

We found comparatively few studies that identified specific tasks that motivate software engineers. A key study

in this area is Alstrum (2003)/[83], who asked the question “What is the attraction to Computing? We have built

on some of the themes identified by Alstrum such as benefit, science and experiment.

Table 9.1 De-motivating Aspects of Software Engineering References # of studies
De-Asp1: Software process/lifecycle – maintenance (note that
maintenance was also found motivating under some conditions)

[12, 85] 2

Table 9.1 shows that only two studies considered what aspects of the lifecycle de-motivated software engineers,

both identified the maintenance task.

Table 9.2. Implementation Factors (as in table 7) References # of studies
IMP1. Job-Fit [15, 20, 37, 82, 85] 5

Similar to our findings relating to research question 2; Table 9.2 highlights that 5 studies found that the degree

that an engineer finds aspects of the software engineering job motivating on de-motivating will depend on his or

her personal job-fit.

4.5 Research Question 5

17 papers were identified as answering Research Question 5, “What models of motivation exist in Software

Engineering?”

We searched for models that try to explain how motivation works or why motivation works the way it does. A

breakdown of the themes we identified in the literature is presented in Table 10.

TABLE 10: Models of Motivation in Software Engineering

Explicit Models of motivation References Frequency
(# of studies)

Mod1: Job Characteristics Theory Model (JCT) of Software
Engineer (SE) Motivation
(development, enhancement or validation)

[14, 15, 89-91] 5

Mod2: Models of leadership influence on SE motivation [7, 59, 91] 3
Mod3: Models of Open Source Developer SE Motivation [59, 87, 88] 3
Mod4: Model of Task Design influence on SE motivation [67] 1
Mod5: Model of Career Progression influence SE on
motivation

[60] 1

Implicit Models of motivation

Rel1: Models focussing on Software Engineer Job Satisfaction [50, 52, 53, 56, 62] 5

Rel2: Model drawing on expectancy theory, goal-setting
theory, and organizational behaviour specific to the software
development process

[92] 1

Rel3: Social support influence on Software Engineer
turnover

[62] 1

The literature presents a disparate set of models that are mostly hypothesised from theoretical studies and

validated through empirical surveys. A commonly-cited model of motivation is the Job Characteristics Theory

(JCT) Model (Hackman and Oldham 1976). The basic tenet of the JCT is that Software Engineers will

experience internal motivation and satisfaction if their Growth Need Strength’s (GNS) are matched by the

Motivational Potential Score (MPS) of the jobs they do. This implies that Software Engineers with low GNS

will be satisfied with low MPS in a job, in much the same way as those with high GNS will need high MPS in a

job. Optimum internal motivation and satisfaction is achieved when Software Engineers’ GNS’s are matched

with the appropriate MPS’s in a job.

Five papers in our review explicitly build upon the JCT to extend it (e.g. with role ambiguity/conflict and

leadership styles), validate it using comparisons with countries outside the USA, such as Japan, or enhance it,

e.g. looking at employment fit (which is similar to job fit, but includes working conditions). A further five

papers presented models that focus on job satisfaction, which is an element of the JCT and is therefore linked to

motivation. For example, [56] suggest that managerial, team member or self-control of tasks influences the level

of job satisfaction felt by an employee.

Three papers explicitly investigate the motivation of open source developers. [87] considers whether two social

science models (one focusing on voluntary action and one focusing on small teams) adequately explain OSS

developers’ motivation. [59] focuses on leadership styles, and [88] investigates the relationship between

intrinsic, extrinsic and internalised extrinsic motivators.

Two papers focus on job or employment fit of some kind. For example, [89] focuses on validating an instrument

to measure employment arrangement fit. This reflects findings from other literature identified under RQ1 where

the influence of career anchor is highlighted. In addition, RQ4 identifies job fit as being a (de)motivator for SEs.

5. DISCUSSION

In this section we discuss how the literature in our systematic literature review assists us to understand the

underlying constructs of motivation in software engineering. Figure 9 shows our enhanced understanding of our

research topics introduced in Figure 6.

5.1 Software engineers as a homogeneous occupational group

Figure 9 shows that the literature is divided as to whether Software Engineers form a distinct occupational

group. However, the majority of included studies support the idea that these practitioners do form a recognisable

group with similar needs. This view is consolidated in the many studies from Couger, Zawacki and colleagues,

e.g. (Couger and Zawacki 1978; Couger and Zawacki 1980; Dittrich et al. 1985; Couger and McIntyre 1987;

Couger and McIntyre 1987-1978; Couger 1988; Couger and Adelsberger 1988; Couger 1989; Couger et al.

1990; Burn et al. 1992; Couger 1992; Couger and Ishikawa 1995) based on a comparison of job perceptions and

needs of more than 6000 people from both Software Engineers [Data Processors/IT professionals] and the

general public. These studies reported that Software Engineers found their work less meaningful and rated their

jobs less favourably than other professionals. Their need to interact with others was negligible. Software

Engineers displayed very high growth needs and were concerned about learning new technology. Myers (1992)

refined the studies of Couger and Zawacki and colleagues to show that although Software Engineers formed a

distinct group, they varied among themselves by job type. More recent work that presents Software Engineers as

a distinct group include: (Kandeel and Wahba 2001; Capretz 2003; Garza et al. 2003; Tanner 2003; Darcy and

Ma 2005; Ramachandran and Rao 2006).

Key Results Definitions
RQ1: What are the characteristics of Software Engineers?
RQ2: What (de)motivates Software Engineers to be more (less) productive?
RQ3: What are the external signs or outcomes of (de)motivated Software

Engineers?
RQ4: What aspects of Software Engineering (de)motivate Software Engineers?
RQ5: What models of motivation exist in Software Engineering?

Table 2-4
Table 5-7
Table 8

Table 9
Table 10

Extrinsic factors: External to the job practitioners do, e.g. work conditions.
These factors just maintain practitioners in their jobs (Baddoo and Hall 2002)
Intrinsic factors: Primary determinant of motivation and satisfaction. Related
to Job itself, e.g. work itself, recognition, achievement (Baddoo and Hall 2002)
Personality: more or less permanent characteristics of an individual’s
state of being, (e.g. shy, extrovert, conscientious) (Chelsom, 2005).

Figure 9: Model of Motivation in Software Engineering

+/-
SW Engineer
Characteristics

Individual
personality
profiles

Context (job
type/culture) A distinct group

Not a distinct group

mediates

RQ2
General
Motivators
(extrinsic and
intrinsic)
in the SW Eng
Literature

RQ4
Motivators/job
satisfiers
Specific to SW
Eng (activity)

RQ3
Outcome
e.g. more/less
absenteeism, job
retention,
increased/decreased
quality.

+/-

RQ1

RQ5

controls

However, several studies take a contrary view. For example, Ferratt and Short (1986; 1988) found that

Software Engineering employees and non-Software Engineering Employees [IS and non-IS employees] could

be motivated equally using the same underlying constructs. Im and Hartland (1990) although disputing Ferratt

and Short’s (1986) methodology, supported their outcome. More recent work that presents Software

Engineers as a group that cannot be distinguished from other occupations when considering motivation

include (Enns et al. 2006; Smith and Speight 2006).

These mixed findings from the 1980’s to today lead us to conclude that whether or not software engineers

form a homogeneous group with similar motivational needs depends on their individual context.

5.2 RQ1 – Software Engineer characteristics

The 43 papers that cover this question provide us with a broad picture of Software Engineer characteristics.

As these characteristics are based on studies from different countries, practitioner roles, personality types,

organisations, development processes and historical periods, we cannot assume that every characteristic

relates to all software engineers. In fact is it clear that this would not be feasible, since some of the

characteristics contradict each other. For example, engineers are seen to be sociable yet introverted, needing

stability on the one hand and liking a variety of new tasks and challenges on the other. Therefore, to apply

these characteristics to any individual we have extracted implicit findings from the papers and identified two

new categories: ‘moderators’ (environmental and demographic influences) and controllers (internal

constructs).

We do not report on the cognitive aspects of a Software Engineer’s personality which go beyond the scope of

this study. However, we note that cognitive processes and personality traits need to be considered and

understood as these internal constructs will determine an individual’s set of characteristics. As Chelsom et al

(2005) note, the differences in people’s personality are greater than the similarities, and “we cannot ignore the

significance of individuality”. The literature also shows that external factors such as career stage and culture

need to be considered as these will ‘moderate’ the strength of each characteristic.

The characteristics cited most often in the literature are the need for growth and independence. The need for

growth may be due to the engineer’s internal make up, and/or the need be ‘marketable’ (another

characteristic) and keep up with the fast changing technology. The need for independence is possibly linked to

the type of person attracted to software engineering that is sometimes seen as a creative task that is not helped

by overbearing management.

Many of the characteristics we identify reflect the findings and views of Couger and Zawacki (1980). This is

not surprising as their job diagnostics survey for data processing personnel (JDS/DP) has been used in several

of the papers included in our literature review, e.g ((Couger and McIntyre 1987; Couger and McIntyre 1987-

1978; Couger 1988; Couger and Adelsberger 1988; Couger 1989; Couger et al. 1990; Couger 1992; Couger

and Ishikawa 1995).

We have extracted from the literature a more structured view of the findings concerning SEs characteristics,

noting that the characteristics of any one individual depend on controllers, such as personality trait and

moderators such as career stage.

5.3 RQ2 – What motivates Software Engineers?

The 62 papers that answer this question create a list of 22 different motivators. The most frequently cited

motivators in the literature are, ‘the need to identify with the task’ such as having clear goals, a personal

interest, understanding the purpose of a task, how it fits in with the whole, having job satisfaction; and

working on an identifiable piece of quality work. Having a clear career path and a variety of tasks is also

found motivating in several papers. The literature suggests it is important to involve the engineer in decision

making, and to participate and work with others, which appears to go against characteristics of independence

and introversion which are cited in many papers. When looking at what Software Engineering activities

motivate Software Engineers we need to consider that some of the findings might not apply today. For

example, we have listed Object Oriented Design as a motivator that is reported as meeting a growth need in

engineers. However, as this finding was reported in the 1990s, it may be that this fulfilled a growth need only

because Object Oriented Design was a new skill at the time –this may no longer apply today. This is just one

example of how a motivator may be context specific, relating to time, role, culture, experience, age, individual

characteristic etc.

An aspect of Software Engineering found both motivating and de-motivating is the maintenance task. This

could be due to several factors. For example this evolutionary phase of software development can consume

between 40 – 80% of software costs. If it is the dominant activity within a group, then it may attract the

recognition and challenges associated with motivation. Alternatively, as 60% of maintenance tasks are in fact

enhancements (Glass 2003) this might also be regarded as problem-solving and challenging. Finally, bug-

fixing may be regarded as motivating if the right person is given the job, i.e. the job-fit is right.

5.3.1 Software Engineer De-motivators

To give a balanced view, we also recorded what the literature reports on Software Engineer de-motivators.

Working conditions and lack of resources are reported as de-motivating in 9 separate studies. These are

classed as hygiene factors by Herzberg et al (1959), who developed the hygiene-motivator theory in the

1950s. This theory asserts that removing the de-motivator will not necessarily translate to motivating

employees. It will simply maintain practitioners in their job and avoid dissatisfaction. Salary or rewards are an

exception to this rule; a good salary can be motivating in unstable environments and early in an engineer’s

career, although salary is usually considered a hygiene factor.

5.3.2 Motivating and de-motivating factors

Finding a factor to be both motivating and de-motivating might be due the temporal effects of motivation as

highlighted by Maslow’s (1954) hierarchy of needs theory. What might motivate someone in the early stages

of their career may end up de-motivating them in the latter stages of their career. For example, the newly

recruited Software Engineer could be highly motivated by job security and close supervision, whereas these

same factors, especially close supervision, could turn out to be de-motivating to a seasoned Software

Engineer. An experienced Software Engineer is more likely to be motivated by challenges, opportunities for

recognition and autonomy.

5.4 RQ3 – The outcomes of motivating Software Engineers

Considering the large body of work on motivation, very little work covers the tangible benefits or outcomes of

motivating engineers. Eighteen studies were found in this category (RQ3), where most dealt with turnover and

absenteeism. Turnover and absenteeism focus on the likelihood of an individual staying in a particular job. As

measures of motivation therefore they suffer from being a management and organisational view of motivation,

i.e. they consider motivation from an organisation’s perspective. We found little work that focused on

understanding or measuring an individual’s motivation to stay in Software Engineering as a profession.

Also, very few studies considered productivity improvements or increase in quality. This is possibly due to

the difficulty in measuring motivation and associating motivation with actual output. Also the themes of

turnover and absenteeism are part of the JDS/DP (Couger and Zawacki 1980) – a survey used by many of the

studies included in this review.

5.5 RQ4 – What is motivating about Software Engineering

Although this review takes in a large range of studies relating to Software Engineering, only a very small

proportion identify what is specifically motivating about this field. When looking at answers to RQ2 (Table 5)

for instance, most of the motivators could apply to many professions.

The literature identifies Software Engineering as a challenging profession and often links challenge to change,

as noted in (Almstrum 2003) “the reason for challenge is the pace of change of the field and the effort it took

to keep pace with the changes. ..If you just want to learn something and do it for the rest of your life …. you

don’t want to go into IT”. Challenge also relates to ‘technical’ challenges (not just coping with change).

Learning, exploring new techniques and problem solving would also appear to be motivating tasks,. ‘Benefit’

is a category identified by Alstrum (2003) and is supported in the work of (Hertel et al. 2003; Roberts et al.

2004; Li et al. 2006), where the three different studies show Software Engineers are motivated by “creating

something that will benefit others”; “the usefulness in supporting other areas/fields”; and creating something

that is “of value to the user”.

As observed above, we found little work that explicitly focused on Software Engineering as a profession, and

hence considering why Software Engineers remain in Software Engineering (even if they change jobs).

5.6 RQ5 – Modelling motivation in Software Engineering

We aimed to synthesise the findings on how motivation in software engineering is modelled in the literature.

However, we found it very difficult to combine all the models as they tend to cover general aspects of

motivation, have few commonalities and only partially cover the Software Engineering domain. exception to

this is found in the recurring theme of models based on the Job Characteristics Theory (JCT) (Hackman and

Oldman 1976) and the JDS/DP (Couger and Zawacki, 1980).

The results from RQ5 though difficult to assimilate fall into one of three camps:

those that use and adapt the JCT model and the JDS/DP e.g. to add leadership considerations,

those that try to provide an alternative to the JCT approach, and

those that take a totally different approach (e.g. using small-team theories to explain Open Source

Development )

According to Couger and Zawacki (1980), the JCT (Hackman and Oldman 1976) was found useful for

management in Software Engineering to analyse individual patterns of motivation. Couger and Zawacki

augmented the underlying constructs of this model in the Job Diagnostics Survey for Data Processing

Personal (JDS/DP) to provide a richer picture of how growth need strength (GNS) relates to Motivation

Potential Score (MPS) in a given job.

Literature that uses the JDS/DP generally aims to validate the theory in different national cultural contexts and

often uses the USA as the benchmark. It is helpful for cross-comparisons to use the same instrument with

other professions and between and within given roles, showing the strength of feeling for certain needs and

identifying differences and similarities. However, the JDS/DP comprises a tick list of factors, and so studies

based on the JDS/DP will only be able to comment on motivating factors contained within the instrument

rather than unearth any new motivators or emerging trends. The nature of the Software Engineer’s job has

changed considerably since the JDS/DP was first devised, and so it is questionable as to whether or not it is as

applicable as it used to be.

We have not found a definitive model of motivation in Software Engineering that adequately captures the

motivators and de-motivators we found in answer to RQ4, “What aspects of Software Engineering

(de)motivate Software Engineers?”, nor the other facets of Software Engineer characteristics and motivation

reported through RQs1-3.

6. LIMITATIONS

6.1 Completeness

We have conducted a very thorough review of the literature eliciting work from 70 different authors including

some secondary studies (where we used the reference in the primary study to lead to another study). We note

however that with the increasing number of works in this area we cannot guarantee to have captured all the

material in this area.

Another area of concern is that few studies have been published on motivation in Software Engineering in

countries such as India that are increasingly involved in Software Engineering (Yourdon 2005), suggesting

that we cannot present a global view of this area. This is not a limitation of our approach, but a reflection of

the limitations imposed on us by the available research in this area.

6.2 Data synthesis

As we have covered different countries and eras in Software Engineering we have grouped all Software

Engineer roles together. Some studies have found that different roles are associated with different

motivational needs and characteristics. By grouping all roles together, we may have lost some of this detail.

7. CONCLUSIONS

Our findings suggest an increasing awareness of motivation in Software Engineering since about 1995, as

compared to the previous 15 years. Most of the studies in this area rely on the use of questionnaires, with 16%

using multiple data collection methods and only 1% using multiple methods without questionnaire. Over half

of the studies (54) were conducted in the USA. In addition, the majority of papers were published in the

Proceedings of SIGCPR Computer Personnel Research rather than mainstream software engineering

conferences or journals. Notwithstanding this, the 92 papers in our systematic literature review provide a

broad understanding of the research conducted into what has motivated Software Engineers in 16 different

countries over the past 26 years.

Mixed findings in the literature lead us to conclude that whether software engineers form a homogeneous

group with similar needs depends on their individual context. Building on the work reported, we have

structured the SE characteristics investigated in the literature into three related categories: ‘raw’

characteristics, moderators and controllers. Whether or not an individual has a particular characteristic

depends on certain controllers, and how strong this characteristic is depends on the moderators.

The literature cites 22 different motivators for Software Engineers. The most frequently cited ones are, ‘the

need to identify with the task’ such as having clear goals, a personal interest, understanding the purpose of a

task, how it fits in with the whole, having job satisfaction; and working on an identifiable piece of quality

work. However some factors are identified as being both motivators and de-motivators. It may be possible to

account for this by considering the career stage of the individual.

Turnover and absenteeism are the most cited outcomes of (de)motivated engineers (maybe because these are

mentioned in the JDS/DP). We found little work that focused on understanding or measuring an individual’s

motivation to stay in Software Engineering as a profession.

Learning, exploring new techniques and problem solving appear to be motivating aspects of SE. However

little work has focused on the specific nature of software engineering itself, or of the impact of the changing

environment in which software engineering is conducted.

Although we found a variety of models of motivation in Software Engineering in the literature, no model

considered all the identified factors in our list of motivators, moderators, controllers and implementers.

Neither did any of the models focus on the nature of the SE’s job itself such as the reliance on tools or

programming languages, the logical nature of problem solving, use of creativity, complex problem-solving,

and so on. Yet ‘the job itself’ continues to be the principal motivator. Therefore, considering the changes in

what the job demands, in terms of new skills and communicating with many different stakeholders, there

appears to be a gap in defining what exactly it is about ‘the job’ that motivates engineers.

It is clear from the literature that there is a need for a comprehensive model of motivation in Software

Engineering that includes what is particularly motivating about the job itself. We also need a better way to

measure motivation, as basing it on turnover only reflects whether an engineer is motivated to stay in an

organisation. It does not shed light on what motivates an individual to stay in the SE profession, to produce

better quality software, increase productivity, and use and share skills in the wider Software Engineering

community.

ACKNOWLEDGMENTS

We thank Dorota Jagielska for helping us pilot this study and David Clover of The Open University for

setting up a collaborative website for the project group.

This research was supported by the UK’s Engineering and Physical Science Research Council, under grant

number EPSRC EP/D057272/1.

REFERENCES

BADDOO, N. and HALL, T. (2002). Motivators of Software Process Improvement: An analysis of practitioners’ views, Journal of
Systems and Software 62 (2): 85-96. Elsevier Science Inc.

BARTOL, K. M. and MARTIN, D. C. (1982). Managing Information Systems Personnel: A Review of the Literature and Managerial
Implications, MIS Quarterly 6 (Special Issue): 49-70.

BEECHAM, S., BADDOO, N., et al. (2006). Protocol of a Systematic Literature Review of Motivation in Software Engineering,
Technical Report No. 452 School of Computer Science, Faculty of Engineering and Information Sciences, University of
Hertfordshire.

BOEHM, B. W. (1981). Software Engineering Economics. Englewood Cliffs, Prentice-Hall, Inc.
BURN, J. M., COUGER, J. D., et al. (1992). Motivating IT professionals. The Hong Kong challenge Information & Management 22 (5):

269-280.
CAPRETZ, L. F. (2003). Personality types in software engineering, International Journal of Human Computer Studies 58 (2): 207-214.

Academic Press.
COUGER, D. J. and MCINTYRE, S. C. (1987-1978). Motivation Norms of Knowledge Engineers compared to those of Software

Engineers, Journal of Management Information Systems 4 (3): 82-93.
COUGER, D. J. and ZAWACKI, R. A. (1980). Motivating and Managing Computer Personnel, John Wiley & Sons.
COUGER, J. D. (1988). Motivators vs. demotivators in the IS environment, Journal of Systems Management 39 (6): 36-41.
COUGER, J. D. (1989). Comparison of motivating environments for programmer/analysts and programmers in the US, Israel and

Singapore, System Sciences, 1989. Vol.IV: Emerging Technologies and Applications Track, Proceedings of the Twenty-Second
Annual Hawaii International Conference on 4: 316-323 vol.4.

COUGER, J. D. (1992). Comparison of motivation norms for programmer /analysts in the Pacific Rim and the U.S International Journal
of Information Systems 1 (3): 16-30.

COUGER, J. D. and ADELSBERGER, H. (1988). Environments: Austria compared to the United States, SIGCPR Comput. Pers. 11 (4):
13-17. ACM Press.

COUGER, J. D., ADELSBERGER, H., et al. (1990). Commonalities in motivating environments for programmer/analysts in Austria,
Israel, Singapore, and the U.S.A, Information & Management 18 (1): 41-46.

COUGER, J. D. and ISHIKAWA, A. (1995). Comparing motivation of Japanese computer personnel versus these of the United States,
System Sciences, 1995. Vol. IV. Proceedings of the Twenty-Eighth Hawaii International Conference on 4: 1012-1019 vol.4.

COUGER, J. D. and MCINTYRE, S. C. (1987). Motivating norms for artifical intelligence personnel, Proceedings of the Twentieth
Hawaii International Conference on System Sciences 1987. Hawaii Int. Conference Syst. Sci. 1987: 370-4 vol.

COUGER, J. D. and ZAWACKI, R. A. (1978). What motivates DP professionals?, Datamation 24 (9): 116.
DARCY, D. P. and MA, M. (2005). Exploring Individual Characteristics and Programming Performance: Implications for Programmer

Selection, International Conference on System Sciences, 2005. HICSS ’05. Proceedings of the 38th Annual Hawaii (03-06
Jan. 2005): 314a-314a. University of Maryland, College Park.

DEMARCO, T. and LISTER, T. (1999). Peopleware – Productive Projects And Teams, Dorset House.
DITTRICH, J. E., DANIEL COUGER, J., et al. (1985). Perceptions of equity, job satisfaction, and intention to quit among data

processing personnel, Information & Management 9 (2): 67-75.
ENNS, H. G., FERRATT, T. W., et al. (2006). Beyond Stereotypes of IT Professionals: Implications for IT HR Practices,

COMMUNICATIONS OF THE ACM 49 (4): 106-109.
FERRATT, T. W. and SHORT, L. E. (1986). Are information systems people different: an investigation of motivational differences,

Management Information Systems Quarterly 10 (4): 377-87.
FERRATT, T. W. and SHORT, L. E. (1988). Are information systems people different? An investigation of how they are and should be

managed, Management Information Systems Quarterly 12 (3): 427-43.
GARZA, A. I., LUNCE, S. E., et al. (2003). Career anchors of Hispanic information systems professionals, Proceedings – Annual

Meeting of the Decision Sciences Institute: 1067-1072. Decision Sciences Institute.

HERZBERG, F., MAUSNER, B., et al. (1959). The Motivation to Work, 2nd Ed. London, Chapman & Hall.
IM, J. H. and HARTMAN, S. (1990). Rethinking the issue of whether IS people are different from non-IS people, MIS Quarterly 14 (1):

1-2. JSTOR.
KANDEEL, H. and WAHBA, K. (2001). Competency models for human resource development: case of Egyptian software industry,

Managing Information Technology in a Global Environment. 2001 Information Resources Management Association
International Conference . Idea Group Publishing. 2001: 117-21.

KITCHENHAM, B. (2004). Procedures for Performing Systematic Reviews, Keele University and National ICT Australia Ltd.: 1 – 28.
KITCHENHAM, B., MENDES, E., et al. (2006). A Systematic Review of Cross- vs. Within-Company Cost Estimation Studies, EASE

2006 10th International Conference on Evaluation and Assessment in Software Engineering: 89-98. Keele University,
Staffordshire, UK.

MCCLELLAND, D.C. (1975). Power: The inner experience. New York: Irvington Press
MCCONNELL, S. (1998). Problem programmers, Software, IEEE 15 (2): 128, 127, 126.
MOMSE(CFS) (2005). Modelling motivation in software engineering Case for Support. (EPSRC Proposal, EPSRC Reference:

EP/D057272/1).
MYERS, M. E. (1992). The information systems profession and the information systems professional – fit or misfit? Proceedings of the

1992 ACM SIGCPR conference on computer personnel research Cincinnati, Ohio: 350-351 (Ed. Al Lederer).
PROCACCINO, J. D., VERNER, J. M., et al. (2005). What do software practitioners really think about project success: An exploratory

study, Journal of Systems and Software 78 (2): 194-203. Elsevier Inc., New York, NY 10010, United States.
PROJECTLINK. (2006). “Motivation House.” http://www.projectlink.co.uk/whoweworkfor.htm, accessed 12.5.2006.
RAMACHANDRAN, S. and RAO, S. V. (2006). An effort towards identifying occupational culture among information systems

professionals, Proceedings of the 2006 ACM SIGMIS CPR conference on computer personnel research: Forty four years of
computer personnel research: achievements, challenges & the future. Claremont, California, USA: 198-204. ACM Press.

SCHEIN, E. H. (1996). Career anchors revisited: Implications for career development in the 21st century. Academy of Management
Executive (Vol) (10) 4 pp 80-88

SMITH, D. C. and SPEIGHT, H. L. (2006). Antecedents of turnover intention and actual turnover among information systems personnel
in South Africa Proceedings of the 2006 ACM SIGMIS CPR conference on computer personnel research: Forty four years of
computer personnel research: achievements, challenges \& the future . Claremont, California, USA: 123-129. ACM Press.

STANDISH REPORT. (1995). “Standish Group Chaos Report.” from URL http://www.scs.carleton.ca/~beau/PM/Standish-Report.html.
TANNER, F. R. (2003). On motivating engineers, Engineering Management Conference, 2003. IEMC ’03. Managing Technologically

Driven Organizations: The Human Side of Innovation and Change: 214-218.
WYNEKOOP, J. L. and WALZ, D. B. (1998). Revisiting the perennial Question: Are IS People Different?, The Database for Advances

in Information Systems 29 (2): 62-72.

APPENDIX 1

Numeric References for the 92 studies included in Systematic Literature Review (Section 4)

1. Agarwal, R., P. De, and T.W. Ferratt, Explaining an IT professional’s preferred employment duration: Empirical tests of a causal
model of antecedents. Proceedings of the ACM SIGCPR Conference, 2002: p. 14-24.

2. Burn, J.M., L.C. Ma, and E.M. Ng Tye, Managing IT professionals in a global environment SIGCPR Comput. Pers, 1995. 16(3):
p. 11-19.

3. Crepeau, R.G., et al., Career Anchors of Information Systems Personnel. Journal of Management Information Systems, 1992.
9(2): p. 145-160.

4. Garza, A.I., S.E. Lunce, and B. Maniam, Career anchors of Hispanic information systems professionals. Proceedings – Annual
Meeting of the Decision Sciences Institute, 2003: p. 1067-1072.

5. Ituma, A., The internal career: an explorative study of the career anchors of information technology workers in Nigeria
Proceedings of the 2006 ACM SIGMIS CPR conference on computer personnel research: Forty four years of computer personnel
research: achievements, challenges & the future. Claremont, California, USA, 2006: p. 205-212.

6. Darcy, D.P. and M. Ma, Exploring Individual Characteristics and Programming Performance: Implications for Programmer
Selection. IEEE International Conference on System Sciences, 2005. HICSS ’05. Proceedings of the 38th Annual Hawaii (03-06
Jan. 2005), 2005: p. 314a-314a.

7. Frangos, S.A., Motivated humans for reliable software products. Microprocessors and Microsystems, 1997. 21(10): p. 605-610.

8. Ferratt, T.W. and L.E. Short, Are information systems people different: an investigation of motivational differences.
Management Information Systems Quarterly, 1986. 10(4): p. 377-87.

9. Burn, J.M., J.D. Couger, and L. Ma, Motivating IT professionals. The Hong Kong challenge Information & Management, 1992.
22(5): p. 269-280.

10. Linberg, K.R., Software developer perceptions about software project failure: a case study. Journal of Systems and Software,
1999. 49(2-3): p. 177-92.

11. Smits, S.J., E.R. McLean, and J.R. Tanner, Managing high achieving information systems professionals. Proceedings of the 1992
ACM SIGCPR Conference on Computer Personnel Research (Cincinnati, Ohio, United States, April 05 – 07, 1992). A. L.
Lederer, Ed. SIGCPR ’92., 1992: p. 314-327.

http://www.projectlink.co.uk/whoweworkfor.htm

http://www.scs.carleton.ca/~beau/PM/Standish-Report.html

12. Couger, J.D., Comparison of motivating environments for programmer/analysts and programmers in the US, Israel and
Singapore. System Sciences, 1989. Vol.IV: Emerging Technologies and Applications Track, Proceedings of the Twenty-Second
Annual Hawaii International Conference on, 1989. 4: p. 316-323 vol.4.

13. Couger, J.D. and H. Adelsberger, Environments: Austria compared to the United States. SIGCPR Comput. Pers., 1988. 11(4): p.
13-17.

14. Couger, J.D., et al., Commonalities in motivating environments for programmer/analysts in Austria, Israel, Singapore, and the
U.S.A. Information & Management, 1990. 18(1): p. 41-46.

15. Couger, J.D. and A. Ishikawa, Comparing motivation of Japanese computer personnel versus these of the United States. System
Sciences, 1995. Vol. IV. Proceedings of the Twenty-Eighth Hawaii International Conference on, 1995. 4: p. 1012-1019 vol.4.

16. Couger, J.D. and S.C. McIntyre, Motivating norms for artifical intelligence personnel. Proceedings of the Twentieth Hawaii
International Conference on System Sciences 1987. Hawaii Int. Conference Syst. Sci. 1987, 1987: p. 370-4 vol.

17. Sumner, M., S. Yager, and D. Franke, Career orientation and organizational commitment of IT personnel. Proceedings of the
2005 ACM SIGMIS CPR Conference on Computer Personnel Research (Atlanta, Georgia, USA, April 14 – 16, 2005). SIGMIS
CPR ’05, 2005: p. 75-80.

18. Mata-Toledo, R.A. and E.A. Unger, Another look at motivating data processing professionals SIGCPR Comput. Pers. , 1985.
10(1): p. 1-7.

19. Capretz, L.F., Personality types in software engineering. International Journal of Human Computer Studies, 2003. 58(2): p. 207-
214.

20. Khalil, O.E.M., et al., What motivates Egyptian IS managers and personnel: Some preliminary results. Proceedings of the ACM
SIGCPR Conference, 1997: p. 187-192.

21. Kym, H. and W.-W. Park, Effect of cultural fit/misfit on the productivity and turnover of is personnel. Proceedings of the 1992
ACM SIGCPR conference on Computer personnel research 1992: p. 184-190.

22. Tanner, F.R., On motivating engineers. Engineering Management Conference, 2003. IEMC ’03. Managing Technologically
Driven Organizations: The Human Side of Innovation and Change, 2003: p. 214-218.

23. Peters, L., Managing software professionals. IEMC ’03 Proceedings. Managing Technologically Driven Organizations: The
Human Side of Innovation and Change (IEEE Cat. No.03CH37502). IEEE. 2003, 2003: p. 61-6.

24. Wynekoop, J.L. and D.B. Walz, Revisiting the perennial Question: Are IS People Different? The Database for Advances in
Information Systems, 1998. 29(2): p. 62-72.

25. Jordan, E. and A.M. Whiteley, HRM practices in information technology management Proceedings of computer personnel
research conference (SIGCPR) on Reinventing IS : managing information technology in changing organizations: managing
information technology in changing organizations. Alexandria, Virginia, United States 1994: p. 57 – 64.

26. Enns, H.G., T.W. Ferratt, and J. Prasad, Beyond Stereotypes of IT Professionals: Implications for IT HR Practices.
COMMUNICATIONS OF THE ACM, 2006. 49(4): p. 106-109.

27. Moore, J.E., Personality characteristics of information systems professionals. Proceedings of the 1991 conference on SIGCPR,
1991: p. 140.

28. Miller, W.C., J.D. Couger, and L.F. Higgins, Comparing innovation styles profile of IS personnel to other occupations. IEEE
International Conference on System Sciences, 1993. HICSS ’93. Proceedings of the 26th Annual Hawaii 1993. iv: p. 378-386.

29. Igbaria, M., G. Meredith, and D.C. Smith, Career orientations of information systems employees in South Africa. The Journal of
Strategic Information Systems, 1995. 4(4): p. 319-340.

30. Kandeel, H. and K. Wahba, Competency models for human resource development: case of Egyptian software industry. Managing
Information Technology in a Global Environment. 2001 Information Resources Management Association International
Conference . Idea Group Publishing. 2001, 2001: p. 117-21.

31. Turley, R.T. and J.M. Bieman, Competencies of exceptional and nonexceptional software engineers. Journal of Systems and
Software, 1995. 28(1): p. 19-38.

32. Agarwal, R. and T.W. Ferratt, Retention and the career motives of IT professionals. Proceedings of the ACM SIGCPR
Conference, 2000: p. 158-166.

33. Cheney, P.H., Effects of Individual Characteristics, Organizational Factors and Task Characteristics on Computer Programmer
Productivity and Job Satisfaction. Information & Management, 1984. 7(4): p. 209-214.

34. Crook, C.W., R.G. Crepeau, and M.E. McMurtrey, Utilization of the career anchor/career orientation constructs for management
of I/S professionals. . SIGCPR Comput. Pers., 1991. 13(2): p. 12-23.

35. Goldstein, D.K., An updated measure of supervisor-rated job performance for programmer/analysis. Proceedings of the ACM
SIGCPR Conference on Management of information Systems Personnel (College park, Maryland, United States, April 07 – 08,
1988), 1988: p. 148-152.

36. Hsu, M.K., et al., Career satisfaction for managerial and technical anchored IS personnel in later career stages SIGMIS Database
2003. 34(4): p. 64-72.

37. Zawacki, R.A., Motivating the IS people of the future. Information systems management (Inf. syst. manage.), 1992. 9(2): p. 73-
75.

38. Smith, D.C. and H.L. Speight, Antecedents of turnover intention and actual turnover among information systems personnel in
South Africa Proceedings of the 2006 ACM SIGMIS CPR conference on computer personnel research: Forty four years of
computer personnel research: achievements, challenges \& the future . Claremont, California, USA, 2006: p. 123-129.

39. Couger, D.J. and S.C. McIntyre, Motivation Norms of Knowledge Engineers compared to those of Software Engineers. Journal
of Management Information Systems, 1987-1978. 4(3): p. 82-93.

40. Im, J.H. and S. Hartman, Rethinking the issue of whether IS people are different from non-IS people. MIS Quarterly, 1990.
14(1): p. 1-2.

41. Myers, M.E., Motivation and performance in the information systems field: a survey of related studies SIGCPR Comput. Pers.,
1991. 13(3): p. 44-49.

42. Ramachandran, S. and S.V. Rao, An effort towards identifying occupational culture among information systems professionals.
Proceedings of the 2006 ACM SIGMIS CPR conference on computer personnel research: Forty four years of computer personnel
research: achievements, challenges & the future. Claremont, California, USA, 2006: p. 198-204.

43. Agarwal, R. and T.W. Ferratt, Crafting an HR strategy to meet the need for IT workers. Communications of the ACM, 2001.
44(7): p. 58-64.

44. Agarwal, R. and T.W. Ferratt, Recruiting, retaining, and developing IT professionals: an empirically derived taxonomy of human
resource practices. Proceedings of the ACM SIGCPR Conference, 1998: p. 292-302.

45. Agarwal, R. and T.W. Ferratt, Enduring practices for managing IT professionals. Communications of the ACM, 2002. 45(9): p.
73-79.

46. Baddoo, N., T. Hall, and D. Jagielska, Software developer motivation in a high maturity company: a case study. Software
Process: Improvement and Practice, 2006. 11(3): p. 219-228.

47. Burn, J.M., et al., Job expectations of IS professionals in Hong Kong. Proceedings of the 1994 ACM SIGCPR Computer
Personnel Research Conference on Reinventing IS : Managing information Technology in Changing Organizations: Managing
information Technology in Changing Organizations (Alexandria, Virginia, United States, March 24 – 26, 1994: p. 231-241.

48. Garden, A., Maintaining the spirit of excitement in growing companies. SIGCPR Comput. Pers., 1988. 11(4): p. 10-12.

49. Klenke, K. and K.-A. Kievit, Predictors of leadership style, organizational commitment and turnover of information systems
professionals. In Proceedings of the 1992 ACM SIGCPR Conference on Computer Personnel Research (Cincinnati, Ohio, United
States, April 05 – 07, 1992). A. L. Lederer, Ed. SIGCPR ’92, 1992: p. 171-183.

50. Mak, B.L. and H. Sockel, A confirmatory factor analysis of IS employee motivation and retention. Information & Management,
2001. 38(5): p. 265-276.

51. Niederman, F. and M.R. Sumner, Job turnover among MIS professionals: an exploratory study of employee turnover.
Proceedings of the 2001 ACM SIGCPR Conference on Computer Personnel Research (San Diego, California, United States). ,
2001: p. 11-20.

52. Ridings, C.M. and L.B. Eder, An Analysis of IS technical career paths and job satisfaction. SIGCPR Comput. Pers., 1999. 20(2):
p. 7-26.

53. Thatcher, J.B., Y. Liu, and L.P. Stepina, The role of the work itself: An empirical examination of intrinsic motivation’s influence
on IT workers attitudes and intentions. Proceedings of the ACM SIGCPR Conference, 2002: p. 25-33.

54. LeDuc, A.L.J., Motivation of programmers. SIGMIS Database, 1980. 11(4): p. 4 – 12.

55. Niederman , F. and M. Sumner, Decision paths affecting turnover among information technology professionals, . Proceedings of
the 2003 SIGMIS conference on Freedom in Philadelphia: leveraging differences and diversity in the IT workforce, April 10-12,
2003, Philadelphia, Pennsylvania 2003: p. 133-142.

56. Santana, M. and D. Robey, Perceptions of control during systems development: effects on job satisfaction of systems
professionals SIGCPR Comput. Pers. , 1995. 16(1): p. 20-34.

57. Garden, A., Behavioural and organisational factors involved in the turnover of high tech professionals. SIGCPR Comput. Pers.,
1988. 11(4): p. 6-9.

58. Checchio, R.A., Creating a motivating environment in software development. Experience with the Management of Software
Projects 1989. Proceedings of the Third IFAC/IFIP Workshop. Pergamon. 1990, 1990: p. 81-6.

59. Li, Y., et al., Motivating open source software developers: influence of transformational and transactional leaderships
Proceedings of the 2006 ACM SIGMIS CPR conference on computer personnel research: Forty four years of computer personnel
research: achievements, challenges & the future. Claremont, California, USA 2006: p. 34-43.

60. Smits, S.J., E.R. McLean, and J.R. Tanner, A longitudinal study of I/S careers: synthesis, conclusion, and recommendations.
Proceedings of the 1997 ACM SIGCPR Conference on Computer Personnel Research (San Francisco, California, United States,
April 03 – 05, 1997: p. 36-48.

61. Andersen, E.S., “Never the twain shall meet”: exploring the differences between Japanese and Norwegian IS professionals.
Proceedings of the 2002 ACM SIGCPR Conference on Computer Personnel Research (Kristiansand, Norway, May 14 – 16,
2002). SIGCPR ’02, 2002: p. 65-71.

62. Lee, P.C., The social context of turnover among information technology professional. Proceedings of the 2002 ACM SIGCPR
Conference on Computer Personnel Research (Kristiansand, Norway, May 14 – 16, 2002). , 2002: p. 145-153.

63. Reid, M.F., et al., Affective commitment in the public sector: the case of IT employees Proceedings of the 2006 ACM SIGMIS
CPR conference on computer personnel research: Forty four years of computer personnel research: achievements, challenges \&
the future . Claremont, California, USA 2006: p. 321-332.

64. Richens, E., HR strategies for IS professionals in the 21st century. Proceedings of the ACM SIGCPR Conference, 1998: p. 289-
291.

65. Morales, A.W., Salary survey 2005. Software Development, 2005. 13(11): p. 32-42.

66. Dittrich, J.E., J. Daniel Couger, and R.A. Zawacki, Perceptions of equity, job satisfaction, and intention to quit among data
processing personnel. Information & Management, 1985. 9(2): p. 67-75.

67. Gambill, S.E., W.J. Clark, and R.B. Wilkes, Toward a holistic model of task design for IS professionals. Information and
Management, 2000. 37(5): p. 217-228.

68. Procaccino, J.D., et al., What do software practitioners really think about project success: An exploratory study. Journal of
Systems and Software, 2005. 78(2): p. 194-203.

69. Carayon, P., et al., Job characteristics and quality of working life in the IT workforce: the role of gender. Proceedings of the 2003
SIGMIS Conference on Computer Personnel Research: Freedom in Philadelphia–Leveraging Differences and Diversity in the IT
Workforce (Philadelphia, Pennsylvania, April 10 – 12, 2003), 2003: p. 58-63.

70. Agarwal, R. and T.W. Ferratt, Toward understanding the relationship between IT human resource management systems and
retention: An empirical analysis based on multiple theoretical and measurement approaches. Proceedings of the ACM SIGCPR
Conference, 2002: p. 126-138.

71. Hsu, M.K., et al., Perceived career incentives and intent to leave. Information & Management, 2003. 40(5): p. 361-369.

72. Rubin, H.I. and E.F. Hernandez, Motivations and behaviors of software professionals Proceedings of the ACM SIGCPR
conference on Management of information systems personnel , College park, Maryland, United States 1988: p. 62-71.

73. Honda, K., et al., Research on work environment for software productivity improvement. Proceedings of COMPSAC 85. The
IEEE Computer Society’s Ninth International Computer Software and Applications Conference (Cat. No.85CH2221-0). IEEE
Comput. Soc. Press. 1985, 1985: p. 241-8.

74. Fujigaki, Y., Stress analysis of software engineers for effective management. Human Factors in Organizational Design and
Management – III. Proceedings of the Third International Symposium. North-Holland. 1990, 1990: p. 255-8.

75. Nelson, A.C. and C. LeRouge, Self esteem: moderator between role stress fit and satisfaction and commitment? Proceedings of
the 2001 ACM SIGCPR Conference on Computer Personnel Research (San Diego, California, United States), 2001: p. 74-77.

76. Lee, P.C., Career plateau and professional plateau: impact on work outcomes of information technology professionals. SIGCPR
Comput. Pers. 20, 4 (Aug. 2002), 2002: p. 25-38.

77. Tanniru, M.R. and S.M. Taylor, Causes of turnover among data processing professionals—some preliminary findings.
Proceedings of the Eighteenth Annual ACM SIGCPR Computer Personnel Research Conference (Washington, D.C., United
States, June 04 – 05, 1981), 1981: p. 224-247.

78. McLean, E.R., S.J. Smits, and J.R. Tanner, The importance of salary on job and career attitudes of information systems
professionals. Information and Management, 1996. 30(6): p. 291-299.

79. Thomas, S.A., S.F. Hurley, and D.J. Barnes, Looking for the human factors in software quality management. 1996 International
Conference on Software Engineering: Education and Practice (SE:EP ’96) 1996: p. 474.

80. Couger, J.D., Comparison of motivation norms for programmer /analysts in the Pacific Rim and the U.S International Journal of
Information Systems, 1992. 1(3): p. 16-30.

81. Couger, J.D., Motivators vs. demotivators in the IS environment. Journal of Systems Management, 1988. 39(6): p. 36-41.

82. Bartol, K.M. and D.C. Martin, Managing Information Systems Personnel: A Review of the Literature and Managerial
Implications. MIS Quarterly, 1982. 6(Special Issue): p. 49-70.

83. Almstrum, V.L., What is the attraction to computing? Communications of the ACM, 2003. 46(9): p. 51-5.

84. Baroudi, J.J. and M.J. Ginzberg, Impact of the technological environment on programmer/analyst job outcomes Communications
of the ACM, 1986. 29(6): p. 546-555.

85. Lending, D. and N.L. Chervany, The changing systems development job: a job characteristics approach. Proceedings of the 1997
ACM SIGCPR Conference on Computer Personnel Research (San Francisco, California, United States, April 03 – 05, 1997: p.
127-137.

86. Mannaro, K., M. Melis, and M. Marchesi, Empirical Analysis on the Satisfaction of IT Employees Comparing XP Practices with
Other Software Development Methodologies. Lecture Notes in Computer Science, 2004: p. 166-174.

87. Hertel, S., S. Niedner, and G. Hermann, Motivation of software developers in Open Source projects: an Internet-based survey of
contributors to the Linux kernel. Research Policy 2003. 32: p. 1159-1177.

88. Roberts, J., I. Hann, and S. Slaughter, Understanding the motivations, participation and performance of Open Source Software
developers: a longitudinal study of the Apache projects. Carnegie Mellon University Working Paper, 2004.

89. Ferratt, T.W., H.G. Enns, and J. Prasad, Instrument Validation for Investigating a Model of Employment Arrangement Fit for IT
Professionals. Proceedings of the ACM SIGMIS CPR Conference, 2003: p. 168-178.

90. Ferratt, T.W., H.G. Enns, and J. Prasad, Employment arrangement fit for IT professionals: An examination of the importance of
fit components. Proceedings of the ACM SIGMIS CPR Conference, 2004: p. 25-29.

91. Goldstein, D.K. and J.F. Rockart, An Examination of Work-related Correlates of Job Satisfaction in Programmer/Analysts. MIS
Quarterly, 1984. 8(2): p. 103-115.

92. Rasch, R.H. and H.L. Tosi, Factors affecting software developers’ performance: an integrated approach. Management
Information Systems Quarterly, 1992. 16(3): p. 395-413.

Moderators
(extern

al)

Demographics
External

Control
Factors

(internal
)

Personality

Characteris
tics

Software
Engineer
characteristics

Controls which
characteristics an individual
has

Moderates power of
each characteristic
Figure 6:
Determinan
ts of
Software
Engineer
Characteris

Figure 6:
Determi
nants
of
Softwar
e

Moderators
(extern

al)

Demographics
External

Control
Factors

(internal
)

Personality

Characteris
tics

Software
Engineer
characteristics

Controls which
characteristics an individual
has

Moderates power of
each characteristic

SAMPLE_SLRs/3.full

REVIEW

Reducing work related psychological ill health and
sickness absence: a systematic literature review
S Michie, S Williams
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

Occup Environ Med 2003;60:3–9

A literature review revealed the following: key work
factors associated with psychological ill health and
sickness absence in staff were long hours worked, work
overload and pressure, and the effects of these on
personal lives; lack of control over work; lack of
participation in decision making; poor social support;
and unclear management and work role. There was
some evidence that sickness absence was associated
with poor management style. Successful interventions
that improved psychological health and levels of
sickness absence used training and organisational
approaches to increase participation in decision making
and problem solving, increase support and feedback,
and improve communication. It is concluded that many
of the work related variables associated with high levels
of psychological ill health are potentially amenable to
change. This is shown in intervention studies that have
successfully improved psychological health and reduced
sickness absence.
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

L evels of ill health, both physical and psycho-logical, and associated sickness absence arehigh among those working in health care in
the UK.1 2 This problem is not unique to the UK.3

Poor psychological health and sickness absence
are likely to lead to problems for patients in that
both the quantity and quality of patient care may
be diminished. Because most health care is
provided by staff working in teams, ill health and
sickness absence in any one individual is likely to
cause increased work and stress for other staff.

Several explanations have been put forward for
this high level of ill health, including the nature of
the work, organisational changes, and the large
amounts and pressure of work.4 A comparison
across UK hospitals in the public sector found
that rates of psychological ill health varied from
17% to 33%, with lower rates in hospitals charac-
terised by smaller size, greater cooperation, better
communication, more performance monitoring, a
stronger emphasis on training, and allowing staff
more control and flexibility in their work.5 This
supports the notion that organisational factors
may contribute to the level of psychological ill
health experienced by staff.

To tackle the problem of work related psycho-
logical ill health, evidence is needed about the
work factors associated with psychological ill
health and sickness absence, and about interven-
tions that have been implemented successfully to

prevent or reduce psychological ill health and
sickness absence. The primary focus of this
review is the association between work factors
and psychological ill health among health care
staff. However, because of the paucity of evidence
in health care,1 evidence was reviewed across all
work settings, although presented separately for
health care workers where appropriate.

METHODS
Our review method was based on that used by the
NHS Centre for Reviews and Dissemination.6 This
method involves a systematic examination of
selected databases using a variety of strategies,
including keywords and subject headings. It
allows the integration of quantitative data across
studies, where they have similar outcome meas-
ures, and the summary of findings where
methods used are diverse.

Identification of papers
Four electronic databases were used: Medline
(1987–99), PsychInfo (1987–99), Embase (1991–
99), and the Cochrane Controlled Trials Register
(1987–99). Relevant papers up to and including
1997 were selected from a larger study.1 The
search strategy in the larger study was of MeSH
key words and text words in each of three catego-
ries: work factors; staff; and ill health/
absenteeism/economic consequences. The search
included all types of employment and all devel-
oped countries but was limited to abstracts in
English. Secondary references were chosen from
the primary paper references and by contacting
academics researching this area. Psychological ill
health included measures of anxiety, depression,
emotional exhaustion, and psychological distress
(“stress” was excluded since it is a mediating
hypothetical construct rather than an outcome
measure of psychological ill health). For the pur-
pose of this review, papers from 1998 and 1999
were identified using the same search strategy,
but excluding physical ill health and economic
consequences.

Selection criteria
Abstracts were selected for retrieval of the paper if
they were judged to include data about both work
factors and psychological ill health or absentee-
ism. Dissertations were excluded, as were studies
of very specific staff groups or settings, work pat-
terns (for example, shift working), or events (for
example, violence). All abstracts were selected
independently by two researchers (three re-
searchers were involved in this activity). The per-
centage of abstracts for which two researchers
agreed about inclusion and exclusion varied

See end of article for
authors’ affiliations
. . . . . . . . . . . . . . . . . . . . . . .

Dr S Michie, Reader in
Clinical Health Psychology,
Centre for Outcomes
Research and Effectiveness,
Department of Psychology,
University College London,
Gower Street, London
WC1E 6BT, UK;
s.michie@ucl.ac.uk

Accepted 14 May 2002
. . . . . . . . . . . . . . . . . . . . . . .

www.occenvmed.com

on July 5, 2020 by guest. P
rotected by copyright.

http://oem
.bm

j.com
/

O
ccup E

nviron M
ed: first published as 10.1136/oem

.60.1.3 on 1 January 2003. D
ow

nloaded from

http://oem.bmj.com/

between 80% and 90%. Disagreements were resolved by
discussion.

Information extraction
Information from papers was extracted and coded within the
following categories: study aim, study design, type of study
population (for example, occupational group), sampling strat-
egy, sample size and response rate, demographic characteris-
tics, type of intervention, type of study measure, main
outcomes, and summary of results.

Further selection criteria
Coded papers excluded from the review were studies with:
volunteer or inadequately described sample; response rate of
less than 60%; no standardised measures of psychological
outcome.

RESULTS
Of the studies identified as part of the larger study,1 40 were
selected for this study (34 associations and six interventions).
A further nine studies meeting the above selection criteria
were identified in the period 1998–99, all of associations. No
studies were found in the Cochrane Controlled Trials Register.
The results are summarised in tables 1–4.

Because these studies were diverse in terms of outcomes
and measures used to assess these outcomes, a meta-analysis
was not appropriate.

Associations with work
The results are presented in three groups: health care workers
in the UK, health care workers in other developed countries,
and non-health care workers. This enabled an assessment of
whether associations between work factors and psychological
ill health are similar across sector and country.

Health care
In the UK, factors associated with psychological ill health in
doctors, from junior to senior grades, are long hours worked,9

high workload and pressure of work,7 16 11 and lack of role
clarity12 (table 1). Pressure of work has also been found to be
associated with poor mental health in dentists.10 In family
doctors, the issues were interruptions during and outside sur-
gery hours and patient demands.16

Among UK nurses, the most frequently reported source of
psychological ill health was workload pressures.17 Distress in
student nurses has been caused by low involvement in
decision making and use of skills, and low social support at
work.13 In a study of health care workers across job type, bul-
lying was found to be prevalent, carried out mainly by manag-
ers and associated with both anxiety and depression.14 Of the
two studies addressing sickness absence, one found a negative
association with job demands,13 while the other found no
association with control over work.15

Similar factors are associated with psychological ill health
in health care workers in the rest of Europe, the USA, and
Australia (table 2). The one study of doctors found an associ-
ation between work control and social support and psycho-
logical distress.22 Among nurses, lack of co-worker
support,24 27 job influence,26 and organisational climate and
role ambiguity28 were associated with psychological distress.
Among other hospital workers, work overload and pressure,
role ambiguity, lack of control over work, and lack of
participation in decision making were all found to be
associated with distress.18 20 25

Sickness absence was associated with work pressures and
lack of training,23 unsupportive management style,21 role
ambiguity, tolerance of absenteeism, and low pay.19

Beyond health care
The picture among non-health care workers in Europe and the
USA was similar to that of health care workers (table 3). The

Table 1 Summary of observational studies of associations between work factors and ill health: health care workers in
UK

Study Design Participants
Response
rate Work factors Outcomes Results

Agius et al,
19967

Cross sectional 375 consultant
doctors

75% Work demands Emotional exhaustion
(MBI)

High academic work demands associated
with low emotional exhaustion (r=−0.14,
p<0.05) Baglioni et al, 19908 Cross sectional 475 senior nurses 80% Workload Mental health (CCEI) No association Baldwin et al, 19979 Longitudinal 142 junior doctors 95% Long hours Psychological distress (GHQ-28) No association overall; association with somatic symptoms, r=0.24 Cooper et al, 198810 Cross sectional 484 dentists 85% Time pressures, pay stressors and technical problems Mental health (CCEI) Time pressures, pay stressors and technical problems associated with poor mental health (respective Bs=0.24, 0.20, 0.12; F=20.54, p<0.001) Deary et al, 199611 Cross sectional 333 consultant doctors 67% Clinical workload Psychological distress (GHQ-28), emotional exhaustion (MBI) High clinical workload associated with emotional exhaustion (χ2 for model=30.31, p=0.11, satisfactory fit) Heyworth et al, 199312 Cross sectional 201 trainee and consultant doctors 72% Task clarity, supportive communication Depression (CES-D) Task clarity and supportive communication associated with lower depression (r=−0.51 and −0.36 respectively, p=0.0001) Parkes, 198213 Experimental 164 student nurses 97% Job demand, discretion, social support Anxiety (GHQ), depression (CCEI), sickness absence Anxiety and depression associated with low job discretion and job support (r=−0.30 and −0.35 for anxiety and −0.26 and −0.36 for depression) and job demand was negatively associated with sickness absence (r=−0.24) Quine, 199914 Cross sectional 1100 health care workers 70% Bullying Anxiety, depression (HADS) Bullying associated with higher anxiety (30% v 9%, p<0.001) and depression (8% v 1%, p<0.001) Rees and Cooper, 199215 Cross sectional 1176 health care workers 67% Control over work (OSI) Sickness absence No association Sutherland and Cooper, 199316 Cross sectional 917 family doctors 61% Job demands Anxiety, depression and somatic anxiety (CCEI) Associations with anxiety (beta=0.17) depression (beta=0.28) and somatic anxiety (beta=0.23) Tyler and Cushway, 199217 Cross sectional 72 nurses 60% Workload, conflict, social support Psychological distress (GHQ 28) GHQ predicted by managing the workload (beta=0.32) 4 Michie, Williams www.occenvmed.com on July 5, 2020 by guest. P rotected by copyright. http://oem .bm j.com / O ccup E nviron M ed: first published as 10.1136/oem .60.1.3 on 1 January 2003. D ow nloaded from http://oem.bmj.com/ key work factors associated with psychological ill health were: work overload and pressure31 34 36 39–40 41 47 48; conflicting de- mands47; lack of control over work and lack of participation in decision making34 36 37 39 40 46–48; poor social support at work31 33 35 38 39 41 47 48; unclear management and work role29 30 41 34 38; interpersonal conflict42 46; and conflict between work and family demands.46 Long hours were found to be associated with depression in women, but not in men.44 Sickness absence was negatively associated with high job demand,45 and positively associated with monotonous work, not learning new skills and low control over work,36 37 45 and non-participation at work.43 Evaluated interventions Six intervention studies met our methodological criteria (table 4). Three were randomised controlled trials,49 50 52 three were conducted in the USA,50 51 54 one in the UK,53 and two in Scandinavia.49 52 Three were of health care workers.50 52 54 Five were training programmes offered as part of the working day and one was an organisational intervention.53 Skills to mobilise support at work and to participate in problem solving and decision making were taught to care staff of people with mental ill health or mental disability in a ran- domised controlled trial.50 Groups of 20 had six sessions of 4–5 hours training over two months, and were trained to train those in their workplace. Compared to those in the control group, the intervention group reported more supportive feed- back, more ability to cope, and better work team functioning and climate. Among those most at risk of leaving, those undergoing the training reported reduced depression. The second randomised controlled trial compared receiving support, advice, and feedback from a psychologist with having the passive presence of the same psychologist at staff meetings in a geriatric hospital facing organisational change.52 Staff were taught skills of stress management, and how to partici- pate in, and control, their work. The intervention was an hour a fortnight during the 10 weeks before, and the 10 weeks after, the organisational change. There was a significant difference between groups, with a decrease of stress hormone levels in the intervention group. Staff of a psychiatric hospital were taught verbal and non- verbal communication and empathy skills.54 Groups of 6–8 had eight hour weekly sessions for four weeks involving infor- mation, videos, modelling, and role playing. Compared to a matched control group, the intervention group showed reduced staff resignations and sick leave, although no statisti- cal tests are reported. Among physically inactive employees of an insurance com- pany, a randomised controlled trial found stress management training and aerobic exercise interventions had mixed Table 2 Summary of observational studies of associations between work factors and ill health: health care workers in developed countries beyond the UK Study Country Design Participants Response rate Work factors Outcomes Results Arsenault et al, 199118 Canada Cross sectional 760 hospital workers Not reported Professional latitude, clinical demands, workload problems, role difficulties Mental strain including depression (Cobb) and anxiety (STAI) Low professional latitude (F=12.7, p<0.001) and high workload problems (F=4.5, p<0.04) and role difficulties (F=31.6, p<0.001) associated with mental strain Brooke and Price, 198919 USA Cross sectional 425 hospital workers 74% Routinisation, centralisation, pay, reward policy, role ambiguity, conflict, overload, organisational tolerance of absenteeism Absenteeism High role ambiguity and tolerance of absenteeism, low pay and low centralisation predicted absenteeism (structural coefficients 0.21, p<0.001; 0.27, p<0.001; −0.11, p<0.05; −0.19, p<0.02, respectively) Estryn-Behar et al, 199020 France Cross sectional 1505 female hospital workers 90% Mental load, insufficient training, time pressure Psychological distress (GHQ-12) Mental load and time pressure associated with psychological distress (ORs 2.9 and 2.2) Gray-Toft and Anderson, 198521 USA Experimental 159 nurses Not reported Open, supportive supervisory style Absenteeism Open supportive supervisory style associated with lower absenteeism (relevant statistics not presented) Johnson et al, 199522 USA Longitudinal 581 doctors 86% Job demands, work control, social support Psychological distress (GHQ-20) Work control and social support negatively associated with psychological distress (B=−0.44, p=0.05 and B=−0.46, p=0.05) Landeweerd and Boumans, 199423 NetherlandsCross sectional 561 nurses 96% Work pressure, job complexity, feedback, autonomy, promotion/training Absence frequency Work pressures associated with absence frequency (B=0.12) and promotion/training negatively associated (B=−0.12) Marshall & Barnett, 199224 USA Cross sectional 362 female nurses and social workers Not reported Work related support, job overload Psychological distress (SCL-90-R) and emotional well-being (Rand Corporation) Co-worker support associated with emotional wellbeing (B=−0.20, p<0.01) Martin, 198425 USA Cross sectional 95 and 140 hospital workers 63% and 70% Work overload and ambiguity, participation in decision making Psychological distress (GHQ-12) Work factors associated with distress (canonical correlations=0.53 and 0.41, p<0.001) Petterson et al, 199526 Sweden Cross sectional 2568 nurses 76% Job influence Emotional exhaustion (MBI) Job influence negatively associated with emotional exhaustion (p<0.001) Pisarski et al, 199827 Australia Cross sectional 172 nurses, aged 21–40 years Not reported Social support Psychological distress (GHQ-12) Co-worker social support directly associated with distress and mediates association with supervisor social support (path coefficients <0.001) Revicki and May, 198928 USA Cross sectional 232 nurses 77% Organisational climate, supervisor behaviour, role ambiguity, social support Depression (Rand corporation) The association of organisational climate and role ambiguity with depression is mediated by stress Reducing work related psychological ill health and sickness absence 5 www.occenvmed.com on July 5, 2020 by guest. P rotected by copyright. http://oem .bm j.com / O ccup E nviron M ed: first published as 10.1136/oem .60.1.3 on 1 January 2003. D ow nloaded from http://oem.bmj.com/ Table 3 Summary of observational studies of associations between work factors and ill health: non-health care workers Study Country Design Participants Response rate Work factors Outcomes Results Bacharach et al, 199129 USA Cross sectional 430 public sector engineers 79% Role conflict, time pressure Emotional exhaustion (MBI) High role conflict (B=0.24) associated with emotional exhaustion Carayon et al, 199530 USA Prospective 148 public sector office workers 71% Job demands, content and control, social support, task clarity and job future ambiguity Worker strain, including anxiety and depression (McNair) Predictors of worker strain at one and two years were task clarity and job future ambiguity Driscoll et al, 199531 USA Cross sectional 4900 public sector workers 70% Job demands, control and social support Anxiety and depression (NIOSH Generic Job Stress Questionnaire) High demand and low support associated with anxiety (ORs 1.46 and 2.01) and depression (ORs 2.02 and 1.69) Ferrie et al, 199832 UK Prospective 10308 public sector workers 80% Job insecurity Psychological distress (GHQ-30) Non-significant association Frese, 199933 Germany Longitudinal 90 male blue collar workers 35–66% Work intensity, uncertainty, organisational problems, environmental problems, social pressure, social support Anxiety, depression Social pressure and support associated with anxiety/depression (r=0.21/0.20 and −0.21/−0.27) Frone et al, 199534 USA Longitudinal 795 employed adults 67% Work pressure, lack of autonomy, role ambiguity Depression (CES-D) Work pressure, lack of autonomy, role ambiguity all associated with depression (Bs=0.10, p<0.01; 0.19, p<0.001; 0.18, p<0.001, respectively) Fusilier et al, 198735 USA Cross sectional 312 police officers and fire fighters 65% Role conflict and ambiguity, social support Depression (Caplan) Low social support associated with depression (B=−0.24, p<0.01) Karasek, 1979: Study 136 Sweden Longitudinal 1896 working males 92% and 85% Decision latitude and job demands Depression (amended from American Health Survey), absenteeism Decision latitude negatively associated with depression and absenteeism (OR=−1.29, p<0.05 and −1.44, p<0.01). Job demands associated with depression (OR=1.45, p<0.001) Study 2 USA Cross sectional 911 working males 76% As above As above Decision latitude negatively associated with depression and absenteeism (OR=−1.41, p<0.01 and –2.04, p<0.001); job demands associated with depression (OR=1.20, p<0.05) Karasek, 199037 Sweden Cross sectional 8504 white collar workers 87% Changes in control over work Depression and absenteeism Decreased control associated with depression (p<0.01) and with absenteeism in men (p<0.01) but not women LaRocco et al, 198038 USA Cross sectional 636 male workers Not reported Supervisor support, participation, future ambiguity, under-utilisation, workload, role conflict Anxiety and depression (Cobb and Kasl) Supervisor support buffers the adverse effect of low participation on depression (p<0.1, significant ) and of future ambiguity on anxiety (p<0.01) Niedhammer et al, 199839 France Prospective 11552 92% Psychological demands, decision latitude and social support Depression (CES-D) High psychological demand, low decision latitude and social support associated with subsequent depression. For men, OR of 1.8, 1.4 and 1.6 respectively and for women, OR of 1.4, 1.4 and 1.3 respectively. Payne and Fletcher, 198340 UK Cross sectional 148 teachers 74% Workload demands, discretion Anxiety and depression (CCEI) Association with workload demands (betas=0.117 for anxiety and 0.176 for depression) and negative association with discretion (betas=−0.222 for anxiety and –0.121 for depression) Reifman et al, 199141 USA Cross sectional and longitudinal 200 married, professional women >90% Social support at work , control over
work, role ambiguity, workload

Depression (SCL-90) Cross sectionally, association with social support at work (r=0.37),
role ambiguity (r=0.35) and workload (r=0.29). No associations
one year later

Romanov et al, 199642 Finland Prospective 15530 employees Not reported Conflict at work Psychiatric morbidity (hospital
discharge registry)

Positive association (RR 2.18, CI95 1.34, 3.54)

Rubenowitz et al,
198343

Sweden Cross sectional 25 departments of 5
companies.
Numbers not
reported

85–90% Perceived participation (individual, group,
representative)

Absenteeism Negative association for individual participation (r=−0.53)

Shields, 199944 Canada Prospective 3830 working
population

80% Long working hours Depression (Composite
International Diagnostic
Interview)

>35 hours per week associated with depression in women
(OR=2.2) but not men

Smulders and Nijuis,
199945

The Netherlands Cross sectional and
prospective

1755 male public
sector workers

70% Job control and job demands Absence rate and absence
frequency

Cross sectionally, job control associated with low absence
frequency (beta=0.10, p<0.01) and job demand associated with low absence rate (beta=−0.08, p<0.05) Sparks and Cooper, 199946 UK Cross sectional 7099 from 13 occupations Not reported Work control, career achievement, organisational climate, job factors, home/work interface, work relationships Mental health (OSI) All associated (r=0.22 to −0.28, p<0.001) Stansfeld et al, 199547 UK Cross sectional 10314 public sector employees 73% Job variety and skill use, control, social support, work pace, conflicting demands Psychological distress (GHQ-30) All significantly associated (intertile trend p values < 0.001) Stansfeld et al, 199848 UK Prospective 7372 public sector workers 72% Job demands, decision latitude, social support and effort-reward imbalance Psychological functioning (SF-36) Low support and effort-reward imbalance associated with poor psychological functioning (OR=1.2 for men and 1.4 for women; 1.8 for men and 2.3 for women respectively). In men, low decision latitude (OR=1.2) and in women, high job demand (OR=2.0) were associated with poor psychological functioning 6 M ichie,W illiam s w w w .occenvm ed.com on July 5, 2020 by guest. Protected by copyright. http://oem.bmj.com/ Occup Environ Med: first published as 10.1136/oem.60.1.3 on 1 January 2003. Downloaded from http://oem.bmj.com/ effects.49 After three sessions a week for 10 weeks, stress man- agement training resulted in improved perceived coping abil- ity but no change in physical or psychological health. Aerobic exercise resulted in improved feelings of wellbeing and decreased complaints of muscle pain. Employees of a fire department underwent one of seven training programmes emphasising one or more aspect of stress management: physiological processes, coping with people, or interpersonal awareness processes.51 Weekly ses- sions for 8–10 people were run over 42 weeks. There was no control group. Compared to baseline, there were reductions in depression, anxiety, psychological strain, and emotional exhaustion immediately after the programme. There was a further reduction in psychological strain and emotional exhaustion at 9–16 months follow up. A structural intervention for local authority staff on long term sickness absence was effective in reducing sickness absence. Referral to occupational health services was triggered after two or three months absence, rather than at six months which was the practice before the intervention. The average duration of sickness absence reduced from 40 to 25 weeks before resumption of work and from 72 to 53 weeks for those staff who left employment for medical reasons. The authors describe large financial savings but no statistical tests are reported.53 DISCUSSION This systematic review of a large number of studies covers a wide range of employment sectors in the developed world and summarises those studies that use rigorous methods. The studies show that, while levels of psychological ill health are higher in health care than in non-health care workers,5 the associations between work factors and psychological ill health are similar. They are also similar across continents. This suggests that a generic approach to reducing work related psychological ill health may be appropriate. The most common work factors associated with psychologi- cal ill health were work demand (long hours, workload, and pressure), lack of control over work, and poor support from managers. These were also associated with sickness absence. The findings of this review, summarised in tables 1–4, are consistent with the demand-control model of job strain.36 Interventions aimed at changing these workplace factors reduced psychological ill health. This review highlights limitations in the research identified. The studies that have been carried out are limited in the ques- tions addressed and in the study designs used. Since most studies are cross sectional, causal relations cannot be shown. It may be that the associations found reflect a tendency for more vulnerable people to choose work in caring roles or other types of job which are well represented in published research studies. The question of what aspects of work lead to ill health and sickness absence can only be addressed by longitudinal studies that are able to investigate the causal relations between work factors and health outcomes and by ran- domised controlled trials of interventions. A longitudinal study that directly addressed the nature of the relation found a causal relation between psychological stress and psychoso- matic complaints.55 There are several practical implications suggested by the studies of association in this review, for both employment Table 4 Summary of studies of interventions Study Participants Design Response rate Intervention Outcomes Results Gronningsaeter et al, 199249 76 physically inactive Norwegian insurance workers Stratified RCT 72% 6 sessions aerobic exercise per week for 10 weeks or 3 sessions stress management training per week for 10 weeks Anxiety (STAI) and health complaints No association of either intervention with anxiety. Aerobic exercise associated with reduced health complaints (F=3.4, p=0.07 compared to controls, and F=4.8, p<0.05 compared to stress management intervention) Heaney et al, 199550 1375 US residential care workers Cluster RCT 62% 6 × 4 hour sessions over 9 weeks to teach skills to enhance social support and problem solving Depression (SCL-90R) For those most at risk of leaving their jobs, R2=0.41, p<0.01 Kagan et al, 199551 373 US fire department workers Randomised, uncontrolled Not reported 42 weeks of 7 psycho-educational programmes, 6 weeks each Anxiety, depression, psychological strain, emotional burnout Compared to baseline, F=52.3, 42.2, 29.1, 10.6 respectively; p<0.001 for all. At 9–16 month follow up, F=4.8 (p<0.05), 8.7 (p<0.01), 21.4 (p<0.001), 45.2 (p<0.001) respectively Lokk and Arnetz, 199752 26 Swedish hospital ward workers RCT 93% 20 weekly 1 hour stress management sessions Stress hormone (prolactin) level Change scores: Intervention group −0.58 Control group +1.85 F=7.3, p<0.01 Malcolm et al, 199353 604 UK long term sick local government workers Observational 100% Early referral to Occupational Health Duration of sickness absence (weeks) 25 weeks in intervention period compared to 40 in control period Smoot and Gonzales, 199554 65 US hospital workers Matched controlled 90% 4 weekly 8 hour sessions of communication training Sick leave (hours) in 6 months after compared to 6 months before % change: −28.2 in experimental group, −6.4 in control group Main messages • Key work factors associated with psychological ill health and sickness absence in staff are long hours worked, work overload and pressure, and the effects of these on personal lives; lack of control over work; lack of participation in decision making; poor social support; and unclear management and work role. • There is some evidence that sickness absence is associated with poor management style. • Successful interventions that improve psychological health and levels of sickness absence use training and organisa- tional approaches to increase participation in decision making and problem solving, increase support and feedback, and improve communication. Reducing work related psychological ill health and sickness absence 7 www.occenvmed.com on July 5, 2020 by guest. P rotected by copyright. http://oem .bm j.com / O ccup E nviron M ed: first published as 10.1136/oem .60.1.3 on 1 January 2003. D ow nloaded from http://oem.bmj.com/ practices and management style. Intervention studies, how- ever, have focused mainly on staff training. There is a need for future studies to evaluate interventions based on employment practices and management style. This would represent primary prevention, reducing sources of psychological ill health, rather than secondary prevention, training individuals who are already experiencing work related stress, to be more robust in the face of such pressures. Only one of the interven- tion studies included an economic evaluation: such evalua- tions are important in facilitating employers to make decisions about whether or not to implement interventions. Future research should adhere to minimum scientific standards absent in many of the studies reviewed, such as adequate design, sufficiently large samples, and valid outcome measures. Lessons that are learnt from rigorously evaluated interventions can then be applied more generally. ACKNOWLEDGEMENTS We are grateful to Shriti Pattani for help with literature searching and to Frédérique Cooper for help with preparing this manuscript. . . . . . . . . . . . . . . . . . . . . . Authors’ affiliations S Michie, Reader in Clinical Health Psychology, Centre for Outcomes Research and Effectiveness, Department of Psychology, University College London, Gower Street, London WC1E 6BT, UK S Williams, Consultant in Occupational Medicine, Royal Free Hampstead NHS Trust, London NW3 2QG, UK REFERENCES 1 Williams S, Michie S, Pattani S. Improving the health of the NHS workforce. London: The Nuffield Trust, 1998. 2 Confederation of British Industry. Managing absence: in sickness and in health. London: CBI, 1997. 3 Whitley TW, Allison Jr EJ, Gallery ME, et al. Work related stress and depression among practicing emergency physicians: an international study. Ann Emerg Med 1994;23:1068–71. 4 Cox T, Griffiths A. The nature and measurement of work stress: theory and practice. In: Wilson, JR, Corlett E, Nigel E, et al, eds. Evaluation of human work: a practical ergonomics methodology, 2nd edn. London: Taylor & Francis, 1995:783–803. 5 Wall TD, Bolden RI, Borrill CS, et al. Minor psychiatric disorder in NHS trust staff: occupational and gender differences. Br J Psychiatry 1997;171:519–23. 6 University of York. Understanding systematic reviews of research on effectiveness. CDR report 4. York: NHS Centre for Reviews and Dissemination, 1996. 7 Agius RM, Blenkin H, Deary IJ, et al. Survey of perceived stress and work demands of consultant doctors. Occup Environ Med 1996;53:217–24. 8 Baglioni Jr AJ, Cooper CL, Hingley P. Job stress, mental health and job satisfaction among UK senior nurses. Stress Medicine 1990;6:9–20. 9 Baldwin PJ, Dodd M, Wrate RM. Young doctors’ health—I. How do working conditions affect attitudes, health and performance. Soc Sci Med 1997;45:35–40. 10 Cooper CL, Watts J, Baglioni Jr AJ, et al. Occupational stress amongst general practice dentists. J Occup Psychol 1988;61:163–74. 11 Deary IJ, Blenkin H, Agius RM, et al. Models of job-related stress and personal achievement among consultant doctors. Br J Psychol 1996;87:3–29. 12 Heyworth J, Whitley, TS, Allison Jr EJ, et al. Correlates of work-related stress amongst consultants and senior registrars in accident and emergency medicine. Arch Emerg Med 1993;10:279–88. 13 Parkes KR. Occupational stress among student nurses: a natural experiment. J Appl Psychol 1982;67:784–96. 14 Quine L. Workplace bullying in NHS community trust: staff questionnaire study. BMJ 1999;318:228–32. 15 Rees D, Cooper CL. Occupational stress in health service workers in the UK. Stress Medicine 1992;8:79–90. 16 Sutherland VJ, Cooper CL. Identifying distress among general practitioners: predictors of psychological ill health and job dissatisfaction. Soc Sci Med 1993;37:575–81. 17 Tyler P, Cushway D. Stress, coping and mental well-being in hospital nurses. Stress Medicine 1992;8:91–8. 18 Arsenault A, Dolan SL, Van Ameringen MR. Stress and mental strain in hospital work: exploring the relationship beyond personality. Journal of Organisational Behaviour 1991;12:483–93. 19 Brooke PP, Price JL. The determinants of employees absenteeism: an empirical test of a causal model. J Occup Psychol 1989;62:1–19. 20 Estryn-Behar M, Kaminski J, Peigne E, et al. Stress at work and mental health status among female hospital workers. Br J Ind Med 1990;47:20–8. 21 Gray-Toft PA, Anderson JG. Organisational stress in the hospital: development of a model for diagnosis and prediction. Health Serv Res 1985;19:753–74. 22 Johnson JV, Stewart W, Hall EM, et al. The psychosocial work environment of physicians. J Occup Environ Med 1995;37:1151–9. 23 Landeweerd JA, Boumans NPG. The effect of work dimensions and need for autonomy on nurses’ work satisfaction and health. J Occup Organ Psychol 1994;67:207–17. 24 Marshall NL, Barnett RC. Work-related support among women in caregiving occupations. J Community Psychol 1992;20:6–42. 25 Martin TN. Role stress and inability to leave as predictors of mental health. Human Relations 1984;37:969–83. 26 Petterson IL, Arnetz BB, Arnetz JE. Predictors of job satisfaction and job influence: results from a national sample of Swedish nurses. Psychother Psychosom 1995;64:9–19. 27 Pisarski A, Bohle P, Callan VJ. Effects of coping strategies, social support and work-nonwork conflict on shift worker’s health. Scand J Work Environ Health 1998;241:41–145. 28 Revicki DA, May HJ. Organisational characteristics, occupational stress, and mental health in nurses. Behav Med 1989;15:30–6. 29 Bacharach SB, Bamberger P, Conley S. Work-home conflict among nurses and engineers: mediating the impact of role stress on burnout and satisfaction at work. Journal of Organisational Behaviour 1991;12:39–53. 30 Carayon P, Yang C, Lim S. Examining the relationship between job design and worker strain over time in a sample of office workers. Ergonomics 1995;38:1199–211. 31 Driscoll RJ, Worthington KA, Hurrell Jr JJ. Workplace assault: an emerging job stressor. Consulting Psychology Journal: Practice and Research 1995;47:205–12. 32 Ferrie JE, Shipley MJ, Marmot MG, et al. An uncertain future: the health effects of threats to employment security in white-collar men and women. Am J Public Health 1998;88:1030–6. 33 Frese M. Social support as a moderator of the relationship between work stressors and psychological dysfunctioning: a longitudinal study with objective measures. J Occup Health Psychol 1999;3:179–92. 34 Frone MR, Russell M, Cooper ML. Job stressors, job involvement and employee health: a test of identity theory. J Occup Psychol 1995;68:1–11. 35 Fusilier MR, Ganster DC, Mayes BT. Effects of social support, role stress, and locus of control on health. Journal of Management 1987;13:517–28. 36 Karasek Jr RA. Job demands, job decision latitude, and mental strain: implications for job redesign. Adm Sci Q 1979;24:285–311. 37 Karasek R. Lower health risk with increased job control among white collar workers. Journal of Organisational Behaviour 1990;11:171–85. 38 LaRocco JM, House JS, French Jr JRP. Social support, occupational stress, and health. J Health Soc Behav 1980;21:202–18. 39 Niedhammer I, Goldberg M, Leclerc A, et al. Psychosocial factors at work and subsequent depressive symptoms in the Gazel cohort. Scand J Work Environ Health 1998;24:197–205. 40 Payne R, Fletcher BC. Job demands, supports, and constraints as predictors of psychological strain among schoolteachers. Journal of Vocational Behaviour 1983;22:136–47. 41 Reifman A, Biernat M, Lang EL. Stress, social support, and health in married professional women with small children. Psychology of Women Quarterly 1991;15:431–45. 42 Romanov K, Appelberg K, Honkasalo M, et al. Recent interpersonal conflict at work and psychiatric morbidity: a prospective study of 15,530 employees aged 24–64. J Psychosom Res 1996;40:169–76. 43 Rubenowitz S, Norrgren F, Tannenbaum AS. Some social psychological effects of direct and indirect participation in ten Swedish companies. Organisation Studies 1983;4:243–59. 44 Shields M. Long working hours and health. Health Reports 1999;11:33–48. 45 Smulders PGW, Nijhuis, FJN. The job demands-job control model and absence behaviour: results of a 3-year longitudinal study. Work and Stress 1999;13:115–31. 46 Sparks K, Cooper CL. Occupational differences in the work-strain relationship: towards the use of situation-specific models. Journal of Occupational Organizational Psychology 1999;72:219–29. 47 Stansfeld SA, North FM, White I, et al. Work characteristics and psychiatric disorder in civil servants in London. J Epidemiol Community Health 1995;49:48–53. 48 Stansfeld SA, Bosma H, Hemingway H, et al. Psychosocial work characteristics and social support as predictors of SF-36 health functioning: the Whitehall II study. Psychosom Med 1998;60:247–55. Policy implications • Many of the work related variables associated with high levels of psychological ill health are potentially amenable to change. • More evaluations of interventions are required, based on randomised or longitudinal research designs. • Interventions for which evidence of effectiveness exists should be piloted and evaluated across different work settings. 8 Michie, Williams www.occenvmed.com on July 5, 2020 by guest. P rotected by copyright. http://oem .bm j.com / O ccup E nviron M ed: first published as 10.1136/oem .60.1.3 on 1 January 2003. D ow nloaded from http://oem.bmj.com/ 49 Gronningsaeter H, Hytten K, Skauli G, et al. Improved health and coping by physical exercise or cognitive behavioural stress management training in a work environment. Psychology and Health 1992;7:147–63. 50 Heany CA, Price RH, Refferty J. Increasing coping resources at work: a field experiment to increase social support, improve work team functioning, and enhance employee mental health. Journal of Organisational Behaviour 1995;16:335–52. 51 Kagan NI, Kagan H, Watson MG. Stress reduction in the workplace: the effectiveness of psychoeducational programs. Journal of Counselling Psychology 1995;42:71–8. 52 Lokk J, Arnetz B. Psychophysiological concomitants of organisational change in health care personnel: effects of a controlled intervention study. Psychother Psychosom 1997;66:74–7. 53 Malcolm RM, Harrison J, Forster H. Effects of changing the pattern of referrals in a local authority. Occup Med 1993;43:211–15. 54 Smoot SL Gonzales JL. Cost-effective communication skills training for state hospital employees. Psychiatr Serv 1995;46:819–22. 55 Frese M. Stress at work and psychosomatic complaints: a causal interpretation. J Appl Psychol 1985;70:314–28. ECHO ................................................................................................................ Air pollution study confirms concerns over childhood rickets A study in India has shown that young children living in areas of high air pollution are in danger ofdeveloping rickets. Two groups of age matched infants and toddlers were compared for serum vitamin D metabolites, cal- cium, alkaline phosphatase (AP), and parathormone (PTH) concentrations. One group lived in a central location in Delhi and the other on the outskirts of the city, where air pollution is much lower. Children from the city centre had significantly lower mean serum total 25-hydroxyvitamin D (25(OH)D)—an indictor of vitamin D status—than children from the outskirts (12.4 ng/ml v 27.1 ng/ml). Their mean serum AP and PTH concentrations were significantly higher, and the inverse relations between 25(OH)D and AP, PTH were also significant. Three children had serum total 25(OH)D low enough to indicate rickets, and nine more below adequate amounts. All children from the outskirts had adequate 25(OH)D. Mean haze score was significantly less at the city centre (2.1 against 2.7). Each group included 34 children aged 9–24 months with similar home conditions, diet, family income, and time spent outside. Blood was taken from 26 children from the city centre and 31 from the outskirts. Haze scores measured at ground level three times daily (0900, 1200, 1600) during February 2000 were taken as a marker for UVB radiation. Concerns are growing that increasing air pollution from industry and motor vehicles blocks out UVB radiation and children’s ability to make vitamin D naturally, leading to rickets. m Archives of Disease in Childhood 2002;87:111–113. Please visit the Occupational and Environmental Medicine website [www. occenvmed.com] for link to this full article. Reducing work related psychological ill health and sickness absence 9 www.occenvmed.com on July 5, 2020 by guest. P rotected by copyright. http://oem .bm j.com / O ccup E nviron M ed: first published as 10.1136/oem .60.1.3 on 1 January 2003. D ow nloaded from http://oem.bmj.com/ SAMPLE_SLRs/4121a006 See discussions, stats, and author profiles for this publication at: https://www.researchgate.net/publication/303696963 Requirements Engineering Visualization: A Systematic Literature Review Conference Paper · September 2016 DOI: 10.1109/RE.2016.61 CITATIONS 27 READS 772 3 authors, including: Some of the authors of this publication are also working on these related projects: Who Should Take This Task?: Dynamic Decision Support for Crowd Workers View project Task Interruptions In Software Development Projects View project Mohammad Noaeen The University of Calgary 9 PUBLICATIONS 51 CITATIONS SEE PROFILE Guenther Ruhe The University of Calgary 264 PUBLICATIONS 4,256 CITATIONS SEE PROFILE All content following this page was uploaded by Zahra Shakeri on 27 September 2016. The user has requested enhancement of the downloaded file. https://www.researchgate.net/publication/303696963_Requirements_Engineering_Visualization_A_Systematic_Literature_Review?enrichId=rgreq-d0def5b66d03e842c0653636795883b7-XXX&enrichSource=Y292ZXJQYWdlOzMwMzY5Njk2MztBUzo0MTEwMDQ0OTczNTA2NTZAMTQ3NTAwMjUxNjkzMw%3D%3D&el=1_x_2&_esc=publicationCoverPdf https://www.researchgate.net/publication/303696963_Requirements_Engineering_Visualization_A_Systematic_Literature_Review?enrichId=rgreq-d0def5b66d03e842c0653636795883b7-XXX&enrichSource=Y292ZXJQYWdlOzMwMzY5Njk2MztBUzo0MTEwMDQ0OTczNTA2NTZAMTQ3NTAwMjUxNjkzMw%3D%3D&el=1_x_3&_esc=publicationCoverPdf https://www.researchgate.net/project/Who-Should-Take-This-Task-Dynamic-Decision-Support-for-Crowd-Workers?enrichId=rgreq-d0def5b66d03e842c0653636795883b7-XXX&enrichSource=Y292ZXJQYWdlOzMwMzY5Njk2MztBUzo0MTEwMDQ0OTczNTA2NTZAMTQ3NTAwMjUxNjkzMw%3D%3D&el=1_x_9&_esc=publicationCoverPdf https://www.researchgate.net/project/Task-Interruptions-In-Software-Development-Projects?enrichId=rgreq-d0def5b66d03e842c0653636795883b7-XXX&enrichSource=Y292ZXJQYWdlOzMwMzY5Njk2MztBUzo0MTEwMDQ0OTczNTA2NTZAMTQ3NTAwMjUxNjkzMw%3D%3D&el=1_x_9&_esc=publicationCoverPdf https://www.researchgate.net/?enrichId=rgreq-d0def5b66d03e842c0653636795883b7-XXX&enrichSource=Y292ZXJQYWdlOzMwMzY5Njk2MztBUzo0MTEwMDQ0OTczNTA2NTZAMTQ3NTAwMjUxNjkzMw%3D%3D&el=1_x_1&_esc=publicationCoverPdf https://www.researchgate.net/profile/Mohammad_Noaeen?enrichId=rgreq-d0def5b66d03e842c0653636795883b7-XXX&enrichSource=Y292ZXJQYWdlOzMwMzY5Njk2MztBUzo0MTEwMDQ0OTczNTA2NTZAMTQ3NTAwMjUxNjkzMw%3D%3D&el=1_x_4&_esc=publicationCoverPdf https://www.researchgate.net/profile/Mohammad_Noaeen?enrichId=rgreq-d0def5b66d03e842c0653636795883b7-XXX&enrichSource=Y292ZXJQYWdlOzMwMzY5Njk2MztBUzo0MTEwMDQ0OTczNTA2NTZAMTQ3NTAwMjUxNjkzMw%3D%3D&el=1_x_5&_esc=publicationCoverPdf https://www.researchgate.net/institution/The_University_of_Calgary?enrichId=rgreq-d0def5b66d03e842c0653636795883b7-XXX&enrichSource=Y292ZXJQYWdlOzMwMzY5Njk2MztBUzo0MTEwMDQ0OTczNTA2NTZAMTQ3NTAwMjUxNjkzMw%3D%3D&el=1_x_6&_esc=publicationCoverPdf https://www.researchgate.net/profile/Mohammad_Noaeen?enrichId=rgreq-d0def5b66d03e842c0653636795883b7-XXX&enrichSource=Y292ZXJQYWdlOzMwMzY5Njk2MztBUzo0MTEwMDQ0OTczNTA2NTZAMTQ3NTAwMjUxNjkzMw%3D%3D&el=1_x_7&_esc=publicationCoverPdf https://www.researchgate.net/profile/Guenther_Ruhe?enrichId=rgreq-d0def5b66d03e842c0653636795883b7-XXX&enrichSource=Y292ZXJQYWdlOzMwMzY5Njk2MztBUzo0MTEwMDQ0OTczNTA2NTZAMTQ3NTAwMjUxNjkzMw%3D%3D&el=1_x_4&_esc=publicationCoverPdf https://www.researchgate.net/profile/Guenther_Ruhe?enrichId=rgreq-d0def5b66d03e842c0653636795883b7-XXX&enrichSource=Y292ZXJQYWdlOzMwMzY5Njk2MztBUzo0MTEwMDQ0OTczNTA2NTZAMTQ3NTAwMjUxNjkzMw%3D%3D&el=1_x_5&_esc=publicationCoverPdf https://www.researchgate.net/institution/The_University_of_Calgary?enrichId=rgreq-d0def5b66d03e842c0653636795883b7-XXX&enrichSource=Y292ZXJQYWdlOzMwMzY5Njk2MztBUzo0MTEwMDQ0OTczNTA2NTZAMTQ3NTAwMjUxNjkzMw%3D%3D&el=1_x_6&_esc=publicationCoverPdf https://www.researchgate.net/profile/Guenther_Ruhe?enrichId=rgreq-d0def5b66d03e842c0653636795883b7-XXX&enrichSource=Y292ZXJQYWdlOzMwMzY5Njk2MztBUzo0MTEwMDQ0OTczNTA2NTZAMTQ3NTAwMjUxNjkzMw%3D%3D&el=1_x_7&_esc=publicationCoverPdf https://www.researchgate.net/profile/Zahra_Shakeri3?enrichId=rgreq-d0def5b66d03e842c0653636795883b7-XXX&enrichSource=Y292ZXJQYWdlOzMwMzY5Njk2MztBUzo0MTEwMDQ0OTczNTA2NTZAMTQ3NTAwMjUxNjkzMw%3D%3D&el=1_x_10&_esc=publicationCoverPdf Requirements Engineering Visualization: A Systematic Literature Review Zahra Shakeri Hossein Abad, Guenther Ruhe Department of Computer Science University of Calgary, Calgary, Canada Email: {zshakeri, ruhe}@ucalgary.ca Mohammad Noaeen Department of Electrical and Computer Engineering University of Calgary, Calgary, Canada Email: mohammad.noaeen@ucalgary.ca Abstract—Requirements Engineering (RE) is a decision-centric activity which is highly data-intensive. The results of this process are known to have key impact on the results of the project. As known from the experience in other fields and disciplines, visualization can potentially provide more insights into data, information and knowledge studied. While research in the area of information visualization and its application to software engi- neering has rapidly increased over the last decade, there is only a limited amount of studies addressing the usage and impact of visualization techniques for RE activities. In this paper, we report on the results of a Systematic Literature Review (SLR) related to RE visualization. Extending the established SLR process by the usage of grounded theory for the encoding of papers, we synthesize 18 usage patterns. Even though there are punctual applications, there is a clear deficit on a holistic perspective across the different RE activities. As another conclusion, we derive the clear need for more research on visualization support in particular for tackling requirements uncertainty, requirements verification, and modelling, as well as Non-Functional Require- ments (NFRs). I. INTRODUCTION AND BACKGROUND Requirements Engineering (RE) activities and tasks such as identifying projects stakeholders, exploring their needs and expectations, communicating requirements and goals, and managing and monitoring requirements changes are the most data-intensive and media-rich activities in every software de- velopment project [1]. In addition, decision-making lies at the heart of early RE activities, and most of these decisions, such as allocating requirements to a specific release or transferring business objectives to technical specifications, are often made with inevitable uncertainties about the final cost, schedule, user needs and expectations, and the systems’ functionalities [2]. The high level of uncertainty in requirements decisions, cou- pled with the massive, heterogeneous, and dynamic volumes of information in the RE process, makes this process the most error-prone activity in every software development project [1]. However, it is believed that visualization techniques which aim to facilitate information flow in RE activities and increase awareness of project stakeholders can improve the RE process, and consequently provide practical solutions to reduce misun- derstandings and communication gaps related to identifying and communicating the requirements [3]. Software Visualization (SV) is a field of Software Engi- neering (SE) that uses computers to visualize the software de- velopment process and its related artifacts (e.g. requirements, design documentations, bug reports), and it can be effectively used to support human reasoning and insight, and to foster the productivity of software development processes [4]. According to the results of an empirical experiment in which 111 re- searchers in SE were asked about the importance of SV, Rainer Koschke [5] reported that 82% of the participants believed SV is important for the success of the software development process. Likewise, Bassil and Keller [6] conducted a study on a SV tool with 107 participants mostly from industry. Based on the results of this study, better understanding of software, increasing productivity and quality of the development process, and complexity management are the main advantages for which practitioners use SV. The importance of RE visualization in effectively commu- nicating and managing stakeholder expectations and conflicts has been extensively discussed by researchers [1], [4], [5], [7]. RE visualization, such as visualizing RE decisions and their consequences and relationships, is critical to alleviate conflicts among stakeholders and to manage requirements changes. While gathering, categorizing, and analyzing the existing visualization techniques can be of use to practitioners and researchers seeking to apply these techniques in their work, there has been no effort to systematically gather, classify, and analyze the RE visualization literature. In order to fulfill this research gap in the literature and to summarize research on the application of visualization techniques in the area of RE, we conducted a Systematic Literature Review (SLR). Moreover, conducting this SLR is in line with the research agenda of our previous work [8] (presented at RENext! 2015) to visualize the uncertainty inherent in requirements debt decisions. This review seeks to analyze, classify, and present the common visualization techniques which have been developed to support various dimensions of RE, such as requirements activities, stakeholders and domains (Table I). Since various paradigms of visualization, such as Data Visualization (DV), Information Visualization (IV), and Knowledge Visualization (KV) differ regarding their ability to transfer and communi- cate data, in this review we classified studies from various visualization perspectives: (1) Data Visualization: graphical representation of unprocessed information [9] (e.g. points, lines or bars). (2) Information Visualization: Card et al. [10] defined IV as “The application of computer supported, visual representations of abstract data to amplify cognition”. (3) 2016 IEEE 24th International Requirements Engineering Conference 2332-6441/16 $31.00 © 2016 IEEE DOI 10.1109/RE.2016.61 6 RE 2016, Beijing, China Research Paper TABLE I VARIOUS DIMENSIONS OF RE RE Stakeholders [12], [13] -Users: who will actually operate and interact with the system -Developers: who design, build, and maintain the system, such as analyst, designer, programmer, tester, and so on -Decision-makers: who have executive power and control over projects decisions to build the system, such as the managers of the development and users organizations, business, and product managers. -Clients: who will pay for the system RE Domain [14] -Problem: requirements that represent the stakeholders’ needs and expec- tations, -Solution: requirements that represent the subsequent layers (e.g. system requirements) RE Activities [12], [15] -Elicitation: Identifying the users and project sponsors needs and expectations, projects stakeholders, sources of requirements, and understanding the application domain -Modelling: Representing the RE process as well as a whole range of RE artifacts. Enterprise, data, behavioural, and domain modelling are some general categories of RE modelling. -Communication: Circulating the identified requirements among project’s stakeholders. -Validation and Verification: Evaluating the completeness and correctness of the elicited requirements and artifacts. -Evolution: Managing and monitoring requirements changes (e.g. adding, deleting, or modifying requirements) Knowledge Visualization: Burkhard [11] defined the concept of KV as an approach that “examines the use of visual representations to improve the transfer of knowledge between at least two persons or group of persons”. The two significant contributions of this review to the RE visualization body of knowledge are: 1) This SLR reports on the process of identifying, gathering, classifying, synthesizing, and analyzing a set of relevant papers on RE visualization techniques based on prede- fined inclusion and exclusion criteria, and rigorous data extraction and analysis methods. 2) This SLR applied the grounded analysis technique to make a body of knowledge for RE visualization. This ev- idence can be of use to researchers and practitioners from various perspectives, such as identifying the research gaps in the area of RE visualization and elaborating on the existing RE visualization techniques and their relation with various visualization paradigms and various RE perspectives. The rest of this paper is structured as follows: Section II describes the details of our review methodology, including research questions, inclusion and exclusion criteria, our search strategies for both manual and automated searches, and the methodology for data extraction and analysis. In Section III, we discuss the results and findings of the review based on our research questions. Threats to the validity of our review results are discussed in Section IV. In Section V, we present related work and discuss the scope and limitations briefly. Finally, in Section VI, we conclude the review and provide recommendations for future research. II. REVIEW METHOD In this review we followed the guidelines provided by Kitchenham and Charters [16] and Zhang et al. [17] for conducting a SLR and identifying relevant studies in software engineering. This section elaborates on the steps we followed during our review process: A. Research Questions This review aims at addressing the following Research Questions (RQs): RQ1: What visualization techniques are used to support various dimensions of RE? The RQ1 aims at investigating how and to what extent the various dimensions of RE (i.e. activities, stakeholders, and domains) are supported by visualization. RQ 2: What are the different purposes of the visualiza- tion techniques applied in RE? This question aims to identify the main functionality of the visualization techniques used to support RE (e.g. coordination, recall, attention). RQ 3: What are the different visualization types used to support RE activities? This question seeks to find out the visualization types (e.g. sketch, diagram, image, map, object, etc.) that are most used to visualize various aspects of RE. RQ 4: How much evaluation is done to analyze the adap- tion of RE visualization techniques in practical contexts? This question guides us to identify the maturity of the ex- isting RE visualization techniques. The results of this question help researchers identify new topics of empirical studies, and enable practitioners to investigate the maturity of the proposed RE visualization techniques. B. Inclusion and Exclusion Criteria The following criteria were used as inclusion and exclusion criteria for the selection of the primary studies in our review: 1) Exclusion Criteria: we excluded publications that meet the following criteria: (1) publications that were not related to RE visualization (2) tutorials, proposals, and position papers and other non-peer reviewed publications (3) peer-reviewed publications that do not meet the minimum requirements of any type of visualizations: (V1) the results of the visualization should be based on the raw data. This requirement implies that image processing and photography, in which the source of data is an image are not treated as visualization. (V2) The outcome of the visualization should be one (or more) image(s) that represent and communicate the data. (V3) The results of the vi- sualization should be readable and understandable by a viewer (e.g. decision makers, requirements engineers) [18]. Visual variables such as size, shape, orientation, colour, value, and texture are basic characteristics of visual representations that must meet these requirements, as stated by Carpendale [19]. Considering these requirements, we excluded publications that only addressed UML and Goal modeling as a visualization technique (27 papers in total). While these techniques are widely being used by researchers and practitioners to model 7 system requirements, they consider only one of these required visual variables (shape), and, as stated by Diehl [4], the visual efficiency of these models is low (V2 and V3 are not met). 2) Inclusion Criteria: We included publications that: (1) addressed visualization techniques to represent various dimen- sions of RE (2) provided some details about their visualization technique and its application in RE (e.g. journal, conference, and workshop papers) (3) considered the visualization prereq- uisites discussed in Section II-B1. C. Search Strategy To retrieve the maximum number of relevant literature for our review, we conducted both manual and automated searches. To this end, we followed the steps provided by Zhang et al. [17] to identify the relevant studies in SE as follows: 1) Step 1- Manual Search: we first identified a set of publication venues and databases related to the areas of SE, RE, and information visualization. Next, these venues were evaluated and finalized by two experts from the areas of SE and information visualization independently, who determined if each of the identified venues should be included for the manual search step. These include: RE, REV (the Require- ments Engineering Visualization workshop, with the main focus on visualizing and representing RE activities), ICSE, IEEE VIS, VisSoft, EASE, FSE, JSS, IST. The included papers from this venue were a great input for the process of frequency analysis (section II-C2) that was performed to identify the search strings required for the automated search step. Next, we manually reviewed the proceedings and journals of these venues and judged their relevance to our research questions based on their title, keywords, and abstract. In situations where the inclusion decision was not possible based only on these components, we reviewed the conclusion, headers/sub-headers, and references of the papers. At the end of this step, we identified 16 papers that addressed RE visualization. To evaluate the reliability of our inclusion and exclusion decisions we used the Cohen’s Kappa statistic [20], which calculates the degree of agreement between two raters, in our case, the first two authors of the paper. The calculated Kappa value was 0.87, which shows significant agreement, as stated by Landis and Koch [21]. All of the identified papers in this step are used to form the Quasi-Gold Standard (QGS), an approach for evaluating the quality of the keywords string [17]. 2) Step 2- Identifying Search Strings: in this step, we used text mining techniques to extract the most frequently used words in the papers identified in the manual search step. To this end, we used the Text Mining Package (tm) 1 of R2. As stated by Keshav [22], the title, abstract, introduction, section/subsection headings (not their content), and conclusion of a paper represent its general idea and main contributions. Thus, before conducting the frequency analysis, we applied the main steps of text pre-processing on these parts of the 16 papers identified in the manual search step (Section II-C1). 1http://tm.r-forge.r-project.org/ 2https://www.r-project.org/ TABLE II SEARCH TERMS Category 1 Category 2 Category 3 Visualization � � Requirements Engineering � Functional � Visualisation � � Requirement � Non-functional � Model(ling) � � Goal � Software Engineering � � Design(ing) � � NFR(s) � Interaction � Release Planning � Communication � Stakeholder� Diagram � � Collaboration � For the purpose of pre-processing, we conducted the following steps iteratively: (1) Manual Transformation (e.g., removing hyphens which appeared after converting the pdf files to plain text). (2) Removing numbers and punctuations (e.g. years and in-text reference numbers). (3) Removing stop words (e.g. articles, conjunctions, and common verbs). (4) Stemming (e.g. reducing visualization, visualizing, and visualize to their root visual. Next, we performed the frequency analysis on the resulting text files and extracted the most frequent words of the 16 identified papers. Table II lists these words and demonstrates how the search terms can be combined to form the search strings using Boolean operations. Each search string can be formed by using a Boolean “AND” operator between search terms of different columns with the same colour, and a Boolean “OR” operator between different search strings. For instance, “Visualization AND Requirements Engineering” OR “Communication AND Stakeholder AND Software engineer- ing” are two valid search strings for the automated search step. 3) Step 3- Automated Search: in this step, we first queried Google Scholar (GS) using the search terms listed in Table II. However, Dean et al. [23] recently performed an experiment to evaluate if GS is sufficient to be used for systematic reviews, and concluded that GS does not meet the required search standards for systematic reviews, and is not able to identify all known and relevant papers in a specific area. Thus, at the next step we searched the following electronic databases to find more relevant papers: ACM Digital Library, ScienceDirect, SpringerLink, and IEEE Xplore. This yielded 559 articles published within the last 16 years (2000- 2016), the studies before 2000 were only focused on UML modeling as a technique of RE visualization. 4) Step 4- Quasi-Gold Standard Evaluation: To evaluate the reliability and quality of the search terms we identified during the manual search step, we followed the guidelines provided by Zhang and Babar [24] and formed the Quasi-Gold Standard for years 2006-2010 (five years of REV workshop), which includes 16 papers. After performing the automated search, we retrieved 15 of these studies. Thus, the quasi- sensitivity is 94%, which represents an acceptable performance for our search process. 5) Step 5- Snowballing: To complement the search results obtained, we used snowballing by scanning the reference list (backward snowballing [25]) of identified papers. Furthermore, there were papers that have been published after the selected 8 Fig. 1. Number of studies per type of visualization they supported per year. DV IV KV 42% 39% 19% #o f Pa pe rs Year of publication total papers, so we performed citation tracking (forward snow- balling [25]) to identify relevant papers citing the papers in the previous step. At the end of this stage, we added three more papers to the list of the primary studies of this review, a total of 26 papers for our analysis. D. Data Extraction and Synthesis Method During this step, we used the following two methods to extract and synthesize data from each of the 26 primary studies included in this review: (1) Grounded Analysis: During this step, we applied a revised version of the grounded theory [26] methodology for conducting a rigorous literature review and followed the steps we implemented in our previous SLR [27] to extract data from each of the 26 primary papers included in this review. We analyzed all of the included papers line by line to extract concepts, which involves high volumes of qualitative data. To help with this process, we used Saturate 3, which is a web- based open coding tool that enables traceability between codes and data. Due to limited space, we only report the results of the open coding and axial coding steps (Table III). (2) Data Extraction Form: To organize the process of data extraction, we created three data collection forms, in which the following data points should be captured by the reviewer of each paper: (1) visualization paradigm (e.g. data, information, or knowledge visualization) (2) various dimensions of RE (i.e. RE activities, stakeholders, and domain) (3) various perspectives of the visualization that have been addressed for RE (i.e. function, audience, goal, and type). More details about these parameters are listed in Tables IV, V, and VI. III. RESULTS AND FINDINGS A. Overview of Papers This section gives an overview on the results of analyzing the primary studies included in this SLR in terms of their published year and visualization paradigm. As illustrated in Figure 1, more than 50% of these studies are published in the period 2010-2012 and there is an increasing trend on supporting knowledge visualization in RE from 2006. 3www.saturateapp.com 1) Visualization Paradigm: Since RE is a high communi- cation and decision intensive activity, knowledge visualization can help to improve the communication among stakeholders and reduce the communication gaps and conflicts among end-users and technical stakeholders of software development projects. However, based on the results of this SLR, only 19% [P22-26] of the studies addressed this visualization paradigm, while 39% ([P12-21] and 42% ([P1-11]) of the studies ad- dressed information and data visualization, respectively. TABLE III OVERVIEW OF OPEN CODES EXTRACTED FROM THE PAPERS INCLUDED IN THIS REVIEW. NUMBERS IN PARENTHESES SHOW THE NUMBER OF OCCURRENCES OF EACH CODE IN THE INCLUDED PAPERS Categories Open codes Requirements Communication (42) Distributed Requirements Engineering (10) [P1, P9, 10], (# of papers: 3) Stakeholders Communication (14), [P1-3, P9-10, P12, P21-22, P25] (# of papers: 9) Communication Gap (18) [P2, P6-7, P9, P12-13, P17, P20, P23, P25-26] (# of papers: 11) Requirements Evolution (47) Requirements Change (23) [P6, P11-12, P14, P21, P24-26], (# of papers: 8) Tracing Requirements (11) [P10, P14-15, P18, P23, P26], (# of papers: 6) Requirements Relationships (13) [P6, P10-11, P18, P23, P25-26], (# of papers: 7) NFRs (19) Non- Functional Requirements (19) [P4-5, P13, P15, P18, P20, P22, P25] (# of papers: 8) Requirements Inspection (27) Requirements Uncertainty (2) [P23-24], (# of papers: 2) Requirements Validation (8) [P6, P10, P17, P26], (# of papers: 4) Requirements Verification(3) [P12, P24], (# of papers: 2) Requirements Planning (27) Requirements Prioritization (15) [P4, P7, P11, P13, P17, P22-25], (# of papers: 9) Requirements Planning (12) [P1, P11, P22-24], (# of papers: 5) With respect to Knowledge Visualization (KV), S. Feather et al. [P1] provided two visualization types (i.e. diagram(s) and Map) to help project stakeholders to understand the how and why of their decisions. For instance, they used a 2D scatter plot to represent the likelihood and impact of a requirement’s risk(s). They also used Kiviat chart to represent the comparison among several risk mitigation decisions. 2) Requirements for RE Visualization: we used grounded analysis to extract data from the primary of this review. By using this technique, we coded all of these papers to explore the main areas (dimensions) of RE in terms of their need of visualization support. After similarity analysis of these codes (open codes and axial codes), we found (Table III): Requirements evolution and communication were among the most RE activities with visualization support, with each of them being 30% and 26% of the extracted codes assigned to each category, respectively. Requirements inspection and planning come next, with 19% of the codes for each category. Visualizing Non-Functional Requirements (NFRs) had less support, with only 12%. More details about the results of applying grounded analysis are available on the website of 9 the first author of the paper 4. Finding 1: Applying grounded analysis revealed that Require- ments Planning and Inspection and NFRs have the strongest deficit of visualization support. B. RQ1- Visualization Support for Various Aspects of RE In this section, we report on the results of our review regarding the current status of visualization support for various aspects of RE. Table IV lists the studies that supported various dimensions of RE, as well as some examples of visualization techniques proposed to support these dimensions. Based on the results of our review, as demonstrated in Fig- ure 2 and Table V, representing a project’s requirements and their related artifacts (what) is addressed in all of the included papers in our SLR. Studies that addressed the process (how) of performing RE activities in their proposed visualization techniques come next with 19 (73%) studies. Representing the motivation of performing a specific RE task or making a requirement decision, as well as representing the roles (who) which are involved in performing the task or making the decision are addressed in 13 (50%) and 8 (31%) studies, respectively. Software development teams and companies are often geographically far from their users and customers. While visual representation of RE tasks and decisions can alleviate the difficulties of communication and coordination in globally distributed RE teams, visualization support for distributed RE is only addressed in 4 (15%) of the studies. 1) RE Activities: As listed and illustrated in Table IV, requirements elicitation is the most supported task by vi- sualization among all RE activities, with 16 (62%) of the papers. Visualization support for other RE activities: require- ments verification, modelling, evolution, and communication are addressed in 10 (38%), 8 (31%), 8 (31%), and 5 (19%) of the papers, respectively. Figure 2 depicts visualization support for various combinations of RE activities. We performed further analysis on the results of this RQ and extracted the common RE Visualization Patterns in the form of < content, focus >, where content shows
each of the RE activities, and focus denotes the What,
How, Why, Where, and Who components of each visualiza-
tion technique. The details of these visualization patterns are
illustrated in Figure 2. As illustrated in this figure, none
of the visualization techniques proposed by the included
studies in our review addressed all of the RE activities.
Further, two combinations of (elicitation,modelling) and
(elicitation, communication, validation) are the most fre-
quent patterns addressed by these studies.

Finding 2: Based on the results of the analysis we performed
on the results of RQ1, we extracted the common RE visualization
patterns in the form of < content, focus >, where content
shows each of the RE activities, and focus denotes the What,

4http://wcm.ucalgary.ca/zshakeri/home

What?

How?

Why?

Where?

Who?

requirements, RE artifacts

representing the process of performing an
RE activity

the motives and causes of RE decisions in
each activity

geographically location (distributed
teams)

peoject’s stakeholders, sources of
requirements

(a) Legend

(b) [P2]

(d) [P1]

(e) [P4]

(f) [P5]

(g) [P6], [P7]

(h) [P8]

(i) [P9]

(j) [P10],
[P11]

(k) [P12],
[P13]

(l) [P14]

(m) [P15],
[P16]

(n) [P17]

(o) [P18],
[P19]

(p) [P20]

(q) [P21],
[P22]

(r) [P23]–[P25]

(s) [P26]

Fig. 2. The common RE visualization patterns (< content, focus >) [E:
Elicitation, M: Modeling, C: Communication, V: Validation, Ev: Evolution]

How, Why, Where, and Who components of each visualization
technique.

This finding can be used as a reference for selecting visu-
alization techniques for a specific combination of RE activities
and visualization goal (focus).

2) Requirements Engineering Stakeholders: The results of
our review show that “developers” and “decision makers” are
the most supported stakeholder group in the studies included
in our review (18 (69%) studies for each category). Among
all the studies, 15 (58%) studies targeted end-users in their
visualization approach, and customers have less visualization
support, with only 8 (31%) of the studies.

3) Requirements Engineering Domain: As illustrated in
Table IV, while 88% of the studies supported the problem
domain in their visualization techniques, only 50% of the
studies addressed the solution domain.

C. RQ2- Different Visualization Functions Applied in RE

To address this RQ, we used data extraction forms to
explore different visualization purposes (functions) proposed
by researchers and practitioners. We used the following func-

TABLE IV
RQ1- VISUALIZATION SUPPORT FOR VARIOUS DIMENSIONS OF RE

Dimensions /Related papers The Proposed Visualization Techniques
Elicitation (E)
[P1-2, P4-6, P8, P10, P12-13,
P16-20, P23-24]

• [P19]: A Stacked Pareto chart and horizontal bar charts to visualize the requirements priority and the correlation
of each stakeholders votes with the resulting priorities, respectively.

Modelling (M)
[P3, P5-7, P10-11, P13, P20] • [P15]: (1) The application of visual objects (glyphs) and graphical variables (i.e. colour, thickness) to model product-line requirements relationships and inter-dependencies. (2) Highlighting the decisions with the greatest number of

inter-dependencies to visualize the strength of the dependencies on the decisions as well as the difficulty of decisions.

Communication (C)
[P9, P11, P19, P22, P26] • [P26]: A visual interactive representation of requirements to involve a large number of geographically distributedusers in the process of RE. This approach improves communicating the requirements in two ways: (1) filtering

requirements based on their geo-coordinate, and (2) filtering requirements based on the user-assigned keywords,
which represents the similarities and dependencies between requirements.

Verification (V)
[P6, P12-15, P17-18, P22-23,
P26]

• [P12]: A visualization technique to represent NFR patterns including problem patterns (e.g. threats, and vulnerable
and undesirable situations) and objective patterns (e.g. different interpretations of NFRs that are provided by a
project’s stakeholders and represents their conflicts in defining these NFRs).

R
E

A
ct

iv
iti

Evolving (Ev)
[P2, P8, P12, P14, P18, P21,
P24, P26]

• [P10]: An interactive visualization technique, which enables a project’s stakeholders to directly manipulate
requirements in real time during their decision making process.

End users (Eu)
[P3, P6-7, P9-10, P12-13, P15,
P17, P19-22, P25-26]

• [P21]: A visualization technique for end-users with different levels of familiarity with computer applications, by
which users can select the features of the system from a list of predefined visual objects and make an image of the
product based on their needs and preferences.

• [P4]: A 3D visualization of requirements to reduce the gap between developers and end users and to validate the
requirements in the earlier stages of the development process.

Developers (D)
[P1-2, P4-6, P8-9, P13-14,
P17-23, P25-26]

• [P7]: Integrating visual objects, specific to scientific software development, into the requirements models to reduce
the learning effort for developers who build scientific applications to successfully create their requirements model
and have concrete ideas of the system before starting the development tasks.

Decision-makers (Dm)
[P1, P3-7, P9, P11, P13, P15,
P19-26]

• [P6]: A three dimensional, space-filling, and growing pyramid, which provides various views of the selected
requirements for each group of projects stakeholders. This visualization technique mainly represents the following
parameters for each of the system requirements: (R = {Sn,W,Ref}), while Sn, W , and Ref denote the
stakeholders to which requirement belongs, requirements scope in terms of its attainment level, and the level of the
requirements refinement, respectively.

St
ak

eh
ol

de
rs

Customers (C)
[P3, P7, P9, P12, P16, P21,
P25-26]

• [P16]: A separate graphical object for the system’s customers to differentiate them from end-users and business
stakeholders and to involve them in RE activities and consequently make their needs and operations more clear and
understandable (Figure 3(e)).

Problem (P)
[P1, P3-7, P9-13, P15-26] • [P17]: A Visual Requirements Analytics (VRA) technique for improving risk assessment decisions by using thefollowing graphical variables on cohesive bar and arc graphs: (1) colour for representing the level of requirements

attainment, (2) height for representing the correlation of a requirement category with other requirements categories
in terms of risk components, (3) width for representing the ratio of related risks to a requirements, and (4) order
for prioritizing requirements based on their impact on the risk components.

D
om

ai
n

Solution (S)
[P2, P4-6, P8-9, P13-14, P18-
19, P21, P23, P25]

• [P23]: A visualization technique to model NFRs, which impacts various requirements decisions, regardless of if
they are system-level or software level.

• [P11]: A visual tool (DREAMER) to represent and model the traceability and coverage of both requirements and
design options during the development process.

tions, proposed by Burkhard [11] to classify the visualization
functions proposed by included studies in our review.

• Coordination: coordinating and managing individuals
during the communication process.

• Recall: improving the understandability and remem-
brance ability of viewers by using conceptual diagrams.

• Maps: following cartographic standards to represent hier-
archy of data (e.g. a ground layer represents the project’s
context and individual elements on this layer represent
the project’s milestones, risks, or resources).

• Motivation: inspiring viewers to take actions.
• Attention: grabing viewer attention by representing

trends, outliers, and in general all the characteristics that
impact viewer decisions.

• Elaboration: providing more clarification about visual
representations.

• New Insights: creating new insights by showing relation-
ships between elements and by demonstrating the context
patterns and details.

Table V lists these functions, the studies that addressed
each function, and some examples of visualization techniques
proposed for each function. Among all the studies, 15 (58%)
studies addressed the attention function, which is the most
addressed function. Elaboration and Motivation come next,
with 13 (50%) studies each. Coordination and insight are
addressed in only 10 (38%), and 9 (35%) studies, respectively.
Recall has the least support with only 4 (15%) studies.

Moreover, we analyzed the proposed RE visualization tech-
niques from their audience perspectives, such as individuals,

TABLE V
RQ2- DIFFERENT PURPOSES OF VISUALIZATION TECHNIQUES APPLIED IN RE

Visualiation Functions Some Samples of the Proposed Visualization Techniques
Coordination
[P9, P11, P19, P22, P26] • [P10]: An interactive Visual Requirements Analytics (VRA) approach which helps stakeholders to overview the systemrequirements, detect inconsistencies and anomalies, and relate heterogeneous artifacts and concerns.

• [P20]: A notation-based visualization technique to represent the flow of requirements by visualizing both informal and
formal communications, and to coordinate the communication among stakeholders in global RE teams.

Attention
[P2, P4-7, P11, P13-15, P18,
P20, P22-24, P26]

• [P23]: The application of visual variables (e.g. size, colour, texture, and value) to represent four quality attributes such
as trustability, performance, feasibility, and certainty, respectively.

Recall
[P3, P5, P8, P10] • [P5]: A set of semantical transparent modelling symbols, which allow end-users without a goal modelling background toparticipate in RE activities.

• [P8]: Feature Survival Charts (FSC) as to represent dynamic scope changes of projects and past project scoping activities.

Motivation
[P1-2, P6, P11, P14-15, P20-
24, P26]

• [P2], [P3]: highlighting the flow of project resources and requirements prioritization options to help project stakeholders
to compare multiple alternative lists of requirements together, and select the next release based on the current status of
resources and to alert teams about requirements changing priorities.

Elaboration
[P6, P8, P10, P15, P17-23,
P25-25]

• [P13]: Highlighting goals and other elements within a conflicting path, and sources of these conflict to help decision-
makers to easily understand conflicts and their intention, and to identify the domain trade-offs.

• [P11]: A bifocal view of diagrams, which allows viewers to focus on a particular object (e.g. requirement, design object,
or artifact) to explore its connection with other objects of the visualization.

Insights
[P1, P13-15, P22-26] • [P14]: The application of visual variables such as colour and value to propose a traceability visualization technique thatanalyzes and represents the candidate links between project requirements, a visual feedback to validate these links, and

represents requirements with high architectural significance and high level of design risk.

group, organization, and network. Based on the results of
this analysis, all the included studies in this SLR targeted
individuals in their visualization techniques (e.g. requirements
analysts, architects, developers, or end-users). Supporting RE
activities in group (or RE teams) comes next with 19 (73%)
studies [P3-7, P9, P11, P13-16, P19-26]. Visualization support
at the organization or network level are addressed by only 7
(27%) [P7, P9, P16, P19-20, P22, P25] and 4 (15%) [P1, P9,
P16, P19] studies, respectively.

Finding 3: Following the results of “grounded analysis”, re-
quirements communication and change require more visualiza-
tion support. However, coordination function, which support this
requirement, has been addressed by only 19% of studies. Thus,
there is a clear need for more visualization techniques that
support this function.

D. RQ3- Different Visualization Types Used to Support RE
Activities

To address this RQ, we used data extraction forms to
explore visualization types proposed to support various dimen-
sions of RE (See Table I for more details about these dimen-
sions). Table VI gives an overview of these visualization types,
the list of studies addressed each type, and some visualization
techniques proposed for each type. Overall, looking at Table
VI, 81% (21) of the included studies in this SLR used diagrams
(e.g. Sankey, Kiviat and bar charts, and 2D scatter plot) to
represent the structural relationships among various activities
and artifacts of RE. Among all the studies, 11 (42%) proposed
their own graphical objects to add more visual intuitiveness to
the visual representations of RE activities and artifacts. Map
and interactive representation of RE activities come next, with

7 (27%) studies for each type. Image and Sketch were used in
only 4 (15%) and 2 (8%) studies, respectively.

With respect to storytelling as a visualization type that
represents a sequence of events, decisions, or changes, each of
which can contain various types of visualization (e.g. image,
diagram, or sketch), text or video or any combination of these
components [28]. Based on the results of our analysis, none
of the included studies (0%) in this review addressed the
application of this technique in RE.

Finding 4: Visualization of Requirements change is not well
supported by literature
Visualization types that can manage various complexities and
challenges inherent in the process of requirements change,
are not well supported by researchers and practitioners. For
example, by using storytelling visualization type, various as-
pects of requirements change can be represented by narrations
explaining the current state and the changes as they occur over
time. Moreover, filling gaps and continuity in the storytelling
approach, as stated by Gershon and Page [28], increases the
audience’s awareness of the transition between various states of
the system as well as the sequence of requirements decisions.

E. RQ4- Evaluation of Proposed RE Visualization Techniques

To answer this research question, we used the following
hierarchy to evaluate the evidence level [29] of the proposed
visualization techniques: (E1) no evidence, evidence obtained
from: (E2) demonstration, (E3) expert opinion, (E4) academic
studies, (E5)industrial studies, (E6) industrial practice. Based
on the results of our analysis, the main strategies for empirical
evaluation are E3 (27%) [P1, P6, P12, P15-16, P20, P22],
E5 (27%) [P4-5, P7-8, P10, P17-18], E6 (19%) [P13, P23-
26], E4 (15%) [P2-3, P9, P21], E1 (8%) [P11, P19], and

TABLE VI
RQ3- DIFFERENT VISUALIZATION TYPES USED TO SUPPORT RE ACTIVITIES

Visualization Types
Image and Sketch
[P2-3, P12, P17] • [P4]: The application of 3D and animation visualization techniques to develop a 3D image of the final product and tovalidate its features with the end users. This approach reduces the total development effort by validating requirements in

the earlier stages of the project.
Diagram
[P1, P4-11, P13-16, P18-20,
P22-26]

• [P3]: Combination of Sankey diagrams and Parallel Coordinates in a multiple tree layout to represent the information
flow of requirements prioritization and release planning processes.

• [P1] The application of bar, Kiviat, 2D scatter plot, and charts and range to convey the risk status of project requirements
and candidate decisions to mitigate these risks and to represent the consequence of uncertainties in the input data about
requirements risks and their mitigation decisions.

Map
[P2, P19, P21-23] • [P1]: TreeMap visualization to show requirements and sub-requirements hierarchy, relative importance, and attainmentstatus of each requirement by using size and colour parameters respectively.

• [P22]: Emotional Density Map (EDM) to demonstrate the intended emotional requirements in designing and developing
video games by using grayscale shading and graphical objects.

Object
[P1-3, P7-9, P11-13, P18, P26] • [P18], [P22]: The application of graphical objects to manage communicating requirements in distributed teams (Fig 3(a,b)) and to capture and represent the primary and secondary emotional requirements.

• [P5]: Proposing a set of semantical transparent symbols for Goal-oriented modeling by conducting an exploratory study
with 104 participants (without any background knowledge of goal modeling). Figure 3 (c,d).

• [P16]: Using semiotic clarity (i.e. defining one notational element for each of the concepts in requirements modeling)
to differentiate business stakeholders from end-users, and to represent the concepts of goal, soft-goal, threats, and
requirements risk mitigation (Figure 3 (e)).

Interactive Visualization
[P6, P10, P12, P17-18, P22,
P26]

• [P3]: An interactive brushing technique to help project decision makers to interactively reveal the relationship in data
(e.g. tapping on each of the elicited requirements highlights all of the releases it belongs to, as well as the representing
stakeholder vote about that specific requirement).

Storytelling –

User
Requirementd

Analyst
Flow Diagram

(a) Graphical objects [P18]
Sad Sheepish Surprised

(b) Graphical objects [P22]

(d) Prototype Symbol Set [P5]

(e) Prototype Symbol Set [P16]

Fig. 3. Application of graphical objects (Glyph) in RE visualization

E2 (4%) [P14]. Given these data and according to the above
classification, most of the studies used preliminary evaluation
methods to evaluate their proposed visualization techniques.
No evidence occurred in two studies [P11, P19]. Additionally,
almost all of these studies employed exactly one type of
evaluation, which implies that there is a clear need for more
evidence to evaluate the quality of the existing RE visualiza-
tion techniques.

IV. THREATS TO VALIDITY

In this SLR, to provide a comprehensive review on the stud-
ies on RE visualization, we followed the guidelines presented
by Kitchenham [16] and Zhang et al. [17] for performing a
SLR in software engineering. However, there are threats to the
validity of the results and findings of our review.

Firstly, the completeness of the list of the primary studies
included in this SLR highly depends on the keywords and
the capability limitations of the employed digital libraries
and search engines. While we used an objective search terms
elicitation approach (i.e. text mining and frequency analysis)
to reduce the risks of subjective search terms, the search
standards and syntax vary among the employed search engines
and libraries in this review, thus we may have missed some
studies related to the research questions of our review.

Secondly, the robustness and comprehensiveness of the
framework we used to classify the studies may affect the
results and findings of this review. We used the framework
proposed by Burkhard [11] to classify the studies based on
their visualization function, goals, types, and audience. To
avoid a framework with insufficient paradigms for the purpose
of classification, we selected a framework related to knowledge
visualization, which covers the other visualization approaches
(i.e. data and information visualization). Moreover, to reduce
the classification bias, all of the classification results were
checked by the first two authors.

V. RELATED WORK

With respect to the related literature reviews on the area
of visualization support for RE, so far, to the best of our
knowledge, there is only one study provided by Amyot and

Mussbacher [30] aimed at describing and analyzing the first
ten years of the User Requirements Notation (URN). The
authors did a study inspired by the systematic literature review
approach, in which 281 research papers related to URN were
analyzed and synthesized to provide a historical overview of
the development of URN together with research implications
and application domains for this notation. However, the review
is limited to URN and does not address other aspects of RE,
such as RE activities and stakeholders. To our knowledge,
the SLR we provided in this paper is the first SLR that
aims to analyze and synthesize the existing evidence on RE
visualization, and to provide insight into the existing research
trend in this area as well as future research implications.

Regarding other types of reviews, such as survey or litera-
ture review, there are different reviews existing that addressed
RE visualization. Cooper Jr et al. [3] conducted a survey of
the research papers that were presented from 2006 to 2008
at the RE Visualization (REV) workshop to represent the
evolution of the research trends in the area of RE over a
specific time period. According to the results of this survey,
many opportunities exist to develop visualization for the early
steps of the RE phase (e.g. understanding the context and
undertaking the groundwork necessary for the RE process)
as well as verification and validation tasks, which is in
line with one of the main findings of this SLR. Gotel et
al. [1], [7] provided two reviews on the primary areas in
which visualization techniques and artifacts can be applied
to support RE activities. The results of this review highlight
the need for visualization techniques to support the multi-
dimensional nature of requirements and to support various
diagnostic activities and decision making tasks during software
development. However, these reviews have been conducted
about 7 to 9 years ago. Thus, new studies are required to
analyze and to synthesize the existing research works in this
area and to define the research trends and implications for
other researchers.

VI. CONCLUSION AND RESEARCH IMPLICATIONS

Visualization techniques are increasingly being used by
researchers and practitioners to understand and to manage RE
decisions and activities. By conducting this SLR, we gath-
ered, classified, and analyzed these techniques from various
perspectives to address the key research questions mentioned
in Section II-A. To this end, we initially identified 559 studies
by querying the employed search engines and digital libraries
(Table VII). After applying the inclusion and exclusion criteria
that were identified at the beginning of our review process
(Section II-B) we selected 26 studies as the primary studies of
our review, while the others were not related to the research
questions. The findings of this review tell us the following:
(1) More investigation and research are needed to support
knowledge visualization in the area of RE, (2) There is
no visualization support for the whole of the RE lifecycle,
(3) Among all RE activities, requirements communication
and evolution (Table IV) have less visualization support, (4)
Visualizing NFRs and requirements uncertainties need more

TABLE VII
AUTOMATED SEARCH RESULTS

Library/publisher #Retrieved #QGS #Selected
IEEE Xplore 311 13 16
ACM DigitalLibrary 28 0 2
SpringerLink 46 1 5
ScienceDirect 31 1 0
Others 143 0 3
Total 559 15 26

investigation, (5) There is a clear need for more effort in
addressing the following visualization types in the context of
RE: storytelling, maps, and interactive visualization, (6) There
is a clear need for more visualization support for distributed
RE, (7) The results of RQ4 show that further evaluation of
the existing visualization methods is mandatory to provide
better evidence regarding the quality and maturity of proposed
visualization methods.

VII. ACKNOWLEDGEMENT

This research was supported by the Natural Sciences and
Engineering Research Council of Canada (NSERC) Discovery
Grant 486565-15 and by an Engage Grant. We appreciate the
constructive comments given by Barbara Kitchenham related
to a former version of the paper.

REFERENCES
[1] O. Gotel, F. Marchese, and S. Morris, “The potential for synergy be-

tween information visualization and software engineering visualiza-
tion,” in Information Visualisation, 2008. IV ’08. 12th International
Conference, 2008, pp. 547–552.

[2] A. Aurum and C. Wohlin, “The fundamental nature of requirements
engineering activities as a decision-making process,” Information and
Software Technology, vol. 45, no. 14, pp. 945 –954, 2003.

[3] J. Cooper, S. W. Lee, R. Gandhi, and O. Gotel, “Requirements Engi-
neering Visualization: A Survey on the State-of-the-Art,” in The 4t̂h
International Workshop on Requirements Engineering Visualization,,
2009, pp. 46–55.

[4] S. Diehl, Software visualization: visualizing the structure, behaviour,
and evolution of software. Springer Science & Business Media, 2007.

[5] R. Koschke, “Software Visualization for Reverse Engineering,” in
Software Visualization, Springer, 2002, pp. 138–150.

[6] S. Bassil and R. K. Keller, “Software Visualization Tools: Survey and
Analysis,” in Program Comprehension, 2001. IWPC 2001. Proceed-
ings. 9th International Workshop on, 2001, pp. 7–17. DOI: 10.1109/
WPC.2001.921708.

[7] O. Gotel, F. Marchese, and S. Morris, “On Requirements Visual-
ization,” in Requirements Engineering Visualization, 2007. Second
International Workshop on, 2007, pp. 11–11.

[8] Z. S. H. Abad and G. Ruhe, “Using Real Options to Manage Techni-
cal Debt in Requirements Engineering,” in Requirements Engineering
Conference (RE), 2015 IEEE 23rd International, 2015, pp. 230–235.

[9] J. Hey, “The Data, Information, Knowledge, Wisdom Chain: the
Metaphorical Link,” Intergovernmental Oceanographic Commission,
2004.

[10] S. K. Card, J. D. Mackinlay, and B. Shneiderman, Readings in
Information Visualization: Using Vision to Think. Morgan Kaufmann,
1999.

[11] R. Burkhard, “Towards a Framework and a Model for Knowledge
Visualization: Synergies Between Information and Knowledge Visu-
alization,” in Knowledge and Information Visualization, ser. Lecture
Notes in Computer Science, S.-O. Tergan and T. Keller, Eds.,
vol. 3426, Springer Berlin Heidelberg, 2005, pp. 238–255.

[12] B. Nuseibeh and S. Easterbrook, “Requirements Engineering: A
Roadmap,” in Proceedings of the Conference on the Future of
Software Engineering, ser. ICSE ’00, ACM, 2000, pp. 35–46.

[13] H. Sharp, A. Finkelstein, and G. Galal, “Stakeholder Identification in
the Requirements Engineering Process,” in Database and Expert Sys-
tems Applications, 1999. Proceedings. Tenth International Workshop
on, 1999, pp. 387–391.

[14] E. Hull, K. Jackson, and J. Dick, Requirements Engineering. Springer
Science & Business Media, 2010.

[15] D. Zowghi and C. Coulin, “Requirements elicitation: a survey of
techniques, approaches, and tools,” in Engineering and managing
software requirements, Springer, 2005, pp. 19–46.

[16] B. Kitchenham and S. Charters, “Guidelines for Performing Sys-
tematic Literature Reviews in Software Engineering,” in Technical
Report, Ver. 2.3 EBSE Technical Report. EBSE, 2007.

[17] H. Zhang, M. A. Babar, and P. Tell, “Identifying relevant studies in
software engineering,” Information and Software Technology, vol. 53,
no. 6, pp. 625 –637, 2011.

[18] R. Kosara, “Visualization Criticism – The Missing Link Between
Information Visualization and Art,” in Information Visualization,
2007. IV ’07. 11th International Conference, 2007, pp. 631–636.

[19] M. Carpendale, “Considering Visual Variables as a Basis for Infor-
mation Visualization,” 2003.

[20] J. Cohen, “Weighted Kappa: Nominal Scale Agreement Provision for
Scaled Disagreement or Partial Credit,” Psychological Bulletin, vol.
70, no. 4, p. 213, 1968.

[21] J. R. Landis and G. G. Koch, “The Measurement of Observer
Agreement for Categorical Data,” Biometrics, pp. 159–174, 1977.

[22] S. Keshav, “How to Read a Paper,” SIGCOMM Comput. Commun.
Rev., vol. 37, no. 3, Jul. 2007.

[23] D. Giustini and M. N. K. Boulos, “Google Scholar is not Enough
to be Used Alone for Systematic Reviews,” Online journal of public
health informatics, vol. 5, no. 2, p. 214, 2013.

[24] H. Zhang and M. Ali Babar, “On Searching Relevant Studies in
Software Engineering,” in Proceedings of the 14th International
Conference on Evaluation and Assessment in Software Engineering,
ser. EASE’10, UK, 2010, pp. 111–120.

[25] T. Greenhalgh, R. Peacock, et al., “Effectiveness and efficiency of
search methods in systematic reviews of complex evidence: audit of
primary sources,” Bmj, vol. 331, no. 7524, pp. 1064–1065, 2005.

[26] J. F. Wolfswinkel, E. Furtmueller, and C. P. Wilderom, “Using
grounded theory as a method for rigorously reviewing literature,”
European Journal of Information Systems, vol. 22, no. 1, pp. 45–55,
2013.

[27] Z. Shakeri Hossein Abad, C. Anslow, and F. Maurer, “Multi Surface
Interactions with Geospatial Data: A Systematic Review,” in Pro-
ceedings of the Ninth ACM International Conference on Interactive
Tabletops and Surfaces, ser. ITS ’14, ACM, 2014, pp. 69–78.

[28] R. Kosara and J. Mackinlay, “Storytelling: The Next Step for Visu-
alization,” Computer, vol. 46, no. 5, pp. 44–50, 2013.

[29] V. Alves, N. Niu, C. Alves, and G. Valena, “Requirements Engineer-
ing for Software Product Lines: A Systematic Literature Review,”
Information and Software Technology, vol. 52, no. 8, pp. 806 –820,
2010.

[30] D. Amyot and G. Mussbacher, “User Requirements Notation: The
First Ten Years, The Next Ten Years,” Journal of software, vol. 6,
no. 5, pp. 747–768, 2011.

PAPERS INCLUDED IN THIS REVIEW
[P1] M. Feather, S. Cornford, J. Kiper, and T. Menzies, “Experiences using

Visualization Techniques to Present Requirements, Risks to Them,
and Options for Risk Mitigation,” in Requirements Engineering Vi-
sualization, 2006. First International Workshop on, 2006, pp. 10–10.

[P2] T. A. Sedbrook, “Visualizing Changing Requirements with Self-
organizing Maps,” The Journal of Computer Information Systems,
vol. 45, no. 2, pp. 63–72, 2004.

[P3] B. A. Aseniero, T. Wun, D. Ledo, G. Ruhe, A. Tang, and S.
Carpendale, “STRATOS: Using Visualization to Support Decisions
in Strategic Software Release Planning,” in Proceedings of the 33rd
Annual ACM Conference on Human Factors in Computing Systems,
ser. CHI ’15, ACM, 2015, pp. 1479–1488.

[P4] A. R. Teyseyre, “3D Requirements Visualization,” Journal of Com-
puter Science and Technology, vol. 3, 2003.

[P5] P. Caire, N. Genon, P. Heymans, and D. Moody, “Visual Notation
Design 2.0: Towards User Comprehensible Requirements Engineering
Notations,” in Requirements Engineering Conference (RE), 2013 21st
IEEE International, 2013, pp. 115–124.

[P6] D. Savio, P. Anitha, A. Patil, and O. Creighton, “Visualizing Re-
quirements in Distributed System Development,” in Requirements
Engineering for Systems, Services and Systems-of-Systems (RES4),
2012 IEEE Second Workshop on, 2012, pp. 14–19.

[P7] Y. Li, M. Harutunian, N. Narayan, B. Bruegge, and G. Buse, “Re-
quirements Engineering for Scientific Computing: A Model-Based
Approach,” in E-Science Workshops (eScienceW), 2011 IEEE Seventh
International Conference on, 2011, pp. 128–134.

[P8] K. Wnuk, B. Regnell, and L. Karlsson, “What Happened to Our
Features? Visualization and Understanding of Scope Change Dynam-
ics in a Large-Scale Industrial Setting,” in Requirements Engineering
Conference. RE ’09. 17th IEEE International, 2009, pp. 89–98.

[P9] T. Ugai, “Visualizing Stakeholder Concerns with Anchored Map,” in
Human Interface and the Management of Information. Interacting
with Information, Springer, 2011, pp. 268–277.

[P10] S. Reddivari, S. Rad, T. Bhowmik, N. Cain, and N. Niu, “Visual Re-
quirements Analytics: A Framework and Case Study,” Requirements
Engineering, vol. 19, no. 3, pp. 257–279, 2014.

[P11] C. Martinie, P. Palanque, M. Winckler, and S. Conversy,
“DREAMER: A Design Rationale Environment for Argumentation,
Modeling and Engineering Requirements,” in Proceedings of the 28th
ACM International Conference on Design of Communication, ser.
SIGDOC ’10, ACM, 2010, pp. 73–80.

[P12] S. Supakkul and L. Chung, “Visualizing Non-Functional Require-
ments Patterns,” in Requirements Engineering Visualization, 2010
Fifth International Workshop on, 2010, pp. 25–34.

[P13] J. Horkoff and E. Yu, “Interactive Goal Model Analysis for Early
Requirements Engineering,” RE, pp. 1–33, 2014.

[P14] C. Duan and J. Cleland-Huang, “Visualization and Analysis in Auto-
mated Trace Retrieval,” in Requirements Engineering Visualization,
2006. First International Workshop on, 2006, pp. 5–5.

[P15] D. Sellier and M. Mannion, “Visualising Product Line Requirement
Selection Decision Inter-dependencies,” in The Second International
Workshop on Requirements Engineering Visualization,, 2007, pp. 7–7.

[P16] B. Berenbach, F. Schneider, and H. Naughton, “The Use of A
Requirements Modelling Language for Industrial Applications,” in
Requirements Engineering Conference (RE), 2012 20th IEEE Inter-
national, 2012, pp. 285–290.

[P17] R. Gandhi and S.-W. Lee, “Visual Analytics for Requirements-driven
Risk Assessment,” in Requirements Engineering Visualization, 2007.
Second International Workshop on, 2007, pp. 6–6.

[P18] P. Laurent, P. MŁder, J. Cleland-Huang, and A. Steele, “A taxonomy
and visual notation for modelling globally distributed requirements
engineering projects,” in Global Software Engineering (ICGSE), 2010
5th IEEE International Conference on, 2010, pp. 35–44.

[P19] B. Regnell, M. Höst, J. N. och Dag, P. Beremark, and T. Hjelm, “An
Industrial Case Study on Distributed Prioritization in Market-driven
Requirements Engineering for Packaged Software,” Requirements
Engineering, vol. 6, no. 1, pp. 51–62, 2001.

[P20] K. Stapel and K. Schneider, “Managing Knowledge on Communi-
cation and Information Flow in Global Software Projects,” Expert
Systems, 2012.

[P21] F. Perez and P. Valderas, “Allowing end-users to actively participate
within the elicitation of pervasive system requirements through im-
mediate visualization,” in The Fourth International Workshop onRe-
quirements Engineering Visualization, 2009, pp. 31–40.

[P22] D. Callele, E. Neufeld, and K. Schneider, “Visualizing emotional re-
quirements,” in Requirements Engineering Visualization, 2009 Fourth
International Workshop on, 2009, pp. 1–10.

[P23] N. Ernst, Y. Yu, and J. Mylopoulos, “Visualizing Non-functional
Requirements,” in Requirements Engineering Visualization, 2006.
First International Workshop on, 2006, pp. 2–2.

[P24] W. Farid and F. Mitropoulos, “NORMATIC: A Visual Tool for
Modelling Non-Functional Requirements in Agile Rrocesses,” in
Southeastcon, 2012 Proceedings of IEEE, 2012, pp. 1–8.

[P25] T. Merten, D. Juppner, and A. Delater, “Improved Representation of
Traceability Links in Requirements Engineering Knowledge Using
Sunburst and Netmap Visualizations,” in Managing Requirements
Knowledge (MARK), 2011 Fourth International Workshop on, 2011,
pp. 17–21.

[P26] S. Lohmann, J. Ziegler, and P. Heim, “Involving End Users in
Distributed Requirements Engineering,” in Engineering Interactive
Systems, Springer, 2008, pp. 221–228.

View publication statsView publication stats

https://www.researchgate.net/publication/303696963

SAMPLE_SLRs/A-Systematic-Literature-Review-of-Servant-Leadership-Theory-in-Organizational-Contexts-_-SpringerLink-1uciawi

5/1/2017 A Systematic Literature Review of Servant Leadership Theory in Organizational Contexts | SpringerLink

https://link.springer.com/article/10.1007/s1055101213226 1/45

A Systematic Literature Review of
Servant Leadership Theory in
Organizational Contexts

Journal of Business Ethics

March 2013, Volume 113, Issue 3, pp 377–393

Denise Linda Parris (1) Email author (deniselparris@gmail.com)
Jon Welty Peachey (2)

1. Barney Barnett School of Business & Free Enterprise, Florida Southern College,
Lakeland, USA
2. Division of Sport Management, Department of Health and Kinesiology, Texas A&M
University, College Station, USA

Article

First Online:
22 April 2012

Received:
20 February 2012

Accepted:
08 April 2012

DOI (Digital Object Identifier): 10.1007/s1055101213226

Cite this article as:
Parris, D.L. & Peachey, J.W. J Bus Ethics (2013) 113: 377. doi:10.1007/s10551
01213226

57 Citations
2 Shares
11k Downloads

Abstract

A new research area linked to ethics, virtues, and morality is servant leadership.
Scholars are currently seeking publication outlets as critics debate whether this new
leadership theory is significantly distinct, viable, and valuable for organizational
success. The aim of this study was to identify empirical studies that explored servant
leadership theory by engaging a sample population in order to assess and synthesize
the mechanisms, outcomes, and impacts of servant leadership. Thus, we sought to
provide an evidenceinformed answer to how does servant leadership work, and how
can we apply it? We conducted a systematic literature review (SLR), a methodology
adopted from the medical sciences to synthesize research in a systematic, transparent,

https://link.springer.com/

https://link.springer.com/journal/10551

https://link.springer.com/journal/10551/113/3/page/1

mailto:deniselparris@gmail.com

http://citations.springer.com/item?doi=10.1007/s10551-012-1322-6

http://www.altmetric.com/details.php?citation_id=724544&domain=link.springer.com

5/1/2017 A Systematic Literature Review of Servant Leadership Theory in Organizational Contexts | SpringerLink

https://link.springer.com/article/10.1007/s1055101213226 2/45

and reproducible manner. A disciplined screening process resulted in a final sample
population of 39 appropriate studies. The synthesis of these empirical studies
revealed: (a) there is no consensus on the definition of servant leadership; (b) servant
leadership theory is being investigated across a variety of contexts, cultures, and
themes; (c) researchers are using multiple measures to explore servant leadership; and
(d) servant leadership is a viable leadership theory that helps organizations and
improves the wellbeing of followers. This study contributes to the development of
servant leadership theory and practice. In addition, this study contributes to the
methodology for conducting SLRs in the field of management, highlighting an
effective method for mapping out thematically, and viewing holistically, new research
topics. We conclude by offering suggestions for future research.

Keywords

Leadership Leadership theory Servant leadership Systematic literature review

Introduction

Leadership is one of the most comprehensively researched social influence processes in
the behavioral sciences. This is because the success of all economic, political, and
organizational systems depends on the effective and efficient guidance of the leaders of
these systems (Barrow 1977). A critical factor to understanding the success of an
organization, then, is to study its leaders. Leadership is a skill used to influence
followers in an organization to work enthusiastically towards goals specifically
identified for the common good (Barrow 1977; Cyert 2006; Plsek and Wilson 2001).
Great leaders create a vision for an organization, articulate the vision to the followers,
build a shared vision, craft a path to achieve the vision, and guide their organizations
into new directions (BanutuGomez and BanutuGomez 2007; Kotter 2001).
According to Schneider (1987), the most important part in building an organization
with a legacy of success is the people in it, which includes the followers (i.e.,
employees and volunteers) as well as the leaders. Leadership theories attempt to
explain and organize the complexity of the nature of leadership and its consequences
(Bass and Bass 2008). Over the years, some leadership scholars have called attention
to the implicit connection between ethics and leadership. A burgeoning new research
area and leadership theory that has been linked to ethics, virtues, and morality is
servant leadership (Graham 1991; Lanctot and Irving 2010; Parolini et al. 2009;
Russell 2001; Whetstone 2002).

Servant leadership theory’s emphasis on service to others and recognition that the role
of organizations is to create people who can build a better tomorrow resonates with
scholars and practitioners who are responding to the growing perceptions that
corporate leaders have become selfish and who are seeking a viable leadership theory
to help resolve the challenges of the twentyfirst century. Despite servant leadership
being coined by Robert K. Greenleaf over three decades ago in 1970, it remains
understudied yet still prominently practiced in boardrooms and organizations (Bass
and Bass 2008; Spears 2005). It has received significant attention in the popular press
(e.g., Fortune magazine and Dateline) (Spears Center 2011) and leading
organizational management authors have discussed the positive effects of servant
leadership on organizational profits and employee satisfaction; see Max DePree
(Leadership is an Art1989), Stephen Covey (Principle Centered Leadership1990),
Peter Senge (The Fifth Discipline: The Art and Styles of the Learning

5/1/2017 A Systematic Literature Review of Servant Leadership Theory in Organizational Contexts | SpringerLink

https://link.springer.com/article/10.1007/s1055101213226 3/45

Organization1990), Peter Block (Stewardship: Choosing Service over Self
Interest1993), and Margaret Wheatley (Finding Our Way: Leadership in an
Uncertain Time2005). However, Greenleaf’s (1970, 1977) conceptualization of servant
leadership as a way of life rather than as a management technique perhaps has slowed
the acceptance of this leadership theory in academia as scholars ask the question: If it
is a way life—a philosophy, how can it be empirically tested? Even Greenleaf admitted
servant leadership is unorthodox and would be difficult to operationalize and apply,
as “it is meant to be neither a scholarly treatise nor a howtodoit manual” (Greenleaf
1977, p. 49). The majority of research to date on servant leadership consists of
developing theoretical frameworks and establishing measurement tools with the
intention that future scholars can apply these tools to explore servant leadership in
practice and as a tenable theory. Only a limited amount of research has empirically
examined this construct.

As an aid in advancing servant leadership theory, we sought to identify these
empirical studies that investigated servant leadership by engaging a sample
population in order to assess and synthesize its mechanisms, outcomes, and impacts.
Currently, there does not exist a comprehensive summary of empirical studies
exploring servant leadership theory in organizational settings (e.g., a systematic
literature review (SLR)), which is a gap in the extant literature. Through exploring
empirical studies investigating servant leadership theory in organizational contexts,
we provide evidence that servant leadership is a tenable theory.

As a promising new field of research, servant leadership faces the challenges once
addressed by the early services marketing and sport management scholars whose new
ideas and concepts were accepted slowly within the conservative culture of academia
(Shannon 1999). Similarly, servant leadership scholars have sought a variety of
publication outlets for their work while they confront a debate on the distinctiveness
and significance of this leadership theory for organizations as well as employees. In
addition, the acceleration of knowledge production in the management field has
resulted in a body of knowledge that is increasingly transdisciplinary, fragmented,
and interdependent from advancement in social sciences. In management research the
literature review is a key tool used to manage the diversity of knowledge for an
academic inquiry; however, a critique of these reviews is that they are typically
descriptive accounts of contributions of selected writers often arbitrarily chosen for
inclusion by the researcher, and that these reviews may lack a critical assessment of
included studies (Tranfield et al. 2003). In contrast, a SLR is different from traditional
narrative reviews in that it adopts a replicable, scientific and transparent process that
aims to mitigate bias through exhaustive literature searches and by providing an
audit trail of the conclusions. A current gap in management research is a discussion of
how to conduct a SLR, how to critically assess studies, and how to integrate the
conclusions. In this SLR, we not only ascertain the current state of the field in servant
leadership research and synthesize divergent studies, but also advance a rigorous
methodology for conducting a SRL in management research.

Thus, the purpose of this study was to systematically examine and organize the
current body of research literature that either quantitatively or qualitatively explored
servant leadership theory in a given organizational setting. In this SRL we only
included empirical studies that investigated servant leadership in an organizational
context and excluded studies with a primary focus on model development or testing
measurement instruments. Earlier reviews on the concept of servant leadership
focused on: identifying key characteristics (Russell and Stone 2002), measurement

5/1/2017 A Systematic Literature Review of Servant Leadership Theory in Organizational Contexts | SpringerLink

https://link.springer.com/article/10.1007/s1055101213226 4/45

development (Barbuto and Wheeler 2006), and proposing a theoretical framework
(Van Dierendonck 2011). Although these reviews help provide insight into how
researchers have attempted to operationalize servant leadership, none of them was
done in a systematic manner (i.e., no methodology to select articles or limit bias), and
none of them specifically explored empirical research.

The following research questions guided this SLR: (a) How was servant leadership
defined? (b) In what contexts was servant leadership theory empirically investigated?
(c) How was servant leadership examined (i.e., the methodology)? and (d) What were
the results of the examination? We begin this paper by summarizing the origin of
servant leadership and follow with a short discussion of the development of servant
leadership as a theory and a new research area. Next, a summary of the method used
for selecting and reviewing the literature is explained, with details on search strategy,
analysis, and assessment of the quality of the reviewed studies. Then, we present our
findings of the SLR on empirical studies that have explored servant leadership theory.
In addition, we discuss the methodological contribution of conducting SLRs in the
field of management as an effective method for mapping out thematically, and
viewing holistically, new research topics. We conclude by offering suggestions for
future research and practice.

Origin of Servant Leadership by Robert K.
Greenleaf

Servant leadership was introduced into an organizational context through Greenleaf’s
three foundational essays—The Servant as Leader (1970), The Institution as Servant
(1972a), and Trustees as Servants (1972b)—all of which he published after retiring
from 40 years of management work at AT&T. Greenleaf (1977) defined servant
leadership as not just a management technique but a way of life which begins with
“the natural feeling that one wants to serve, to serve first” (p. 7). Greenleaf (1977)
conceptualized the servant as leader from his impressions of Journey to the East by
Hesse (1956) and used the character Leo to describe a true servant: “Leadership was
bestowed upon a man who was by nature a servant… His servant nature was the real
man, not bestowed, not assumed, and not to be taken away” (p. 21). Servant leaders
are distinguished by both their primary motivation to serve (what they do) and their
selfconstruction (who they are), and from this conscious choice of ‘doing’ and ‘being’
they aspire to lead (Sendjaya and Sarros 2002). Greenleaf (1977) believed servant
leadership was an inward lifelong journey.

Upon retirement in 1964, Greenleaf launched a second career, which spanned
25 years, in which he articulated his new leadership paradigm—servant leadership. He
promoted servant leadership in many publications and presentations, including
lectures at Massachusetts Institute of Technology’s (M.I.T.) Sloan School of
Management, Harvard Business School, Dartmouth College, and the University of
Virginia; and served as leadership consultant to institutions such as Ford
Foundation, Lilly Endowment, M.I.T., R.K. Mellon Foundation, and the American
Foundation for Management. In 1964 he founded the Center for Applied Ethics,
renamed the Robert K. Greenleaf Center for Servant Leadership in 1985, which helps
people understand the principles and practices of servant leadership (Greenleaf Center

5/1/2017 A Systematic Literature Review of Servant Leadership Theory in Organizational Contexts | SpringerLink

https://link.springer.com/article/10.1007/s1055101213226 5/45

2011). Over 20 % of Fortune magazine top 100 companies have sought guidance from
the Greenleaf Center for Servant Leadership, such as Starbucks, Vanguard Investment
Group, Southwest Airlines, and ID Industries (Greenleaf Center).

Although the contemporary study of servant leadership evolved largely from Greenleaf
(1970, 1977), the practice of servant leadership is not a new concept, with roots dating
back to ancient teachings of the world’s great religions, as well as to statements of
numerous great leaders and thinkers (Sendjaya and Sarros 2002). The concept of
servant leadership echoes the messages of Mother Theresa, Moses, Harriet Tubman,
Laotzu, Mohandas Gandhi, Martin Luther King, Jr., Confucius, and many other
religious, historic, and current leaders (Keith 2008). Many scholars model Jesus
Christ’s teachings to his disciples as the ultimate example of servant leadership
(Ebener and O’Connell 2010; Lanctot and Irving 2010; Winston 2004). Whereas other
leadership theories are traditionally defined only by what the leader does, servant
leaders are defined by their character and by demonstrating their complete
commitment to serve others. This creates one of the core challenges for theorists; how
to construct models that encompass Greenleaf’s theoretical message of “servanthood
throughleadershipthroughpractice” (Prosser 2010, p. 28) that operates not only on
a surfacelevel but deep within a person’s being. Although scholars have agreed
theories, frameworks, and models will increase our understanding of the meaning,
implications, and applications of servant leadership, it is important to remain aware
of the more abstract, underlying principles and concepts of a servant as a leader
(Spears 1998; Keith 2008; Prosser 2010).

Servant Leadership as a Theory

Although servant leadership is a growing trend being practiced by private and non
profit organizations alike, there is still a lack of research in this area (Farling et al.
1999). The majority of research in servant leadership has streamed from Greenleaf’s
(1977) foundational texts and the Greenleaf Center (see Akuchie 1993; Bordas 1995;
Brody 1995; Buchen 1998; Chamberlain 1995; Frick 1995; Gaston 1987; Kelley 1995;
Kiechel 1995; Kuhnert and Lewis 1987; Lee and Zemke 1995; Lloyd 1996; Lopez 1995;
McCollum 1995; McGeeCooper and Trammell 1995; Rasmussen 1995; Rieser 1995;
Senge 1995; Smith 1995; Snodgrass 1993; Spears 1995, 1996; Tatum 1995; Vanourek
1995). Many of these writers present narrative examples of how servant leadership is
being used in organizational settings; however, this is also the primary limitation of
much of the servant leadership literature, which is anecdotal in nature instead of
empirical (Bowman 1997; Northouse 1997; Sendjaya and Sarros 2002). Bass (2000)
acknowledged that servant leadership requires extensive research, emphasizing that
“the strength of the servant leadership movement and its many links to encouraging
follower learning, growth, and autonomy, suggests that the untested theory will play
a role in the future leadership of the learning organization” (p. 33). The promise of
servant leadership has since motivated scholars and practitioners to explore the
possibilities of the servantfirst paradigm.

Since Farling et al.’s (1999) call for empirical studies, there have emerged three
streams of research (Van Dierendonck and Patterson 2011): (a) a conceptual stream
(Spears 1998; Laub 1999; Patterson 2003); (b) a measurement stream (Page and
Wong 2000; Wong and Page 2003; Ehrhart 2004; Barbuto and Wheeler 2006; Dennis
and Bocarnea 2005; Liden et al. 2008; Sendjaya et al. 2008; Van Dierendonck and

5/1/2017 A Systematic Literature Review of Servant Leadership Theory in Organizational Contexts | SpringerLink

https://link.springer.com/article/10.1007/s1055101213226 6/45

Nuijte 2011); and (c) model development (Russell and Stone 2002; Van Dierendonck
2011). Notably absent from the above streams are empirical studies that explore
servant leadership theory in a given organizational setting. In addition, in spite of the
growing amount of research on servant leadership, the theory is still underdefined,
with various authors grappling with definitions (Anderson 2009). This is as Greenleaf
(1977) predicted, when he warned that servant leadership would be difficult to apply
and operationalize. He did not provide a management howtodoitmanual; instead,
he challenged readers to reflect, ponder, and grow (Frick 2004; Spears 1995).

To date, three reviews of servant leadership have been conducted, which help provide
insight into how researchers have organized the complexity of Greenleaf’s concepts on
servant leadership into a theoretical framework. Russell and Stone’s (2002) review
revealed the following nine functional attributes, or operative qualities and distinctive
characteristics of servant leaders; vision, honesty, integrity, trust, service, modeling,
pioneering, appreciation of others, and empowerment. In addition, Russell and Stone
determined 11 accompanying attributes, which are interrelated and supportive of the
nine core attributes listed above: communication, credibility, competence,
stewardship, visibility, influence, persuasion, listening, encouragement, teaching,
and delegation. From this assimilation of attributes, Russell and Stone developed a
model of servant leadership to spark future application and research. While their
review provides a conceptual overview of servant leadership, it lacks a methodology.
Barbuto and Wheeler (2006) developed an integrated model of servant leadership
after conducting a literature review, which synthesized the attributes of servant
leadership into five factors; altruistic calling, emotional healing, persuasive mapping,
wisdom, and organizational stewardship. The third review by Van Dierendonck (2011)
also concludes with another conceptual model, which identifies six key characteristics
of servant leadership: empowering and developing people, humility, authenticity,
interpersonal acceptance, providing direction, and stewardship. All of these reviews
exemplify different interpretations of Greenleaf’s writings employing different
terminologies; however, all include the fundamental dimension of servanthood or the
willingness to serve others. These reviews highlight the plurality of servant leadership
theory, leaving the researcher, student, or practitioner to ponder exactly what servant
leadership theory is. As DiMaggio (1995) pointed out “there is more than one kind of
good theory” (p. 391).

Given that previous reviews have examined the development of conceptual
frameworks and measurement tools for servant leadership, the present review focuses
only on empirical studies that have explored servant leadership theory in an
organizational context. As such, the current study is the first review to provide a
synthesis, based upon evidence in published peerreviewed journals, of empirical
studies conducted on servant leadership theory in organizational settings.

Methodology

The SLR is often contrasted with traditional literature reviews because systematic
reviews are objective, replicable, systematic, comprehensive, and the process is
reported in the same manner as for reporting empirical research (Weed 2005). The
origin of SLRs is in the medical, health care, and policy fields, where they have been
used to assemble the best evidence to make clinical and policy decisions (Cook et al.
1997; Tranfield et al. 2003). SLRs in management are used to provide transparency,

5/1/2017 A Systematic Literature Review of Servant Leadership Theory in Organizational Contexts | SpringerLink

https://link.springer.com/article/10.1007/s1055101213226 7/45

clarity, accessibility, and impartial inclusive coverage on a particular area (Thorpe et
al. 2006). Klassen et al (1998) define SLR as “a review in which there is a
comprehensive search for relevant studies on a specific topic, and those identified are
then appraised and synthesized according to a predetermined explicit method” (p.
700). This SRL specifically explored research studies that have examined servant
leadership theory in a given organizational setting. Since our focus was gaining
insight on the empirical investigation of servant leadership theory, we excluded
studies with a primary focus on model development or testing measurement
instruments. The approach of this review entailed extensive searches of relevant
databases with the intention of ensuring, as far as possible, that all literature on
servant leadership was identified while maintaining the focus on literature of greatest
pertinence to the research questions (i.e., empirical studies that have investigated
servant leadership theory in organizational settings). Next, we discuss our search
methods, inclusion and exclusion criteria, sample, and data analysis.

Search Methods

Published studies were identified through searches of electronic databases accessible
through the authors’ university library system. Databases included in this review
were: PsycInfo, Eric, Sociological Abstracts, PAIS International, Social Services,
Communication Abstracts, International Bibliography of the Social Sciences (IBSS),
Physical Education Index, World Wide Political Abstracts from the vendor CSA,
Academic Search Complete, Business Source Complete, Communication and Mass
Media Complete, Education and Administration Abstracts, Gender Studies, CINAHL,
Health Source: Nursing/Academic Edition, Human Resources Abstracts, and Medline
through the vendor EBSCO. All results were limited to Englishonly peerreviewed
journal articles. The searches for published studies were conducted in a systematic
manner, following the order of the databases listed above.

Inclusion and Exclusion Criteria

The initial search required that articles included in the review were studies that must:
(a) be published in a peerreviewed journal; (b) be in the English language; and (c) use
the keyword “servant leadership.” No restriction was placed on year of publication.
The number of articles containing the keyword “servant leadership” retrieved from
each database was recorded. Next, we examined if there were any external duplicates
from the current database being searched and the previous databases that had already
been searched. We recorded the number of external duplicates, and then deleted the
duplicated journal articles from the last database searched while keeping a running
total of new articles found.

Once all possible studies had been identified, we conducted a second screening to
assess eligibility against inclusion criteria and then full text articles were retrieved for
those that met the inclusion criteria. The inclusion criteria for the second screening
required that the published peerreviewed article meet all of the following four
specifications: (a) be in the English language; (b) be an empirical study (i.e., not an
essay, book review, letter, literature review, editorial, opinion, journalistic or antidotal
article); (c) discuss servant leadership as the main topical theme; and (d) examine
servant leadership theory either quantitatively or qualitatively. Articles were excluded
if any of these four components was not addressed in the abstract, results, or

5/1/2017 A Systematic Literature Review of Servant Leadership Theory in Organizational Contexts | SpringerLink

https://link.springer.com/article/10.1007/s1055101213226 8/45

discussion sections of the respective study. Finally, additional articles meeting the
inclusion criteria were found by examining the bibliographies of resources identified
through the secondary screening.

Sample

Peerreviewed publications were identified using the key terms outlined in the
inclusion and exclusion criteria section above. In all, a total of 381 articles where
retrieved; however, after duplicates were deleted there remained 255 articles meeting
the initial inclusion criteria. After the secondary search process was conducted, a final
sample of 44 appropriate studies was obtained. Upon retrieving full text articles, an
additional five articles were excluded after further examination because they did not
satisfy the screening criteria. The final sample of articles constituted 39 empirical
studies. Peerreviewed articles meeting the outlined criteria were published between
2004 and 2011. The 39 published articles were drawn from a variety of peerreviewed
journals (n = 27). Table 1 depicts the list of journals included in the study, the number
of articles included from each journal, and the database they were accessed through.

Table 1

Database and journals included in systematic literature review

5/1/2017 A Systematic Literature Review of Servant Leadership Theory in Organizational Contexts | SpringerLink

https://link.springer.com/article/10.1007/s1055101213226 9/45

Database Journal Count

Eric Alberta Journal of Educational Research 1

PsycInfo Business Ethics: A European Review 1

Eric Catholic Education: A Journal of Inquiry & Practice 1

Eric
Educational Management Administration &
Leadership

PsycInfo
European Journal of Work and Organizational
Psychology

Scopus Global Virtue Ethics Review 1

CINAHL Health Care Management Review 1

PsycInfo Home Health Care Management & Practice 1

Business Source
Complete

International Journal of Business Research 1

Eric International Journal of Leadership in Education 2

Scopus International Journal of Leadership Studies 2

PsycInfo International Journal of Sports Science & Coaching 1

PsycInfo Journal of Applied Psychology 3

Business Source
Complete

Journal of Business & Economics Research 1

Academic Search
Complete

Journal of Interprofessional Care 1

5/1/2017 A Systematic Literature Review of Servant Leadership Theory in Organizational Contexts | SpringerLink

https://link.springer.com/article/10.1007/s1055101213226 10/45

Database Journal Count

PsycInfo Journal of Management Development 1

Academic Search
Complete

Journal of Park & Recreation Administration 1

PsycInfo Journal of Personal Selling & Sales Management 2

Business Source
Complete

Journal of the Academy of Business & Economics 1

Eric Journal of Women in Educational Leadership 1

PsycInfo Leadership 1

PsycInfo Leadership & Organization Development Journal 6

PsycInfo Nonprofit Management and Leadership 1

PsycInfo Personnel Psychology 1

Business Source
Complete

Review of Business Research 2

Business Source
Complete

Services Marketing Quarterly 1

PsycInfo
The International Journal of Human Resource
Management

We grouped the journals by their area of focus, which showed a concentration of
research taking place in leadership (n = 9), education (n = 7), business (n = 6), and
psychology (n = 6), with the fields of nursing (n = 3), management (n = 2), personal
selling and sales management (n = 2), ethics (n = 1), parks and recreation
administration (n = 1), services marketing (n = 1), and sports (n = 1) representing a
smaller number of empirical studies.

Data Analysis

The Matrix Method (Garrard 1999) was utilized as the strategy for organizing and
abstracting pertinent information from these publications. For this study, the

5/1/2017 A Systematic Literature Review of Servant Leadership Theory in Organizational Contexts | SpringerLink

https://link.springer.com/article/10.1007/s1055101213226 11/45

following information was abstracted from each article: (a) How was servant
leadership defined? (b) In what contexts was servant leadership theory empirically
investigated? (c) How was servant leadership examined? and (d) What were the
results of the examination? Last, for each publication, the methodology used to
examine servant leadership was evaluated. For qualitative studies, we used a critical
appraisal tool designed by Letts et al. (2007), and for quantitative studies we used a
critical appraisal tool designed by the Institute for Public Health Sciences (2002). In
addition to these two appraisal assessments we used Stoltz et al.’s (2004) critical
appraisal tool, which assessed both quantitative and qualitative studies. We adopted
these three critical appraisal tools to create a threepoint scale to reflect the quality of
studies: high (I); medium (II)—used if studies did not meet criteria for high (I) or low
quality; and low (III). Table 2 describes our classification for high to low quality
studies, which was based on the three critical appraisal tools mentioned above.

Table 2

Classification and quality assessment of studies

5/1/2017 A Systematic Literature Review of Servant Leadership Theory in Organizational Contexts | SpringerLink

https://link.springer.com/article/10.1007/s1055101213226 12/45

I = High II = Medium III = Low

QNT

Study using quantitative analysis of data.
Clearly focused study, sufficient background
provided, well planned, method appropriate,
measures validated, applicable and adequate
number of participants, data analysis
sufficiently rigorous with adequate statistical
methods, findings clearly stated

–

Not focused
study,
insufficient
background
provided,
poorly planned,
inappropriate
method,
invalidated
measures,
inapplicable
and inadequate
number of
participants,
data analysis
insufficiently
rigorous, with
inadequate
statistical
methods,
unclear
findings

QAL

Study using qualitative analysis of data.
Purpose stated clearly, relevant background
literature reviewed, design appropriate,
identified researcher’s theoretical or
philosophical perspective, relevant and well
described selection of participants and context,
procedural rigor in data collection strategies
and analysis, evidence of the four components
of trustworthiness (credibility, transferability,
dependability, and confirmability) results are
comprehensive and well described

–

Vaguely
formulated
purpose,
insufficient
background,
few or
unsatisfactory
descriptions of
participants
and context,
trustworthiness
inadequately
addressed,
lacks in
description of
data collection,
data analysis,
and results

QNT quantitative study, QAL qualitative study, I high quality, II medium quality, III
low quality

5/1/2017 A Systematic Literature Review of Servant Leadership Theory in Organizational Contexts | SpringerLink

https://link.springer.com/article/10.1007/s1055101213226 13/45

The findings from these studies were summarized and placed into matrixes (i.e.,
tables). Our SLR findings consist of a synthesis of the results from all 39 empirical
studies along with the assessment of quality for each study. Further, we assess the
level of supporting evidence for thematic conclusions drawn from combining the
results of multiple studies.

Findings

Overall, this review highlights that servant leadership theory is being researched and
tested across a variety of contexts, cultures, disciplines, and themes. Our sample
included 11 qualitative studies, 27 quantitative studies, and one mixed method study,
all empirically assessing servant leadership theory. Thus, this review illustrates that
servant leadership is being explored both quantitatively and qualitatively, and the
topic has an international appeal with studies being conducted in 11 countries. In the
quality assessment, 22 studies were classified as high, 12 as medium, and five as low
quality. Conclusive statements were made based upon the synthesis of findings from
each article. The conclusions (see Table 3) were classified as A (strong evidence) or B
(moderate evidence) based on scientific strength.

Table 3

Overview of conclusions

5/1/2017 A Systematic Literature Review of Servant Leadership Theory in Organizational Contexts | SpringerLink

https://link.springer.com/article/10.1007/s1055101213226 14/45

Result
themes

Conclusion Evidence References

Crosscultural
applicability

SL is accepted
and practiced in
various cultures;
however,
components of SL
have different
weights

Strong
evidence in
favor of
statement
(A)

Cerit (2009, 2010) (QNT I, QNT I);
Hamilton and Bean (2005) (QAL
III); Hale and Fields (2007) (QNT I);
Han et al. (2010) (QAL II); Pekerti
and Sendjaya (2010) (QNT I)

SL attributes

Spears’ (1998) 10
characteristics
are representative
of a servant
leader applied in
different context

Strong
evidence in
favor of
statement
(A)

Boroski and Greif (2009) (QAL III);
Crippen (2004) (QAL II); Crippen
and Wallin (2008a) (QAL II);
Crippen and Wallin (2008b) (QAL
II); Sturm (2009) (QAL I)

Patterson (2003)
and Winston
(2003) models of
SL are supported

Strong
evidence in
favor of
statement
(A)

Winston (2004) (QAL I); Dingman
and Stone (2007) (QAL II)

Team level
effectiveness

SL leads to
increased leader
trust and
organizational
trust

Strong
evidence in
favor of
statement
(A)

Joseph and Winston (2005) (QNT I);
Reinke (2004) (QNT II); Senjaya and
Pekerti (2010) (QNT I); Washington
et al (2006) (QNT I)

SL fosters
organizational
citizenship
behavior

Strong
evidence in
favor of
statement
(A)

Ebener and O’Connell (2010) (QAL
I); Hu and Liden (2011) (QNT I);
Ehrhart (2004) (QNT I); Walumbwa
et al (2010) (QNT I)

Procedural justice
is positively
associated with
SL

Strong
evidence in
favor of
statement
(A)

Ehrhart (2004) (QNT I); Walumbwa
et al (2010) (QNT I); Chung et al.
(2010) (QNT II)

5/1/2017 A Systematic Literature Review of Servant Leadership Theory in Organizational Contexts | SpringerLink

https://link.springer.com/article/10.1007/s1055101213226 15/45

Result
themes

Conclusion Evidence References

SL increases team
effectiveness

Strong
evidence in
favor of
statement
(A)

Irving and Longbotham (2007) (QNT
I); Schaubroeck et al (2011) (QNT I);
Hu and Liden (2011) (QNT I)

SL is associated
with greater
leadership
effectiveness

Strong
evidence in
favor of
statement
(A)

Taylor et al. (2007) (QNT II); Mayer
et al. (2008) (QNT I); McCuddy and
Cavin (2008) (QNT III)

SL enhances
collaboration

Moderate
evidence in
favor of
statement
(B)

Garber et al. (2009) (QNT II); Sturm
(2009) (QAL I); Irving and
Longbotham (2007) (QNT I)

Followers’ well
being SL increases

employee job
satisfaction

Strong
evidence in
favor of
statement
(A)

Cerit (2009) (QNT I); Jenkins and
Stewart (2010) (QNT I); Mayer et al.
(2008) (QNT I); Chung et al. (2010)
(QNT II)

SL creates a
positive work
climate

Strong
evidence in
favor of
statement
(A)

Neubert et al. (2008) (QNT I); Black
(2010) (Mixed Method: QNT II and
QAL III); Jaramillo et al. (2009a)
(QNT I)

SL supports
employee
creativity and
helping behaviors

Strong
evidence in
favor of
statement
(A)

Jaramillo et al. (2009b) (QNT I);
Neubert et al. (2008) (QNT I)

SL improves
followers well
being

Strong
evidence in
favor of
statement
(A)

Jaramillo et al. (2009b) (QNT I);
Rieke et al. (2008) (QNT I)

5/1/2017 A Systematic Literature Review of Servant Leadership Theory in Organizational Contexts | SpringerLink

https://link.springer.com/article/10.1007/s1055101213226 16/45

Result
themes

Conclusion Evidence References

SL lowers
employee
turnover

Strong
evidence in
favor of
statement
(A)

Jaramillo et al. (2009a) (QNT I);
Babakus et al. (2011) (QNT I)

SL increases
commitment

Strong
evidence in
favor of
statement
(A)

Cerit (2010) (QNT I); Hamilton and
Bean (2005) (QAL III); Hale and
Fields (2007) (QNT I); Han et al.
(2010) (QAL II); Pekerti and
Sendjaya (2010) (QNT I) Jaramillo et
al. (2009a) (QNT I); Jaramillo et al.
(2009b) (QNT I)

Spirituality
SL is associated
with workplace
spirituality

Insufficient
evidence

Herman (2010) (QNT II)

Demographics

Propensity
toward engaging
in SL is
associated with
demographic
variables

Insufficient
evidence

Fridell et al. (2009) (QNT II);
McCuddy and Cavin (2009) (QNT
III); Taylor et al. (2007) (QNT II)

Implementation
of SL

Knowledge and
framing of SL can
affect adoption

Insufficient
evidence

Hamilton and Bean (2005) (QAL
III); SavageAustin and Honeycutt
(2011) (QAL III)

Positive
relationship
between
succession
planning and SL

Insufficient
evidence

Dingman and Stone (2007) (QAL II)

SL servant leadership, QNT quantitative study, QAL qualitative study, I high quality,
II medium quality, III low quality

If two or more studies of high quality supported a conclusion or one study of high
quality in addition to two or more studies of medium quality supported the
conclusion, we assigned it an (A) rating. On the other hand, conclusions with one
study of high quality and one study of medium quality or two studies of medium
quality were assigned a (B) rating. If a conclusion(s) did not fall under (A) strong
evidence in favor of conclusion or (B) moderate evidence in favor of conclusion, we

5/1/2017 A Systematic Literature Review of Servant Leadership Theory in Organizational Contexts | SpringerLink

https://link.springer.com/article/10.1007/s1055101213226 17/45

classified it as insufficiently supported and labeled insufficient evidence. The
following discussion of our findings is organized around the four central research
questions.

How was Servant Leadership Defined?

Servant leadership theory was introduced to readers by authors of empirical studies by
citing one or all three of the following: Greenleaf (1977), Spears (1995, 1998, 2004),
and Laub (1999). Generally, authors described servant leadership by quoting one of
these three authors in addition to citing multiple other authors, including, but not
limited to: Barbuto and Wheeler (2006), Graham (1991), Ehrhart (2004), Liden et al.
(2008), Page and Wong (2000), and Patterson (2003). Here, we discuss the three
most cited authors on servant leadership that have provided definitions.

Greenleaf (1970, 1972a, b, 1977), the grandfather of servant leadership, was cited by
37 of the 39 empirical studies. The majority of authors used part or all of Greenleaf’s
description from his original essay, The Servant as Leader (1970):

It begins with the natural feeling one wants to serve, to serve first. Then
conscious choice brings one to aspire to lead. That person is sharply
different from one who is leader first.

… The difference manifests itself in the care taken by the servantfirst to
make sure that other people’s highest priority needs are being served.
The best test, and difficult to administer, is this: Do those served grow
as persons? Do they, while being served, become healthier, wiser, freer,
more autonomous, more likely themselves to become servants? And,
what is the effect on the least privileged in society? Will they benefit or at
least not be further deprived? (Greenleaf 1970 as cited in Greenleaf 1977,
p. 27).

The majority of authors in our sample, like Greenleaf himself, defined servant
leadership theory in a descriptive manner. These descriptions usually cited multiple
scholarly works in the conceptual and measurement research streams, in addition to
citing leading organizational management authors.

The second most referenced author defining servant leadership theory was Larry
Spears. Like Greenleaf, Spears gained his knowledge from practice with most of his
works being nonempirical. He served for 17 years as the head of the Greenleaf Center,
has authored more than 10 books on servant leadership, and in 2008 established the
Larry C. Spears Center for Servant Leadership, Inc. (Spears Center 2011). Spears (1995,
1998, 2004) identified 10 characteristics of servant leaders from Greenleaf’s writings:
listening, empathy, healing, awareness, persuasion, philosophy, conceptualization,
foresight, stewardship, commitment to the growth of people, and building
community. These attributes are described in Table 4.

Table 4

Spears’ (1998) 10 characteristics of a servant leader

5/1/2017 A Systematic Literature Review of Servant Leadership Theory in Organizational Contexts | SpringerLink

https://link.springer.com/article/10.1007/s1055101213226 18/45

Characteristic Description

Listening
Automatically responding to any problem by receptively listening
to what is said, which allows them to identify the will of the group
and help clarify that will

Empathy
Striving to accept and understand others, never rejecting them,
but sometimes refusing to recognize their performance as good
enough

Healing
Recognizing as human beings they have the opportunity to make
themselves and others ‘whole’

Awareness
Strengthened by general awareness and above all selfawareness,
which enables them to view situations holistically

Persuasion Relying primarily on convincement rather than coercion

Conceptualization
Seeking to arouse and nurture theirs’ and others’ abilities to
‘dream great dreams’

Foresight
Intuitively understanding the lessons from the past, the present
realities, and the likely outcome of a decision for the future

Stewardship Committing first and foremost to serving others needs

Commitment to
the growth of
people

Nurtures the personal, professional, and spiritual growth of each
individual

Building
community

Identifies means of building communities among individuals
working within their institutions, which can give the healing love
essential for health

Four of the qualitative studies in our sample used Spear’s 10 characteristics to inform
their analysis (Crippen 2004; Crippen and Wallin 2008a, b; Sturm 2009).

The third most cited author in defining servant leadership theory is Laub (1999). His
Organizational Leadership Assessment (OLA) was an outcome of his dissertation. The
OLA assesses an organization’s health based upon the six key areas of an effective

5/1/2017 A Systematic Literature Review of Servant Leadership Theory in Organizational Contexts | SpringerLink

https://link.springer.com/article/10.1007/s1055101213226 19/45

servantminded organization by exploring the perceptions of top leaders, managers
and supervisors, and the workforce; however, it does not assess the servant leadership
of individual leaders (OLA Group 2011). Authors in our sample used Laub’s definition,
which terms the practice of servant leadership as placing “the good of those led over
the selfinterest of the leader” (1999, p. 81). In addition, authors would list and
describe Laub’s six key variables of an effective servantled organization: (a) values
people—believing, serving, and nonjudgmentally listening to others; (b) develops
people—providing learning, growth, encouragement and affirmation; (c) builds
community—developing strong collaborative and personal relationships; (d) displays
authenticity—being open, accountable, and willing to learn from others; (e) provides
leadership—foreseeing the future, taking initiative, and establishing goals; and (f)
shares leadership—facilitating and sharing power. The OLA has been widely used in
health organizations (OLA Group), and was used in six quantitative studies in our
sample (Herman 2010; Black 2010; Cerit 2010; Cerit 2009; Irving and Longbotham
2007; Joseph and Winston 2005).

In summary, our results confirm Anderson’s (2009) and Van Dierendonck’s (2011)
assessments that servant leadership theory remains underdefined with no consensus
on its definition or theoretical framework. Scholars are still seeking to articulate
Greenleaf’s conceptualization of servant leadership by using a variety of definitions
sourced from multiple works.

In What Contexts was Servant Leadership Theory
Empirically Investigated?

Our sample illustrates servant leadership theory is being studied across cultures,
contexts, and across a diversity of research foci. Overall, the sample consisted of
studies in 11 countries, which included four crosscultures studies. These findings
demonstrate that servant leadership is being practiced in various cultures,
specifically: U.S. (n = 23), Canada (n = 4), China (n = 2), Turkey (n = 2), Indonesia
(n = 1), New Zealand (n = 1), Kenya (n = 1), and the Republic of Trinidad (n = 1), with
five crossculture studies comparing U.S. and Ghana, U.S. and UK, U.S. and China
(n = 2), and Indonesia and Australia.

A contextual analysis of the sample revealed that servant leadership theory is being
applied in the following organizational settings: education (n = 17), which consisted
of religious schools (n = 6) and secular schools (n = 11); secular for profit
organizations (n = 17), which notably included financial services (n = 4) and nursing
(n = 3); public organizations (n = 2); religious organizations (n = 1); nonprofit
organizations (n = 1); and in a historical context (n = 1). It is important to note that
servant leadership was examined in a religious context in seven of the 39 studies, and
that the education field represents 44 % of the contextual environment for the entire
sample.

This synthesis also revealed seven key research themes, with some studies containing
more than one area of focus. The themes and their associated studies are presented in
Table 3. An overall count and description of each theme is as follows: (a) crosscultural
applicability—acceptance, practices, and different weights of servant leadership in a
variety of cultures (n = 7); (b) servant leadership attributes—conceptual models
characteristics were studied (n = 7); (c) team level effectiveness—effects of servant
leadership explored at the unit level (n = 20); (d) followers’ wellbeing—effects on

5/1/2017 A Systematic Literature Review of Servant Leadership Theory in Organizational Contexts | SpringerLink

https://link.springer.com/article/10.1007/s1055101213226 20/45

employees in a servantled environment (n = 20); (e) spirituality—connection between
spiritual workplace and servantled workplace was investigated (n = 1); (f)
demographics (n = 3); and (g) implementation of servant leadership (n = 3). We
discuss a synthesis of these themes below in the last section of our findings, where we
provide an overview of the results of studies included in the sample.

How was Servant Leadership Examined (i.e., the
Methodology)?

All of the 27 quantitative studies used surveys as the data collection method. The two
most popular measures of servant leadership theory used by these empirical studies
were Laub’s (1999) OLA instrument—used by six studies (Herman 2010; Black 2010;
Cerit 2009, 2010; Irving and Longbotham 2007; Joseph and Winston 2005) and the
Servant Leadership Scale developed by Ehrhart (2004)—used by six studies (Ehrhart
2004; Jaramillo et al. 2009a, b; Mayer et al. 2008; Neubert et al. 2008; Walumbwa et
al. 2010). Instruments that were utilized by two studies included: Barbuto and
Wheeler’s (2006) instrument (Jenkins and Stewart 2010; Garber et al. 2009); Liden et
al.’s (2008) instrument (Hu and Liden 2011; Schaubroeck et al. 2011); and Sendjaya et
al.’s (2008) survey (Pekerti and Sendjaya 2010; Sendjaya and Pekerti 2010). Taylor et
al. (2007) used Page and Wong’s (1998) selfassessment measure. Washington et al.
(2006) used Dennis and Winston’s (2003) instrument, which was an adopted version
of Page and Wong’s (2000) instrument. Rieke et al. (2008) used Hammermeister et
al.’s (2008) instrument, which was also an adopted version of Page and Wong’s
instrument. Babakus et al. (2011) and Hale and Fields (2007) used lesser known
scales, those of Lytle et al. (1998) and Dennis (2004), respectively. One study tapped
a survey designed by the U.S. Office of Personal Management (OPM). Four studies
used surveys developed specifically for the research: Fridell et al. (2009), Reinke
(2004), and McCuddy and Cavin (2008, 2009). In summary, out of 27 survey studies,
there were 14 different measures used. It is important to note that the majority of
authors combined multiple measurement scales to construct their surveys. In
addition, the majority of these measures explored servant leadership theory at the
unit level of analysis (i.e., group or team performance) while only a few examined it at
the individual level of analysis (i.e., individual performance).

Similarly, the 11 qualitative studies used a variety of servant leadership frameworks to
inform their analyses, while three studies did not provide any information on
frameworks. Four of the qualitative studies used Spears (1998) 10 characteristics to
inform their analyses (Crippen 2004; Crippen and Wallin 2008a, b; Sturm 2009). Two
studies used Patterson (2003) and Winsten’s (2003) models—Dingman and Stone
(2007) and Winston (2004). Han et al. (2010) used multiple dimensions and
definitions of servant leadership in Western literature including but not limited to:
Barbuto and Wheeler (2006); Liden et al. (2008); Ehrhart (2004); and Sendjaya et al.
(2008). The multiple quantitative and qualitative measures used by the studies in our
sample reinforce our findings for research question one, where it was found that
authors have defined servant leadership in various ways. Similarly, as this review
demonstrates, there is still not an agreed upon measurement strategy for servant
leadership theory.

What were the Results of the Examination?

5/1/2017 A Systematic Literature Review of Servant Leadership Theory in Organizational Contexts | SpringerLink

https://link.springer.com/article/10.1007/s1055101213226 21/45

Our sample of empirical studies illustrates that servant leadership is a tenable theory.
It is viable and valuable on an individual and an organization level, which can lead to
increased overall effectiveness of individuals and teams. In Table 3, a synthesis of the
conclusions from our sample of articles is divided by theme, with a rating of the
evidence to support each individual conclusion. We discuss the results of these
empirical studies by theme below.

CrossCultural Applicability

The crosscultural studies (Hamilton and Bean 2005—U.S. and UK; Hale and Fields
2007—U.S. and Ghana; Han et al. 2010—U.S. and China; Pekerti and Sendjaya 2010—
Indonesia and Australia; Schaubroeck et al. (2011)—U.S. and China) all indicate
servant leadership’s acceptability across a variety of cultures. However, these studies
also show that the different attributes perceived to make up servant leadership are not
weighted equally across cultures. For example: Hale and Fields (2007) found that
vision had a significantly stronger relationship with leader effectiveness for Ghanaians
in comparison to North Americans; Han et al. (2009) found “being dutiful” to be an
extended form of servant leadership in China; Hamilton and Bean (2005) discovered
that introducing servant leadership within a Christian context was perceived as
obtrusive in the United Kingdom; and Pekerti and Sendjaya (2010) found that
Australian leaders exhibited more behaviors with authentic self (leadership flows out
of who we are as opposed to what we do), while Indonesian leaders exhibited more
behaviors with responsible morality (reflective moral reasoning employed to assess
whether or not the process and outcomes of one’s leadership are ethical) and
transforming influence (articulation and implementation of a shared vision which
provides inspiration, meaning to one’s work, and creates a positive work
environment). In contrast to these findings, Schaubroeck et al. (2011) found no
significant differences in perceptions of servant leadership between Hong Kong and
the U.S. These crosscultural studies, along with studies conducted in different
countries, imply that servant leadership might be practiced across a variety of
cultures, but culturespecific perceptions of servant leadership exist based on
socialization and national context.

Servant Leader Attributes

Seven studies explored the conceptual definitions of servant leadership, and found
Spears (1998), Patterson’s (2003), and Winston’s (2003) attributes to be
representative of servant leadership in different contexts. Five studies (Boroski and
Greif 2009; Crippen 2004; Crippen and Wallin 2008a, b; Sturm 2009) within three
different contexts (schools, community, and nursing) supported Spears 10
characteristics (see Table 3). Two studies (Winston 2004; Dingman and Stone 2007)
provided support for Patterson’s (2003) leadertofollower and Winston’s (2003)
followertoleader models of servant leadership. Patterson’s model of leader–follower
interaction starts with the leaders’ agapaó (love for others) which she conceptualizes
as a collection of the following seven values: being teachable; showing concern for
others; demonstrating discipline; seeking the greatest good for the organization;
showing mercy in actions and beliefs with all people; meeting the needs of followers
and the organization; and creating a place where peace grows within the organization.
These seven values are based upon the biblical concept of the seven beatitudes from
Matthew 5 (Patterson 2003; Winston 2003, 2004). Instead of focusing on leader
follower interaction as Patterson’s model does, Winston’s model focuses on the

5/1/2017 A Systematic Literature Review of Servant Leadership Theory in Organizational Contexts | SpringerLink

https://link.springer.com/article/10.1007/s1055101213226 22/45

followertoleader interactions. Winston’s followertoleader model starts with the
followers’ agapaó and then shows how the followers are servant leaders themselves by
utilizing the same variables as Patterson’s model. As stated above, studies confirm the
applicability of the variables in both of these models: trust, empowerment, vision,
altruism, intrinsic motivation, commitment, and service (Winston 2004; Dingman
and Stone 2007). Thus, the attributes identified by Spears, Patterson, and Winston
were represented within the measurement instruments discussed above.

Team Level Effectiveness

Sixteen empirical studies explored servant leadership theory at a unit level. Overall,
these studies found that a servantled organization enhances leader trust and
organizational trust, organizational citizenship behavior, procedural justice, team and
leader effectiveness, and the collaboration between team members. Several studies
found that a servantled environment provided affirmation of justice and fair
treatment, which is positively associated with procedural justice, or the perception of
how a work group as a whole is treated (Ehrhart 2004; Walumbwa et al. 2010; Chung
et al. 2010). Procedural justice fosters trust in the servant leader and in the servantled
organization (Joseph and Winston 2005; Reinke 2004; Sendjaya and Pekerti 2010;
Washington et al. 2006). This creates an open and trusting environment, which can
enhance collaboration among team members (Garber et al. 2009; Sturm 2009; Irving
and Longbotham 2007). Collaboration in a servantled organization creates a helping
culture (i.e., a spirit of willingness), which increases team members’ organizational
citizenship behavior, defined as prosocial and altruistic behaviors that have been
shown to improve organizational performance (Ebener and O’Connell 2010; Hu and
Liden 2011; Ehrhart 2004; Walumbwa et al. 2010). Servant leadership also improves
overall team effectiveness (Taylor et al. 2007; Mayer et al. 2008; McCuddy and Cavin
2008) and can enhance leaders’ effectiveness (Irving and Longbotham 2007;
Schaubroeck et al. 2011; Hu and Liden 2011). In summary, servant leadership creates
a trusting, fair, collaborative, and helping culture that can result in greater individual
and organizational effectiveness.

Followers’ WellBeing

Findings from 15 empirical studies illustrate that servant leadership enhances
followers’ wellbeing. These studies showed conceptually and empirically how servant
leadership influences followers’ wellbeing by creating a positive work climate
(Neubert et al. 2008; Black 2010; Jaramillo et al. 2009a), which is related to greater
organizational commitment (Cerit 2010; Hamilton and Bean 2005; Hale and Fields
2007; Han et al. 2010; Pekerti and Sendjaya 2010). Greater commitment to the
organization increases employee job satisfaction (Cerit 2009; Jenkins and Stewart
2010; Mayer et al. 2008; Chung et al. 2010) and consequently decreases employee
turnover (Jaramillo et al. 2009b; Babakus et al. 2011). Servant leaders create these
positive outcomes by developing trust while nurturing followers, which encourages
the creativity, helping behaviors, and wellbeing of followers (Jaramillo et al. 2009a;
Babakus et al. 2011; Rieke et al. 2008). Overall, these studies support the notion that
servant leadership can improve followers’ wellbeing.

Spirituality

5/1/2017 A Systematic Literature Review of Servant Leadership Theory in Organizational Contexts | SpringerLink

https://link.springer.com/article/10.1007/s1055101213226 23/45

One study (Herman 2010) found a positive connection between workplace spirituality
and servant leadership, while six studies explored servant leadership within religious
intuitions. In addition, many scholars described servant leadership using the
teachings of Jesus Christ as a reference (Ebener and O’Connell 2010; Hamilton and
Bean 2005; Winston 2004). Although there appears to be a relationship between
spirituality and servant leadership, there was insufficient evidence to draw
conclusions for this review.

Demographics

Three studies (Fridell et al. 2009; McCuddy and Cavin 2009; Taylor et al. 2007)
attempted to identify demographic characteristics conducive to practicing servant
leadership. However, these studies lacked methodological quality sufficient to support
any conclusions. In addition, many of the findings of these studies contradicted each
other as well as other studies within our sample. For example, one study found
significant differences between men and women’s servant leadership style usage—
female leaders were more likely to practice daily reflection and consensus building,
foster self worth, and engage in healing relationships (Fridell et al. 2009), while
another study found no difference (McCuddy and Cavin 2009). Also, one study found
that socioeconomic factors were positively related to servant behaviors (McCuddy
and Cavin 2009), while another study found that no demographic variables were
significantly related to servant leadership (Taylor et al. 2007) Therefore, it remains to
be discovered if there are in fact demographic characteristics that are related to servant
leadership.

Implementation of Servant Leadership

Three studies examined servant leadership in various organizational processes
(Hamilton and Bean 2005—leadership development; SavageAustin and Honeycutt
2011—organizational change; Dingman and Stone 2007—succession planning).
Nevertheless, these studies were not supported by other empirical studies nor were
their methodological quality sufficient to provide any conclusions.

Limitations

Although this SLR was conducted in a disciplined manner, potential limitations must
be acknowledged. We limited the search process to indexed journals available through
the authors’ university library system that were peerreviewed published articles
written in the English language. Thus, this review did not include nonindexed
journals or dissertations because they are not peerreviewed, or peerreviewed servant
leadership articles published in a language other than English. Given the apparent
universal interest in servant leadership, as identified in our review, perhaps there are
more empirical studies being published in other languages that would complement or
contradict some of the conclusions drawn from this review. The methodology and
findings of the studies included in the review were assessed by two independent
reviewers aided by a critical assessment tool, which was utilized to make the
evaluation phase more accurate. However, our attempt to integrate results conducted
with qualitative as well as quantitative data analysis may have limited the ability to
sufficiently explore all methodological considerations when fusing the findings of both

5/1/2017 A Systematic Literature Review of Servant Leadership Theory in Organizational Contexts | SpringerLink

https://link.springer.com/article/10.1007/s1055101213226 24/45

types of empirical studies into a coherent text. In order to guide future scholars in
conducting SLRs, more work is needed on how to assess the quality of qualitative and
quantitative research in the field of management. Given SLRs origins are in the
medical field, which conduct controlled trial studies, there are few critical appraisal
tools that are applicable to the research methods used in other disciplines, such as
qualitative inquiry and crosssectional studies.

Conclusion

This SLR demonstrates servant leadership theory is applicable in a variety of cultures,
contexts, and organizational settings. Even though Greenleaf first coined the
philosophy in the 1970s, it has taken until 2004 for servant leadership to be explored
in an empirical manner. This SLR did not place any limitation on the publication year
of peerreviewed journal articles; however, no empirical studies were found across all
the databases searched before 2004. To date, the majority of research in servant
leadership is either attempting to conceptually define and model the theory or develop
measurement tools to empirical test it. Thus, the greater part of research on servant
leadership is addressing one of the major criticisms of the theoretical construct, which
is the difficulty of operationalizing its concepts and principles (Brumback 1999; Wong
and Davey 2007). Quay is not alone in his sentiments on Greenleaf’s works: “For all
his good advice and many practical ideas, he is a Don Quixote trying to convince
managers to pursue good and eschew evil” (1997, p. 83). By Greenleaf’s own
admission, his ideas are unorthodox, yet the value of this review illustrates that
servant leadership works and is a tenable theory.

The first question of this review sought to discover how servant leadership is being
defined. Although our findings indicated the majority of authors use Greenleaf (1970,
1972a, b, 1997), Spears (1998), and Laub (1999) to help define servant leadership,
there still does not exist an accepted consensus over its definition. This lack of
consensus creates confusion (Van Dierendonck 2011) amongst researchers, as they
create their own variations of definitions and theoretical models. Perhaps one day
there will be a generally accepted theory of servant leadership, but the empirical cross
cultural studies in this review highlight that while servant leadership has been
researched in a variety of cultures, it has different meanings based on socialization
and national context. In addition, Greenleaf (1977) argued that servant leadership is
an inward lifelong journey, implying that the meaning of servant leadership could
change throughout one’s life time. Therefore, this review does not conclude with a
model or another definition of servant leadership; however, it does provide an
overview of multiple definitions of servant leadership currently being used in
empirical studies in order to further our conceptual understanding.

Second, this review explored the contexts in which servant leadership is being
empirically investigated. Our review illustrates the diversity of cultures,
organizational settings, and research foci in which servant leadership is being
explored. There seems to be pronounced interest in investigating servant leadership in
the U.S. and throughout the Asia Pacific region; however, there is a paucity of studies
being conducted in other parts of the world. Currently, the majority of studies are
exploring servant leadership in an educational setting (44 % of our sample).
Organizational settings that have received less attention from researchers include
medical institutions, public organizations, nonprofit organizations, and community

5/1/2017 A Systematic Literature Review of Servant Leadership Theory in Organizational Contexts | SpringerLink

https://link.springer.com/article/10.1007/s1055101213226 25/45

level organizations. Research on servant leadership is concentrated in the fields of
leadership, education, business and psychology; whereas, there is only a small
number of studies in the fields of nursing, management, personal selling and sales,
ethics, parks and recreation administration, services marketing, and sports. The
research themes being explored the least are: spirituality, demographics, and
implementation of servant leadership. Thus, this review helps researchers identify
areas and contexts which are relatively unexplored in relation to servant leadership
and thus ripe for further investigation.

Third, this review examined the tools that can be used to measure the existence and
outcomes of servant leadership. The multiple quantitative and qualitative measures
used by the studies point to the fact that there is currently not an agreed upon
measurement instrument of the theoretical construct. This review can point
researchers towards the current measurement tools available, how they are being used,
and in what contexts they are being applied. Last, this review synthesized the findings
of these empirical studies (see Table 3). Seven research themes emerged: crosscultural
applicability, servant leadership attributes, teamlevel effectiveness, followers’ well
being, spirituality, demographics, and implementation of servant leadership. This
synthesis can help researchers identify the current findings in the extant literature and
to discover research foci that remain relatively underexplored.

Several intriguing directions for future research emerged from our SLR. First, this SLR
only identified 39 empirical studies that explored servant leadership theory in
organizational settings, highlighting the need for researchers to empirically
investigate the construct of servant leadership in a variety of organizational contexts.
In the burgeoning field of entrepreneurship, researchers could explore how to build a
servantled organization, or in the field of organizational change, studies could
explore how to implement servant leadership in an established organization or during
a merger or acquisition. Second, there is a need to investigate the antecedents of
servant leadership development, such as personal attributes of the leader, background
of the leader, and organizational history and trajectory. Third, researchers can
examine other outcomes of servant leadership, such as voluntarily organizational
turnover, succession planning, affective organizational commitment, and employee
wellbeing through generative growth. Last, there is a need to develop critical
appraisal tools for quantitative and quantitative research used in the field of
management to conduct SLRs. Perhaps our integration of several appraisal tools can
serve as a template, as we assessed the level of supporting evidence for thematic
conclusions drawn from combining the results of multiple studies.

This SLR is the first synthesis of empirical studies exploring servant leadership theory
in organizational contexts that utilizes a rigorous methodology to mitigate bias
through exhaustive literature searches and by providing an audit trail of the
conclusions. This review enhances our understanding of the definition(s) of servant
leadership, illustrates the diversity of cultures, organizational settings, and research
foci in which it is being examined, identifies tools that can be used to measure its
existence and outcomes, and shows that servant leadership is a viable leadership
theory that helps organizations and the wellbeing of followers. Our findings
synthesize empirical research on servant leadership theory across the
multidisciplinary fields of business, medicine, psychology, religion, leisure,
education, and economics and law. Scholars exploring servant leadership are using
theories from other disciplines to build upon existing theory and to develop theory
that is uniquely applicable to their field (e.g., organizational behavior, sport, gender).

5/1/2017 A Systematic Literature Review of Servant Leadership Theory in Organizational Contexts | SpringerLink

https://link.springer.com/article/10.1007/s1055101213226 26/45

Thus, this SLR validated servant leadership as a viable and valuable theory, and
therefore, illustrates how servant leadership theory can be used to inform future
empirical studies. In addition, and importantly, this SLR contributes to advancing
the methodology of conducting a SLR in the management context. Here, we showcase
how a SRL can provide an effective method for mapping out thematically the current
body of research literature that empirically explores servant leadership theory in
organizational contexts. However, this type of systematic review with rigorous
methodology can be applied to other research streams within management as an aid
in holistically synthesizing the state of the field in various topical areas.

As a viable leadership theory, servant leadership can perhaps provide the ethical
grounding and leadership framework needed to help address the challenges of the
twentyfirst century: technological advancements, economic globalization, increased
communications, the Internet, rising terrorism, environmental degradation, war and
violence, disease and starvation, threat of global warming, intensifying gap between
the poor and rich worldwide, as well as many other unsolved issues. Servant
leadership contrasts, traditional leaderfirst paradigms, which applaud a Darwinism,
individualistic, and capitalist approach to life, implicating that only the strong will
survive. Sadly, this belief system is operating at the heart of most organizations and is
the consequence of most of our modern tragedies: Arthur Andersen and Enron, Dennis
Kozlowski and Tyco, and Bernard Ebbers and WorldCom (Forbes 2010). Servant
leaders believe “the world does not have to be like this” (Keith 2008, p. ix) and actively
work at changing society for the better. In short, this review shows servant leadership
can help address these ethical dilemmas.

References

Akuchie, N. D. (1993). The servants and the superstars: An examination of servant
leadership in light of Matthew 20: 20–29. Christian Education Journal, 16(1), 39–
43.
Google Scholar (http://scholar.google.com/scholar_lookup?
title=The%20servants%20and%20the%20superstars%3A%20An%20examination%2
0of%20servant%20leadership%20in%20light%20of%20Matthew%2020%3A%2020%
E2%80%9329&author=ND.%20Akuchie&journal=Christian%20Education%20Journ
al&volume=16&issue=1&pages=3943&publication_year=1993)

Anderson, J. A. (2009). When a servantleader comes knocking …. Leadership &
Organization Development Journal, 30(1), 4–15.
CrossRef (http://dx.doi.org/10.1108/01437730910927070)
Google Scholar (http://scholar.google.com/scholar_lookup?
title=When%20a%20servant
leader%20comes%20knocking%20%E2%80%A6&author=JA.%20Anderson&journal=
Leadership%20%26%20Organization%20Development%20Journal&volume=30&issu
e=1&pages=415&publication_year=2009)

Babakus, E., Yavas, U., & Ashill, N. J. (2011). Service worker burnout and turnover
intentions: Roles of personjob fit, servant leadership, and customer orientation.
Services Marketing Quarterly, 32(1), 17–31.
CrossRef (http://dx.doi.org/10.1080/15332969.2011.533091)
Google Scholar (http://scholar.google.com/scholar_lookup?
title=Service%20worker%20burnout%20and%20turnover%20intentions%3A%20Role
s%20of%20person
job%20fit%2C%20servant%20leadership%2C%20and%20customer%20orientation&a

http://scholar.google.com/scholar_lookup?title=The%20servants%20and%20the%20superstars%3A%20An%20examination%20of%20servant%20leadership%20in%20light%20of%20Matthew%2020%3A%2020%E2%80%9329&author=ND.%20Akuchie&journal=Christian%20Education%20Journal&volume=16&issue=1&pages=39-43&publication_year=1993

http://dx.doi.org/10.1108/01437730910927070

http://scholar.google.com/scholar_lookup?title=When%20a%20servant-leader%20comes%20knocking%20%E2%80%A6&author=JA.%20Anderson&journal=Leadership%20%26%20Organization%20Development%20Journal&volume=30&issue=1&pages=4-15&publication_year=2009

http://dx.doi.org/10.1080/15332969.2011.533091

http://scholar.google.com/scholar_lookup?title=Service%20worker%20burnout%20and%20turnover%20intentions%3A%20Roles%20of%20person-job%20fit%2C%20servant%20leadership%2C%20and%20customer%20orientation&author=E.%20Babakus&author=U.%20Yavas&author=NJ.%20Ashill&journal=Services%20Marketing%20Quarterly&volume=32&issue=1&pages=17-31&publication_year=2011

5/1/2017 A Systematic Literature Review of Servant Leadership Theory in Organizational Contexts | SpringerLink

https://link.springer.com/article/10.1007/s1055101213226 27/45

uthor=E.%20Babakus&author=U.%20Yavas&author=NJ.%20Ashill&journal=Service
s%20Marketing%20Quarterly&volume=32&issue=1&pages=17
31&publication_year=2011)

BanutuGomez, M. B., & BanutuGomez, S. M. T. (2007). Leadership and
organizational change in a competitive environment. Business Renaissance
Quarterly, 2(2), 69–91.
Google Scholar (http://scholar.google.com/scholar_lookup?
title=Leadership%20and%20organizational%20change%20in%20a%20competitive%
20environment&author=MB.%20BanutuGomez&author=SMT.%20Banutu
Gomez&journal=Business%20Renaissance%20Quarterly&volume=2&issue=2&pages
=6991&publication_year=2007)

Barbuto, J. E., & Wheeler, D. W. (2006). Scale development and construct
clarification of servant leadership. Group & Organization Management, 31, 300–326.
CrossRef (http://dx.doi.org/10.1177/1059601106287091)
Google Scholar (http://scholar.google.com/scholar_lookup?
title=Scale%20development%20and%20construct%20clarification%20of%20servant
%20leadership&author=JE.%20Barbuto&author=DW.%20Wheeler&journal=Group%
20%26%20Organization%20Management&volume=31&pages=300
326&publication_year=2006)

Barrow, J. C. (1977). The variables of leadership: A review and conceptual framework.
Academy of Management Review, 2, 233–251.
Google Scholar (http://scholar.google.com/scholar_lookup?
title=The%20variables%20of%20leadership%3A%20A%20review%20and%20concept
ual%20framework&author=JC.%20Barrow&journal=Academy%20of%20Managemen
t%20Review&volume=2&pages=233251&publication_year=1977)

Bass, B. M. (2000). The future of leadership in the learning organization. Journal of
Leadership Studies, 7(3), 18–38.
CrossRef (http://dx.doi.org/10.1177/107179190000700302)
Google Scholar (http://scholar.google.com/scholar_lookup?
title=The%20future%20of%20leadership%20in%20the%20learning%20organization
&author=BM.%20Bass&journal=Journal%20of%20Leadership%20Studies&volume=
7&issue=3&pages=1838&publication_year=2000)

Bass, B., & Bass, R. (2008). The bass handbook of leadership: Theory, research, and
managerial applications (4th ed.). New York: The Free Press.

Black, G. L. (2010). Correlational analysis of servant leadership and school climate.
Catholic Education: A Journal of Inquiry & Practice, 13(4), 437–466.
Google Scholar (http://scholar.google.com/scholar_lookup?
title=Correlational%20analysis%20of%20servant%20leadership%20and%20school%
20climate&author=GL.%20Black&journal=Catholic%20Education%3A%20A%20Jou
rnal%20of%20Inquiry%20%26%20Practice&volume=13&issue=4&pages=437
466&publication_year=2010)

Block, P. (1993). Stewardship: Choosing service over self interest. San Francisco:
BerrettKoehler Publishers.
Google Scholar (http://scholar.google.com/scholar_lookup?
title=Stewardship%3A%20Choosing%20service%20over%20self%20interest&author=
P.%20Block&publication_year=1993)

Bordas, J. (1995). Power and passion: Finding personal purpose. In L. Spears (Ed.),
Reflections of leadership: How Robert K. Greenleaf’s theory of servant leadership
influenced today’s top management thinkers (pp. 179–193). New York: Wiley.

http://scholar.google.com/scholar_lookup?title=Leadership%20and%20organizational%20change%20in%20a%20competitive%20environment&author=MB.%20Banutu-Gomez&author=SMT.%20Banutu-Gomez&journal=Business%20Renaissance%20Quarterly&volume=2&issue=2&pages=69-91&publication_year=2007

http://dx.doi.org/10.1177/1059601106287091

http://scholar.google.com/scholar_lookup?title=Scale%20development%20and%20construct%20clarification%20of%20servant%20leadership&author=JE.%20Barbuto&author=DW.%20Wheeler&journal=Group%20%26%20Organization%20Management&volume=31&pages=300-326&publication_year=2006

http://scholar.google.com/scholar_lookup?title=The%20variables%20of%20leadership%3A%20A%20review%20and%20conceptual%20framework&author=JC.%20Barrow&journal=Academy%20of%20Management%20Review&volume=2&pages=233-251&publication_year=1977

http://dx.doi.org/10.1177/107179190000700302

http://scholar.google.com/scholar_lookup?title=The%20future%20of%20leadership%20in%20the%20learning%20organization&author=BM.%20Bass&journal=Journal%20of%20Leadership%20Studies&volume=7&issue=3&pages=18-38&publication_year=2000

http://scholar.google.com/scholar_lookup?title=Correlational%20analysis%20of%20servant%20leadership%20and%20school%20climate&author=GL.%20Black&journal=Catholic%20Education%3A%20A%20Journal%20of%20Inquiry%20%26%20Practice&volume=13&issue=4&pages=437-466&publication_year=2010

http://scholar.google.com/scholar_lookup?title=Stewardship%3A%20Choosing%20service%20over%20self%20interest&author=P.%20Block&publication_year=1993

5/1/2017 A Systematic Literature Review of Servant Leadership Theory in Organizational Contexts | SpringerLink

https://link.springer.com/article/10.1007/s1055101213226 28/45

Google Scholar (http://scholar.google.com/scholar_lookup?
title=Power%20and%20passion%3A%20Finding%20personal%20purpose&author=J
.%20Bordas&pages=179193&publication_year=1995)

Boroski, E., & Greif, T. B. (2009). Servantleaders in community colleges: Their
values, beliefs, and implications. Review of Business Research, 9(4), 113–120.
Google Scholar (http://scholar.google.com/scholar_lookup?title=Servant
leaders%20in%20community%20colleges%3A%20Their%20values%2C%20beliefs%2
C%20and%20implications&author=E.%20Boroski&author=TB.%20Greif&journal=R
eview%20of%20Business%20Research&volume=9&issue=4&pages=113
120&publication_year=2009)

Bowman, M. A. (1997). Popular approaches to leadership. In P. G. Northhouse (Ed.),
Leadership: Theory and practice (pp. 239–260). Thousand Oaks, CA: Sage.
Google Scholar (http://scholar.google.com/scholar_lookup?
title=Popular%20approaches%20to%20leadership&author=MA.%20Bowman&pages
=239260&publication_year=1997)

Brody, D. (1995). First among equals: A corporate executive’s vision and the
reemerging philosophy of trustees as servantleaders. In L. Spears (Ed.), Reflections of
leadership: How Robert K. Greenleaf’s theory of servant leadership influenced
today’s top management thinkers (pp. 129–132). New York: Wiley.
Google Scholar (http://scholar.google.com/scholar_lookup?
title=First%20among%20equals%3A%20A%20corporate%20executive%E2%80%99s
%20vision%20and%20the%20reemerging%20philosophy%20of%20trustees%20as%2
0servantleaders&author=D.%20Brody&pages=129132&publication_year=1995)

Buchen, I. H. (1998). Servant leadership: A model for future faculty and future
institutions. The Journal of Leadership Studies, 5(1), 125–134.
CrossRef (http://dx.doi.org/10.1177/107179199800500111)
Google Scholar (http://scholar.google.com/scholar_lookup?
title=Servant%20leadership%3A%20A%20model%20for%20future%20faculty%20an
d%20future%20institutions&author=IH.%20Buchen&journal=The%20Journal%20of
%20Leadership%20Studies&volume=5&issue=1&pages=125
134&publication_year=1998)

Cerit, Y. (2009). The effects of servant leadership behaviors of school principals on
teachers’ job satisfaction. Educational Management Administration & Leadership,
37(5), 600–623.
CrossRef (http://dx.doi.org/10.1177/1741143209339650)
Google Scholar (http://scholar.google.com/scholar_lookup?
title=The%20effects%20of%20servant%20leadership%20behaviors%20of%20school%
20principals%20on%20teachers%E2%80%99%20job%20satisfaction&author=Y.%20
Cerit&journal=Educational%20Management%20Administration%20%26%20Leaders
hip&volume=37&issue=5&pages=600623&publication_year=2009)

Cerit, Y. (2010). The effects of servant leadership on teachers’ organizational
commitment in primary schools in Turkey. International Journal of Leadership in
Education, 13(3), 301–317.
CrossRef (http://dx.doi.org/10.1080/13603124.2010.496933)
Google Scholar (http://scholar.google.com/scholar_lookup?
title=The%20effects%20of%20servant%20leadership%20on%20teachers%E2%80%9
9%20organizational%20commitment%20in%20primary%20schools%20in%20Turke
y&author=Y.%20Cerit&journal=International%20Journal%20of%20Leadership%20i
n%20Education&volume=13&issue=3&pages=301317&publication_year=2010)

Chamberlain, P. (1995). Teambuilding and servantleadership. In L. Spears (Ed.),
Reflections of leadership: How Robert K. Greenleaf’s theory of servant leadership

http://scholar.google.com/scholar_lookup?title=Power%20and%20passion%3A%20Finding%20personal%20purpose&author=J.%20Bordas&pages=179-193&publication_year=1995

http://scholar.google.com/scholar_lookup?title=Servant-leaders%20in%20community%20colleges%3A%20Their%20values%2C%20beliefs%2C%20and%20implications&author=E.%20Boroski&author=TB.%20Greif&journal=Review%20of%20Business%20Research&volume=9&issue=4&pages=113-120&publication_year=2009

http://scholar.google.com/scholar_lookup?title=Popular%20approaches%20to%20leadership&author=MA.%20Bowman&pages=239-260&publication_year=1997

http://scholar.google.com/scholar_lookup?title=First%20among%20equals%3A%20A%20corporate%20executive%E2%80%99s%20vision%20and%20the%20reemerging%20philosophy%20of%20trustees%20as%20servant-leaders&author=D.%20Brody&pages=129-132&publication_year=1995

http://dx.doi.org/10.1177/107179199800500111

http://scholar.google.com/scholar_lookup?title=Servant%20leadership%3A%20A%20model%20for%20future%20faculty%20and%20future%20institutions&author=IH.%20Buchen&journal=The%20Journal%20of%20Leadership%20Studies&volume=5&issue=1&pages=125-134&publication_year=1998

http://dx.doi.org/10.1177/1741143209339650

http://scholar.google.com/scholar_lookup?title=The%20effects%20of%20servant%20leadership%20behaviors%20of%20school%20principals%20on%20teachers%E2%80%99%20job%20satisfaction&author=Y.%20Cerit&journal=Educational%20Management%20Administration%20%26%20Leadership&volume=37&issue=5&pages=600-623&publication_year=2009

http://dx.doi.org/10.1080/13603124.2010.496933

http://scholar.google.com/scholar_lookup?title=The%20effects%20of%20servant%20leadership%20on%20teachers%E2%80%99%20organizational%20commitment%20in%20primary%20schools%20in%20Turkey&author=Y.%20Cerit&journal=International%20Journal%20of%20Leadership%20in%20Education&volume=13&issue=3&pages=301-317&publication_year=2010

5/1/2017 A Systematic Literature Review of Servant Leadership Theory in Organizational Contexts | SpringerLink

https://link.springer.com/article/10.1007/s1055101213226 29/45

influenced today’s top management thinkers (pp. 169–178). New York: Wiley.
Google Scholar (http://scholar.google.com/scholar_lookup?title=Team
building%20and%20servantleadership&author=P.%20Chamberlain&pages=169
178&publication_year=1995)

Chung, J. Y., Jung, C. S., Kyle, G. T., & Petrick, J. F. (2010). Servant leadership and
procedural justice in the U.S. national park service: The antecedents of job
satisfaction. Journal of Park & Recreation Administration, 28(3), 1–15.
Google Scholar (http://scholar.google.com/scholar_lookup?
title=Servant%20leadership%20and%20procedural%20justice%20in%20the%20U.S.
%20national%20park%20service%3A%20The%20antecedents%20of%20job%20satisf
action&author=JY.%20Chung&author=CS.%20Jung&author=GT.%20Kyle&author=
JF.%20Petrick&journal=Journal%20of%20Park%20%26%20Recreation%20Administ
ration&volume=28&issue=3&pages=115&publication_year=2010)

Cook, D. J., Mulrow, C. D., & Haynes, R. B. (1997). Systematic reviews: Synthesis of
best evidence for clinical decisions. Annals of Internal Medicine, 126(5), 376–380.
CrossRef (http://dx.doi.org/10.7326/00034819126519970301000006)
Google Scholar (http://scholar.google.com/scholar_lookup?
title=Systematic%20reviews%3A%20Synthesis%20of%20best%20evidence%20for%2
0clinical%20decisions&author=DJ.%20Cook&author=CD.%20Mulrow&author=RB.
%20Haynes&journal=Annals%20of%20Internal%20Medicine&volume=126&issue=5
&pages=376380&publication_year=1997)

Covey, S. (1990). Principle centered leadership. New York: Simon and Schuster.
Google Scholar (http://scholar.google.com/scholar_lookup?
title=Principle%20centered%20leadership&author=S.%20Covey&publication_year=1
990)

Crippen, C. L. (2004). Pioneer women in Manitoba: Evidence of servantleadership.
Journal of Women in Educational Leadership, 2(4), 257–271.
Google Scholar (http://scholar.google.com/scholar_lookup?
title=Pioneer%20women%20in%20Manitoba%3A%20Evidence%20of%20servant
leadership&author=CL.%20Crippen&journal=Journal%20of%20Women%20in%20E
ducational%20Leadership&volume=2&issue=4&pages=257
271&publication_year=2004)

Crippen, C., & Wallin, D. (2008a). First conversations with Manitoba
superintendents: Talking their walk. Alberta Journal of Educational Research, 54(2),
147–160.
Google Scholar (http://scholar.google.com/scholar_lookup?
title=First%20conversations%20with%20Manitoba%20superintendents%3A%20Talk
ing%20their%20walk&author=C.%20Crippen&author=D.%20Wallin&journal=Albert
a%20Journal%20of%20Educational%20Research&volume=54&issue=2&pages=147
160&publication_year=2008)

Crippen, C., & Wallin, D. (2008b). Manitoba superintendents: Mentoring and
leadership. Educational Management Administration & Leadership, 36(4), 546–
565.
CrossRef (http://dx.doi.org/10.1177/1741143208095793)
Google Scholar (http://scholar.google.com/scholar_lookup?
title=Manitoba%20superintendents%3A%20Mentoring%20and%20leadership&autho
r=C.%20Crippen&author=D.%20Wallin&journal=Educational%20Management%20
Administration%20%26%20Leadership&volume=36&issue=4&pages=546
565&publication_year=2008)

Cyert, R. (2006). Defining leadership and explicating the process. Nonprofit
Management and Leadership, 1(1), 29–38.

http://scholar.google.com/scholar_lookup?title=Team-building%20and%20servant-leadership&author=P.%20Chamberlain&pages=169-178&publication_year=1995

http://scholar.google.com/scholar_lookup?title=Servant%20leadership%20and%20procedural%20justice%20in%20the%20U.S.%20national%20park%20service%3A%20The%20antecedents%20of%20job%20satisfaction&author=JY.%20Chung&author=CS.%20Jung&author=GT.%20Kyle&author=JF.%20Petrick&journal=Journal%20of%20Park%20%26%20Recreation%20Administration&volume=28&issue=3&pages=1-15&publication_year=2010

http://dx.doi.org/10.7326/0003-4819-126-5-199703010-00006

http://scholar.google.com/scholar_lookup?title=Systematic%20reviews%3A%20Synthesis%20of%20best%20evidence%20for%20clinical%20decisions&author=DJ.%20Cook&author=CD.%20Mulrow&author=RB.%20Haynes&journal=Annals%20of%20Internal%20Medicine&volume=126&issue=5&pages=376-380&publication_year=1997

http://scholar.google.com/scholar_lookup?title=Principle%20centered%20leadership&author=S.%20Covey&publication_year=1990

http://scholar.google.com/scholar_lookup?title=Pioneer%20women%20in%20Manitoba%3A%20Evidence%20of%20servant-leadership&author=CL.%20Crippen&journal=Journal%20of%20Women%20in%20Educational%20Leadership&volume=2&issue=4&pages=257-271&publication_year=2004

http://scholar.google.com/scholar_lookup?title=First%20conversations%20with%20Manitoba%20superintendents%3A%20Talking%20their%20walk&author=C.%20Crippen&author=D.%20Wallin&journal=Alberta%20Journal%20of%20Educational%20Research&volume=54&issue=2&pages=147-160&publication_year=2008

http://dx.doi.org/10.1177/1741143208095793

http://scholar.google.com/scholar_lookup?title=Manitoba%20superintendents%3A%20Mentoring%20and%20leadership&author=C.%20Crippen&author=D.%20Wallin&journal=Educational%20Management%20Administration%20%26%20Leadership&volume=36&issue=4&pages=546-565&publication_year=2008

5/1/2017 A Systematic Literature Review of Servant Leadership Theory in Organizational Contexts | SpringerLink

https://link.springer.com/article/10.1007/s1055101213226 30/45

CrossRef (http://dx.doi.org/10.1002/nml.4130010105)
Google Scholar (http://scholar.google.com/scholar_lookup?
title=Defining%20leadership%20and%20explicating%20the%20process&author=R.
%20Cyert&journal=Nonprofit%20Management%20and%20Leadership&volume=1&is
sue=1&pages=2938&publication_year=2006)

De Pree, M. (1989). Leadership is an art. New York: Doubleday Business.
Google Scholar (http://scholar.google.com/scholar_lookup?
title=Leadership%20is%20an%20art&author=M.%20Pree&publication_year=1989)

Dennis, R. (2004). Servant leadership theory: Development of the servant leadership
assessment instrument. Unpublished PhD Thesis. Regent University, Virginia Beach,
Virginia, USA.

Dennis, R. S., & Bocarnea, M. C. (2005). Development of the servant leadership
assessment instrument. Leadership and Organization Development Journal, 26(8),
600–615.
CrossRef (http://dx.doi.org/10.1108/01437730510633692)
Google Scholar (http://scholar.google.com/scholar_lookup?
title=Development%20of%20the%20servant%20leadership%20assessment%20instru
ment&author=RS.%20Dennis&author=MC.%20Bocarnea&journal=Leadership%20an
d%20Organization%20Development%20Journal&volume=26&issue=8&pages=600
615&publication_year=2005)

Dennis, R. S., & Winston, B. (2003). A factor analysis of Page and Wong’s servant
leadership assessment instrument. Leadership & Organization Development
Journal, 24(8), 455–459.
CrossRef (http://dx.doi.org/10.1108/01437730310505885)
Google Scholar (http://scholar.google.com/scholar_lookup?
title=A%20factor%20analysis%20of%20Page%20and%20Wong%E2%80%99s%20ser
vant%20leadership%20assessment%20instrument&author=RS.%20Dennis&author=
B.%20Winston&journal=Leadership%20%26%20Organization%20Development%20
Journal&volume=24&issue=8&pages=455459&publication_year=2003)

DiMaggio, P. (1995). Comments on “What Theory is Not”. Administrative Science
Quarterly, 40, 391–397.
CrossRef (http://dx.doi.org/10.2307/2393790)
Google Scholar (http://scholar.google.com/scholar_lookup?
title=Comments%20on%20%E2%80%9CWhat%20Theory%20is%20Not%E2%80%9
D&author=P.%20DiMaggio&journal=Administrative%20Science%20Quarterly&volu
me=40&pages=391397&publication_year=1995)

Dingman, W. W., & Stone, A. G. (2007). Servant leadership’s role in the succession
planning process: A case study. International Journal of Leadership Studies, 2(2),
98–113.
Google Scholar (http://scholar.google.com/scholar_lookup?
title=Servant%20leadership%E2%80%99s%20role%20in%20the%20succession%20p
lanning%20process%3A%20A%20case%20study&author=WW.%20Dingman&author
=AG.%20Stone&journal=International%20Journal%20of%20Leadership%20Studies
&volume=2&issue=2&pages=98113&publication_year=2007)

Ebener, D. R., & O’Connell, D. J. (2010). How might servant leadership work?
Nonprofit Management and Leadership, 20(3), 315–335.
CrossRef (http://dx.doi.org/10.1002/nml.256)
Google Scholar (http://scholar.google.com/scholar_lookup?
title=How%20might%20servant%20leadership%20work%3F&author=DR.%20Ebene
r&author=DJ.%20O%E2%80%99Connell&journal=Nonprofit%20Management%20a
nd%20Leadership&volume=20&issue=3&pages=315335&publication_year=2010)

http://dx.doi.org/10.1002/nml.4130010105

http://scholar.google.com/scholar_lookup?title=Defining%20leadership%20and%20explicating%20the%20process&author=R.%20Cyert&journal=Nonprofit%20Management%20and%20Leadership&volume=1&issue=1&pages=29-38&publication_year=2006

http://scholar.google.com/scholar_lookup?title=Leadership%20is%20an%20art&author=M.%20Pree&publication_year=1989

http://dx.doi.org/10.1108/01437730510633692

http://scholar.google.com/scholar_lookup?title=Development%20of%20the%20servant%20leadership%20assessment%20instrument&author=RS.%20Dennis&author=MC.%20Bocarnea&journal=Leadership%20and%20Organization%20Development%20Journal&volume=26&issue=8&pages=600-615&publication_year=2005

http://dx.doi.org/10.1108/01437730310505885

http://scholar.google.com/scholar_lookup?title=A%20factor%20analysis%20of%20Page%20and%20Wong%E2%80%99s%20servant%20leadership%20assessment%20instrument&author=RS.%20Dennis&author=B.%20Winston&journal=Leadership%20%26%20Organization%20Development%20Journal&volume=24&issue=8&pages=455-459&publication_year=2003

http://dx.doi.org/10.2307/2393790

http://scholar.google.com/scholar_lookup?title=Comments%20on%20%E2%80%9CWhat%20Theory%20is%20Not%E2%80%9D&author=P.%20DiMaggio&journal=Administrative%20Science%20Quarterly&volume=40&pages=391-397&publication_year=1995

http://scholar.google.com/scholar_lookup?title=Servant%20leadership%E2%80%99s%20role%20in%20the%20succession%20planning%20process%3A%20A%20case%20study&author=WW.%20Dingman&author=AG.%20Stone&journal=International%20Journal%20of%20Leadership%20Studies&volume=2&issue=2&pages=98-113&publication_year=2007

http://dx.doi.org/10.1002/nml.256

http://scholar.google.com/scholar_lookup?title=How%20might%20servant%20leadership%20work%3F&author=DR.%20Ebener&author=DJ.%20O%E2%80%99Connell&journal=Nonprofit%20Management%20and%20Leadership&volume=20&issue=3&pages=315-335&publication_year=2010

5/1/2017 A Systematic Literature Review of Servant Leadership Theory in Organizational Contexts | SpringerLink

https://link.springer.com/article/10.1007/s1055101213226 31/45

Ehrhart, M. G. (2004). Leadership and procedural justice climate as antecedents of
unit level organizational citizenship behavior. Personnel Psychology, 57(1), 61–94.
CrossRef (http://dx.doi.org/10.1111/j.17446570.2004.tb02484.x)
Google Scholar (http://scholar.google.com/scholar_lookup?
title=Leadership%20and%20procedural%20justice%20climate%20as%20antecedents
%20of%20unit%20level%20organizational%20citizenship%20behavior&author=MG.
%20Ehrhart&journal=Personnel%20Psychology&volume=57&issue=1&pages=61
94&publication_year=2004)

Farling, M. L., Stone, A. G., & Winston, B. E. (1999). Servant leadership: Setting the
stage for empirical research. Journal of Leadership Studies, 6, 49–62.
CrossRef (http://dx.doi.org/10.1177/107179199900600104)
Google Scholar (http://scholar.google.com/scholar_lookup?
title=Servant%20leadership%3A%20Setting%20the%20stage%20for%20empirical%2
0research&author=ML.%20Farling&author=AG.%20Stone&author=BE.%20Winston
&journal=Journal%20of%20Leadership%20Studies&volume=6&pages=49
62&publication_year=1999)

Forbes.: 2010, The corporate scandal sheet. Retrieved November 25th from
http://www.forbes.com/2002/07/25/accountingtracker.html
(http://www.forbes.com/2002/07/25/accountingtracker.html).

Frick, D. M. (1995). Pyramids, circles, and gardens: Stories of implementing servant
leadership. In L. Spears (Ed.), Reflections of leadership: How Robert K. Greenleaf’s
theory of servantleadership influenced today’s top management thinkers (pp. 241–
256). New York: Wiley.
Google Scholar (http://scholar.google.com/scholar_lookup?
title=Pyramids%2C%20circles%2C%20and%20gardens%3A%20Stories%20of%20imp
lementing%20servant%20leadership&author=DM.%20Frick&pages=241
256&publication_year=1995)

Frick, D. M. (2004). Robert K. Greenleaf: A life of servant leadership. San Francisco:
BerrettKoehler.
Google Scholar (http://scholar.google.com/scholar_lookup?
title=Robert%20K.%20Greenleaf%3A%20A%20life%20of%20servant%20leadership&
author=DM.%20Frick&publication_year=2004)

Fridell, M., Belcher, R. N., & Messner, P. E. (2009). Discriminate analysis gender
public school principal servant leadership differences. Leadership & Organization
Development Journal, 30(8), 722–736.
CrossRef (http://dx.doi.org/10.1108/01437730911003894)
Google Scholar (http://scholar.google.com/scholar_lookup?
title=Discriminate%20analysis%20gender%20public%20school%20principal%20serv
ant%20leadership%20differences&author=M.%20Fridell&author=RN.%20Belcher&a
uthor=PE.%20Messner&journal=Leadership%20%26%20Organization%20Developm
ent%20Journal&volume=30&issue=8&pages=722736&publication_year=2009)

Garber, J. S., Madigan, E. A., Click, E. R., & Fitzpatrick, J. J. (2009). Attitudes
towards collaboration and servant leadership among nurses, physicians and
residents. Journal of Interprofessional Care, 23(4), 331–340.
CrossRef (http://dx.doi.org/10.1080/13561820902886253)
Google Scholar (http://scholar.google.com/scholar_lookup?
title=Attitudes%20towards%20collaboration%20and%20servant%20leadership%20a
mong%20nurses%2C%20physicians%20and%20residents&author=JS.%20Garber&a
uthor=EA.%20Madigan&author=ER.%20Click&author=JJ.%20Fitzpatrick&journal=
Journal%20of%20Interprofessional%20Care&volume=23&issue=4&pages=331
340&publication_year=2009)

http://dx.doi.org/10.1111/j.1744-6570.2004.tb02484.x

http://scholar.google.com/scholar_lookup?title=Leadership%20and%20procedural%20justice%20climate%20as%20antecedents%20of%20unit%20level%20organizational%20citizenship%20behavior&author=MG.%20Ehrhart&journal=Personnel%20Psychology&volume=57&issue=1&pages=61-94&publication_year=2004

http://dx.doi.org/10.1177/107179199900600104

http://scholar.google.com/scholar_lookup?title=Servant%20leadership%3A%20Setting%20the%20stage%20for%20empirical%20research&author=ML.%20Farling&author=AG.%20Stone&author=BE.%20Winston&journal=Journal%20of%20Leadership%20Studies&volume=6&pages=49-62&publication_year=1999

http://www.forbes.com/2002/07/25/accountingtracker.html

http://scholar.google.com/scholar_lookup?title=Pyramids%2C%20circles%2C%20and%20gardens%3A%20Stories%20of%20implementing%20servant%20leadership&author=DM.%20Frick&pages=241-256&publication_year=1995

http://scholar.google.com/scholar_lookup?title=Robert%20K.%20Greenleaf%3A%20A%20life%20of%20servant%20leadership&author=DM.%20Frick&publication_year=2004

http://dx.doi.org/10.1108/01437730911003894

http://scholar.google.com/scholar_lookup?title=Discriminate%20analysis%20gender%20public%20school%20principal%20servant%20leadership%20differences&author=M.%20Fridell&author=RN.%20Belcher&author=PE.%20Messner&journal=Leadership%20%26%20Organization%20Development%20Journal&volume=30&issue=8&pages=722-736&publication_year=2009

http://dx.doi.org/10.1080/13561820902886253

http://scholar.google.com/scholar_lookup?title=Attitudes%20towards%20collaboration%20and%20servant%20leadership%20among%20nurses%2C%20physicians%20and%20residents&author=JS.%20Garber&author=EA.%20Madigan&author=ER.%20Click&author=JJ.%20Fitzpatrick&journal=Journal%20of%20Interprofessional%20Care&volume=23&issue=4&pages=331-340&publication_year=2009

5/1/2017 A Systematic Literature Review of Servant Leadership Theory in Organizational Contexts | SpringerLink

https://link.springer.com/article/10.1007/s1055101213226 32/45

Garrard, J. (1999). Health sciences literature review made easy: The matrix method.
Sudbury, MA: Jones and Barlett Publishers.
Google Scholar (http://scholar.google.com/scholar_lookup?
title=Health%20sciences%20literature%20review%20made%20easy%3A%20The%20
matrix%20method&author=J.%20Garrard&publication_year=1999)

Gaston, H. G. (1987). A model for leadership: Servant stewardship ministry.
Southwestern Journal of Theology, 37(2), 35–43.
Google Scholar (http://scholar.google.com/scholar_lookup?
title=A%20model%20for%20leadership%3A%20Servant%20stewardship%20ministry
&author=HG.%20Gaston&journal=Southwestern%20Journal%20of%20Theology&vo
lume=37&issue=2&pages=3543&publication_year=1987)

Graham, J. (1991). Servantleadership in organizations: Inspirational and moral.
Leadership Quarterly, 2(2), 105–119.
CrossRef (http://dx.doi.org/10.1016/10489843(91)90025W)
Google Scholar (http://scholar.google.com/scholar_lookup?title=Servant
leadership%20in%20organizations%3A%20Inspirational%20and%20moral&author=
J.%20Graham&journal=Leadership%20Quarterly&volume=2&issue=2&pages=105
119&publication_year=1991)

Greenleaf, R. K. (1970). The servant as leader. Indianapolis: The Robert K. Greenleaf
Center.
Google Scholar (http://scholar.google.com/scholar_lookup?
title=The%20servant%20as%20leader&author=RK.%20Greenleaf&publication_year=
1970)

Greenleaf, R. K. (1972a). The institution as servant. Indianapolis: The Robert K.
Greenleaf Center.
Google Scholar (http://scholar.google.com/scholar_lookup?
title=The%20institution%20as%20servant&author=RK.%20Greenleaf&publication_y
ear=1972)

Greenleaf, R. K. (1972b). Trustees as servants. Indianapolis: The Robert K. Greenleaf
Center.
Google Scholar (http://scholar.google.com/scholar_lookup?
title=Trustees%20as%20servants&author=RK.%20Greenleaf&publication_year=1972
)

Greenleaf, R. K. (1977). Servant leadership: A journey into the nature of legitimate
power and greatness. New York: Paulist Press.
Google Scholar (http://scholar.google.com/scholar_lookup?
title=Servant%20leadership%3A%20A%20journey%20into%20the%20nature%20of%
20legitimate%20power%20and%20greatness&author=RK.%20Greenleaf&publication
_year=1977)

Greenleaf Center, Inc. (2011). Retrieved July 10, 2011 from http://www.greenleaf.org/
(http://www.greenleaf.org/).

Hale, J. R., & Fields, D. L. (2007). Exploring servant leadership across cultures: A
study of followers in Ghana and the USA. Leadership, 3(4), 397–417.
CrossRef (http://dx.doi.org/10.1177/1742715007082964)
Google Scholar (http://scholar.google.com/scholar_lookup?
title=Exploring%20servant%20leadership%20across%20cultures%3A%20A%20study
%20of%20followers%20in%20Ghana%20and%20the%20USA&author=JR.%20Hale&
author=DL.%20Fields&journal=Leadership&volume=3&issue=4&pages=397
417&publication_year=2007)

http://scholar.google.com/scholar_lookup?title=Health%20sciences%20literature%20review%20made%20easy%3A%20The%20matrix%20method&author=J.%20Garrard&publication_year=1999

http://scholar.google.com/scholar_lookup?title=A%20model%20for%20leadership%3A%20Servant%20stewardship%20ministry&author=HG.%20Gaston&journal=Southwestern%20Journal%20of%20Theology&volume=37&issue=2&pages=35-43&publication_year=1987

http://dx.doi.org/10.1016/1048-9843(91)90025-W

http://scholar.google.com/scholar_lookup?title=Servant-leadership%20in%20organizations%3A%20Inspirational%20and%20moral&author=J.%20Graham&journal=Leadership%20Quarterly&volume=2&issue=2&pages=105-119&publication_year=1991

http://scholar.google.com/scholar_lookup?title=The%20servant%20as%20leader&author=RK.%20Greenleaf&publication_year=1970

http://scholar.google.com/scholar_lookup?title=The%20institution%20as%20servant&author=RK.%20Greenleaf&publication_year=1972

http://scholar.google.com/scholar_lookup?title=Trustees%20as%20servants&author=RK.%20Greenleaf&publication_year=1972

http://scholar.google.com/scholar_lookup?title=Servant%20leadership%3A%20A%20journey%20into%20the%20nature%20of%20legitimate%20power%20and%20greatness&author=RK.%20Greenleaf&publication_year=1977

Home

http://dx.doi.org/10.1177/1742715007082964

http://scholar.google.com/scholar_lookup?title=Exploring%20servant%20leadership%20across%20cultures%3A%20A%20study%20of%20followers%20in%20Ghana%20and%20the%20USA&author=JR.%20Hale&author=DL.%20Fields&journal=Leadership&volume=3&issue=4&pages=397-417&publication_year=2007

5/1/2017 A Systematic Literature Review of Servant Leadership Theory in Organizational Contexts | SpringerLink

https://link.springer.com/article/10.1007/s1055101213226 33/45

Hamilton, F., & Bean, C. J. (2005). The importance of context, beliefs and values in
leadership development. Business Ethics: A European Review, 14(4), 336–347.
CrossRef (http://dx.doi.org/10.1111/j.14678608.2005.00415.x)
Google Scholar (http://scholar.google.com/scholar_lookup?
title=The%20importance%20of%20context%2C%20beliefs%20and%20values%20in%
20leadership%20development&author=F.%20Hamilton&author=CJ.%20Bean&journ
al=Business%20Ethics%3A%20A%20European%20Review&volume=14&issue=4&pa
ges=336347&publication_year=2005)

Hammermeister, J., Burton, D., Pickering, M. A., Westro, K., Baldwin, N., & Chase,
M. (2008). Servant leadership in sport: A Philosophy whose time has arrived.
International Journal of Servant Leadership, 4, 185–215.
Google Scholar (http://scholar.google.com/scholar_lookup?
title=Servant%20leadership%20in%20sport%3A%20A%20Philosophy%20whose%20
time%20has%20arrived&author=J.%20Hammermeister&author=D.%20Burton&aut
hor=MA.%20Pickering&author=K.%20Westro&author=N.%20Baldwin&author=M.%
20Chase&journal=International%20Journal%20of%20Servant%20Leadership&volu
me=4&pages=185215&publication_year=2008)

Han, Y., Kakabadse, N. K., & Kakabadse, A. (2010). Servant leadership in the People’s
Republic of China: A case study of the public sector. Journal of Management
Development, 29(3), 265–281.
CrossRef (http://dx.doi.org/10.1108/02621711011025786)
Google Scholar (http://scholar.google.com/scholar_lookup?
title=Servant%20leadership%20in%20the%20People%E2%80%99s%20Republic%20
of%20China%3A%20A%20case%20study%20of%20the%20public%20sector&author
=Y.%20Han&author=NK.%20Kakabadse&author=A.%20Kakabadse&journal=Journa
l%20of%20Management%20Development&volume=29&issue=3&pages=265
281&publication_year=2010)

Herman, R. (2010). The promise of servant leadership for workplace spirituality.
International Journal of Business Research, 10(6), 83–102.
Google Scholar (http://scholar.google.com/scholar_lookup?
title=The%20promise%20of%20servant%20leadership%20for%20workplace%20spiri
tuality&author=R.%20Herman&journal=International%20Journal%20of%20Busines
s%20Research&volume=10&issue=6&pages=83102&publication_year=2010)

Hesse, H. (1956). The journey of the east. New York: Noonday Press.
Google Scholar (http://scholar.google.com/scholar_lookup?
title=The%20journey%20of%20the%20east&author=H.%20Hesse&publication_year
=1956)

Hu, J., & Liden, R. C. (2011). Antecedents of team potency and team effectiveness: An
examination of goal and process clarity and servant leadership. Journal of Applied
Psychology 1–12. doi:10.1037/a0022465 (http://dx.doi.org/10.1037/a0022465).

Institute for Public Health Sciences. (2002). 11 questions to help you make sense of
descriptive/crosssectional studies. New York: Yeshiva University.
Google Scholar (http://scholar.google.com/scholar_lookup?
title=11%20questions%20to%20help%20you%20make%20sense%20of%20descriptiv
e%2Fcrosssectional%20studies&publication_year=2002)

Irving, J. A., & Longbotham, G. J. (2007). Team effectiveness and six essential servant
leadership themes: A regression model based on items in the organizational leadership
assessment. International Journal of Leadership Studies, 2(2), 98–113.
Google Scholar (http://scholar.google.com/scholar_lookup?
title=Team%20effectiveness%20and%20six%20essential%20servant%20leadership%
20themes%3A%20A%20regression%20model%20based%20on%20items%20in%20th

http://dx.doi.org/10.1111/j.1467-8608.2005.00415.x

http://scholar.google.com/scholar_lookup?title=The%20importance%20of%20context%2C%20beliefs%20and%20values%20in%20leadership%20development&author=F.%20Hamilton&author=CJ.%20Bean&journal=Business%20Ethics%3A%20A%20European%20Review&volume=14&issue=4&pages=336-347&publication_year=2005

http://scholar.google.com/scholar_lookup?title=Servant%20leadership%20in%20sport%3A%20A%20Philosophy%20whose%20time%20has%20arrived&author=J.%20Hammermeister&author=D.%20Burton&author=MA.%20Pickering&author=K.%20Westro&author=N.%20Baldwin&author=M.%20Chase&journal=International%20Journal%20of%20Servant%20Leadership&volume=4&pages=185-215&publication_year=2008

http://dx.doi.org/10.1108/02621711011025786

http://scholar.google.com/scholar_lookup?title=Servant%20leadership%20in%20the%20People%E2%80%99s%20Republic%20of%20China%3A%20A%20case%20study%20of%20the%20public%20sector&author=Y.%20Han&author=NK.%20Kakabadse&author=A.%20Kakabadse&journal=Journal%20of%20Management%20Development&volume=29&issue=3&pages=265-281&publication_year=2010

http://scholar.google.com/scholar_lookup?title=The%20promise%20of%20servant%20leadership%20for%20workplace%20spirituality&author=R.%20Herman&journal=International%20Journal%20of%20Business%20Research&volume=10&issue=6&pages=83-102&publication_year=2010

http://scholar.google.com/scholar_lookup?title=The%20journey%20of%20the%20east&author=H.%20Hesse&publication_year=1956

http://dx.doi.org/10.1037/a0022465

http://scholar.google.com/scholar_lookup?title=11%20questions%20to%20help%20you%20make%20sense%20of%20descriptive%2Fcross-sectional%20studies&publication_year=2002

http://scholar.google.com/scholar_lookup?title=Team%20effectiveness%20and%20six%20essential%20servant%20leadership%20themes%3A%20A%20regression%20model%20based%20on%20items%20in%20the%20organizational%20leadership%20assessment&author=JA.%20Irving&author=GJ.%20Longbotham&journal=International%20Journal%20of%20Leadership%20Studies&volume=2&issue=2&pages=98-113&publication_year=2007

5/1/2017 A Systematic Literature Review of Servant Leadership Theory in Organizational Contexts | SpringerLink

https://link.springer.com/article/10.1007/s1055101213226 34/45

e%20organizational%20leadership%20assessment&author=JA.%20Irving&author=G
J.%20Longbotham&journal=International%20Journal%20of%20Leadership%20Stud
ies&volume=2&issue=2&pages=98113&publication_year=2007)

Jaramillo, F., Grisaffe, D. B., Chonko, L. B., & Roberts, J. A. (2009a). Examining the
impact of servant leadership on sales force performance. Journal of Personal Selling &
Sales Management, 29(3), 257–275.
CrossRef (http://dx.doi.org/10.2753/PSS08853134290304)
Google Scholar (http://scholar.google.com/scholar_lookup?
title=Examining%20the%20impact%20of%20servant%20leadership%20on%20sales
%20force%20performance&author=F.%20Jaramillo&author=DB.%20Grisaffe&author
=LB.%20Chonko&author=JA.%20Roberts&journal=Journal%20of%20Personal%20S
elling%20%26%20Sales%20Management&volume=29&issue=3&pages=257
275&publication_year=2009)

Jaramillo, F., Grisaffe, D. B., Chonko, L. B., & Roberts, J. A. (2009b). Examining the
impact of servant leadership on salesperson’s turnover intention. Journal of Personal
Selling & Sales Management, 29(4), 351–365.
CrossRef (http://dx.doi.org/10.2753/PSS08853134290404)
Google Scholar (http://scholar.google.com/scholar_lookup?
title=Examining%20the%20impact%20of%20servant%20leadership%20on%20sales
person%E2%80%99s%20turnover%20intention&author=F.%20Jaramillo&author=D
B.%20Grisaffe&author=LB.%20Chonko&author=JA.%20Roberts&journal=Journal%
20of%20Personal%20Selling%20%26%20Sales%20Management&volume=29&issue=
4&pages=351365&publication_year=2009)

Jenkins, M., & Stewart, A. C. (2010). The importance of a servant leader orientation.
Health Care Management Review, 35(1), 46–54.
CrossRef (http://dx.doi.org/10.1097/HMR.0b013e3181c22bb8)
Google Scholar (http://scholar.google.com/scholar_lookup?
title=The%20importance%20of%20a%20servant%20leader%20orientation&author=
M.%20Jenkins&author=AC.%20Stewart&journal=Health%20Care%20Management%
20Review&volume=35&issue=1&pages=4654&publication_year=2010)

Joseph, E. E., & Winston, B. E. (2005). A correlation of servant leadership, leader
trust, and organizational trust. Leadership & Organization Development Journal,
26(1), 6–22.
CrossRef (http://dx.doi.org/10.1108/01437730510575552)
Google Scholar (http://scholar.google.com/scholar_lookup?
title=A%20correlation%20of%20servant%20leadership%2C%20leader%20trust%2C%
20and%20organizational%20trust&author=EE.%20Joseph&author=BE.%20Winsto
n&journal=Leadership%20%26%20Organization%20Development%20Journal&volu
me=26&issue=1&pages=622&publication_year=2005)

Keith, K. (2008). The case for servant leadership. Westfield, IN: Greenleaf Center for
Servant Leadership.

Kelley, M. (1995). The new leadership. In L. Spears (Ed.), Reflections of leadership:
How Robert K. Greenleaf’s theory of servantleadership influenced today’s top
management thinkers (pp. 194–197). New York: Wiley.
Google Scholar (http://scholar.google.com/scholar_lookup?
title=The%20new%20leadership&author=M.%20Kelley&pages=194
197&publication_year=1995)

Kiechel, W. III (1995). The leader as servant. In L. Spears (Ed.), Reflections of
leadership: How Robert K. Greenleaf’s theory of servantleadership influenced
today’s top management thinkers (pp. 121–125). New York: Wiley.

http://dx.doi.org/10.2753/PSS0885-3134290304

http://scholar.google.com/scholar_lookup?title=Examining%20the%20impact%20of%20servant%20leadership%20on%20sales%20force%20performance&author=F.%20Jaramillo&author=DB.%20Grisaffe&author=LB.%20Chonko&author=JA.%20Roberts&journal=Journal%20of%20Personal%20Selling%20%26%20Sales%20Management&volume=29&issue=3&pages=257-275&publication_year=2009

http://dx.doi.org/10.2753/PSS0885-3134290404

http://scholar.google.com/scholar_lookup?title=Examining%20the%20impact%20of%20servant%20leadership%20on%20salesperson%E2%80%99s%20turnover%20intention&author=F.%20Jaramillo&author=DB.%20Grisaffe&author=LB.%20Chonko&author=JA.%20Roberts&journal=Journal%20of%20Personal%20Selling%20%26%20Sales%20Management&volume=29&issue=4&pages=351-365&publication_year=2009

http://dx.doi.org/10.1097/HMR.0b013e3181c22bb8

http://scholar.google.com/scholar_lookup?title=The%20importance%20of%20a%20servant%20leader%20orientation&author=M.%20Jenkins&author=AC.%20Stewart&journal=Health%20Care%20Management%20Review&volume=35&issue=1&pages=46-54&publication_year=2010

http://dx.doi.org/10.1108/01437730510575552

http://scholar.google.com/scholar_lookup?title=A%20correlation%20of%20servant%20leadership%2C%20leader%20trust%2C%20and%20organizational%20trust&author=EE.%20Joseph&author=BE.%20Winston&journal=Leadership%20%26%20Organization%20Development%20Journal&volume=26&issue=1&pages=6-22&publication_year=2005

http://scholar.google.com/scholar_lookup?title=The%20new%20leadership&author=M.%20Kelley&pages=194-197&publication_year=1995

5/1/2017 A Systematic Literature Review of Servant Leadership Theory in Organizational Contexts | SpringerLink

https://link.springer.com/article/10.1007/s1055101213226 35/45

Google Scholar (http://scholar.google.com/scholar_lookup?
title=The%20leader%20as%20servant&author=W.%20Kiechel&pages=121
125&publication_year=1995)

Klassen, T. P., Jahad, A. R., & Moher, D. (1998). Guides for reading and interpreting
systematic reviews. Archives of Pediatric & Adolescent Medicine, 157(7), 700–704.
Google Scholar (http://scholar.google.com/scholar_lookup?
title=Guides%20for%20reading%20and%20interpreting%20systematic%20reviews&
author=TP.%20Klassen&author=AR.%20Jahad&author=D.%20Moher&journal=Arch
ives%20of%20Pediatric%20%26%20Adolescent%20Medicine&volume=157&issue=7&
pages=700704&publication_year=1998)

Kotter, J. P. (2001). What leaders really do? Harvard Business Review, December, 3–
12.

Kuhnert, K. W., & Lewis, P. (1987). Transactional and transformational leadership: A
constructive/developmental analysis. Academy of Management Review, 12(4), 648–
657.
Google Scholar (http://scholar.google.com/scholar_lookup?
title=Transactional%20and%20transformational%20leadership%3A%20A%20constr
uctive%2Fdevelopmental%20analysis&author=KW.%20Kuhnert&author=P.%20Lewi
s&journal=Academy%20of%20Management%20Review&volume=12&issue=4&pages
=648657&publication_year=1987)

Lanctot, J. D., & Irving, J. A. (2010). Character and leadership: Situating servant
leadership in a proposed virtues framework. International Journal of Leadership
Studies, 6(1), 28–50.
Google Scholar (http://scholar.google.com/scholar_lookup?
title=Character%20and%20leadership%3A%20Situating%20servant%20leadership%
20in%20a%20proposed%20virtues%20framework&author=JD.%20Lanctot&author=
JA.%20Irving&journal=International%20Journal%20of%20Leadership%20Studies&v
olume=6&issue=1&pages=2850&publication_year=2010)

Laub, J. (1999). Assessing the servant organization: Development of the Servant
Organizational Leadership (SOLA) instrument. Dissertation Abstracts International,
60(2), 308 (UMI No. 9921922).

Lee, C., & Zemke, R. (1995). The search for spirit in the workplace. In L. Spears (Ed.),
Reflections of leadership: How Robert K. Greenleaf’s theory of servant leadership
influenced today’s top management thinkers (pp. 99–112). New York: Wiley.
Google Scholar (http://scholar.google.com/scholar_lookup?
title=The%20search%20for%20spirit%20in%20the%20workplace&author=C.%20Lee
&author=R.%20Zemke&pages=99112&publication_year=1995)

Letts, L., Wilkins, S., Law, M., Stewart, D., Bosch, J., & Westmorland, M. (2007).
Critical review form: Qualitative studies (version 2.0). Retrieved from
http://www.sph.nhs.uk/sphfiles/caspappraisaltools/Qualitative%20Appraisal%20Tool
(http://www.sph.nhs.uk/sphfiles/caspappraisaltools/Qualitative%20Appraisal%20
Tool ).

Liden, R. C., Wayne, S. J., Zhao, H., & Henderson, D. (2008). Servant leadership:
Development of a multidimensional measure and multilevel assessment. Leadership
Quarterly, 19, 161–177.
CrossRef (http://dx.doi.org/10.1016/j.leaqua.2008.01.006)
Google Scholar (http://scholar.google.com/scholar_lookup?
title=Servant%20leadership%3A%20Development%20of%20a%20multidimensional
%20measure%20and%20multi
level%20assessment&author=RC.%20Liden&author=SJ.%20Wayne&author=H.%20

http://scholar.google.com/scholar_lookup?title=The%20leader%20as%20servant&author=W.%20Kiechel&pages=121-125&publication_year=1995

http://scholar.google.com/scholar_lookup?title=Guides%20for%20reading%20and%20interpreting%20systematic%20reviews&author=TP.%20Klassen&author=AR.%20Jahad&author=D.%20Moher&journal=Archives%20of%20Pediatric%20%26%20Adolescent%20Medicine&volume=157&issue=7&pages=700-704&publication_year=1998

http://scholar.google.com/scholar_lookup?title=Transactional%20and%20transformational%20leadership%3A%20A%20constructive%2Fdevelopmental%20analysis&author=KW.%20Kuhnert&author=P.%20Lewis&journal=Academy%20of%20Management%20Review&volume=12&issue=4&pages=648-657&publication_year=1987

http://scholar.google.com/scholar_lookup?title=Character%20and%20leadership%3A%20Situating%20servant%20leadership%20in%20a%20proposed%20virtues%20framework&author=JD.%20Lanctot&author=JA.%20Irving&journal=International%20Journal%20of%20Leadership%20Studies&volume=6&issue=1&pages=28-50&publication_year=2010

http://scholar.google.com/scholar_lookup?title=The%20search%20for%20spirit%20in%20the%20workplace&author=C.%20Lee&author=R.%20Zemke&pages=99-112&publication_year=1995

http://www.sph.nhs.uk/sphfiles/caspappraisaltools/Qualitative%20Appraisal%20Tool

http://dx.doi.org/10.1016/j.leaqua.2008.01.006

http://scholar.google.com/scholar_lookup?title=Servant%20leadership%3A%20Development%20of%20a%20multidimensional%20measure%20and%20multi-level%20assessment&author=RC.%20Liden&author=SJ.%20Wayne&author=H.%20Zhao&author=D.%20Henderson&journal=Leadership%20Quarterly&volume=19&pages=161-177&publication_year=2008

5/1/2017 A Systematic Literature Review of Servant Leadership Theory in Organizational Contexts | SpringerLink

https://link.springer.com/article/10.1007/s1055101213226 36/45

Zhao&author=D.%20Henderson&journal=Leadership%20Quarterly&volume=19&pa
ges=161177&publication_year=2008)

Lloyd, B. (1996). A new approach to leadership. Leadership and Organizational
Development Journal, 17(7), 29–32.
CrossRef (http://dx.doi.org/10.1108/01437739610148358)
Google Scholar (http://scholar.google.com/scholar_lookup?
title=A%20new%20approach%20to%20leadership&author=B.%20Lloyd&journal=Le
adership%20and%20Organizational%20Development%20Journal&volume=17&issue
=7&pages=2932&publication_year=1996)

Lopez, I. O. (1995). Becoming a servantleader: The personal development path. In L.
Spears (Ed.), Reflections of leadership: How Robert K. Greenleaf’s theory of servant
leadership influenced today’s top management thinkers (pp. 179–193). New York:
Wiley.
Google Scholar (http://scholar.google.com/scholar_lookup?
title=Becoming%20a%20servant
leader%3A%20The%20personal%20development%20path&author=IO.%20Lopez&pa
ges=179193&publication_year=1995)

Lytle, R. S., Hom, P. W., & Mokwa, M. P. (1998). SERV_OR: A managerial measure of
organizational serviceorientation. Journal of Retailing, 74, 455–489.
CrossRef (http://dx.doi.org/10.1016/S00224359(99)801043)
Google Scholar (http://scholar.google.com/scholar_lookup?
title=SERV_OR%3A%20A%20managerial%20measure%20of%20organizational%20s
ervice
orientation&author=RS.%20Lytle&author=PW.%20Hom&author=MP.%20Mokwa&j
ournal=Journal%20of%20Retailing&volume=74&pages=455
489&publication_year=1998)

Mayer, D. M., Bardes, M., & Piccolo, R. F. (2008). Do servantleaders help satisfy
follower needs? An organizational justice perspective. European Journal of Work and
Organizational Psychology, 17(2), 180–197.
CrossRef (http://dx.doi.org/10.1080/13594320701743558)
Google Scholar (http://scholar.google.com/scholar_lookup?title=Do%20servant
leaders%20help%20satisfy%20follower%20needs%3F%20An%20organizational%20j
ustice%20perspective&author=DM.%20Mayer&author=M.%20Bardes&author=RF.%
20Piccolo&journal=European%20Journal%20of%20Work%20and%20Organizational
%20Psychology&volume=17&issue=2&pages=180197&publication_year=2008)

McCollum, J. (1995). Chaos, complexity, and servantleadership. In L. Spears (Ed.),
Reflections of leadership: How Robert K. Greenleaf’s theory of servant leadership
influenced today’s top management thinkers (pp. 241–256). New York: Wiley.
Google Scholar (http://scholar.google.com/scholar_lookup?
title=Chaos%2C%20complexity%2C%20and%20servant
leadership&author=J.%20McCollum&pages=241256&publication_year=1995)

McCuddy, M. K., & Cavin, M. C. (2008). Fundamental moral orientations, servant
leadership, and leadership effectiveness: An empirical test. Review of Business
Research, 8(4), 107–117.
Google Scholar (http://scholar.google.com/scholar_lookup?
title=Fundamental%20moral%20orientations%2C%20servant%20leadership%2C%2
0and%20leadership%20effectiveness%3A%20An%20empirical%20test&author=MK.
%20McCuddy&author=MC.%20Cavin&journal=Review%20of%20Business%20Resea
rch&volume=8&issue=4&pages=107117&publication_year=2008)

McCuddy, M. K., & Cavin, M. C. (2009). The demographic context of servant
leadership. Journal of the Academy of Business & Economics, 9(2), 129–139.

http://dx.doi.org/10.1108/01437739610148358

http://scholar.google.com/scholar_lookup?title=A%20new%20approach%20to%20leadership&author=B.%20Lloyd&journal=Leadership%20and%20Organizational%20Development%20Journal&volume=17&issue=7&pages=29-32&publication_year=1996

http://scholar.google.com/scholar_lookup?title=Becoming%20a%20servant-leader%3A%20The%20personal%20development%20path&author=IO.%20Lopez&pages=179-193&publication_year=1995

http://dx.doi.org/10.1016/S0022-4359(99)80104-3

http://scholar.google.com/scholar_lookup?title=SERV_OR%3A%20A%20managerial%20measure%20of%20organizational%20service-orientation&author=RS.%20Lytle&author=PW.%20Hom&author=MP.%20Mokwa&journal=Journal%20of%20Retailing&volume=74&pages=455-489&publication_year=1998

http://dx.doi.org/10.1080/13594320701743558

http://scholar.google.com/scholar_lookup?title=Do%20servant-leaders%20help%20satisfy%20follower%20needs%3F%20An%20organizational%20justice%20perspective&author=DM.%20Mayer&author=M.%20Bardes&author=RF.%20Piccolo&journal=European%20Journal%20of%20Work%20and%20Organizational%20Psychology&volume=17&issue=2&pages=180-197&publication_year=2008

http://scholar.google.com/scholar_lookup?title=Chaos%2C%20complexity%2C%20and%20servant-leadership&author=J.%20McCollum&pages=241-256&publication_year=1995

http://scholar.google.com/scholar_lookup?title=Fundamental%20moral%20orientations%2C%20servant%20leadership%2C%20and%20leadership%20effectiveness%3A%20An%20empirical%20test&author=MK.%20McCuddy&author=MC.%20Cavin&journal=Review%20of%20Business%20Research&volume=8&issue=4&pages=107-117&publication_year=2008

5/1/2017 A Systematic Literature Review of Servant Leadership Theory in Organizational Contexts | SpringerLink

https://link.springer.com/article/10.1007/s1055101213226 37/45

Google Scholar (http://scholar.google.com/scholar_lookup?
title=The%20demographic%20context%20of%20servant%20leadership&author=MK.
%20McCuddy&author=MC.%20Cavin&journal=Journal%20of%20the%20Academy%
20of%20Business%20%26%20Economics&volume=9&issue=2&pages=129
139&publication_year=2009)

McGeeCooper, A., & Trammell, D. (1995). Servant leadership: Is there really time for
it? In L. Spears (Ed.), Reflections of leadership: How Robert K. Greenleaf’s theory of
servant leadership influenced today’s top management thinkers (pp. 113–120). New
York: Wiley.
Google Scholar (http://scholar.google.com/scholar_lookup?
title=Servant%20leadership%3A%20Is%20there%20really%20time%20for%20it%3F
&author=A.%20McGeeCooper&author=D.%20Trammell&pages=113
120&publication_year=1995)

Neubert, M. J., Kacmar, K. M., Carlson, D. S., Chonko, L. B., & Roberts, J. A. (2008).
Regulatory focus as a mediator of the influence of initiating structure and servant
leadership on employee behavior. Journal of Applied Psychology, 93(6), 1220–1233.
CrossRef (http://dx.doi.org/10.1037/a0012695)
Google Scholar (http://scholar.google.com/scholar_lookup?
title=Regulatory%20focus%20as%20a%20mediator%20of%20the%20influence%20of
%20initiating%20structure%20and%20servant%20leadership%20on%20employee%
20behavior&author=MJ.%20Neubert&author=KM.%20Kacmar&author=DS.%20Carl
son&author=LB.%20Chonko&author=JA.%20Roberts&journal=Journal%20of%20Ap
plied%20Psychology&volume=93&issue=6&pages=1220
1233&publication_year=2008)

Northouse, P. G. (1997). Leadership: Theory and practice. Thousand Oaks, CA: Sage.
Google Scholar (http://scholar.google.com/scholar_lookup?
title=Leadership%3A%20Theory%20and%20practice&author=PG.%20Northouse&pu
blication_year=1997)

OLA Group (Organizational Leadership Assessment Group). (2011). Retrieved
September 4, 2011, from http://www.olagroup.com/ (http://www.olagroup.com/).

Page, D., & Wong, T. P. (1998). A philosophy conceptual framework for measuring
servant leadership. Unpublished manuscript (Langley, Canada: Trinity Western
University).

Page, D., & Wong, T. P. (2000). A philosophy conceptual framework for measuring
servant leadership. In S. Adjibolosoo (Ed.), The Human factor in shaping the course
of history and development. Lanham, MD: University Press of America.
Google Scholar (http://scholar.google.com/scholar_lookup?
title=A%20philosophy%20conceptual%20framework%20for%20measuring%20serva
nt%20leadership&author=D.%20Page&author=TP.%20Wong&publication_year=200
0)

Parolini, J., Patterson, K., & Winston, B. (2009). Distinguishing between
transformational and servant leadership. Leadership & Organizational Development
Journal, 30(3), 274–291.
Google Scholar (http://scholar.google.com/scholar_lookup?
title=Distinguishing%20between%20transformational%20and%20servant%20leader
ship&author=J..%20Parolini&author=K..%20Patterson&author=B..%20Winston&jo
urnal=Leadership%20%26%20Organizational%20Development%20Journal&volume
=30&issue=3&pages=274291&publication_year=2009)

Patterson, K. (2003). Servant leadership: A theoretical model. Dissertation Abstracts
International, 64(2), 570 (UMI No. 3082719).

http://scholar.google.com/scholar_lookup?title=The%20demographic%20context%20of%20servant%20leadership&author=MK.%20McCuddy&author=MC.%20Cavin&journal=Journal%20of%20the%20Academy%20of%20Business%20%26%20Economics&volume=9&issue=2&pages=129-139&publication_year=2009

http://scholar.google.com/scholar_lookup?title=Servant%20leadership%3A%20Is%20there%20really%20time%20for%20it%3F&author=A.%20McGee-Cooper&author=D.%20Trammell&pages=113-120&publication_year=1995

http://dx.doi.org/10.1037/a0012695

http://scholar.google.com/scholar_lookup?title=Regulatory%20focus%20as%20a%20mediator%20of%20the%20influence%20of%20initiating%20structure%20and%20servant%20leadership%20on%20employee%20behavior&author=MJ.%20Neubert&author=KM.%20Kacmar&author=DS.%20Carlson&author=LB.%20Chonko&author=JA.%20Roberts&journal=Journal%20of%20Applied%20Psychology&volume=93&issue=6&pages=1220-1233&publication_year=2008

http://scholar.google.com/scholar_lookup?title=Leadership%3A%20Theory%20and%20practice&author=PG.%20Northouse&publication_year=1997

http://www.olagroup.com/

http://scholar.google.com/scholar_lookup?title=A%20philosophy%20conceptual%20framework%20for%20measuring%20servant%20leadership&author=D.%20Page&author=TP.%20Wong&publication_year=2000

http://scholar.google.com/scholar_lookup?title=Distinguishing%20between%20transformational%20and%20servant%20leadership&author=J..%20Parolini&author=K..%20Patterson&author=B..%20Winston&journal=Leadership%20%26%20Organizational%20Development%20Journal&volume=30&issue=3&pages=274-291&publication_year=2009

5/1/2017 A Systematic Literature Review of Servant Leadership Theory in Organizational Contexts | SpringerLink

https://link.springer.com/article/10.1007/s1055101213226 38/45

Pekerti, A. A., & Sendjaya, S. (2010). Exploring servant leadership across cultures:
Comparative study in Australia and Indonesia. The International Journal of Human
Resource Management, 21(5), 754–780.
CrossRef (http://dx.doi.org/10.1080/09585191003658920)
Google Scholar (http://scholar.google.com/scholar_lookup?
title=Exploring%20servant%20leadership%20across%20cultures%3A%20Comparati
ve%20study%20in%20Australia%20and%20Indonesia&author=AA.%20Pekerti&aut
hor=S.%20Sendjaya&journal=The%20International%20Journal%20of%20Human%2
0Resource%20Management&volume=21&issue=5&pages=754
780&publication_year=2010)

Plsek, P., & Wilson, T. (2001). Complexity, leadership, and management in healthcare
organizations. British Medical Journal (BMJ), 323, 746–749.
CrossRef (http://dx.doi.org/10.1136/bmj.323.7315.746)
Google Scholar (http://scholar.google.com/scholar_lookup?
title=Complexity%2C%20leadership%2C%20and%20management%20in%20healthc
are%20organizations&author=P.%20Plsek&author=T.%20Wilson&journal=British%
20Medical%20Journal%20%28BMJ%29&volume=323&pages=746
749&publication_year=2001)

Prosser, S. (2010). Servant leadership: More philosophy, less theory. Westfield, IN:
The Greenleaf Center for Servant Leadership.
Google Scholar (http://scholar.google.com/scholar_lookup?
title=Servant%20leadership%3A%20More%20philosophy%2C%20less%20theory&au
thor=S.%20Prosser&publication_year=2010)

Rasmussen, T. (1995). Creating a culture of servant leadership: A real life story. In L.
Spears (Ed.), Reflections of leadership: How Robert K. Greenleaf’s theory of servant
leadership influenced today’s top management thinkers (pp. 282–307). New York:
Wiley.
Google Scholar (http://scholar.google.com/scholar_lookup?
title=Creating%20a%20culture%20of%20servant%20leadership%3A%20A%20real%2
0life%20story&author=T.%20Rasmussen&pages=282307&publication_year=1995)

Reinke, S. J. (2004). Service before self: Towards a theory of servantleadership. Global
Virtue Ethics Review, 5, 30–57.
Google Scholar (http://scholar.google.com/scholar_lookup?
title=Service%20before%20self%3A%20Towards%20a%20theory%20of%20servant
leadership&author=SJ.%20Reinke&journal=Global%20Virtue%20Ethics%20Review&
volume=5&pages=3057&publication_year=2004)

Rieke, M., Hammermeister, J., & Chase, M. (2008). Servant leadership in sport: A new
paradigm for effective coach behavior. International Journal of Sports Science &
Coaching, 3(2), 227–239.
CrossRef (http://dx.doi.org/10.1260/174795408785100635)
Google Scholar (http://scholar.google.com/scholar_lookup?
title=Servant%20leadership%20in%20sport%3A%20A%20new%20paradigm%20for
%20effective%20coach%20behavior&author=M.%20Rieke&author=J.%20Hammerme
ister&author=M.%20Chase&journal=International%20Journal%20of%20Sports%20S
cience%20%26%20Coaching&volume=3&issue=2&pages=227
239&publication_year=2008)

Rieser, C. (1995). Claiming servantleadership as your heritage. In L. Spears (Ed.),
Reflections of leadership: How Robert K. Greenleaf’s theory of servant leadership
influenced today’s top management thinkers (pp. 49–60). New York: Wiley.
Google Scholar (http://scholar.google.com/scholar_lookup?
title=Claiming%20servant

http://dx.doi.org/10.1080/09585191003658920

http://scholar.google.com/scholar_lookup?title=Exploring%20servant%20leadership%20across%20cultures%3A%20Comparative%20study%20in%20Australia%20and%20Indonesia&author=AA.%20Pekerti&author=S.%20Sendjaya&journal=The%20International%20Journal%20of%20Human%20Resource%20Management&volume=21&issue=5&pages=754-780&publication_year=2010

http://dx.doi.org/10.1136/bmj.323.7315.746

http://scholar.google.com/scholar_lookup?title=Complexity%2C%20leadership%2C%20and%20management%20in%20healthcare%20organizations&author=P.%20Plsek&author=T.%20Wilson&journal=British%20Medical%20Journal%20%28BMJ%29&volume=323&pages=746-749&publication_year=2001

http://scholar.google.com/scholar_lookup?title=Servant%20leadership%3A%20More%20philosophy%2C%20less%20theory&author=S.%20Prosser&publication_year=2010

http://scholar.google.com/scholar_lookup?title=Creating%20a%20culture%20of%20servant%20leadership%3A%20A%20real%20life%20story&author=T.%20Rasmussen&pages=282-307&publication_year=1995

http://scholar.google.com/scholar_lookup?title=Service%20before%20self%3A%20Towards%20a%20theory%20of%20servant-leadership&author=SJ.%20Reinke&journal=Global%20Virtue%20Ethics%20Review&volume=5&pages=30-57&publication_year=2004

http://dx.doi.org/10.1260/174795408785100635

http://scholar.google.com/scholar_lookup?title=Servant%20leadership%20in%20sport%3A%20A%20new%20paradigm%20for%20effective%20coach%20behavior&author=M.%20Rieke&author=J.%20Hammermeister&author=M.%20Chase&journal=International%20Journal%20of%20Sports%20Science%20%26%20Coaching&volume=3&issue=2&pages=227-239&publication_year=2008

http://scholar.google.com/scholar_lookup?title=Claiming%20servant-leadership%20as%20your%20heritage&author=C.%20Rieser&pages=49-60&publication_year=1995

5/1/2017 A Systematic Literature Review of Servant Leadership Theory in Organizational Contexts | SpringerLink

https://link.springer.com/article/10.1007/s1055101213226 39/45

leadership%20as%20your%20heritage&author=C.%20Rieser&pages=49
60&publication_year=1995)

Russell, R. (2001). The role of values in servant leadership. Leadership &
Organization Development Journal, 22(2), 76–83.
CrossRef (http://dx.doi.org/10.1108/01437730110382631)
Google Scholar (http://scholar.google.com/scholar_lookup?
title=The%20role%20of%20values%20in%20servant%20leadership&author=R.%20R
ussell&journal=Leadership%20%26%20Organization%20Development%20Journal&
volume=22&issue=2&pages=7683&publication_year=2001)

Russell, R., & Stone, A. G. (2002). A review of servant leadership attributes:
Developing a practical model. Leadership and Organizational Development Journal,
23(3), 145–157.
CrossRef (http://dx.doi.org/10.1108/01437730210424)
Google Scholar (http://scholar.google.com/scholar_lookup?
title=A%20review%20of%20servant%20leadership%20attributes%3A%20Developing
%20a%20practical%20model&author=R.%20Russell&author=AG.%20Stone&journal
=Leadership%20and%20Organizational%20Development%20Journal&volume=23&i
ssue=3&pages=145157&publication_year=2002)

SavageAustin, A., & Honeycutt, A. (2011). Servant leadership: A phenomenological
study of practices, experiences, organizational effectiveness, and barriers. Journal of
Business & Economics Research, 9(1), 49–54.
Google Scholar (http://scholar.google.com/scholar_lookup?
title=Servant%20leadership%3A%20A%20phenomenological%20study%20of%20pra
ctices%2C%20experiences%2C%20organizational%20effectiveness%2C%20and%20ba
rriers&author=A.%20Savage
Austin&author=A.%20Honeycutt&journal=Journal%20of%20Business%20%26%20
Economics%20Research&volume=9&issue=1&pages=4954&publication_year=2011)

Schaubroeck, J., Lam, S. S. K., & Peng, A. C. (2011). Cognitionbased and affectbased
trust as mediators of leader behavior influences on team performance. Journal of
Applied Psychology, 96(4), 863–871.
CrossRef (http://dx.doi.org/10.1037/a0022625)
Google Scholar (http://scholar.google.com/scholar_lookup?title=Cognition
based%20and%20affect
based%20trust%20as%20mediators%20of%20leader%20behavior%20influences%20o
n%20team%20performance&author=J.%20Schaubroeck&author=SSK.%20Lam&auth
or=AC.%20Peng&journal=Journal%20of%20Applied%20Psychology&volume=96&is
sue=4&pages=863871&publication_year=2011)

Schneider, B. (1987). The people make the place. Personnel Psychology, 40, 437–453.
CrossRef (http://dx.doi.org/10.1111/j.17446570.1987.tb00609.x)
Google Scholar (http://scholar.google.com/scholar_lookup?
title=The%20people%20make%20the%20place&author=B.%20Schneider&journal=P
ersonnel%20Psychology&volume=40&pages=437453&publication_year=1987)

Sendjaya, S., & Pekerti, A. (2010). Servant leadership as antecedent of trust in
organizations. Leadership & Organization Development Journal, 31(7), 643–663.
CrossRef (http://dx.doi.org/10.1108/01437731011079673)
Google Scholar (http://scholar.google.com/scholar_lookup?
title=Servant%20leadership%20as%20antecedent%20of%20trust%20in%20organizat
ions&author=S.%20Sendjaya&author=A.%20Pekerti&journal=Leadership%20%26%2
0Organization%20Development%20Journal&volume=31&issue=7&pages=643
663&publication_year=2010)

http://scholar.google.com/scholar_lookup?title=Claiming%20servant-leadership%20as%20your%20heritage&author=C.%20Rieser&pages=49-60&publication_year=1995

http://dx.doi.org/10.1108/01437730110382631

http://scholar.google.com/scholar_lookup?title=The%20role%20of%20values%20in%20servant%20leadership&author=R.%20Russell&journal=Leadership%20%26%20Organization%20Development%20Journal&volume=22&issue=2&pages=76-83&publication_year=2001

http://dx.doi.org/10.1108/01437730210424

http://scholar.google.com/scholar_lookup?title=A%20review%20of%20servant%20leadership%20attributes%3A%20Developing%20a%20practical%20model&author=R.%20Russell&author=AG.%20Stone&journal=Leadership%20and%20Organizational%20Development%20Journal&volume=23&issue=3&pages=145-157&publication_year=2002

http://scholar.google.com/scholar_lookup?title=Servant%20leadership%3A%20A%20phenomenological%20study%20of%20practices%2C%20experiences%2C%20organizational%20effectiveness%2C%20and%20barriers&author=A.%20Savage-Austin&author=A.%20Honeycutt&journal=Journal%20of%20Business%20%26%20Economics%20Research&volume=9&issue=1&pages=49-54&publication_year=2011

http://dx.doi.org/10.1037/a0022625

http://scholar.google.com/scholar_lookup?title=Cognition-based%20and%20affect-based%20trust%20as%20mediators%20of%20leader%20behavior%20influences%20on%20team%20performance&author=J.%20Schaubroeck&author=SSK.%20Lam&author=AC.%20Peng&journal=Journal%20of%20Applied%20Psychology&volume=96&issue=4&pages=863-871&publication_year=2011

http://dx.doi.org/10.1111/j.1744-6570.1987.tb00609.x

http://scholar.google.com/scholar_lookup?title=The%20people%20make%20the%20place&author=B.%20Schneider&journal=Personnel%20Psychology&volume=40&pages=437-453&publication_year=1987

http://dx.doi.org/10.1108/01437731011079673

http://scholar.google.com/scholar_lookup?title=Servant%20leadership%20as%20antecedent%20of%20trust%20in%20organizations&author=S.%20Sendjaya&author=A.%20Pekerti&journal=Leadership%20%26%20Organization%20Development%20Journal&volume=31&issue=7&pages=643-663&publication_year=2010

5/1/2017 A Systematic Literature Review of Servant Leadership Theory in Organizational Contexts | SpringerLink

https://link.springer.com/article/10.1007/s1055101213226 40/45

Sendjaya, S., & Sarros, J. (2002). Servant leadership: Its origin, development, and
application in organizations. Journal of Leadership and Organizational Studies,
9(2), 57–64.
CrossRef (http://dx.doi.org/10.1177/107179190200900205)
Google Scholar (http://scholar.google.com/scholar_lookup?
title=Servant%20leadership%3A%20Its%20origin%2C%20development%2C%20and
%20application%20in%20organizations&author=S.%20Sendjaya&author=J.%20Sarr
os&journal=Journal%20of%20Leadership%20and%20Organizational%20Studies&vo
lume=9&issue=2&pages=5764&publication_year=2002)

Sendjaya, S., Sarros, J., & Santora, J. (2008). Defining and measuring servant
leadership behavior in organizations. Journal of Management Studies, 45(2), 402–
424.
CrossRef (http://dx.doi.org/10.1111/j.14676486.2007.00761.x)
Google Scholar (http://scholar.google.com/scholar_lookup?
title=Defining%20and%20measuring%20servant%20leadership%20behavior%20in%
20organizations&author=S.%20Sendjaya&author=J.%20Sarros&author=J.%20Santo
ra&journal=Journal%20of%20Management%20Studies&volume=45&issue=2&pages
=402424&publication_year=2008)

Senge, P. (1990). The Fifth discipline: The art and styles of the learning organization.
New York: Doubleday Business.
Google Scholar (http://scholar.google.com/scholar_lookup?
title=The%20Fifth%20discipline%3A%20The%20art%20and%20styles%20of%20the
%20learning%20organization&author=P.%20Senge&publication_year=1990)

Senge, P. M. (1995). Robert’s Greenleaf’s legacy: A new foundation for twentyfirst
century institutions. In L. Spears (Ed.), Reflections of leadership: How Robert K.
Greenleaf’s theory of servantleadership influenced today’s top management
thinkers (pp. 217–240). New York: Wiley.
Google Scholar (http://scholar.google.com/scholar_lookup?
title=Robert%E2%80%99s%20Greenleaf%E2%80%99s%20legacy%3A%20A%20new
%20foundation%20for%20twenty
first%20century%20institutions&author=PM.%20Senge&pages=217
240&publication_year=1995)

Shannon, R. (1999). Sport marketing: An examination of academic marketing
publication. Journal of Services Marketing, 13(6), 517–534.
CrossRef (http://dx.doi.org/10.1108/08876049910298775)
Google Scholar (http://scholar.google.com/scholar_lookup?
title=Sport%20marketing%3A%20An%20examination%20of%20academic%20marke
ting%20publication&author=R.%20Shannon&journal=Journal%20of%20Services%2
0Marketing&volume=13&issue=6&pages=517534&publication_year=1999)

Smith, W. (1995). Servantleadership: A pathway to the emerging territory. In L.
Spears (Ed.), Reflections of leadership: How Robert K. Greenleaf’s theory of servant
leadership influenced today’s top management thinkers (pp. 198–213). New York:
Wiley.
Google Scholar (http://scholar.google.com/scholar_lookup?title=Servant
leadership%3A%20A%20pathway%20to%20the%20emerging%20territory&author=
W.%20Smith&pages=198213&publication_year=1995)

Snodgrass, K. R. (1993). Your slaves—on account of Jesus’ servant leadership in the
New Testament. In J. R. Hawkinson & R. K. Johnston (Eds.), Servant leadership (pp.
7–19). Chicago: Covenant Publications.
Google Scholar (http://scholar.google.com/scholar_lookup?
title=Your%20slaves%E2%80%94on%20account%20of%20Jesus%E2%80%99%20se

http://dx.doi.org/10.1177/107179190200900205

http://scholar.google.com/scholar_lookup?title=Servant%20leadership%3A%20Its%20origin%2C%20development%2C%20and%20application%20in%20organizations&author=S.%20Sendjaya&author=J.%20Sarros&journal=Journal%20of%20Leadership%20and%20Organizational%20Studies&volume=9&issue=2&pages=57-64&publication_year=2002

http://dx.doi.org/10.1111/j.1467-6486.2007.00761.x

http://scholar.google.com/scholar_lookup?title=Defining%20and%20measuring%20servant%20leadership%20behavior%20in%20organizations&author=S.%20Sendjaya&author=J.%20Sarros&author=J.%20Santora&journal=Journal%20of%20Management%20Studies&volume=45&issue=2&pages=402-424&publication_year=2008

http://scholar.google.com/scholar_lookup?title=The%20Fifth%20discipline%3A%20The%20art%20and%20styles%20of%20the%20learning%20organization&author=P.%20Senge&publication_year=1990

http://scholar.google.com/scholar_lookup?title=Robert%E2%80%99s%20Greenleaf%E2%80%99s%20legacy%3A%20A%20new%20foundation%20for%20twenty-first%20century%20institutions&author=PM.%20Senge&pages=217-240&publication_year=1995

http://dx.doi.org/10.1108/08876049910298775

http://scholar.google.com/scholar_lookup?title=Sport%20marketing%3A%20An%20examination%20of%20academic%20marketing%20publication&author=R.%20Shannon&journal=Journal%20of%20Services%20Marketing&volume=13&issue=6&pages=517-534&publication_year=1999

http://scholar.google.com/scholar_lookup?title=Servant-leadership%3A%20A%20pathway%20to%20the%20emerging%20territory&author=W.%20Smith&pages=198-213&publication_year=1995

http://scholar.google.com/scholar_lookup?title=Your%20slaves%E2%80%94on%20account%20of%20Jesus%E2%80%99%20servant%20leadership%20in%20the%20New%20Testament&author=KR.%20Snodgrass&pages=7-19&publication_year=1993

5/1/2017 A Systematic Literature Review of Servant Leadership Theory in Organizational Contexts | SpringerLink

https://link.springer.com/article/10.1007/s1055101213226 41/45

rvant%20leadership%20in%20the%20New%20Testament&author=KR.%20Snodgras
s&pages=719&publication_year=1993)

Spears Center. (2011). Retrieved September 5, 2011 from
http://www.spearscenter.org/ (http://www.spearscenter.org/).

Spears, L. (1995). Introduction: Servantleadership and the Greenleaf legacy. In L.
Spears (Ed.), Reflections of leadership: How Robert K. Greenleaf’s theory of servant
leadership influenced today’s top management thinkers (pp. 1–16). New York: Wiley.
Google Scholar (http://scholar.google.com/scholar_lookup?
title=Introduction%3A%20Servant
leadership%20and%20the%20Greenleaf%20legacy&author=L.%20Spears&pages=1
16&publication_year=1995)

Spears, L. (1996). Reflections on Robert K. Greenleaf and servantleadership.
Leadership & Organization Development, 17(7), 33–35.
CrossRef (http://dx.doi.org/10.1108/01437739610148367)
Google Scholar (http://scholar.google.com/scholar_lookup?
title=Reflections%20on%20Robert%20K.%20Greenleaf%20and%20servant
leadership&author=L.%20Spears&journal=Leadership%20%26%20Organization%20
Development&volume=17&issue=7&pages=3335&publication_year=1996)

Spears, L. (1998). Insights on leadership: Service, stewardship, spirit, and servant
leadership. New York: Wiley.
Google Scholar (http://scholar.google.com/scholar_lookup?
title=Insights%20on%20leadership%3A%20Service%2C%20stewardship%2C%20spir
it%2C%20and%20servant%20leadership&author=L.%20Spears&publication_year=1
998)

Spears, L. C. (2004). Practicing servantleadership. Leader to Leader, 34, 7–11.
CrossRef (http://dx.doi.org/10.1002/ltl.94)
Google Scholar (http://scholar.google.com/scholar_lookup?
title=Practicing%20servant
leadership&author=LC.%20Spears&journal=Leader%20to%20Leader&volume=34&p
ages=711&publication_year=2004)

Spears, L. C. (2005). On character and servantleadership: Ten characteristics of
effective, caring leaders. Greenleaf Center for ServantLeadership. Retrieved July 12,
2011, from http://www.greenleaf.org/leadership/readaboutit/articles/On
CharacterandServantLeadershipTenCharacteristics.htm
(http://www.greenleaf.org/leadership/readaboutit/articles/OnCharacterand
ServantLeadershipTenCharacteristics.htm).

Stoltz, P., Udén, G., & Willman, A. (2004). Support for family careers who care for an
elderly person at home—A systematic literature review. Scandinavian Journal of
Caring Sciences, 18, 111–118.
CrossRef (http://dx.doi.org/10.1111/j.14716712.2004.00269.x)
Google Scholar (http://scholar.google.com/scholar_lookup?
title=Support%20for%20family%20careers%20who%20care%20for%20an%20elderly
%20person%20at%20home%E2%80%94A%20systematic%20literature%20review&a
uthor=P.%20Stoltz&author=G.%20Ud%C3%A9n&author=A.%20Willman&journal=
Scandinavian%20Journal%20of%20Caring%20Sciences&volume=18&pages=111
118&publication_year=2004)

Sturm, B. A. (2009). Principles of servantleadership in community health nursing:
Management issues and behaviors discovered in ethnographic research’. Home Health
Care Management & Practice, 21(2), 82–89.
CrossRef (http://dx.doi.org/10.1177/1084822308318187)

http://www.spearscenter.org/

http://scholar.google.com/scholar_lookup?title=Introduction%3A%20Servant-leadership%20and%20the%20Greenleaf%20legacy&author=L.%20Spears&pages=1-16&publication_year=1995

http://dx.doi.org/10.1108/01437739610148367

http://scholar.google.com/scholar_lookup?title=Reflections%20on%20Robert%20K.%20Greenleaf%20and%20servant-leadership&author=L.%20Spears&journal=Leadership%20%26%20Organization%20Development&volume=17&issue=7&pages=33-35&publication_year=1996

http://scholar.google.com/scholar_lookup?title=Insights%20on%20leadership%3A%20Service%2C%20stewardship%2C%20spirit%2C%20and%20servant%20leadership&author=L.%20Spears&publication_year=1998

http://dx.doi.org/10.1002/ltl.94

http://scholar.google.com/scholar_lookup?title=Practicing%20servant-leadership&author=LC.%20Spears&journal=Leader%20to%20Leader&volume=34&pages=7-11&publication_year=2004

http://www.greenleaf.org/leadership/read-about-it/articles/On-Character-and-Servant-Leadership-Ten-Characteristics.htm

http://dx.doi.org/10.1111/j.1471-6712.2004.00269.x

http://scholar.google.com/scholar_lookup?title=Support%20for%20family%20careers%20who%20care%20for%20an%20elderly%20person%20at%20home%E2%80%94A%20systematic%20literature%20review&author=P.%20Stoltz&author=G.%20Ud%C3%A9n&author=A.%20Willman&journal=Scandinavian%20Journal%20of%20Caring%20Sciences&volume=18&pages=111-118&publication_year=2004

http://dx.doi.org/10.1177/1084822308318187

5/1/2017 A Systematic Literature Review of Servant Leadership Theory in Organizational Contexts | SpringerLink

https://link.springer.com/article/10.1007/s1055101213226 42/45

Google Scholar (http://scholar.google.com/scholar_lookup?
title=Principles%20of%20servant
leadership%20in%20community%20health%20nursing%3A%20Management%20iss
ues%20and%20behaviors%20discovered%20in%20ethnographic%20research%E2%8
0%99&author=BA.%20Sturm&journal=Home%20Health%20Care%20Management
%20%26%20Practice&volume=21&issue=2&pages=8289&publication_year=2009)

Tatum, J. G. (1995). Meditations on servantleadership. In L. Spears (Ed.),
Reflections of leadership: How Robert K. Greenleaf’s theory of servantleadership
influenced today’s top management thinkers (pp. 308–312). New York: Wiley.
Google Scholar (http://scholar.google.com/scholar_lookup?
title=Meditations%20on%20servantleadership&author=JG.%20Tatum&pages=308
312&publication_year=1995)

Taylor, T., Martin, B. N., Hutchinson, S., & Jinks, M. (2007). Examination of
leadership practices of principals identified as servant leaders. International Journal
of Leadership in Education, 10(4), 401–419.
CrossRef (http://dx.doi.org/10.1080/13603120701408262)
Google Scholar (http://scholar.google.com/scholar_lookup?
title=Examination%20of%20leadership%20practices%20of%20principals%20identifi
ed%20as%20servant%20leaders&author=T.%20Taylor&author=BN.%20Martin&aut
hor=S.%20Hutchinson&author=M.%20Jinks&journal=International%20Journal%20
of%20Leadership%20in%20Education&volume=10&issue=4&pages=401
419&publication_year=2007)

Thorpe, R., Holt, R., Pittaway, L., & Macpherson, A. (2006). Knowledge within small
and medium sized firms: A systematic review of the evidence. International Journal
of Management Reviews, 7(4), 257–281.
CrossRef (http://dx.doi.org/10.1111/j.14682370.2005.00116.x)
Google Scholar (http://scholar.google.com/scholar_lookup?
title=Knowledge%20within%20small%20and%20medium%20sized%20firms%3A%2
0A%20systematic%20review%20of%20the%20evidence&author=R.%20Thorpe&auth
or=R.%20Holt&author=L.%20Pittaway&author=A.%20Macpherson&journal=Intern
ational%20Journal%20of%20Management%20Reviews&volume=7&issue=4&pages=
257281&publication_year=2006)

Tranfield, D., Denyer, D., & Smart, P. (2003). Towards a methodology for developing
evidenceinformed management knowledge by means of systematic review. British
Journal of Management, 14, 207–222.
CrossRef (http://dx.doi.org/10.1111/14678551.00375)
Google Scholar (http://scholar.google.com/scholar_lookup?
title=Towards%20a%20methodology%20for%20developing%20evidence
informed%20management%20knowledge%20by%20means%20of%20systematic%20
review&author=D.%20Tranfield&author=D.%20Denyer&author=P.%20Smart&journ
al=British%20Journal%20of%20Management&volume=14&pages=207
222&publication_year=2003)

Van Dierendonck, D. (2011). Servant leadership: A review and syntheses. Journal of
Management, 27(4), 1228–1261.
CrossRef (http://dx.doi.org/10.1177/0149206310380462)
Google Scholar (http://scholar.google.com/scholar_lookup?
title=Servant%20leadership%3A%20A%20review%20and%20syntheses&author=D.%
20Dierendonck&journal=Journal%20of%20Management&volume=27&issue=4&page
s=12281261&publication_year=2011)

Van Dierendonck, D., & Nuijte, K. (2011). The ServantLeadership Survey (SLS):
Development and validation of a multidimensional measure. Journal of Business in

http://scholar.google.com/scholar_lookup?title=Principles%20of%20servant-leadership%20in%20community%20health%20nursing%3A%20Management%20issues%20and%20behaviors%20discovered%20in%20ethnographic%20research%E2%80%99&author=BA.%20Sturm&journal=Home%20Health%20Care%20Management%20%26%20Practice&volume=21&issue=2&pages=82-89&publication_year=2009

http://scholar.google.com/scholar_lookup?title=Meditations%20on%20servant-leadership&author=JG.%20Tatum&pages=308-312&publication_year=1995

http://dx.doi.org/10.1080/13603120701408262

http://scholar.google.com/scholar_lookup?title=Examination%20of%20leadership%20practices%20of%20principals%20identified%20as%20servant%20leaders&author=T.%20Taylor&author=BN.%20Martin&author=S.%20Hutchinson&author=M.%20Jinks&journal=International%20Journal%20of%20Leadership%20in%20Education&volume=10&issue=4&pages=401-419&publication_year=2007

http://dx.doi.org/10.1111/j.1468-2370.2005.00116.x

http://scholar.google.com/scholar_lookup?title=Knowledge%20within%20small%20and%20medium%20sized%20firms%3A%20A%20systematic%20review%20of%20the%20evidence&author=R.%20Thorpe&author=R.%20Holt&author=L.%20Pittaway&author=A.%20Macpherson&journal=International%20Journal%20of%20Management%20Reviews&volume=7&issue=4&pages=257-281&publication_year=2006

http://dx.doi.org/10.1111/1467-8551.00375

http://scholar.google.com/scholar_lookup?title=Towards%20a%20methodology%20for%20developing%20evidence-informed%20management%20knowledge%20by%20means%20of%20systematic%20review&author=D.%20Tranfield&author=D.%20Denyer&author=P.%20Smart&journal=British%20Journal%20of%20Management&volume=14&pages=207-222&publication_year=2003

http://dx.doi.org/10.1177/0149206310380462

http://scholar.google.com/scholar_lookup?title=Servant%20leadership%3A%20A%20review%20and%20syntheses&author=D.%20Dierendonck&journal=Journal%20of%20Management&volume=27&issue=4&pages=1228-1261&publication_year=2011

5/1/2017 A Systematic Literature Review of Servant Leadership Theory in Organizational Contexts | SpringerLink

https://link.springer.com/article/10.1007/s1055101213226 43/45

Psychology. doi:10.1007/s1086901091941 (http://dx.doi.org/10.1007/s10869
01091941).

Van Dierendonck, D., & Patterson, K. (2011). Servant leadership, recent development
in theory and research. Dallas, TX: Presented at Greenleaf Center’s annual
international conference.
Google Scholar (http://scholar.google.com/scholar_lookup?
title=Servant%20leadership%2C%20recent%20development%20in%20theory%20and
%20research&author=D.%20Dierendonck&author=K.%20Patterson&publication_yea
r=2011)

Vanourek, R. A. (1995). Servantleadership and the future. In L. Spears (Ed.),
Reflections of leadership: How Robert K. Greenleaf’s theory of servant leadership
influenced today’s top management thinkers (pp. 298–307). New York: Wiley.
Google Scholar (http://scholar.google.com/scholar_lookup?title=Servant
leadership%20and%20the%20future&author=RA.%20Vanourek&pages=298
307&publication_year=1995)

Walumbwa, F. O., Hartnell, C. A., & Oke, A. (2010). Servant leadership, procedural
justice climate, service climate, employee attitudes, and organizational citizenship
behavior: A crosslevel investigation. Journal of Applied Psychology, 95(3), 517–529.
CrossRef (http://dx.doi.org/10.1037/a0018867)
Google Scholar (http://scholar.google.com/scholar_lookup?
title=Servant%20leadership%2C%20procedural%20justice%20climate%2C%20servic
e%20climate%2C%20employee%20attitudes%2C%20and%20organizational%20citiz
enship%20behavior%3A%20A%20cross
level%20investigation&author=FO.%20Walumbwa&author=CA.%20Hartnell&autho
r=A.%20Oke&journal=Journal%20of%20Applied%20Psychology&volume=95&issue
=3&pages=517529&publication_year=2010)

Washington, R. R., Sutton, C. D., & Feild, H. S. (2006). Individual differences in
servant leadership: The roles of values and personality. Leadership & Organization
Development Journal, 27(8), 700–716.
CrossRef (http://dx.doi.org/10.1108/01437730610709309)
Google Scholar (http://scholar.google.com/scholar_lookup?
title=Individual%20differences%20in%20servant%20leadership%3A%20The%20role
s%20of%20values%20and%20personality&author=RR.%20Washington&author=CD.
%20Sutton&author=HS.%20Feild&journal=Leadership%20%26%20Organization%2
0Development%20Journal&volume=27&issue=8&pages=700
716&publication_year=2006)

Weed, M. (2005). “Meta interpretation”: A method for interpretive synthesis of
qualitative research. Forum: Qualitative Social Research (FQS), 6(1), 1–21.
Google Scholar (http://scholar.google.com/scholar_lookup?
title=%E2%80%9CMeta%20interpretation%E2%80%9D%3A%20A%20method%20fo
r%20interpretive%20synthesis%20of%20qualitative%20research&author=M.%20Wee
d&journal=Forum%3A%20Qualitative%20Social%20Research%20%28FQS%29&volu
me=6&issue=1&pages=121&publication_year=2005)

Wheatley, M. (2005). Finding our way: Leadership in an uncertain times. San
Francisco: BerrettKoehler Publishers.
Google Scholar (http://scholar.google.com/scholar_lookup?
title=Finding%20our%20way%3A%20Leadership%20in%20an%20uncertain%20tim
es&author=M.%20Wheatley&publication_year=2005)

Whetstone, J. (2002). Personalism and moral leadership: the servant leader with a
transforming vision. Leadership and Organizational Development Journal, 25(3/4),
349–359.

http://dx.doi.org/10.1007/s10869-010-9194-1

http://scholar.google.com/scholar_lookup?title=Servant%20leadership%2C%20recent%20development%20in%20theory%20and%20research&author=D.%20Dierendonck&author=K.%20Patterson&publication_year=2011

http://scholar.google.com/scholar_lookup?title=Servant-leadership%20and%20the%20future&author=RA.%20Vanourek&pages=298-307&publication_year=1995

http://dx.doi.org/10.1037/a0018867

http://scholar.google.com/scholar_lookup?title=Servant%20leadership%2C%20procedural%20justice%20climate%2C%20service%20climate%2C%20employee%20attitudes%2C%20and%20organizational%20citizenship%20behavior%3A%20A%20cross-level%20investigation&author=FO.%20Walumbwa&author=CA.%20Hartnell&author=A.%20Oke&journal=Journal%20of%20Applied%20Psychology&volume=95&issue=3&pages=517-529&publication_year=2010

http://dx.doi.org/10.1108/01437730610709309

http://scholar.google.com/scholar_lookup?title=Individual%20differences%20in%20servant%20leadership%3A%20The%20roles%20of%20values%20and%20personality&author=RR.%20Washington&author=CD.%20Sutton&author=HS.%20Feild&journal=Leadership%20%26%20Organization%20Development%20Journal&volume=27&issue=8&pages=700-716&publication_year=2006

http://scholar.google.com/scholar_lookup?title=%E2%80%9CMeta%20interpretation%E2%80%9D%3A%20A%20method%20for%20interpretive%20synthesis%20of%20qualitative%20research&author=M.%20Weed&journal=Forum%3A%20Qualitative%20Social%20Research%20%28FQS%29&volume=6&issue=1&pages=1-21&publication_year=2005

http://scholar.google.com/scholar_lookup?title=Finding%20our%20way%3A%20Leadership%20in%20an%20uncertain%20times&author=M.%20Wheatley&publication_year=2005

5/1/2017 A Systematic Literature Review of Servant Leadership Theory in Organizational Contexts | SpringerLink

https://link.springer.com/article/10.1007/s1055101213226 44/45

Google Scholar (http://scholar.google.com/scholar_lookup?
title=Personalism%20and%20moral%20leadership%3A%20the%20servant%20leader
%20with%20a%20transforming%20vision&author=J.%20Whetstone&journal=Leade
rship%20and%20Organizational%20Development%20Journal&volume=25&issue=3
%2F4&pages=349359&publication_year=2002)

Winston, B. E. (2003). Extending Patterson’s servant leadership model: Explaining
how leaders and followers interact in a circular model. Virginia Beach, VA: Servant
Leadership Research Roundtable.
Google Scholar (http://scholar.google.com/scholar_lookup?
title=Extending%20Patterson%E2%80%99s%20servant%20leadership%20model%3
A%20Explaining%20how%20leaders%20and%20followers%20interact%20in%20a%2
0circular%20model&author=BE.%20Winston&publication_year=2003)

Winston, B. E. (2004). Servant leadership at Heritage Bible College: A singlecase
study. Leadership & Organization Development Journal, 25(7), 600–617.
CrossRef (http://dx.doi.org/10.1108/01437730410561486)
Google Scholar (http://scholar.google.com/scholar_lookup?
title=Servant%20leadership%20at%20Heritage%20Bible%20College%3A%20A%20si
ngle
case%20study&author=BE.%20Winston&journal=Leadership%20%26%20Organizat
ion%20Development%20Journal&volume=25&issue=7&pages=600
617&publication_year=2004)

Wong, P. T., & Davey, D. (2007). Best practices of servant leadership Servant
Leadership Research Roundtable. Virginia Beach, VA: Regent University.
Google Scholar (http://scholar.google.com/scholar_lookup?
title=Best%20practices%20of%20servant%20leadership%20Servant%20Leadership%
20Research%20Roundtable&author=PT.%20Wong&author=D.%20Davey&publicatio
n_year=2007)

Wong, P. T., & Page, D. (2003). Servant leadership: An opponentprocess model and
the revised servant leadership profile. Virginia Beach, VA: Servant Leadership
Research Roundtable.
Google Scholar (http://scholar.google.com/scholar_lookup?
title=Servant%20leadership%3A%20An%20opponent
process%20model%20and%20the%20revised%20servant%20leadership%20profile&a
uthor=PT.%20Wong&author=D.%20Page&publication_year=2003)

Personalised recommendations

1. Conceptualising spiritual leadership in secular organizational contexts and its
relation to transformational, servant and environmental leadership
Crossman, Joanna
Leadership & Organization Development Journal (2010)

http://scholar.google.com/scholar_lookup?title=Personalism%20and%20moral%20leadership%3A%20the%20servant%20leader%20with%20a%20transforming%20vision&author=J.%20Whetstone&journal=Leadership%20and%20Organizational%20Development%20Journal&volume=25&issue=3%2F4&pages=349-359&publication_year=2002

http://scholar.google.com/scholar_lookup?title=Extending%20Patterson%E2%80%99s%20servant%20leadership%20model%3A%20Explaining%20how%20leaders%20and%20followers%20interact%20in%20a%20circular%20model&author=BE.%20Winston&publication_year=2003

http://dx.doi.org/10.1108/01437730410561486

http://scholar.google.com/scholar_lookup?title=Servant%20leadership%20at%20Heritage%20Bible%20College%3A%20A%20single-case%20study&author=BE.%20Winston&journal=Leadership%20%26%20Organization%20Development%20Journal&volume=25&issue=7&pages=600-617&publication_year=2004

http://scholar.google.com/scholar_lookup?title=Best%20practices%20of%20servant%20leadership%20Servant%20Leadership%20Research%20Roundtable&author=PT.%20Wong&author=D.%20Davey&publication_year=2007

http://scholar.google.com/scholar_lookup?title=Servant%20leadership%3A%20An%20opponent-process%20model%20and%20the%20revised%20servant%20leadership%20profile&author=PT.%20Wong&author=D.%20Page&publication_year=2003

http://recommended.springernature.com/recommended/

https://recommendations.springernature.com/v1181/redirect/004bM9w-ENjUDU5H2pVULybxHk-rLyrGdwBVTxVCIrsooPkPPUcZrIlInW7qO3JgbVLEHMytidkACsh4x15NNW0AQaN9lf5VRCFrTwf_oGnRB0bLNLdBKEwK0SjOGxHMSwosYByDMlm3yjfudFnHfypOt_dHAte8858kqGyNzHs6GRSUsNZn587m7C1iUvBIJRCrVXiN27JKOjKEx5kBWmqi0DjvwyAu62rSs59Mn0XszUoLGytqeE7QphHIZDkmHk4iLssV-cjabRrj7jusel17GZRIEui0azRyCCXZLuwVdOczhIDkOAEmNVNkYNOQ5b-7nk5IFwDziP72B_meJi41rFFECJ7DucQmLMpUObn15rRu7PoUg9Y3M65ynGHYJsb2QtPQdJF5FP8cl24Tp8eKMhPWqLO2QbaoQno8mAlveJV8we8CSr-eO0vOTXtMUDmwnbJWsBhj6qsxTH7G_1JLQYtNuUFcF0dMV7kCkzhmAOOU4BcxQq-mnpPUVtTPltFKsRLeehvteDJHTQ100NXa4IJ6MxjAlSTlJpOxKdFYJVCyn7aorFPb6Mu3SytfYjucFMs4O3B37kfA3DQQOEPm88h-ItsLTpB4qfzL_mQJBAby8svc-3YRVpNYT4vXK0NsFzGU-yUZMclVB-hVl4X–GtwG0Zmfc4A6Tushw8LpGBN7wCrUXvW314owS3-XMPOtvEUek9IINjgTAsqvueWHmcah_YcGA2LU2hCzyhcqh4oP9kFzJo8uXnKqs3hys_Kdh12aILf8o0YK1HWwOIGIICNfXM2gElMN-78MRAhA2ry30GvRKaOqHYyqiMix08YLFbG0YKOOcncNiEPahpDwJ_opl1DtS0yM08rt9-xQAzGPzNqFqFeSCcIuw30T5Fh-88pIR6gVX-9-5rPKPCFF4ukBdUmKCAO1cvAraBiY4NyPsEu9ymkW27HTBWGcv0xQm5k0S33Gymgv-V5Aw%3D%3D

5/1/2017 A Systematic Literature Review of Servant Leadership Theory in Organizational Contexts | SpringerLink

https://link.springer.com/article/10.1007/s1055101213226 45/45

Not logged in Charles Sturt University CSU Division of Library Services (3000175634) CAUL/GO8 Consortium
(3001018290) 137.166.200.155

About this article

Publisher Name Springer Netherlands
Print ISSN 01674544
Online ISSN 15730697

About this journal
Reprints and Permissions

https://www.springernature.com/

https://www.springer.com/journal/10551/about

https://s100.copyright.com/AppDispatchServlet?publisherName=Springer&orderBeanReset=true&orderSource=SpringerLink&author=Denise+Linda+Parris&authorEmail=deniselparris%40gmail.com&issueNum=3&contentID=10.1007%2Fs10551-012-1322-6&openAccess=false&endPage=393&publicationDate=2012&startPage=377&volumeNum=113&title=A+Systematic+Literature+Review+of+Servant+Leadership+Theory+in+Organizational+Contexts&imprint=Springer+Science%2BBusiness+Media+B.V.&publication=0167-4544&authorAddress=111+Lake+Hollingsworth+Drive%2C+Lakeland%2C+FL%2C+33801-5698%2C+USA

SAMPLE_SLRs/Manikas – Software ecosystems – 2013

K
D

A
R
R
A
A

K
S
S
S

a
s
b
t
b
p

i
b
A
b
a
3
e
t
t

e
a
t
s

(

0
h

The Journal of Systems and Software 86 (2013) 1294– 1306

Contents lists available at SciVerse ScienceDirect

The Journal of Systems and Software

j ourna l ho me page: www.elsev ier .com/ locate / j ss

oftware ecosystems – A systematic literature review

onstantinos Manikas ∗, Klaus Marius Hansen
epartment of Computer Science (DIKU), University of Copenhagen, Denmark

r t i c l e i n f o

rticle history:
eceived 28 March 2012
eceived in revised form 8 December 2012
ccepted 8 December 2012
vailable online 20 December 2012

a b s t r a c t

A software ecosystem is the interaction of a set of actors on top of a common technological platform
that results in a number of software solutions or services. Arguably, software ecosystems are gaining
importance with the advent of, e.g., the Google Android, Apache, and Salesforce.com ecosystems. How-
ever, there exists no systematic overview of the research done on software ecosystems from a software
eywords:
oftware ecosystems
oftware ecosystem
ystematic literature review

engineering perspective. We performed a systematic literature review of software ecosystem research,
analyzing 90 papers on the subject taken from a gross collection of 420. Our main conclusions are that
while research on software ecosystems is increasing (a) there is little consensus on what constitutes a
software ecosystem, (b) few analytical models of software ecosystems exist, and (c) little research is done
in the context of real-world ecosystems. This work provides an overview of the field, while identifying
areas for future research.
. Introduction

It has recently been suggested that software ecosystems (SECOs)
re an effective way to construct large software systems on top of a
oftware platform by composing components developed by actors
oth internal and external (Bosch, 2009; te Molder et al., 2011). In
his setting, software engineering is spread outside the traditional
orders of software companies to a group of companies, private
ersons, or other legal entities.

This differs from traditional outsourcing techniques in that the
nitiating actor does not necessarily own the software produced
y contributing actors and does not hire the contributing actors.
ll actors, however, coexist in an interdependent way, an example
eing the iOS ecosystem in which Apple provides review of and

platform for selling applications in return for a yearly fee and
0% of revenues of application sale.1 This is a parallel to natural
cosystems where the different members of the ecosystems (e.g.,
he plants, animals, or insects) are part of a food network where
he existence of one species depends on the rest.

In addition to iOS, Google’s Android ecosystem is a prominent
xample of a (smartphone) software ecosystem. Such ecosystems

re arguably gaining importance commercially: it is, e.g., estimated
hat in 2012, more smartphones than personal computers will be
old.2

∗ Corresponding author. Tel: +45 23839917.
E-mail addresses: kmanikas@diku.dk (K. Manikas), klausmh@diku.dk

K.M. Hansen).
1 http://developer.apple.com/programs/ios/distribute.html.
2 http://www.slideshare.net/CMSummit/ms-internet-trends060710final.

164-1212/$ – see front matter © 2012 Elsevier Inc. All rights reserved.
ttp://dx.doi.org/10.1016/j.jss.2012.12.026
© 2012 Elsevier Inc. All rights reserved.

While software ecosystems are thus arguably gaining impor-
tance, research in software ecosystems is in its infancy, starting
in 2005 with Messerschmitt and Szyperski (2005) and now with a
dedicated workshop in its third year.3 Our own literature search
(see Section 3) revealed a gross list of 420 published papers
on software ecosystems. However, until now there has been no
systematic literature review (SLR) of the research literature on soft-
ware ecosystems, leading to potential issues in identifying research
gaps and contributions.

In the context of this, we have conducted a systematic litera-
ture review in the field of software ecosystems using the approach
of Kitchenham and Charters (2007). As such, the purpose of this lit-
erature review is to provide an overview of the research reported in
the field and identify possible issues that existing literature is not
addressing adequately. This work is intended to function as a snap-
shot of the research in the field by (i) identifying and analyzing the
different definitions of SECOs, (ii) analyzing the growth in research
reported per year, (iii) classifying the research by type of result, (iv)
defining and analyzing the software architecture and structure of
SECOs, and (v) analyzing to which extent research is connected to
SECO industry.

1.1. Article structure

The rest of this article is organized as following: in Section 2 we

specify the review protocol, in Section 3 we document the extrac-
tion of the literature, in Section 4 we analyze the literature and
answer the research questions, in Section 5 we list possible threats

3 http://www.softwareecosystems.org/workshop/.

dx.doi.org/10.1016/j.jss.2012.12.026

http://www.sciencedirect.com/science/journal/01641212

http://www.elsevier.com/locate/jss

mailto:kmanikas@diku.dk

mailto:klausmh@diku.dk

http://developer.apple.com/programs/ios/distribute.html

Internet Trends 2010 by Morgan Stanley Research from FM Signal

http://www.softwareecosystems.org/workshop/

dx.doi.org/10.1016/j.jss.2012.12.026

f Syste

t
l

K
p
a
i
w
r
d
t
l

a
o

K. Manikas, K.M. Hansen / The Journal o

o the validity of this work and identify areas not covered from the
iterature and in Section 6 we conclude.

. Review protocol

The applied review protocol is based on the guidelines of
itchenham and Charters (2007). The establishment of the review
rotocol is necessary to ensure that the literature review is system-
tic and to minimize researcher bias. As such, the literature review
s focused on a set of research questions that serve the aim of this

ork and derive from the reasons that initiated this review. The
eview protocol is organized in a way that the research questions
efine the main areas this study is focusing on. Section 2.2 defines
he paper literature extraction strategy including the list of resource
ibraries, the search query and inclusion/exclusion criteria.

.1. Research questions

The purpose of this systematic literature review is to provide
n overview of the research reported in the field of SECO. In this
verview, we intent to address the following research questions:

Q 1: How is the term ‘software ecosystem’ defined?
In order to be able to analyze the field of SECOs, we should

first define the SECO as object of study. Thus, the first objec-
tive of this work is to provide an overview of how the
research community defines the term ‘software ecosystem’.
We achieve that by looking into the SECO definitions in the
literature and comparing them. This will create an under-
standing of what the research community means by the term
SECO.

Q 2: What is the research output per year in the SECO field?
By grouping the literature per publication year we are able

to identify possible trends in the research invested in the field
of SECOs. An increase in the number of publications per year,
for example, would imply the increase in importance of the
field while a decrease in the number of publications might
have as a possible reason the research in the field reaching
a dead end. Analyzing the trends might give an idea of how
the importance of the field of SECOs is changing with time.

Q 3: What is the type of result that software ecosystem research
reports?

After having defined the term SECO, a question that we
want to address is what kind of research this field reports.
Therefore, it is of interest to classify the papers according
to the contribution they make. From a software engineering
perspective, Shaw’s classification of research results (Shaw,
2003) has been chosen. The classification contains the fol-
lowing categories:

Procedure or technique: This category includes papers that
are providing a concrete and implementable way to solve a
SECO problem. The solutions should be in the form of a proce-
dure or technique that can be applied and not general rules of
thumb or reported experiences. For example, Kazman et al.
(2012) analyze a series of traditional software design and
software architecture principles and methods in the perspec-
tive of the SECOs (or software-intensive ecosystems as they
are called in the paper). This results in some new or adapted
methods for the software design and architecture of these
software-intensive ecosystems.
Qualitative or descriptive model: Papers using models

based on qualitative analysis of data or well argumentation
of existing cases. Papers in this category provide an analyti-
cal or descriptive model for the problem area. As an example
the analysis of two different kinds of SECO: the “as-a-service”
ms and Software 86 (2013) 1294– 1306 1295

and “on-premise” software ecosystems that derived from a
comparative study of two existing SECOs presented in Hilkert
et al. (2010).
Empirical model: This category includes papers that use
models derived from the quantitative data collection of the
problem area. A paper of this category studies empirical data
and concludes some analysis or predicting model. For exam-
ple, Yu et al. (2008) extract information from open source
systems to assess the evolvability of software.
Analytic model: Papers using models based on automatic
or mathematical manipulation for solving a specific prob-
lem. For example the paper of Capuruç o and Capretz (2010)
that propose a prediction of recommendations and interac-
tion between the members of a social ecosystem based on a
mathematical analysis of the member relationships.
Tool or notation: A tool or notation created or implemented
applying some method or technique. For example, a tool
for recovering components and their relationships in free or
open source projects, proposed by Lungu (2008)
Specific solution, prototype, answer, or judgment: Papers
documenting a complete solution, evaluation of a theory or
comparison of different theories based on a software engi-
neering problem. The result is addressing a specific problem.
An example would be Pettersson and Gil (2010) who address
reusability and adaptability issues in mobile learning sys-
tems
Report: Papers documenting knowledge and experience
obtained, rules of thumb or checklists but not systematic
enough to be a descriptive model. For example, the analy-
sis of the hybrid business and revenue models that software
companies can have (Popp, 2011).

RQ 4: What is the role of architecture in software ecosystem research?
For single systems, software architecture is seen as impor-

tant in determining the quality of a system being built (Bass
et al., 2003; Hansen et al., 2011). In relation to this, we
analyze the extent to which SECO literature stresses soft-
ware architecture. We evaluate the literature in whether it
is documenting any considerations towards SECO software
architecture. In doing so, our concept of software architec-
ture is in line with Bass et al. (2003):

“The software architecture of a program or computing
system is the structure or structures of the system, which
comprise software elements, the externally visible prop-
erties of those elements, and the relationships among
them.”

We here extend the definition to concern software ecosys-
tems, i.e., we define ‘software ecosystem architecture’ as the
structure or structures of the software ecosystem in terms
of elements, the properties of these elements, and the rela-
tionships among these elements. The SECO elements can be
systems, system components, and actors. Relationships then
include software architecture-related relationships as well
as actor-related relationships such the relationship between
two actors.

RQ 5: How is the connection between research and industry in the
area of software ecosystems?

It is of interest to know how close industry and research
are in the field of software ecosystems. Research benefits
from realism of problems when connected to the industry
while industry arguably may become more innovative and

efficient when connected to research. In the case of SECOs
research results are more valid when they are concerning
existing SECOs, while studies of problems in existing SECO
can help the industry improve.

1 f Syste

k
o

1
2
3
4

s
e
a
c
t
f
d
w
s
d
t
s
o
T
e

•

needed to address the research questions. The information extrac-
tion is in the form of descriptive text enclosed by identifying labels
296 K. Manikas, K.M. Hansen / The Journal o

We investigate how connected the research world is with
the industry by examining how much of the literature has
focused on real-world SECOs. We accept that a paper has
focus on a real-world SECO when it either presents an exist-
ing SECO as an object of study or uses the data from the study
of one to support a claim or result. For example, this could be
a paper that is deducting information of the external actors
of an ecosystem by studying the relationships between the
actors of one or more existing SECOs. However, we do not
include papers that merely mention a SECO, e.g., in order to
support their definition of SECOs, and that thus present no
study of the SECO.

.2. Defining the literature body

The strategy for collecting the relevant literature is twofold: (i) a
eyword search in a list of scientific libraries and (ii) the collection
f the papers from the SECO workshop series.

With respect to (i), the scientific libraries included in the search
re:

. The ACM Digital Library4

. IEEE Explore5

. Springer Verlags’ digital library, SpingerLink6

. ScienceDirect.7 An online collection of published scientific
research operated by the publisher Elsevier.

. Thomson Reuters’ Web of Science.8 An online academic citation
index.

The literature extraction consists of two separate keyword
earches with the search terms “software ecosystem” and “software
cosystems” in the libraries above. The search query is intention-
lly kept simple so we can extract the maximum number of papers
ontaining the terms. We specifically define SECO(s) as the keyword
o underline the differentiation of the field of software ecosystem
rom different ecosystems like business, digital or social. The bor-
ers of the SECO field can be sometimes vaguely defined especially
hen overlapping with other kinds of ecosystems. For example

ome SECOs in the literature can also fit in a digital ecosystem
efinition and there are several studies on business ecosystems
hat produce software. The purpose of this work, however, is to
tudy software ecosystems and any possible intersections with
ther ecosystems should be studied from the SECO point of view.
herefore, this study does not include studies on other kinds of
cosystems.

With respect to (ii), we include the papers from the International
orkshops on Software Ecosystems (IWSECO).
The selected literature body collected from both (i) and (ii)

hould commit to a set of inclusion criteria:

The literature should address software ecosystems as an area
of research, either main or secondary. Therefore, the keywords
“software ecosystem(s)” should exist as a whole and continuously
in at least one of the fields: title, keywords or abstract. Addition-
ally, possible composites of the keywords should be examined,

e.g., software-intensive ecosystems.
Be research papers, i.e., being published in a scientific peer
reviewed venue.
Be written in English.

4 http://dl.acm.org/.
5 http://ieeexplore.ieee.org.
6 http://www.springerlink.com/.
7 http://www.sciencedirect.com/.
8 http://apps.webofknowledge.com/.
ms and Software 86 (2013) 1294– 1306

• Have a document body that is more than one page long.

Consequently, the literature does not contain books, extended
abstracts, presentations, presentation notes, keynotes or papers
written in other language than english.

The literature body is the results of the following steps:

1. Collecting all the literature. The literature collection is the com-
bination of the scientific library search and the IWSECO papers.
The library search, at this point includes a search of the keywords
in the whole text body in order to include the maximum amount
of papers.

2. Applying inclusion/exclusion criteria. The literature collection
resulting from the previous step are searched for the keywords
in the fields title, abstract, keywords.

3. Verifying rejected papers. The rejected literature from the pre-
vious step is searched for only the terms “ecosystem(s)” and
“software” in the fields title, abstract, keywords and evalu-
ated if they are related literature. This would avoid rejecting
papers with different combinations of the keywords, for example
“software-intensive ecosystems”.

4. Verifying included papers. The included literature that resulted
from the two previous steps is verified manually by reading the
abstract and conclusion. In this step, we make sure that the
papers included in the review provide results that are directly
or indirectly related to the field of SECO.

3. Collecting the literature body

To obtain the literature body of our review, we apply the sys-
tematic literature review (SLR) protocol described in Section 2 with
the extraction date of June 11, 2012. The four steps for defining
the literature body described in Section 2.2 can be seen in Table 1.
The literature collection starts with 420 papers extracted from the
libraries. All the IWSECO papers are included in this collection. After
applying the inclusion/exclusion criteria, we reject 297 paper. Out
of the 297 rejected, we apply step 3 and included six papers with key
words ”open ecosystems”, ”software-intensive ecosystems”, ”ERP
ecosystems”, ”information ecosystem”, ”source code ecosystems”,
”Eclipse ecosystem”. In step 4 we went through 129 papers (123
from step 2 plus 6 from step 3) and find 90 papers relevant. We
contribute the high number of rejected papers in step 2 to two
reasons: (i) some libraries would search in the whole paper text
body and thus retrieve papers mentioning SECO but not reporting
research on that field and (ii) Science Direct does not recognize the
quotation marks in “software ecosystem” or “software ecosystems”
so it retrieves results that the words are not adjacent to each other
but in different locations in the texts, therefore there were many
papers not related to software engineering. We also note that from
the six papers selected in step 3, only one (Kazman et al., 2012) is
part of the included papers.

During the data extraction process, we read the papers found
relevant and extracted interesting information and information
for automated sorting. In continuation, a set of custom scripts
export the requested information.

Table 1
The steps and included papers to define the literature body.

Step Nr of papers

1. Collecting the literature 420
2. Applying inclusion/exclusion criteria 123
3. Verifying rejected papers (included) 6
4. Verifying included papers 90

http://dl.acm.org/

http://ieeexplore.ieee.org

http://www.springerlink.com/

http://www.sciencedirect.com/

http://apps.webofknowledge.com/

f Syste

r
i

g
a
t
f
n
b
p
o
(
h
m
d
i
i
d

o
p
w
u
a
d

T
T

K. Manikas, K.M. Hansen / The Journal o

. Analysis

In this section we analyze the literature and the results of the
eview. The section is organized according to the research questions
n Section 2.1.

.1. Defining SECO

During this literature review, we obtained an overview of the
eneral field referred to as software ecosystems. One of our initial
ims was to define the term SECO by summarizing the definitions in
he literature. Looking into the literature, our first remark is that we
ound a large number of papers (40 out of the total of 90) that did
ot define the term SECO. This is, either because the authors are
asing their work on previous research (own or not) that would
rovide the background and definition or because the main focus
f the paper is not in the general field of SECO. For example, Bosch
2010a) is not providing any definition, but he is referring back to
is own work (Bosch, 2009) where he provides a definition and
ore detailed analysis of the field. On the other hand, Popp (2011)

efines the business and revenue models for SECOs. In his paper, he
s providing definitions for the business and revenue models that
s the main focus, instead of a definition of a SECO. This, however,
oes not make it of less value to the research field of SECOs.

Taking the papers that provide a definition, we notice that few
f them are defining the SECO with their own words. Two of these
apers are also citing more definitions from the literature along
ith their own. The rest of the papers, are defining the field by
sing one or more definitions from the existing literature. When we
nalyzed the definitions, we found that we can group the quoted
efinitions in four groups according to the source of the definition:

Messerschmitt and Szyperski (2005) is the oldest definition of
SECO in the found literature referring to the book on SECO pub-
lished in 2005.

“Traditionally, a software ecosystem refers to a collection of
software products that have some given degree of symbiotic
relationships.” (Messerschmitt and Szyperski, 2005)

Jansen et al. (2009b) mainly refer to the following definition:

“We define a software ecosystem as a set of businesses func-
tioning as a unit and interacting with a shared market for
software and services, together with the relationships among
them. These relationships are frequently under-pinned by
a common technological platform or market and operate
through the exchange of information, resources and artifacts.”
(Jansen et al., 2009b)

Bosch (2009) and Bosch and Bosch-Sijtsema (2010b,c) provide two
definitions in their papers. The papers quoting his definitions are

taking one of the following:

“ A software ecosystem consists of the set of software solu-
tions that enable, support and automate the activities and
transactions by the actors in the associated social or business

able 2
he papers belonging to each group of SECO definition.

Definition Papers

Not available [19, 1, 45, 39, 43, 2, 5, 6, 9, 11, 48, 49, 59
26, 55, 32, 24, 63, 82, 74, 75, 69, 62, 64,

Jansen et al. [3, 4, 10, 16, 13, 28, 37, 44, 14, 29, 12, 6
Bosch et al. [40, 41, 10, 13, 20, 23, 44, 14, 17, 12, 89
Own [38, 8, 30, 58, 56, 47, 12, 34, 73]
Lungu et al. [7, 15, 18, 80, 68, 81]
Messerschmitt et al. [40, 50, 37, 57, 85]
ms and Software 86 (2013) 1294– 1306 1297

ecosystem and the organizations that provide these solutions.”
(Bosch, 2009)
“A software ecosystem consists of a software platform, a set of
internal and external developers and a community of domain
experts in service to a community of users that compose rel-
evant solution elements to satisfy their needs.” (Bosch and
Bosch-Sijtsema, 2010b,c)

Lungu et al. (2010a) are presenting a different definition of the
SECOs that is adopted by a number of papers:

“A software ecosystem is a collection of software projects
which are developed and evolve together in the same envi-
ronment.” (Lungu et al., 2010a)

In Table 2 we show the different groupings and the papers
belonging to each group. The in the column Papers refer to the
literature body listed in Appendix A.

Not surprisingly, if we look at the definitions we can see that
they have two things in common: they concern software in some
form (software systems, products, services, or a software platform)
and they are all including some kind of relationships either “symbi-
otic”, “common evolution”, “business” or “technical”. If we look at
what perspective the authors have in the definitions, we note that
Messerschmitt and Lungu et al. have a pure technical perspective by
talking about software and its symbiosis/co-existence, while Bosch
et al. and Jansen et al. include, apart from the technical, a social
and business perspective to their definition and the symbiosis is
not only on the technical level. Taking the two wider-perspective
definitions of Bosch et al. and Jansen, which are referenced by the
majority of the papers that provide a definition for SECO (65%), we
can identify three main elements in their definitions:

Common Software The software appears either as a “common
technological platform” (Jansen et al., 2009b), “software
solutions” (Bosch, 2009) or “software platform” (Bosch
and Bosch-Sijtsema, 2010b,c)

Business This is expressed as either “a set of business” (Jansen
et al., 2009b), “business ecosystem” (Bosch, 2009), a
“community of users that have needs to be satisfied”
(Bosch and Bosch-Sijtsema, 2010b,c). In this element, the
term “Business” is implying a wider sense than the profit
or revenue models. This element also includes possible
benefits other than financial revenues, e.g., the benefits
an actor would get from the involvement in an free or
open source project.

Connecting Relationships “a set of businesses (. . .) together with
the relationships among them ” (Jansen et al., 2009b),
“actors in the associated social ecosystem” (Bosch, 2009),
“community of domain experts” and “community of
users” (Bosch and Bosch-Sijtsema, 2010b,c)
Combining the definitions above with the three elements iden-
tified, we define a software ecosystem as the interaction of a set of
actors on top of a common technological platform that results in a
number of software solutions or services. Each actor is motivated

Total

, 52, 51, 54, 42, 53, 36, 46, 31, 22, 21, 35, 33,
70, 78, 67, 83]

, 27, 87, 86, 72, 76, 71, 61, 65, 66, 60, 90, 84] 24
, 77, 79] 13

9
6
5

1298 K. Manikas, K.M. Hansen / The Journal of Systems and Software 86 (2013) 1294– 1306

Table 3
Papers published per year.

Year Papers Total

2007 [31, 46, 57] 3
2008 [50, 53, 54] 3
2009 [9, 10, 11, 20, 6, 42, 51, 52, 56, 58] 10
2010 [1, 12, 13, 14, 15, 16, 17, 18, 19, 21, 22, 23,24, 27, 28, 29, 30, 33,34, 35, 36, 37, 38, 39,41, 43, 45, 47, 48, 49,55, 59] 32
2011 [2, 3, 4, 5, 6, 7, 8, 26,32, 40, 44, 87, 89, 88,73, 72, 74, 68, 71, 69,64, 61, 65, 70, 79, 66,60, 67, 83, 85, 90, 84] 32

b
t
s
a
o
fi
m
I
e
A
p
a
a
m
t
a
o
r
m
(
u
(

i
t
S
d
w
c
e
t
h
f
a
v
m
i
o
r
c
D
i

T
T

2012 [63, 80, 86, 82, 76, 77,75, 62, 78, 81]

y a set of interests or business models and connected to the rest of
he actors and the ecosystem as a whole with symbiotic relation-
hips, while, the technological platform is structured in a way that
llows the involvement and contribution of the different actors. In
ther words, the SECO provides possibilities for the actors to bene-
t from their participation in the ecosystem. The types of benefits
ight vary depending on the actor and the nature of the ecosystem.

n a commercial ecosystem the actors might gain direct revenues,
.g., developers making apps for iPhone and selling them in the
pp Store, while in a non-commercial ecosystem the actors might
articipate for non-monetary benefits (fame, knowledge, ideology
nd so on), e.g., the developers contributing to Apache. Addition-
lly, the actors’ relationships to the ecosystem as a whole are of
utual interest (mutualism): the actors’ benefits increase by the

hriving of the ecosystem and the ecosystem benefits by increased
ctor activity. The relationships among the actos in a SECO, on the
ther hand, are characterized by the wider spectrum of symbiotic
elationships. Depending on the actors and their activity, two actors
ight have mutual benefits (mutualism), be in direct competition

competition/antagonism), be unaffected (neutralism) or one being
naffected while the other is benefiting (amensalism) or harmed
parasitism) by their relationship.

When looking at the rest of the papers, we note that there
s a number of papers that assist in the conceptualization of
he field in a wider sense than just providing the definition of
ECOs. These papers are used as a conceptual base of succee-
ing work. In this concept, Bosch (2009) proposes a taxonomy
here he divides SECOs in three categories: operating system-

entric, application-centric and end-user programming software
cosystems. In continuation he discusses the steps needed for
he transition to a SECO and implications this transition might
ave. Jansen et al. (2009a), apart from providing the definition

or SECO seen above, propose three scopes to study SECOs that
re also explained briefly in Boucharas et al. (2009): an external
iew on ecosystems that studies the SECOs themselves and the
arkets around them, an internal view of a SECO that is focus-

ng on software supply networks and their relationships, and an
rganization-centric perspective that studies the actors and their

elationships. Campbell and Ahmed (2010) propose a view of SECO
onsisted of three dimensions: business, architectural and social.
hungana et al. (2010) make a comparison of the SECO with biolog-

cal ecosystems from the perspective of resource management and

able 4
he papers grouped according to the result groups.

Result Papers

Report [19, 43, 2, 6, 10, 8, 59, 52, 56, 51, 36
73,86, 82, 76, 77, 71, 75,64, 61, 65,

Tool or notation [9, 48, 58, 54, 15, 22,47, 6, 80, 68, 6
Procedure or technique [40, 41, 5, 11, 30, 53, 16, 13, 28, 18,
Qualitative or descriptive model [38, 39, 3, 4, 7, 21, 37, 29, 34, 74]
Empirical model [45, 50, 44, 57, 55, 27, 89]
Analytic model [1, 42, 63, 72, 90]
Specific solution [49, 23, 33, 88]
10

biodiversity and underline the importance of diversity, monitoring
of health and supporting social interaction for the field of SECO.dos
Santos and Werner (2011b) collect the concepts appearing in the
papers from IWSECO 2009 to 2010 and organize them in three
views: SECO architecture, SECO strategies and tactics and SECO
social networks. Finally, Barbosa and Alves (2011) conduct a sys-
tematic mapping in the field of SECO and categorize the research
in eight fields unfolded around open source software, ecosystem
modeling, and business issues.

4.2. Yearly activity

Another point of study in this work, is the analysis of the year
of publication. We order the papers according to their publication
year as can be seen in Table 3. The literature on SECO starts in 2007
(although (Messerschmitt and Szyperski, 2005) dates back to 2005,
it was excluded from this study for being a book and not a research
paper). The first two years – 2007 and 2008 – provide an equally
low number of papers. However, an increase appears in 2009 and
continues to 2010 with 2010 and 2011 having the same amount of
papers.

The increase of papers gives us a clear sign that the field of SECO
is gaining in importance among the published research. This is
also underlined with the establishment of a workshop dedicated
to SECOs, the International Workshop on Software Ecosystems
(IWSECO), in 2009. While this does not give insight into software
ecosystems in themselves, it stresses the potential significance of
the concept.

4.3. Research results

As noted in research question in Section 2.1, it is of interest to
examine what kind of results the papers are reporting. We have
classified the papers in the categories listed in research question
in Section 2.1 and can be seen in Table 4. As it can be seen from
the table, the majority of the papers fall under the Report cate-
gory. This means that these papers have as contribution knowledge
and experience obtained, rules of thumb or checklists or interest-

ing observations but they are not systematic enough, nor generic
enough to be applied to different domains or too abstract to provide
a concrete contribution. An example of a paper falling under this
category is the paper by Dhungana et al. (2010) that compares

% of total

, 46, 31,20, 35, 12, 26, 32, 24,14, 29, 33, 17, 87,
70, 78, 66,67, 83, 85, 84]

9, 62, 79,81] 15
14,17] 13

11
8
5
4

K. Manikas, K.M. Hansen / The Journal of Systems and Software 86 (2013) 1294– 1306 1299

Table 5
The papers according to the SECO Architecture groups.

SECO architecture group Papers % of total

SECO SE [40, 19, 45, 38, 39, 43, 11, 49, 58, 36, 46, 15, 22, 47, 35, 29, 17, 57, 87, 63, 68, 69, 70, 79, 66, 81] 35
SECO business and management [2, 4, 5, 10, 58, 56, 16, 20, 23, 37, 44, 14, 33, 12, 6, 27, 87, 88, 86, 76, 77, 71, 75, 62, 66, 60, 67, 83, 90] 39

, 86, 8

S
r
o
p

m
e
d
i
n
a
n
s
p

c
S
2
s
l
i
i
h
w
p
n
h
fi
t
P

p
r
t
t
f
d
g

s
n
b
t
t
p
f

T
t
f
a

SECO relationships [3, 4, 6, 30, 13, 31, 21, 33, 17, 87, 73

ECOs to the natural ecosystem and reports observations and a
esearch agenda. This paper does not report any concrete method
f some kind and the data used is not systematic enough for the
aper to be included in the qualitative model.

Looking at the distribution, we note that the category with the
ost papers after Report is Tool or Notation. The papers of this cat-

gory are implementing tools or notations that are mostly using
ata from FOSS SECOs. This, as we will discuss more in Section 4.5,

s related to the fact that FOSS SECOs provide access to a lot of tech-
ical data, e.g., commit history or bug reports that are not easy to
ccess in proprietary SECOs. The third category, Procedure or Tech-
ique, includes papers that report an implementable technique to
olve a specific task. For example the paper by Fricker (2009), that
roposes a technique for requirement management in SECOs.

When examining the percentage of papers that fall under each
ategory, we can make the following observations. The field of
ECOs is a new research field, with the first papers appearing in
007. This implies that there is an amount of research resources
pent in defining the field and its limits, for example the papers ana-
yzed in Section 4.1. In addition, as it is shown in Section 4.5, there
s a relatively small amount of research spent in examining SECOs
n the industry. These two reasons result in the Report category
aving a bigger percentage to all the other categories. Additionally,
e recognize that the field of SECO is wide and can have multi-
le research perspectives, such as software engineering (SE), social
etworks or technical management. In connections to this, there
ave been several papers focusing on some specific aspect of the
eld providing specific and implementable techniques. This poten-
ially explains the high percentage in the Tool or Notation and
rocedure or Technique categories.

.4. SECO architecture

To address RQ in Section 2.1, we separated and analyzed the
apers that are addressing the SECO architecture as defined in the
esearch question. During the analysis of the papers, we could iden-
ify three logical groups of SECO architecture papers. Table 5 shows
he distribution of the papers according to their main research
ocus. Below we elaborate on the three SECO architectural groups
escribing them in more detail. Papers used in the description of a
roup might not represent their main research focus.

.4.1. SECO software engineering
Software ecosystems, having as a product one or several

oftware systems have problems that belongs to the software engi-
eering field. A part of the SECO literature is focusing on SE either
y using SE practices directly or by adapting existing SE practices to
he SECO context. This category consists of papers focusing on more
echnical issues related directly or indirectly to the technological
latform of a SECO. It contains 26 papers, i.e., 35% of the literature
ocusing on SECO architecture aspects.

One important aspect of this category is software architecture.

he software architecture of a SECO should support the nature of
he ecosystem (i.e., be adapted to the needs of the specific SECO),
ollow the SECO management, business rules and restrictions and
llow the integration and existence of multiple functionality in
2, 72, 76, 61, 65, 85, 84] 26

a secure and reliable manner. A modular and flexible architec-
ture would allow integration and interoperability of the developed
software (Viljainen and Kauppinen, 2011; Bosch, 2009). Interfaces
allow external development on a SECO platform. The stability
and translucency of the platform interfaces are essential for the
component integration and interaction (Cataldo and Herbsleb,
2010; Bosch, 2010a). Changes to existing interfaces or components
might create inconsistencies to dependent components (Robbes
and Lungu, 2011; Lungu et al., 2010a,b). Process-centric approaches
are not effective in managing large scale software, instead system
architecture should be used as a coordination mechanism (Bosch
and Bosch-Sijtsema, 2010a). Constantly evolving software requires
the adaptation of the software development processes. Develop-
ment should be integration-centric, independent deployment and
releases should be organized in a release grouping and release train
fashion (Bosch and Bosch-Sijtsema, 2010b; Bosch, 2010a). Architec-
tural design and analysis techniques are based on a set of principles
as identifying business goals, describing architectural significan
requirement, tactics and architectural evaluation. These principles
are used in defining the software architecture of a SECO (Kazman
et al., 2012).

Apart from software architecture, in the wider SE related sub-
jects, requirement elicitation appears as an interesting challenge
in the SECO concept as the stakeholders are multiple and distant
from the central ecosystem management. The use of “requirement
value chain” is proposed to propagate requirements (Fricker, 2009,
2010).

4.4.2. SECO business and management
This category contains papers focusing on the business, organi-

zational and management aspects of SECOs. Independently of how
each SECO is organized, there is an organizational and management
entity that is responsible for monitoring, operational and decision
making part of the SECO whether it being a proprietary company,
an open source community or a hybrid of the two. This category
is sub-divided into two groups: organizational & management and
business.

The organizational and management group includes papers that
are focusing on the organizational actions in a SECO. These actions
are initiated from decisions, rules and processes or controlling
mechanisms. The main activities of this group are summarized in:
monitoring the SECO, evaluating and decision making, and taking
actions.

In order to ensure that a SECO is functioning well, specific mea-
surements need to be introduced that would provide an overview
of the state of the SECO while at the same time raise attention
for actions and allow comparison of SECOs. The literature is refer-
ring to the concept of the health of a software ecosystem (van
Ingen et al., 2011; van Angeren et al., 2011; van den Berk et al.,
2010; dos Santos and Werner, 2011a,b; Kilamo et al., 2012; Jansen
et al., 2012, 2009a; Viljainen and Kauppinen, 2011; Mizushima and
Ikawa, 2011; McGregor, 2010; Dhungana et al., 2010; Boucharas

et al., 2009). This concept has been introduced by Iansiti et al. as a
way to measure the performance of a business ecosystem (BECO).
In more detail they measure the “extent to which an ecosystem as
a whole is durably growing opportunities for its members and those

1 f Syste

w
b
t
(
t
e
A
2
M
W
h
o
K
o
o
s
d
s

s
S
p
a
t
a
t
m
e
m
b
h
a
t

n
a
o
n
i
2
t
i
r
a
2
t
B
e
b
m
(
2
2
r
i
o
m
r

e
e
i
t
t
u
d

300 K. Manikas, K.M. Hansen / The Journal o

ho depend on it” (Iansiti and Levien, 2004a) and inspired from
iological ecosystems define the health of a (business) ecosys-
em as an analogy to robustness, productivity and niche creation
Iansiti and Levien, 2004b,a). These studies, although excluded from
he collected literature, are referenced by the majority of the lit-
rature elaborating on SECO health (van Ingen et al., 2011; van
ngeren et al., 2011; van den Berk et al., 2010; Kilamo et al.,
012; Jansen et al., 2012, 2009a; Viljainen and Kauppinen, 2011;
izushima and Ikawa, 2011; McGregor, 2010; dos Santos and
erner, 2011b; Boucharas et al., 2009). An additional study on the

ealth of business ecosystems that is referenced by several papers
f the literature (van Angeren et al., 2011; van den Berk et al., 2010;
ilamo et al., 2012; Jansen et al., 2012, 2009a) that are elaborating
n SECO health, is the paper of den Hartigh et al. (2006) that, based
n the Iasiti et al studies mentioned above, applies health mea-
urement to Dutch IT business ecosystems. In the SECO field, van
en Berk et al. (2010) base their work on BECO health to create a
trategy assessment model.

The proper evaluation of SECO measurements, such as health,
upports and encourages correcting or improving actions in the
ECO. This requires a management entity that would have the
ower and possibility to apply changes both in the technical but
lso in the organizational aspects of the SECO. To our knowledge
here is no study in the SECO literature on the different man-
gement entities and the decision making mechanisms applied
o drive the SECO. This might be because of high variability in

anagement models or the disclosure of information in propri-
tary SECOs. It would be possible to study the decision making
echanisms of a FOSS project where changes are applied, e.g.,

ased on online member voting, but it is challenging to study
ow a proprietary SECO canalizes information from the peripheral
ctors, evaluates this information and decides on actions based on
hat.

After monitoring the SECO and concluding in a set of decisions, a
ext step is to execute these decisions. One of the ways of applying
ctions that appears in the literature is communication. A clear view
n the direction that the ecosystem would evolve and the commu-
ication of this view to the ecosystem actors and involved parties

s underlined as a necessity (Bosch, 2009; Viljainen and Kauppinen,
011). Creating roadmaps, visions or long-term strategic planning of
he ecosystem allows the actors to plan, in their turn, their activity
n the ecosystem and align their business models with the SECO
oadmaps (Kakola, 2010; Bosch, 2009; Hanssen, 2011; Viljainen
nd Kauppinen, 2011; Jansen et al., 2012; van den Berk et al.,
010). At the same time the ecosystem can set the requirement of
he SECO actors to commit to the published roadmaps (Bosch and
osch-Sijtsema, 2010b,c). From a more practical perspective, the
cosystem orchestrators can organize the component composition
y providing a long-term plan of organized releases in a release
anagement or release trains that the actors can coordinate with

Bosch and Bosch-Sijtsema, 2010b,c; Fricker, 2010; Jansen et al.,
012; van den Berk et al., 2010; van der Schuur et al., 2011; Bosch,
009). Bosch and Bosch-Sijtsema (2010c) analyzed the concept of
elease grouping where different groups of components are released
n different times allowing less coordination and communication
verhead. Kilamo et al. (2012) introduce the release readiness assess-
ent where proprietary software is assessed on its ability to be

eleased as open source/ open ecosystem.
An important part of the SECO business and management cat-

gory is related to the business perspective of the ecosystem. As
xplained in the definition analysis, the business perspective is
mportant as without a solid business and business model serving

he SECO and its actors, the SECO might lose its actors to competi-
ive businesses or ecosystems and risk extinction. It is essential to
nderline that the business and business model as mentioned here
o not necessarily imply monetary benefits. The business model
ms and Software 86 (2013) 1294– 1306

that would serve the SECO actors, as mentioned in the definition,
might imply value in other forms, for example fame or experience
in the case of a FOSS SECO actor. The same applies to the SECO itself.
A SECO might include other benefits than revenues in its business
model. An example would be advantage over competitors or “vis-
ibility within the market” (van Angeren et al., 2011). This implies
that the traditional software company business models where the
revenues are a result of software license selling cannot be fully
applied in the ecosystem concept. Popp (2011) provides an anal-
ysis of business models that are applied in three ecosystems and
makes a separation between the business models and the revenue
models of a SECO. He underlines the importance of revenue mod-
els and states that “revenue models (. . .) often containing one or more
non-monetary compensations, can be a source of competitive advan-
tage” (Popp, 2011). Burkard et al. (2012) refer to revenue models
from two perspectives: actors or niche players provide their prod-
ucts for a fee and the SECO orchestrator or hub requires a fee from
the actors. This fee can be base either on fixed or variable price
models.

Although, selling software licenses might not be a main rev-
enue venue for a SECO, the issue of software licenses is still of
interest in the SECOs. SECOs collect code developed by different
developers or companies with different policies and many times
even in an combination of proprietary and open source. Addressing
or avoiding possible intellectual property right (IPR) or licens-
ing violations would ease the software integration, allow possible
reuse that might lead to more niche creation, clarify possible busi-
ness models and avoid legal complications that demand heavy
resources. Licensing and IPR issues appear in a number of papers
(Alspaugh et al., 2009; Jansen et al., 2012, 2009b; Mizushima and
Ikawa, 2011; te Molder et al., 2011; Kilamo et al., 2012; Scacchi
and Alspaugh, 2012) in the literature. In relation to this, Alspaugh
et al. (2009) and Scacchi and Alspaugh (2012) discuss the issue of
software licensing in open architecture systems, recognize changes
in licenses on different versions of the same component or in the
evolution of a software system and propose a structure for mod-
eling software licenses. Mizushima and Ikawa (2011) analyze the
IP management process of Eclipse called the “Eclipse Legal Pro-
cess” and state that this process was a reason for vendors to join
Eclipse. Anvaari and Jansen (2010) analyze the mobile software
platforms and evaluate their level of openness taking into con-
sideration also their licensing policies. Finally, Popp (2011) names
three roles in the intellectual property (IP) business utilization: the
IP distributors that sell IPR from the inventors or usage rights to
the customers, the IP lessors that “rents” IPs or products of IP (e.g.,
software) for a specific time and the IP brokers that matches the
needs of an IP requestor to an IP owner. For example an IP bro-
ker might facilitate a startup software company to find software
vendors.

4.4.3. SECO relationships
An open technological platform in combination with a set of

management processes and business models, cannot create a SECO
without the social aspect. A community, social network or a set of
actors weaved around a platform and sets of rules communicat-
ing and interacting both among themselves and with the platform
is essential. Because of the existence of this interaction, the soft-
ware architecture of the platform has to be designed with different
considerations than a proprietary platform. The management pro-
cess, business models and IPR issues become more complicated
while at the same time the evolution of the system is faster and

towards several directions while the SECO gains privileged posi-
tion in the market. There are several actors that might be part of
a SECO. The following list gives an overview of the most common
actors encountered in the literature.

f Syste

(

(
M
K
e
(

(

a
J
(

a
D
S

(
J
V

K. Manikas, K.M. Hansen / The Journal o

rchestrator 9, “keystone (player, organization)”, 10 “hub”,11

“shaper”,12 “management (unit)”,13 or “platform
owner”14 is a company, department of a company,
actor or set of actors, community or independent entity
that is responsible for the well-functioning of the SECO.
This unit is typically managing the SECO by running the
platform, creating and applying rules, processes, business
procedures, setting and monitoring quality standards
and/or orchestrating the SECO actor relationships.

iche player 15“influencer”,16 or “component
developer/builder/team”,17 is the SECO actor that
contributes to the SECO by typically developing or adding
components to the platform, producing functionality
that customers require. This actor is part of the SECO
and complements the work of the keystone by providing
value to the ecosystem. Depending on the management
model of the ecosystem the niche players might influence
the decision making in the management of the SECO.

xternal actor 18“external developer (team)”,19 “third party
developers/community”,20 “external parties”,21 “exter-
nal partner”,22 “external entities”,23 “participant”,24 or
“external adopter”,25 is the actor (company, person,
entity) that makes use of the possibilities the ecosys-
tem provides and thus providing indirect value to the
ecosystem. This actor is external to the SECO manage-
ment and usually has an activity limited to the actor’s
interest. Depending on the nature of the ecosystem, the
external actor might be developing on top of or parallel to
the SECO platform, identify bugs, promote the SECO and
its products or propose improvements. This type of actor
includes the role of the participant or follower in FOSS

SECOs. An actor that is member of the SECO with either
participation of limited responsibility or simply observing
the evolution of the SECO from the inside.

9 Used in: van Angeren et al. (2011, 2011), Jansen et al. (2009b,a), Hilkert et al.
2010), Idu et al. (2011), van der Schuur et al. (2011).
10 Used in: van Angeren et al. (2011), Burkard et al. (2012), Campbell and Ahmed
2010), Hanssen (2011), Jansen et al. (2009a, 2012), Kabbedijk and Jansen (2011),

cGregor (2010), Pettersson et al. (2010), Riis and Schubert (2012), Viljainen and
auppinen (2011), Idu et al. (2011), dos Santos and Werner (2011a,b), te Molder
t al. (2011), van den Berk et al. (2010), van Ingen et al. (2011), van der Schuur et al.
2011).
11 Used in: dos Santos and Werner (2011a,b), Burkard et al. (2012), Hilkert et al.
2010), Riis and Schubert (2012), van den Berk et al. (2010).
12 Used in: Jansen et al. (2009a), Viljainen and Kauppinen (2011), van der Schuur
t al. (2011)
13 Used in: Campbell and Ahmed (2010).
14 Used in: van Angeren et al. (2011).
15 Used in: Jansen et al. (2009a), Viljainen and Kauppinen (2011, 2011), dos Santos
nd Werner (2011a,b), Burkard et al. (2012), Yu and Deng (2011), Kabbedijk and
ansen (2011), Riis and Schubert (2012), te Molder et al. (2011), van den Berk et al.
2010), van Ingen et al. (2011), van der Schuur et al. (2011).
16 Used in: dos Santos and Werner (2011a,b), van den Berk et al. (2010).
17 Used in: Jansen et al. (2009a, 2012), Viljainen and Kauppinen (2011, 2011), Bosch
nd Bosch-Sijtsema (2010c), Bosch (2009).
18 Used in: Pettersson and Gil (2010), Hansen et al. (2011), Pettersson et al. (2010).
19 Used in: Bosch (2010a, 2009, 2010b), Pettersson et al. (2010), dos Santos
nd Werner (2011a,b), Bosch and Bosch-Sijtsema (2010b,c,a), Jansen et al. (2012),
raxler and Stevens (2011), Kilamo et al. (2012), Viljainen and Kauppinen (2011),
cacchi and Alspaugh (2012), van Ingen et al. (2011), Weiss (2011).
20 Used in: Anvaari and Jansen (2010), Bosch and Bosch-Sijtsema (2010b,c), Bosch
2009, 2010b), Campbell and Ahmed (2010), Dhungana et al. (2010), Hanssen (2011),
ansen et al. (2009a, 2012), Mizushima and Ikawa (2011), Seichter et al. (2010),
iljainen and Kauppinen (2011).

21 Used in: Bosch and Bosch-Sijtsema (2010b,c).
22 Used in: Bosch (2010b), Draxler and Stevens (2011).
23 Used in: Campbell and Ahmed (2010).
24 Used in: Jansen et al. (2009a).
25 Used in: Viljainen and Kauppinen (2011).
ms and Software 86 (2013) 1294– 1306 1301

Vendor “independent software vendor (ISV)”,26, “reseller” or
“value-added reseller (VAR)”,27 is mainly the company
or business unit that makes profit from selling the
products of the SECO to customers, end-users or other
vendors/VARs. The products might be complete integra-
tions, components, selling or leasing of licenses or support
agreements. A vendor that is modifying the SECO prod-
uct by, e.g., adding functionality or combining different
components together is called VAR.

Customer or “end user” is the person, company, entity that either
purchases or obtains a complete or partial product of the
SECO or a niche player either directly from the SECO/niche
player or through a vendor/VAR.

A different characterization of the social network of a SECO
appears in (Jansen et al., 2012; Scacchi and Alspaugh, 2012) where
they characterize the SECO niche as a software supply network of
producers, integrators and customers.

An interesting perspective of SECO relationships is the actor
participation model that SECOs follow. Different ecosystems apply
different models for allowing actors to contribute to the ecosystem.
These models are many times related to the nature of the platform
and to what extent it allows/supports different kinds of collabora-
tion, but mostly to the business model behind the ecosystem. To
explain this better, we take the actor participation model of three
ecosystems as an example: a traditional FOSS project that is often
open to any participant willing to join, the Eclipse ecosystem where
developers can join freely but have to go through the Eclipse Legal
Process every time they commit code (Mizushima and Ikawa, 2011)
and the the Open Design Alliance (ODA) where actors have to pay an
annual fee to be part of the ecosystem (van Angeren et al., 2011). The
openness or closeness of a SECO describes how easy it is for an actor
to participate in an ecosystem. The measurement of the openness
of a SECO is an interesting perspective that affects the social net-
work of an ecosystem. As already mentioned, the level of openness
depends on parameters outside of the SECO social network per-
spective, however, it is analyzed as part of this perspective since it
affects heavily the social networks. te Molder et al. (2011) claim that
the openness and closeness of a platform is not binary, but there
are many different levels. In their paper they introduce the concept
of “clopeness” and propose a model for assessing the clopeness of
a SECO. Jansen et al. (2012) state that the complicity of opening
or closing the SECO as “multi-facet and cannot be judged without
extensive study”. They also explain that the benefits of opening
up the ecosystem are often not clear, while a post-evaluation of
whether the ecosystem was ready for the changes will be reflected
in the SECO health after the changes have been applied. Finally they
make a separation between the supply and demand of a SECO and
mention that a SECO can choose to open either of them or both.

In the software supply network, Riis and Schubert (2012) ana-
lyze how the relationships evolve in an ERP SECO when the SECO
vendor (orchestrator) is pushing an upgrade to a newer version. It
is notable that the relations can be push-oriented, i.e., the orches-
trator pushes a new version to the ISVs and VARs and eventually
the customer, but also pull-oriented, i.e., the customer requests

an older version from the ISVs/VARs end eventually the orches-
trator. Jansen et al. (2012) referring to Popp (2010) numbers three
distribution channels: (i) direct through VAR, (ii) indirect through

26 Used in: Jansen et al. (2009b,a, 2012), Bosch (2009), Boucharas et al. (2009),
Draxler and Stevens (2011), Hilkert et al. (2010), Hunink et al. (2010), Riis and
Schubert (2012), te Molder et al. (2011), van den Berk et al. (2010), Viljainen and
Kauppinen (2011), Scacchi and Alspaugh (2012), Janner et al. (2008).

27 Used in: Riis and Schubert (2012), Jansen et al. (2012, 2009b,a), Janner et al.
(2008), Boucharas et al. (2009), Hanssen (2011), Popp (2011).

1302 K. Manikas, K.M. Hansen / The Journal of Systems and Software 86 (2013) 1294– 1306

Table 6
The papers using existing SECOs.

SECO type Papers % of total

Proprietary [45, 41, 2, 4, 10, 30, 54, 53, 16, 37, 44, 33, 6, 26, 87, 63, 82, 72, 62, 65] 22
0, 76,
, 56, 1

s
(
s
t
t
s
s
b
u
s
p
r

C
b
t
p
t
c
K
i
f
t
b
J
G
w
t
o
p

d
t
a
o

n
r
o
F
t
o
c
t
i
t
l
c
d
t
e
e
(
a

FOSS [6, 7, 48, 51, 50, 42, 31, 18, 57, 27, 89, 88, 73, 8
No SECO [40, 19, 1, 38, 39, 43, 3, 5, 9, 11, 8, 49, 59, 58, 52

12, 34, 55, 32, 24, 86, 71, 69, 61, 78, 79, 90]

ervice organization and (iii) direct to customer. Yu et al. (2008), Yu
2011) adopt the natural ecology types of symbiotic relationships to
oftware symbiosis: mutualism, where both systems benefit from
heir relations, commensalism, where one system benefits from
he relations while the other is unaffected, parasitism, where one
ystem benefits and the other is harmed, amensalism, where one
ystem is harmed and the other unaffected, competition, where
oth systems are harmed and neutralism where both systems are
naffected. Although, the symbiotic relations were described in the
oftware symbiosis context rather than the social network, in our
erspective, they could also be used to reflect SECO social network
elations.

When looking into the niche player relationships, Kazman and
hen (2010) proposes the Metropolis model for the relationships
etween the actors in a SECO where it is consisted of the kernel
hat is responsible for platform and fundamental functionality, the
eriphery that is consisted of the prosumers building on top of
he kernel’s platform, and the masses that are the end-users. This
an be parallelized to the “onion model” (Jergensen et al., 2011;
ilamo et al., 2012) appearing in FOSS projects, where the member

nvolvement is similar to the layers of an onion: a member starts
rom the external layers having tasks with low responsibility, e.g.,
ranslation, and slowly moves to the inner layers gaining responsi-
ilities. In another study of the developer behavior, Kabbedijk and
ansen (2011) studied the interaction of developers within the Ruby
ithub SECO and noted three different roles: the “lone wolf” that
orks mainly alone and produces big part of the system used by

he rest of the users, the “networker” that is connected to several
ther developers and the “one day flies” that have created only one
opular component without significant activity afterwards.

Communication among the different roles is also of interest. van
er Schuur et al. (2011) study how knowledge is transferred within
he different roles of a SECO while Fricker (2010) proposes the prop-
gation of information in terms of requirements from the end-users
r customers to the ecosystem with the requirement value chains.

.5. Connection with industry

From the research questions that are mentioned in the begin-
ing of this article, question 2.1 is investigating the use of
eal-world SECOs in the research. The purpose is to give a view
n how close the connection of the research is to the industry.
rom the data collection process, we have compiled a list with all
he papers that are using an existing SECO in their research as an
bject of study. Analyzing this list, we end up with the results that
an be seen in Table 6. Going through the results, we notice that
he slight majority of the papers (53%) is using an existing SECO
n their research. The existing ecosystems are appearing in mainly
wo ways: (i) one or more SECOs are studied and the paper pub-
ishes study results, conclusions, interesting remarks as it is the
ase with Hanssen (2011) that describe the transition of a tra-
itional waterfall-based software company to a SECO and (ii) a
heory, framework, taxonomy or tool is developed based on lit-

rature, hypothesis or experience and then applied to one or more
xisting ecosystems to prove it, as it is the case in te Molder et al.
2011) where the Clopennes Assessment model is applied to an
nonymized SECO to support the theory. In both of the cases,
74, 68, 77, 75, 64, 70, 66, 60, 67, 83, 81, 85, 84] 31
3, 36, 46, 20, 23, 15, 22, 47, 21, 28, 35, 14, 29, 17, 47

we argue that the use of existing ecosystems as objects of study
increases the ‘external’ validity of the results.

Table 6 is separating the papers that study existing ecosystems
in papers studying proprietary and free or open source software
(FOSS) ecosystems. We separate the two kinds of ecosystems as
they have significant differences. In a strict proprietary ecosystem,
the source code and other artifacts produced are protected, as they
are the products that would yield revenues to the ecosystem, while
new actors would probably have to be certified in some way so
they would be allowed to participate in the ecosystem. In a tradi-
tional FOSS ecosystem, the actors do not necessarily participate to
obtain direct revenues from their activity in the ecosystem, while
it is often much easier for an actor to participate in a FOSS than
a proprietary SECO, since FOSS SECOs typically do no require any
verification of new actors. Naturally, this simplistic way of sepa-
rating proprietary and FOSS SECOs is only used to underline the
differences of the two kinds of ecosystems. A majority of the SECOs
would probably be categorized as a hybrid, combining elements
from the two kinds. However, in the literature we note that papers
studying FOSS SECOs are mostly concerned with problems of tech-
nical or social nature, while the papers studying proprietary SECOs
include business and strategic problems. This is only natural, since
FOSS projects allow the mining and processing of several details
(like source code, commit logs, etc.) but they do not necessarily
have a clear business model for the whole SECO or the participat-
ing actors (or at least it does not apear so in the literature). This
underlines the importance of the research focusing on FOSS SECOs
to include business and strategic perspectives. On the other hand,
papers in the proprietary SECO group can get information about
SECO strategies and positioning in the market, but it is harder to
get access to proprietary information like source code, developer
commits and so on.

Table 7 lists the existing SECOs used in the literature. The lit-
erature is studying 43 SECOs in total, out of which, 30 are studied
in only one paper each. We note that out of the 12 SECOs stud-
ied in more than one paper (in this count we do not include
the “Anonymized/not named” category), only two (GX Software
and SAP) do not belong to the FOSS group and Eclipse being
the most studied SECO (appearing in seven papers). Additionally,
18 out of the 43 studied SECOs are of proprietary nature. We
explained this, by the additional challenge posed in gaining access
to information in a proprietary SECOs in contradiction to a FOSS
where data are usually accessed by mining a publicly available
repository.

5. Discussion

The purpose of this study is to provide an overview of the field
of software ecosystems by reviewing and analyzing the published
literature. This work has been done based on the review protocol
explained in Section 2.

In this work we did not include any evaluation of the quality of
the relevant literature. The only consideration relating to the qual-

ity of a paper is the number of papers within the literature citing
this paper, if any. It could be argued that a possible assessment of
the quality of the literature could be undertaken to set focus on the
gravity each paper should have in the analysis sections, e.g., 4.4.

K. Manikas, K.M. Hansen / The Journal of Syste

Table 7
The SECOs appearing in the literature.

SECO name Papers

Eclipse, Eclipse Foundation 6, 89, 73, 76, 67, 83, 85
GNOME 7, 51, 80, 74
Open Design Alliance 6, 76, 10, 16
Anonymized/not named 65, 82, 45
Brazilian Public Software (BPS) 88, 64, 33
Linux, Linux Kernel 50, 57, 70
Android 27, 66
GX Software 76, 10
Evince 7, 18
FOSS 42, 31
FreeBSD 50, 57
iPhone/iPad App Store 27, 72
SAP 53, 2
Apache Web Server 70
Artop 67
Brasero 7
CAS Software AG 37
CSoft 44
CubicEyes 6
Debian 60
Google Chrome 75
Google
Gurux 2
Firefox 77
HIS GmbH 75
HISinOne 63
Mac App Store 26
Microsoft 72
Nokia Siemens Networks 2
Nautilus 87
Pharo 81
Ruby 68
S. Chand Edutech 84
SOOPS BV 62
Squeak 54
Symbian 68
TFN 200 67
UniImprove 41
Unity 30 75
US Department of Defense 30
WattDepot 48
WinMob 27
World of Worcraft 89

o
c

fi
a
o
a
n
p
t

d
S
a
a
s
s

4
t
l
p
b

1. Capuruç o and Capretz (2010)
Apart from addressing the research questions and providing an
verview in the field, we also identified several areas that are not
overed in the literature body.

As already noted, the field of software ecosystems is not the only
eld inspired by the natural ecosystems. There has been significant
mount of work done in other ecosystems like the business, social
r natural ecosystems themselves. The SECO literature does not
ppear to examine work done in other ecosystems apart from a
umber of papers mentioned in Section 4. Possible intersections or
arallelizations of the fields would allow the use of theories from
he other fields or different perspectives in SECO problems.

An important ingredient of the success of an ecosystem is
iversity. The differentiation of actors would allow niche creation.
tatements similar to this have appeared several times in the liter-
ture. However, no concrete studies have been provided to prove

statement of this kind. Technical, organizational, business and
ocial variability in harmonic symbiosis settings could bring more
tability and possibly contribute to a healthier ecosystem.

The concept of health of an ecosystem, as explained in Section
, section has been introduced to SECO from the business ecosys-
em theory. Measuring the health of an ecosystem would provide
arge benefits for the SECO industry and research. The health would

rovide indications on the future of the ecosystem and give possi-
le feedback on applied changes in the ecosystem. However, apart
ms and Software 86 (2013) 1294– 1306 1303

from referring to SECO health, very few studies elaborate, analyze
or measure the health of a software ecosystem.

The intellectual property rights and licensing issues are a focus
point of a small part of the literature. Finding effective ways to
address issues of this kind is of more importance than the atten-
tion it has been receiving in the literature. Issues of this kind are
of importance both to the organizational perspectives of a SECO –
how to organize the development in the ecosystem– but also in the
business – how to develop the proper business/revenue models.

Quality assurance (QA) is a field that has also not been efficiently
addressed in the literature. The adoption of traditional QA meth-
ods might not necessarily work in a SECO, because of the separation
of platform and actors. Possibly, the proper QA strategies depend
on the orchestration of the ecosystem and solutions might be spe-
cific to each SECO, however, there is a need for SECO specific QA
strategies.

Finally, a field that has not been covered in the literature, is
the organization of and decision making in SECOs. We recognize
the high differentiation in the management models existing SECOs
apply that would probably give reasons to why this field is not
addressed in the literature. However, we argue that studies on that
aspect of SECOs would assist, providing a more complete picture of
the field.

6. Conclusion

Software ecosystems is an area that has been gaining in popu-
larity the last five years. The software industry is moving towards
software ecosystems, with platforms like Google Android and Apple
iOS increasing in popularity, while research has increasing inter-
est in the field, with the fourth year of a dedicated workshop
(IWSECO 2012). This article is documenting a systematic literature
review held on the field of software ecosystems. The purpose of
this work was to provide an overview of the field and identify pos-
sible research issues or areas not covered. We found and analyzed
90 relevant papers from a gross total of 420 extracted from a list
of scientific libraries. Based on this, we provided an overview of
the definition of SECOs as it is defined in the literature including
finding patterns in the different definitions provided and list the
common main items that consist a SECO. We reported an increase
in the research from 2007 to today. Additionally, we classified the
research papers according to the result they reported and identified
a lack in analytical models and an excess in report papers. More-
over, we defined “SECO architecture” and identified and analyzed
the three main components that is consisted of: SECO Software
Engineering, SECO Business and Management and SECO Relations.
Finally, we examined the intersection of research and industry and
found that half of the papers relate to the industry while at the same
time most of them are focusing on FOSS SECOs. In conclusion, we
identify the field of software ecosystems as a new field of growing
importance and potential both in research and industry.

Acknowledgements

The authors would like to thank the anonymous reviewers for
their comments that greatly improved the quality of this paper.

This work has been partially funded by the Net4Care project
within Caretech Innovation (http://www.caretechinnovation.dk/
projekter/net4care/).

Appendix A. Literature body
2. Popp (2011)
3. Yu and Deng (2011)

http://www.caretechinnovation.dk/projekter/net4care/

1 f Syste

1
1
1
1
1
1
1
1
1
1
1
2
2
2
2
2
2
2
2
2
3
3
3
3
3
3
3
3
3
3
4
4
4
4
4
4
4
4
4
4
5
5
5
5
5
5
5
5
5
5
6
6
6
6
6
6
6
6
6

304 K. Manikas, K.M. Hansen / The Journal o

4. te Molder et al. (2011)
5. dos Santos and Werner (2011b)
6. van Angeren et al. (2011)
7. Mens and Goeminne (2011)
8. Barbosa and Alves (2011)
9. Alspaugh et al. (2009)
0. Jansen et al. (2009a)
1. Fricker (2009)
2. Campbell and Ahmed (2010)
3. Seichter et al. (2010)
4. Dhungana et al. (2010)
5. Lungu et al. (2010b)
6. van den Berk et al. (2010)
7. Cataldo and Herbsleb (2010)
8. Goeminne and Mens (2010)
9. Bosch (2010a)
9. Bosch (2009)
1. Kazman and Chen (2010)
2. Lungu and Lanza (2010)
3. Pettersson et al. (2010)
4. Scacchi (2010b)
5. Boucharas et al. (2009)
6. Brummermann et al. (2011)
7. Anvaari and Jansen (2010)
8. Hunink et al. (2010)
9. dos Santos and Werner (2010)
0. McGregor (2010)
1. Scacchi (2007a)
2. Krishna and Srinivasa (2011)
3. Alves and Pessôa (2010)
4. Briscoe (2010)
5. Fricker (2010)
6. Scacchi (2010a)
7. Hilkert et al. (2010)
8. Bosch and Bosch-Sijtsema (2010c)
9. Bosch and Bosch-Sijtsema (2010a)
0. Kazman et al. (2012)
1. Schneider et al. (2010)
2. Yu and Woodard (2009)
3. Bosch (2010b)
4. Hanssen (2011)
5. Bosch and Bosch-Sijtsema (2010b)
6. Scacchi (2007b)
7. Lungu et al. (2010a)
8. Brewer and Johnson (2010)
9. Pettersson and Gil (2010)
0. Yu et al. (2008)
1. Lungu et al. (2009)
2. An (2009)
3. Janner et al. (2008)
4. Lungu (2008)
5. Hindle et al. (2010)
6. Jansen et al. (2009b)
7. Yu et al. (2007)
8. Schugerl et al. (2009)
9. Kakola (2010)
0. Ververs et al. (2011)
1. van Angeren et al. (2011)
2. Scholten et al. (2012)
3. Brummermann et al. (2012)
4. Stefanuto et al. (2011)
5. van der Schuur et al. (2011)

6. van Ingen et al. (2011)
7. Weiss (2011)
8. Robbes and Lungu (2011)
ms and Software 86 (2013) 1294– 1306

69. Schmerl et al. (2011)
70. Yu (2011)
71. dos Santos and Werner (2011a)
72. Idu et al. (2011)
73. Draxler et al. (2011a)
74. Jergensen et al. (2011)
75. Scacchi and Alspaugh (2012)
76. Jansen et al. (2012)
77. Kilamo et al. (2012)
78. Pettersson and Vogel (2012)
79. Kajan et al. (2011)
80. Pérez et al. (2012)
81. Neu et al. (2011)
82. Riis and Schubert (2012)
83. Mizushima and Ikawa (2011)
84. Kabbedijk and Jansen (2011)
85. Draxler and Stevens (2011)
86. Burkard et al. (2012)
87. Viljainen and Kauppinen (2011)
88. Alves et al. (2011)
89. Draxler et al. (2011b)
90. Widjaja and Buxmann (2011)

References

Alspaugh, T., Asuncion, H., Scacchi, W., 2009. The role of software licenses in open
architecture ecosystems. In: First International Workshop on Software Ecosys-
tems (IWSECO-2009), Citeseer, pp. 4–18.

Alves, A.M., Pessôa, M., 2010. Brazilian public software: beyond sharing. In: in:
Proceedings of the International Conference on Management of Emergent Digital
EcoSystems, ACM, New York, NY, USA, pp. 73–80.

Alves, A.M., Pessoa, M., Salviano, C.F., 2011. Towards a systemic maturity model
for public software ecosystems. In: O’Connor, R.V., Rout, T., McCaffery, F., Dor-
ling, A. (Eds.), Software Process Improvement and Capability Determination,
vol. 155 of Communications in Computer and Information Science. Springer,
Berlin/Heidelberg, pp. 145–156, 10.1007/978-3-642-21233-8 13.

An, H., 2009. Research on software problems based on ecological angle. In: Inter-
national Conference on Environmental Science and Information Application
Technology 2009 (ESIAT 2009), pp. 11–14.

van Angeren, J., Blijleven, V., Jansen, S., 2011. Relationship intimacy in software
ecosystems: a survey of the dutch software industry. In: Proceedings of the
International Conference on Management of Emergent Digital EcoSystems, ACM,
New York, NY, USA, pp. 68–75.

van Angeren, J., Kabbedijk, J., Popp, K.M., 2011. A survey of associate models used
within large software ecosystems. In: Third International Workshop on Software
Ecosystems (IWSECO-2011), CEUR-WS, pp. 27–39.

Anvaari, M., Jansen, S., 2010. Evaluating architectural openness in mobile soft-
ware platforms. In: Proceedings of the Fourth European Conference on Software
Architecture: Companion Volume, ACM, New York, NY, USA, pp. 85–92.

Barbosa, O., Alves, C., 2011. A systematic mapping study on software ecosystems. In:
Third International Workshop on Software Ecosystems (IWSECO-2011), CEUR-
WS, pp. 15–26.

Bass, L., Clements, P., Kazman, R., 2003. Software Architecture in Practice. Addison-
Wesley Longman Publishing Co., Inc., Boston.

van den Berk, I., Jansen, S., Luinenburg, L., 2010. Software ecosystems: a software
ecosystem strategy assessment model. In: Proceedings of the Fourth European
Conference on Software Architecture: Companion Volume, ACM, New York, NY,
USA, pp. 127–134.

Bosch, J., 2009. From software product lines to software ecosystems. In: Proceedings
of the 13th International Software Product Line Conference, Carnegie Mellon
University, Pittsburgh, PA, USA, pp. 111–119.

Bosch, J., 2010a. Architecture challenges for software ecosystems. In: Proceedings of
the Fourth European Conference on Software Architecture: Companion Volume,
ACM, New York, NY, USA, pp. 93–95.

Bosch, J., 2010b. Architecture in the age of compositionality. In: Babar, M., Gor-
ton, I. (Eds.), Software Architecture, volume 6285 of Lecture Notes in Computer
Science. Springer Berlin/Heidelberg, pp. 1–4, http://dx.doi.org/10.1007/978-3-
642-15114-9 1

Bosch, J., Bosch-Sijtsema, P., 2010a. Coordination between global agile teams:
from process to architecture. In: Smite, D., Moe, N.B., Agerfalk, P.J. (Eds.),
Agility Across Time and Space. Springer, Berlin/Heidelberg, pp. 217–233,
http://dx.doi.org/10.1007/978-3-642-12442-6 15

Bosch, J., Bosch-Sijtsema, P., 2010b. From integration to composition. On the impact

of software product lines global development and ecosystems. Journal of Sys-
tems and Software 83, 67–76, SI: Top Scholars.

Bosch, J., Bosch-Sijtsema, P.M., 2010c. Softwares product lines, global development
and ecosystems: Collaboration in software engineering. In: Mistrik, I., van der
Hoek, A., Grundy, J., Whitehead, J. (Eds.), Collaborative Software Engineering.

f Syste

K. Manikas, K.M. Hansen / The Journal o

Springer, Berlin/Heidelberg, pp. 77–92, http://dx.doi.org/10.1007/978-3-642-
10294-3 4

oucharas, V., Jansen, S., Brinkkemper, S., 2009. Formalizing software ecosystem
modeling. In: in: Proceedings of the 1st International Workshop on Open Com-
ponent Ecosystems, ACM, New York, NY, USA, pp. 41–50.

rewer, R., Johnson, P., 2010. Wattdepot: an open source software ecosystem for
enterprise-scale energy data collection storage analysis and visualization. In:
in: First IEEE International Conference on Smart Grid Communications (Smart-
GridComm), pp. 91–95.

riscoe, G., 2010. Complex adaptive digital ecosystems. In: Proceedings of the Inter-
national Conference on Management of Emergent Digital EcoSystems, ACM, New
York, NY, USA, pp. 39–46.

rummermann, H., Keunecke, M., Schmid, K., 2011. Variability issues in the evolu-
tion of information system ecosystems. In: Proceedings of the 5th Workshop on
Variability Modeling of Software-Intensive Systems, ACM, New York, NY, USA,
pp. 159–164.

rummermann, H., Keunecke, M., Schmid, K., 2012. Formalizing distributed evolu-
tion of variability in information system ecosystems. In: Proceedings of the Sixth
International Workshop on Variability Modeling of Software-Intensive Systems,
ACM, New York, NY, USA, pp. 11–19.

urkard, C., Widjaja, T., Buxmann, P., 2012. Software ecosystems. Business
and Information Systems Engineering 4, 41–44, http://dx.doi.org/10.1007/
s12599-011-0199-8.

ampbell, P.R.J., Ahmed, F., 2010. A three-dimensional view of software
ecosystems. In: in: Proceedings of the Fourth European Conference on
Software Architecture: Companion Volume, ACM, New York, NY, USA,
pp. 81–84.

apuruç o, R.A.C., Capretz, L.F., 2010. Integrating recommender information in social
ecosystems decisions. In: in: Proceedings of the Fourth European Conference
on Software Architecture: Companion Volume, ACM, New York, NY, USA, pp.
143–150.

ataldo, M., Herbsleb, J.D., 2010. Architecting in software ecosystems: interface
translucence as an enabler for scalable collaboration. In: in: Proceedings of
the Fourth European Conference on Software Architecture: Companion Volume,
ACM, New York, NY, USA, pp. 65–72.

hungana, D., Groher, I., Schludermann, E., Biffl, S., 2010. Software ecosystems
vs natural ecosystems learning from the ingenious mind of nature. In: in:
Proceedings of the Fourth European Conference on Software Architecture: Com-
panion Volume, ACM, New York, NY, USA, pp. 96–102.

raxler, S., Jung, A., Boden, A., Stevens, G., 2011a. Workplace warriors: identifying
team practices of appropriation in software ecosystems. In: in: Proceedings of
the 4th International Workshop on Cooperative and Human Aspects of Software
Engineering, ACM, New York, NY, USA, pp. 57–60.

raxler, S., Jung, A., Stevens, G., 2011b. Managing software portfolios: a com-
parative study. In: Costabile, M., Dittrich, Y., Fischer, G., Piccinno, A. (Eds.),
End-User Development, vol. 6654 of Lecture Notes in Computer Science.
Springer, Berlin/Heidelberg, pp. 337–342, http://dx.doi.org/10.1007/978-3-642-
21530-8 36

raxler, S., Stevens, G., 2011. Supporting the collaborative appropriation of an
open software ecosystem. Computer Supported Cooperative Work (CSCW) 20,
403–448, http://dx.doi.org/10.1007/s10606-011-9148-9

ricker, S., 2009. Specification and analysis of requirements negotiation strategy in
software ecosystems. In: in: First International Workshop on Software Ecosys-
tems (IWSECO-2009), Citeseer, pp. 19–33.

ricker, S., 2010. Requirements value chains: stakeholder management and require-
ments engineering in software ecosystems. In: Wieringa, R., Persson, A. (Eds.),
Requirements Engineering: Foundation for Software Quality, vol. 6182 of
Lecture Notes in Computer Science. Springer, Berlin/Heidelberg, pp. 60–66,
http://dx.doi.org/10.1007/978-3-642-14192-8 7

oeminne, M., Mens, T., 2010. A framework for analysing and visualising open source
software ecosystems. In: in: Proceedings of the Joint ERCIM Workshop on Soft-
ware Evolution (EVOL) and International Workshop on Principles of Software
Evolution (IWPSE), ACM, New York, NY, USA, pp. 42–47.

ansen, K.M., Jonasson, K., Neukirchen, H., 2011. Controversy corner. An empirical
study of software architectures’ effect on product quality. Journal of Systems
and Software 84, 1233–1243.

anssen, G.K., 2011. A longitudinal case study of an emerging software ecosys-
tem: Implications for practice and theory. Journal of Systems and Software,
http://dx.doi.org/10.1016/j.jss.2011.04.020

en Hartigh, E., Tol, M., Visscher, W., 2006. The health measurement of a business
ecosystem. In: in: Proceedings of the European Network on Chaos and Complex-
ity Research and Management Practice Meeting.

ilkert, D., Wolf, C.M., Benlian, A., Hess, T., 2010. The “as-a-service”-paradigm and
its implications for the software industry – insights from a comparative case
study in crm software ecosystems. In: Aalst, W., Mylopoulos, J., Sadeh, N.M.,
Shaw, M.J., Szyperski, C., Tyrvainen, P., Jansen, S., Cusumano, M.A. (Eds.), Soft-
ware Business, vol. 51 of Lecture Notes in Business Information Processing.
Springer, Berlin/Heidelberg, pp. 125–137, http://dx.doi.org/10.1007/978-3-642-
13633-7 11

indle, A., Herraiz, I., Shihab, E., Jiang, Z.M., 2010. Mining challenge 2010: Freebsd
gnome desktop and debian/ubuntu. In: in: 7th IEEE Working Conference on

Mining Software Repositories (MSR), pp. 82–85.

unink, I., van Erk, R., Jansen, S., Brinkkemper, S., 2010. Industry taxonomy engi-
neering: the case of the european software ecosystem. In: in: Proceedings of
the Fourth European Conference on Software Architecture: Companion Volume,
ACM, New York, NY, USA, pp. 111–118.
ms and Software 86 (2013) 1294– 1306 1305

Iansiti, M., Levien, R., 2004a. The Keystone Advantage: What the New Dynamics of
Business Ecosystems Mean for Strategy Innovation and Sustainability. Harvard
Business Press, Boston.

Iansiti, M., Levien, R., 2004b. Strategy as ecology. Harvard Business Review 82,
68–81.

Idu, A., van de Zande, T., Jansen, S., 2011. Multi-homing in the apple ecosystem:
why and how developers target multiple apple app stores. In: in: Proceedings of
the International Conference on Management of Emergent Digital EcoSystems,
ACM, New York, NY, USA, pp. 122–128.

van Ingen, K., van Ommen, J., Jansen, S., 2011. Improving activity in communities
of practice through software release management. In: in: Proceedings of the
International Conference on Management of Emergent Digital EcoSystems, ACM,
New York, NY, USA, pp. 94–98.

Janner, T., Schroth, C., Schmid, B., 2008. Modelling service systems for collaborative
innovation in the enterprise software industry – the st. gallen media reference
model applied. In: in: IEEE International Conference on Services Computing 2008
(SCC’08), pp. 145–152.

Jansen, S., Brinkkemper, S., Finkelstein, A., 2009a. Business network management as
a survival strategy: a tale of two software ecosystems. In: in: First International
Workshop on Software Ecosystems (IWSECO-2009), Citeseer, pp. 34–48.

Jansen, S., Brinkkemper, S., Souer, J., Luinenburg, L., 2012. Shades of gray: opening
up a software producing organization with the open software enterprise model.
Journal of Systems and Software 85, 1495–1510, Software Ecosystems.

Jansen, S., Finkelstein, A., Brinkkemper, S., 2009b. A sense of community: A
research agenda for software ecosystems. In: in: 31st International Conference
on Software Engineering – Companion Volume, 2009. ICSE-Companion 2009,
pp. 187–190.

Jergensen, C., Sarma, A., Wagstrom, P., 2011. The onion patch: migration in open
source ecosystems. In: in: Proceedings of the 19th ACM SIGSOFT Symposium
and the 13th European Conference on Foundations of Software Engineering,
ACM, New York, NY, USA, pp. 70–80.

Kabbedijk, J., Jansen, S., 2011. Steering insight: An exploration of the ruby soft-
ware ecosystem. In: Regnell, B., Weerd, I., Troyer, O., Aalst, W., Mylopoulos, J.,
Rosemann, M., Shaw, M.J., Szyperski, C. (Eds.), Software Business, vol. 80 of Lec-
ture Notes in Business Information Processing. Springer, Berlin/Heidelberg, pp.
44–55, http://dx.doi.org/10.1007/978-3-642-21544-5 5

Kajan, E., Lazic, L., Maamar, Z., 2011. Software engineering framework for digital
service-oriented ecosystem. In: in: 19th Telecommunications Forum (TELFOR),
pp. 1320–1323.

Kakola, T., 2010. Standards initiatives for software product line engineering and
management within the international organization for standardization. In:
in: 43rd Hawaii International Conference on System Sciences (HICSS), 2010,
pp. 1–10.

Kazman, R., Chen, H.M., 2010. The metropolis model and its implications for the
engineering of software ecosystems. In: in: Proceedings of the FSE/SDP Work-
shop on Future of Software Engineering Research, ACM, New York, NY, USA, pp.
187–190.

Kazman, R., Gagliardi, M., Wood, W., 2012. Scaling up software architecture analysis.
Journal of Systems and Software 85, 1511–1519, Software Ecosystems.

Kilamo, T., Hammouda, I., Mikkonen, T., Aaltonen, T., 2012. From proprietary to open
source-growing an open source ecosystem. Journal of Systems and Software 85,
1467–1478.

Kitchenham, B., Charters, S., 2007. Guidelines for performing systematic literature
reviews in software engineering. Engineering 2.

Krishna, R.P.M., Srinivasa, K.G., 2011. Analysis of projects and volunteer participa-
tion in large scale free and open source software ecosystem. SIGSOFT Software
Engineering Notes 36, 1–5.

Lungu, M., 2008. Towards reverse engineering software ecosystems. In:
in: IEEE International Conference on Software Maintenance, ICSM 2008,
pp. 428–431.

Lungu, M., Lanza, M., 2010. The small project observatory: a tool for reverse engineer-
ing software ecosystems. In: in: Proceedings of the 32nd ACM/IEEE International
Conference on Software Engineering – Volume 2, ACM, New York, NY, USA, pp.
289–292.

Lungu, M., Lanza, M., Gîrba, T., Robbes, R., 2010a. The small project observatory: visu-
alizing software ecosystems. Science of Computer Programming 75, 264–275,
Experimental Software and Toolkits (EST 3): A special issue of the Work-
shop on Academic Software Development Tools and Techniques (WASDeTT
2008).

Lungu, M., Malnati, J., Lanza, M., 2009. Visualizing gnome with the small project
observatory. In: in: Proceedings of the 6th IEEE International Working Confer-
ence on Mining Software Repositories, 2009 (MSR’09), pp. 103–106.

Lungu, M., Robbes, R., Lanza, M., 2010b. Recovering inter-project dependencies in
software ecosystems. In: in: Proceedings of the IEEE/ACM International Con-
ference on Automated Software Engineering, ACM, New York, NY, USA, pp.
309–312.

McGregor, J.D., 2010. A method for analyzing software product line ecosystems. In:
in: Proceedings of the Fourth European Conference on Software Architecture:
Companion Volume, ACM, New York, NY, USA, pp. 73–80.

Mens, T., Goeminne, M., 2011. Analysing the evolution of social aspects of open
source software ecosystems. In: in: Third International Workshop on Software

Ecosystems (IWSECO-2011), CEUR-WS, pp. 1–14.

Messerschmitt, D., Szyperski, C., 2005. Software Ecosystem: Understanding An Indis-
pensable Technology and Industry. MIT Press Books 1, London, England.

Mizushima, K., Ikawa, Y., 2011. A structure of co-creation in an open source software
ecosystem: a case study of the eclipse community. In: in: 2011 Proceedings of

dx.doi.org/10.1007/s12599-011-0199-8

1 f Syste

306 K. Manikas, K.M. Hansen / The Journal o

PICMET’11, Technology Management in the Energy Smart World (PICMET), pp.
1–8.

e Molder, J., van Lier, B., Jansen, S., 2011. Clopenness of systems: The interwo-
ven nature of ecosystems. In: in: Third International Workshop on Software
Ecosystems (IWSECO-2011), CEUR-WS, pp. 52–64.

eu, S., Lanza, M., Hattori, L., D’Ambros, M., 2011. Telling stories about gnome with
complicity. In: in: 2011 6th IEEE International Workshop on Visualizing Software
for Understanding and Analysis (VISSOFT), pp. 1–8.

ettersson, O., Gil, D., 2010. On the issue of reusability and adaptability in m-learning
systems. In: in: 2010 6th IEEE International Conference on Wireless, Mobile and
Ubiquitous Technologies in Education (WMUTE), pp. 161–165.

ettersson, O., Svensson, M., Gil, D., Andersson, J., Milrad, M., 2010. On the role of
software process modeling in software ecosystem design. In: in: Proceedings of
the Fourth European Conference on Software Architecture: Companion Volume,
ACM, New York, NY, USA, pp. 103–110.

ettersson, O., Vogel, B., 2012. Reusability and interoperability in mobile learning:
a study of current practices. In: in: 2012 IEEE Seventh International Confer-
ence onWireless, Mobile and Ubiquitous Technology in Education (WMUTE),
pp. 306–310.

érez, J., Deshayes, R., Goeminne, M., Mens, T., 2012. Seconda: software ecosystem
analysis dashboard. In: in: 16th European Conference on Software Maintenance
and Reengineering (CSMR), pp. 527–530.

opp, K., 2010. Goals of software vendors for partner ecosystems – a practitioner’s
view. Software Business, 181–186.

opp, K.M., 2011. Hybrid revenue models of software companies and their rela-
tionship to hybrid business models. In: in: Third International Workshop on
Software Ecosystems (IWSECO-2011), CEUR-WS, pp. 77–88.

iis, P., Schubert, P., 2012. Upgrading to a new version of an erp system: a
multilevel analysis of influencing factors in a software ecosystem. In: in:
45th Hawaii International Conference on System Science (HICSS), pp. 4709–
4718.

obbes, R., Lungu, M., 2011. A study of ripple effects in software ecosystems (nier
track). In: in: Proceedings of the 33rd International Conference on Software
Engineering, ACM, New York, NY, USA, pp. 904–907.

os Santos, R.P., Werner, C., 2011a. Treating business dimension in soft-
ware ecosystems. In: in: Proceedings of the International Conference on
Management of Emergent Digital EcoSystems, ACM, New York, NY, USA,
pp. 197–201.

os Santos, R.P., Werner, C.M.L., 2010. Revisiting the concept of components in soft-
ware engineering from a software ecosystem perspective. In: in: Proceedings of
the Fourth European Conference on Software Architecture: Companion Volume,
ACM, New York, NY, USA, pp. 135–142.

os Santos, R.P., Werner, C.M.L., 2011b. A proposal for software ecosystem
engineering. In: in: Third International Workshop on Software Ecosystems
(IWSECO-2011), CEUR-WS, pp. 40–51.

cacchi, W., 2007a. Free/open source software development: recent research results
and emerging opportunities. In: in: The 6th Joint Meeting on European Software
Engineering Conference and the ACM SIGSOFT Symposium on the Foundations
of Software Engineering: Companion Papers, ACM, New York, NY, USA, pp.
459–468.

cacchi, W., 2007b. Free/open source software development: recent research results
and methods. In: Zelkowitz, M.V. (Ed.), Architectural Issues, Elsevier, vol. 69 of
Advances in Computers. , pp. 243–295.

cacchi, W., 2010a. Collaboration practices and affordances in free/open source
software development. In: Mistrík, I., van der Hoek, A., Grundy, J., Whitehead,
J. (Eds.), Collaborative Software Engineering. Springer, Berlin/Heidelberg, pp.
307–327, http://dx.doi.org/10.1007/978-3-642-10294-3 15

cacchi, W., 2010b. The future of research in free/open source software development.
In: in: Proceedings of the FSE/SDP Workshop on Future of Software Engineering
Research, ACM, New York, NY, USA, pp. 315–320.

cacchi, W., Alspaugh, T.A., 2012. Understanding the role of licenses and evolution
in open architecture software ecosystems. Journal of Systems and Software 85,

1479–1494.

chmerl, B., Garlan, D., Dwivedi, V., Bigrigg, M.W., Carley, K.M., 2011. Sorascs: a case
study in soa-based platform design for socio-cultural analysis. In: Proceedings
of the 33rd International Conference on Software Engineering, ACM, New York,
NY, USA, pp. 643–652.
ms and Software 86 (2013) 1294– 1306

Schneider, K., Meyer, S., Peters, M., Schliephacke, F., Mörschbach, J., Aguirre, L., 2010.
Feedback in context: supporting the evolution of it-ecosystems. In: Ali Babar, M.,
Vierimaa, M., Oivo, M. (Eds.), Product-Focused Software Process Improvement,
vol. 6156 of Lecture Notes in Computer Science. Springer, Berlin/Heidelberg, pp.
191–205, 10.1007/978-3-642-13792-1 16.

Scholten, U., Fischer, R., Zirpins, C., 2012. The dynamic network notation: harness-
ing network effects in paas-ecosystems. In: Proceedings of the Fourth Annual
Workshop on Simplifying Complex Networks for Practitioners, ACM, New York,
NY, USA, pp. 25–30.

Schugerl, P., Rilling, J., Witte, R., Charland, P., 2009. A quality perspective of soft-
ware evolvability using semantic analysis. In: IEEE International Conference on
Semantic Computing, ICSC’09, pp. 420–427.

van der Schuur, H., Jansen, S., Brinkkemper, S., 2011. The power of propagation:
on the role of software operation knowledge within software ecosystems. In:
Proceedings of the International Conference on Management of Emergent Digital
EcoSystems, ACM, New York, NY, USA, pp. 76–84.

Seichter, D., Dhungana, D., Pleuss, A., Hauptmann, B., 2010. Knowledge management
in software ecosystems: software artefacts as first-class citizens. In: Proceedings
of the Fourth European Conference on Software Architecture: Companion Vol-
ume, ACM, New York, NY, USA, pp. 119–126.

Shaw, M., 2003. Writing good software engineering research papers. In: Mini-
tutorial for Proc ICSE”03.

Stefanuto, G., Spiess, M., Alves, A.M., Castro, P.F.D., 2011. Quality in software digital
ecosystems the users perceptions. In: Proceedings of the International Confer-
ence on Management of Emergent Digital EcoSystems, ACM, New York, NY, USA,
pp. 85–88.

Ververs, E., van Bommel, R., Jansen, S., 2011. Influences on developer participation in
the debian software ecosystem. In: Proceedings of the International Conference
on Management of Emergent Digital EcoSystems, ACM, New York, NY, USA, pp.
89–93.

Viljainen, M., Kauppinen, M., 2011. Software ecosystems: a set of management prac-
tices for platform integrators in the telecom industry. In: Regnell, B., Weerd, I.,
Troyer, O., Aalst, W., Mylopoulos, J., Rosemann, M., Shaw, M.J., Szyperski, C. (Eds.),
Software Business, vol. 80 of Lecture Notes in Business Information Processing.
Springer, Berlin/Heidelberg, pp. 32–43, 10.1007/978-3-642-21544-5 4.

Weiss, M., 2011. Economics of collectives. In: Proceedings of the 15th Interna-
tional Software Product Line Conference, vol. 2, ACM, New York, NY, USA, pp.
39:1–39:8.

Widjaja, T., Buxmann, P., 2011. Compatibility of software platforms. In: Heinzl, A.,
Buxmann, P., Wendt, O., Niche Weitzel, T. (Eds.), Theory-Guided Modeling and
Empiricism in Information Systems Research. Physica-Verlag HD, Berlin Heidel-
berg, Germany, pp. 15–41, http://dx.doi.org/10.1007/978-3-7908-2781-1 2

Yu, E., Deng, S., 2011. Understanding software ecosystems: A strategic modeling
approach. In: Third International Workshop on Software Ecosystems (IWSECO-
2011), CEUR-WS, pp. 65–76.

Yu, L., 2011. Coevolution of information ecosystems: a study of the statistical rela-
tions among the growth rates of hardware, system software, and application
software. SIGSOFT Software Engineering Notes 36, 1–5.

Yu, L., Ramaswamy, S., Bush, J., 2007. Software evolvability: an ecosystem point of
view. In: in: Third International IEEE Workshop on Software Evolvability, pp.
75–80.

Yu, L., Ramaswamy, S., Bush, J., 2008. Symbiosis and software evolvability. IT Pro-
fessional 10, 56–62.

Yu, S., Woodard, C., 2009. Innovation in the programmable web: charac-
terizing the mashup ecosystem. In: Feuerlicht, G., Lamersdorf, W. (Eds.),
Service-Oriented Computing – ICSOC 2008 Workshops, vol. 5472 of Lec-
ture Notes in Computer Science. Springer, Berlin/Heidelberg, pp. 136–147,
http://dx.doi.org/10.1007/978-3-642-01247-1 13.

Konstantinos Manikas is a PhD scholar at the Department of Computer Science
of Copenhagen University. His main research areas are software architecture and
software ecosystems with interest in telemedicince and healthcare IT.
Klaus Marius Hansen is a professor of Software Development at the University of
Copenhagen. He received a Ph.D. degree in Computer Science from Aarhus University
in 2002 and his research focuses on software technology and use in particular in
relation to pervasive and dependable computing.

Software ecosystems – A systematic literature review
1 Introduction
1.1 Article structure

2 Review protocol
2.1 Research questions
2.2 Defining the literature body

3 Collecting the literature body
4 Analysis
4.1 Defining SECO
4.2 Yearly activity
4.3 Research results
4.4 SECO architecture
4.4.1 SECO software engineering
4.4.2 SECO business and management
4.4.3 SECO relationships

4.5 Connection with industry

5 Discussion
6 Conclusion
Acknowledgements
Appendix A Literature body
References

SAMPLE_SLRs/Meri18a

A Systematic Literature Review of Software Visualization Evaluation

L. Merinoa, M. Ghafaria, C. Anslowb, O. Nierstrasza

aSoftware Composition Group, University of Bern, Switzerland
bSchool of Engineering and Computer Science, Victoria University of Wellington, New Zealand

Abstract

Context: Software visualizations can help developers to analyze multiple aspects of complex software systems, but their effec-
tiveness is often uncertain due to the lack of evaluation guidelines.

Objective: We identify common problems in the evaluation of software visualizations with the goal of formulating guidelines
to improve future evaluations.

Method: We review the complete literature body of 387 full papers published in the SOFTVIS/VISSOFT conferences, and
study 181 of those from which we could extract evaluation strategies, data collection methods, and other aspects of the evaluation.

Results: Of the proposed software visualization approaches, 62% lack a strong evaluation. We argue that an effective software
visualization should not only boost time and correctness but also recollection, usability, engagement, and other emotions.

Conclusion: We call on researchers proposing new software visualizations to provide evidence of their effectiveness by con-
ducting thorough (i) case studies for approaches that must be studied in situ, and when variables can be controlled, (ii) experiments
with randomly selected participants of the target audience and real-world open source software systems to promote reproducibility
and replicability. We present guidelines to increase the evidence of the effectiveness of software visualization approaches, thus
improving their adoption rate.

Published in: Journal of Systems and Software, https://doi.org/10.1016/j.jss.2018.06.027

Keywords: software visualisation, evaluation, literature review

1. Introduction

Software visualizations are useful for analyzing multiple
aspects of complex software systems. Software visualization
tools have been proposed to help analysts make sense of multi-
variate data [25], to support programmers in comprehending the
architecture of systems [31], to help researchers analyze version
control repositories [9], and to aid developers of software prod-
uct lines [16]. However, most developers are still unaware of
which existing visualization approaches are suitable to adopt
for their needs. We conjecture that the low adoption of software
visualization results from their unproved effectiveness and lack
of evaluations. Indeed, researchers adopt varying strategies to
evaluate software visualization approaches, and therefore the
quality of the evidence of their effectiveness varies. We believe
that a characterization of the evaluation of software visualiza-
tion approaches will (i) assist researchers in the field to improve
the quality of evaluations, and (ii) increase the adoption of vi-
sualization among developers.

We consider previous research to be an important step to
characterizing the evidence of the effectiveness of software vi-
sualization approaches. However, we reflect that previous re-
search has failed to define what is an effective software visu-
alization, and consequently comparing the effectiveness of vi-
sualization approaches is not possible. Moreover, we believe
that some studies have used a loose definition of “case studies”
and include many usage scenarios of visualization instead that

present little evidence of the effectiveness of an approach. In
our investigation we perform a subtler analysis of the character-
istics of evaluations to elucidate these concerns. Consequently,
we formulated the following research questions:

RQ1.) What are the characteristics of evaluations that validate
the effectiveness of software visualization approaches?

RQ2.) How appropriate are the evaluations that are conducted
to validate the effectiveness of software visualization?

We believe that answering these questions will assist re-
searchers in the software visualization field to improve the qual-
ity of evaluations by identifying evaluation strategies and meth-
ods and their common pitfalls. In particular, we reviewed 181
full papers of the 387 papers published in SOFTVIS/VISSOFT.
We identified evaluation strategies such as surveys, case studies,
and experiments, as well as characteristics such as tasks, par-
ticipants, and systems used in evaluations. We found that 62%
(i.e., 113) of the proposed software visualization approaches ei-
ther do not include any evaluation, or include a weak evaluation
(i.e., anecdotal evidence, usage scenarios). Almost all of them
(i.e., 110) introduce a new software visualization approach. The
remaining three discuss an existing approach but without pro-
viding a stronger evaluation. We also found that 29% of the
studies (i.e., 53) conducted experiments in which 30% (i.e., 16)
corresponded to visualizations that target the novice developer

Preprint submitted to The Journal of Systems and Software June 21, 2018

https://doi.org/10.1016/j.jss.2018.06.027

audience, and included appropriate participants. The remaining
70% proposed visualizations for developers with various lev-
els of experience. However, amongst them only 30% included
experienced developers, and the remaining 70% (i.e., 37) in-
cluded in experiments only students and academics of a conve-
nience sample who are vulnerable to selection bias and hence
hinder generalization. We found that 7% (i.e., 12) of the studies
conducted a case study that involved (i) professional develop-
ers from industry, and (ii) real-world software systems. Finally,
3% (i.e., 4) of studies conducted a survey. Even though we are
not aware of a similar quantitative report of the state of the art
in information visualization, a review of the practice of evalua-
tion [12] found similar issues.

We believe that for software visualization approaches to be
adopted by developers, visualizations not only must prove their
effectiveness via evaluations, but evaluations should also in-
clude participants of the target audience, and be based on real-
world software systems. Finally, we recommend researchers in
the field to conduct surveys that can help them to identify what
are the frequent and complex problems that affect developers.

This paper makes the following contributions:

1. A study of the characteristics of evaluations performed in
the literature of software visualization.

2. Guidelines for researchers in the visualization field who
need to evaluate software visualization approaches.

3. A publicly available data set including the information of
the studies and classifications.1

The remainder of the paper is structured as follows: Sec-
tion 2 presents related work. Section 3 describes the main
concepts that are addressed in the characterization. Section 4
describes the methodology that we followed to collect and se-
lect relevant studies proposed in the software visualization field.
Section 5 presents our results by classifying evaluations based
on adopted strategies, methods and their characteristics. Sec-
tion 6 discusses our research questions and threats to validity of
our findings, and Section 7 concludes and presents future work.

2. Related Work

A few studies have attempted to characterize the evaluation
of software visualization approaches via a literature review. For
instance, Schots and Werner [35] reviewed 36 papers published
between 1993 and 2012 and proposed an extended taxonomy
that includes evidence of the applicability of a software visu-
alization as a dimension [34]. They found that papers lacked
a clear description of information related to the evidence on
the use of visualization. Seriai et al. [38] analyzed 87 papers
published between 2000 and 2012. They found that most vi-
sualizations are evaluated via case studies (i.e., 78.16%), and
only a few researchers conducted experiments (i.e., 16.09%).
They observed that even though the proportion of publications

1http://scg.unibe.ch/research/softvis-eval

that include an evaluation is fairly constant over time, they lack
rigor. Mattila et al. [19] included 83 papers published between
2010 and 2015 in their analysis. They also found that only
a few researchers conducted experiments (i.e., 13.25%), some
performed case studies (i.e., 22.89%), and the rest used other
evaluation methods. In our investigation we cover a much larger
body of literature (i.e., 181 full papers) that spans up to 2017.
We not only characterize the state-of-the-art in software visual-
ization evaluation, but we also propose guidance to researchers
in the field by detecting common pitfalls, and by elaborating
on guidelines to conduct evaluation of software visualization
approaches.

Other studies have opted to evaluate software visualization
tools and have reported guidelines. For example, Storey et
al. [41] evaluated 12 software visualization tools, and proposed
an evaluation framework based on intent, information, presen-
tation, interaction, and effectiveness. Sensalire et al. [36, 37]
evaluated 20 software visualization tools proposed for main-
tenance based via experiments, and elaborated various lessons
learned. They identified a number of dimensions that are criti-
cal for organizing an evaluation, and then analyzing the results.
Müller et al. [27] proposed a structured approach for conduct-
ing controlled experiments in envisioned 3D software visual-
ization tools. Instead of concentrating on rather limited number
of tools, we chose a meta analysis by analyzing the reports of
the evaluation of proposed visualization tools. In this way we
could analyze the state-of-the-art in the practice of software vi-
sualization evaluation, and consequently elaborate guidelines
for defining what is an effective software visualization.

A few reviews of the software visualization literature that
focus on various domains have tangentially analyzed the eval-
uation aspect. Lopez-Herrejon et al. [16] analyzed evaluation
strategies used in visualizations proposed for software product
line engineering, and they found that most approaches used case
studies. They also found that only a few performed experi-
ments, and a few others did not explicitly describe an evalu-
ation. Shahin et al. [39] discussed the evaluation of visualiza-
tion approaches proposed to support software architecture, and
classified the evidence of the evaluation using a 5-step scale [1].
The analysis of the results showed that almost half of the eval-
uations represent toy examples or demonstrations. The other
half correspond to industrial case studies, and a very few others
described experiments and anecdotal evidence of tool adoption.
Novais et al. [30] investigated the evaluations of approaches
that proposed visualization to analyze software evolution. In
most of the analyzed studies evaluation consisted in usage ex-
amples that were demonstrated by the authors of the study. In
a few of them, the demonstration was carried out by external
users. Evaluation strategies based on experiments were found
to be extremely rare. In almost 20% of the studies they did not
find an explicit evaluation. Since the main focus of these men-
tioned studies is not on evaluation (as opposed to ours), they
only characterize the evaluation of the analyzed studies, and of-
fer little advice for researchers who need to perform their own
evaluations of software visualizations.

Similar efforts have been made in the information visualiza-
tion field. Amar and Stasko [2] proposed a task-based frame-

http://scg.unibe.ch/research/softvis-eval

work for the evaluation of information visualizations. Forsell [8]
proposed a guide to scientific evaluation of information visual-
ization that focuses on quantitative experimental research. The
guide contains recommendations for (a) designing, (b) conduct-
ing, (c) analyzing results, and (d) reporting on experiments.
Lam et al. [15] proposed seven scenarios for empirical stud-
ies in information visualization. Isenberg et al. [12] reviewed
581 papers to analyze the practice of evaluating visualization.
Some of the pitfalls they found are that in some evaluations
(i) participants do not belong to the target audience, (ii) goals
are not explicit, (iii) the strategy and analysis method is not ap-
propriate, and (iv) the level of rigor is low. Elmqvist and Yi [6]
proposed patterns for visualization evaluation that present solu-
tions to common problems encountered when evaluating a visu-
alization system. We observed that advice given in the context
of information visualization can also be applied to software vi-
sualization evaluation; however, we also observed that there are
particularities in software visualization that require a tailored
analysis, which is an objective of our investigation.

3. Background

The strategies that researchers adopt to evaluate the effec-
tiveness of a software visualization approach can be classified
into two main categories:

i) Theoretical principles from information visualization that
provide researchers support to justify a chosen visual en-
coding [28]. For instance, the effectiveness of perceptual
channels depends on the data type (i.e., categorical, or-
dered, or quantitative) [17].

ii) Empirical evidence gathered from the evaluation of a tech-
nique, method or tool. Amongst them we find a) ex-
ploratory evaluations that involve high-level real-world tasks,
for which identifying the aspects of the tool that boosted
the effectiveness is complex; and b) explanatory evalua-
tions in which high-level tasks are dissected into low-level
(but less realistic) tasks that can be measured in isolation
to identify the cause of an increase in the effectiveness of
an approach [44].

Amongst the strategies used in empirical evaluations we find
(a) surveys [45] that allow researchers to collect data from de-
velopers who are the users of a system, and hence analyze the
collected data to generalize conclusions; (b) experiments [40]
that provide researchers with a high level of control to manip-
ulate some variables while controlling others (i.e., controlled
experiments) with randomly assigned subjects (when it is not
possible to ensure randomness the strategy is called “quasi-
experiment”); and (c) case studies [33] that help researchers to
investigate a phenomenon in its real-life context (i.e., the case),
hence giving researchers a lower level of control than an exper-
iment but enabling a deeper analysis.

Several methods exist for collecting data in each evaluation
strategy. The two most common methods [7] are (i) question-
naires in which the researcher provides instructions to partici-
pants to answer a set of questions that can range from loosely

structured (e.g., exploratory survey) to closed and fully struc-
tured (e.g., to collect data of the background of participants in
an experiment), and (ii) interviews in which a researcher can
ask a group of subjects a set of closed questions in a fixed or-
der (i.e., fully structured), a mix of open and closed questions
(i.e., semi-structured), and open-ended questions (i.e., unstruc-
tured). Less frequent methods for collecting data are observa-
tional ones such as (iii) think-aloud in which researchers ask
participants to verbalize their thoughts while performing the
evaluation. Besides, recent experiments have collected data us-
ing (iv) video recording to capture the behavior of participants
during the evaluation; (v) sketch drawing to evaluate recollec-
tion; and (vi) eye tracking to measure the browsing behavior of
eye’s movement.

Finally, there are several statistical tests that are usually
used to analyze quantitative data collected from an experiment.
For discrete or categorical data, tests such as Chi-square and
Cohen’s kappa are suitable. For questions that analyze the re-
lationships of independent variables, regression analysis can be
applied. For correlation analysis of dependent variables one has
to first analyze if the parametric assumptions holds. That is, if
the data is (i) collected from independent and unbiased sam-
ples, (ii) normally distributed (Shapiro-Wilk test is suggested
and proven more powerful than Kolmogorov-Smirnov [32]), and
(iii) present equal variances (e.g., Levene’s test, Mauchly’s test).
Parametric data can be analyzed with Pearson’s r, while non-
parametric with Spearman’s Rank Correlation. For the analy-
sis of differences of parametric data collected from two groups
Student’s unpaired t-test, Paired t-test, and Hotelling’s T-square
are appropriate. For the non-parametric case Mann-Whitney U
and Wilcoxon Rank sum test are suitable. In the case of analysis
that involves more than two groups of parametric data ANOVA
is a frequent choice, which is usually followed by a post-hoc
test such as Tukey HSD. When data is non-parametric Kruskal-
Wallis test and Friedman test are suitable as well.

4. Methodology

We applied the Systematic Literature Review approach, a
rigorous and auditable research methodology for Evidence-Based
Software Engineering. We followed Keele’s comprehensive guide-
lines [14], which make it less likely that the results of the lit-
erature survey will be biased. The method offers a means for
evaluating and interpreting relevant research to a topic of inter-
est by evidence, which is robust and transferable. We defined
a review protocol to ensure rigor and reproducibility, in which
we determine (i) research questions, (ii) data sources and search
strategy, (iii) inclusion and exclusion criteria, (iv) quality as-
sessment, (v) data extraction, and (vi) selected studies.

4.1. Data sources and search strategy
Systematic literature reviews often define as their data source

digital libraries such as ACM DL2 or IEEE Xplore.3 To find

2http://dl.acm.org/
3http://ieeexplore.ieee.org

http://dl.acm.org/

http://ieeexplore.ieee.org

suitable primary studies for analysis, they define a search strat-
egy that typically is based on keywords. Instead, we decided
to adopt as data source the complete set of papers published
by the SOFTVIS and VISSOFT conferences. We believe the
sixteen editions and hundreds of papers dedicated especially to
software visualization offer a sound body of literature used in
previous studies [26]. We based our decision on (i) the good
B classification that they obtain in the CORE ranking4 (which
considers citation rates, paper submission and acceptance rates
among other indicators), (ii) related work that concluded that
results from the analysis of software visualization evaluation in
papers published by other venues do not differ from those pub-
lished by SOFTVIS/VISSOFT [19, 38]. Although we observe
that publications in better ranked venues might require stronger
evaluations, we believe that analyzing a collection of studies
that have been accepted for publication according to fairly sim-
ilar criteria will support a more objective comparison, and will
provide a suitable baseline for future investigations.

4.2. Inclusion and exclusion criteria
We reviewed the proceedings and programs of the venues

to include full papers and exclude other types of papers that
due to limited space are unlikely to contain enough detail. In
particular, from the 387 papers we excluded 178 papers that
corresponded to: (i) 61 poster. (ii) 52 new ideas and emerging
results (NIER), (iii) 44 tool demo (TD), (iv) 8 keynote, (v) 8
position, and (vi) 5 challenge papers,

4.3. Quality assessment
We then assessed the quality of the remaining 209 papers.

We classified the studies according to the categories proposed
by Munzner [28], in which a visualization paper can be classi-
fied into one of five categories:

a) Evaluations describe how a visualization is used to deal with
tasks in a problem domain. Evaluations are often conducted
via user studies in laboratory settings in which participants
solve a set of tasks while variables are measured.

b) Design studies show how existing visualization techniques
can be usefully combined to deal with a particular problem
domain. Typically, design studies are evaluated through case
studies and usage scenarios.

c) Systems elaborate on the architectural design choices of a
proposed visualization tool and the lessons learned from ob-
serving its use.

d) Techniques focus on novel algorithms that improve the ef-
fectiveness of visualization.Techniques are often evaluated
using benchmarks that measure performance.

e) Models include Commentary papers in which an expert in
the field advocate a position and argue to support it; For-
malism papers present new models, definitions or terminol-
ogy to describe techniques; and Taxonomy papers propose
categories that help researchers to analyze the structure of a
domain.

4http://portal.core.edu.au/conf-ranks/

For each paper, we first read the abstract, second the conclusion,
and finally, in the cases where we still were not sure of their
main contribution, we read the rest of the paper. Although some
papers might exhibit characteristics of more than one type, we
classified them by focusing on their primary contribution.

We observed that model papers in which the main contribu-
tion is a commentary, a formalism or a taxonomy, usually do
not describe explicit evaluations. Consequently, we excluded
twenty-eight papers that we classified in those categories: (i) six
commentary, (ii) seven taxonomy, and (iii) fifteen formalism
papers.

Figure 1a provides an overview of the selection process.
Figure 1b summarizes the 387 collected papers and highlights
the 181 included in the study. Figure 1c shows the outcome
of our classification. We observe that the two venues have
a slightly different focus. SOFTVIS papers focus mostly on
design studies, while VISSOFT papers focus mainly on tech-
niques. A frequent critique of visualization papers is a lack of
evaluation. Indeed, papers in which the main contribution is an
evaluation are unusual (i.e., 10%). The chart also shows that
the two main paper types in visualization are design study and
technique.

The collection of 181 full papers includes studies from six
to eleven pages in length. Initially, we were reluctant to include
six-page papers, but we observed that in two editions of the con-
ferences all full papers were of that length. Consequently, we
analyzed the distribution of research strategies used to evaluate
software visualization approaches by paper length. We did not
find any particular trend, and so decided to include them.

4.4. Data extraction

To accelerate the process of finding and extracting the data
from the studies, we collected keywords that authors commonly
use to describe evaluations iteratively. That is, we started the
process by searching for the following keywords in each pa-
per: “evaluation”, “survey”, “experiment”, “case study”, and
“user study”. When we did not find these keywords, we man-
ually inspected the paper and looked for other new representa-
tive keywords to expand our set. During the manual inspection
when we did not find an explicit evaluation we labeled the pa-
pers accordingly. In the end, we collected the following set of
keywords:

{evaluation, survey, [case|user] stud[y|ies], [application |
usage | analysis] example[s], use case[s], application

scenario[s], [controlled | user] experiment, demonstration,
user scenario[s], example of use, usage scenario[s],

example scenario[s], demonstrative result[s]}

We investigated whether evaluations that involve users are
conducted with end users from the expected target audience
(i.e., representative sample) to ensure the generality of results.
Therefore, in studies that used this type of evaluation, we ex-
tracted who conducted the evaluation, and what subject sys-
tems were involved. We extracted these data by scanning the
evaluation section of papers. In particular, we extracted (i) data

http://portal.core.edu.au/conf-ranks/

SOFTVIS [N=148]
VISSOFT [N=239]

Inclusion Criteria

N = 387

Keynote [N=8]
Challenge [N=5]

NIER [N=52]
TD [N=44]

Position [N=8]
Poster [N=61]

Exclusion Criteria

N = 209

Commentary [N=6]
Taxonomy [N=7]

Formalism [N=15]

Quality Assessment

N = 181

(a) Stages of the search process and number of selected studies in each
stage.

10
19

4
12

16
12
12

16
7

18
9
9
10
11

7
9

VISSOFT’02
SOFTVIS’03
VISSOFT’03
SOFTVIS’05
VISSOFT’05
SOFTVIS’06
VISSOFT’07
SOFTVIS’08
VISSOFT’09
SOFTVIS’10
VISSOFT’11
VISSOFT’13
VISSOFT’14
VISSOFT’15
VISSOFT’16
VISSOFT’17

0 10 20 30 40 50

Included Total

(b) The 181 included papers from the collection of 387 papers published
in SOFTVIS/VISSOFT venues.

Design Study Technique System Evaluation

VISSOFT SOFTVIS Total

Figure 1: The 181 SOFTVIS/VISSOFT full papers included.

Design Study Evaluation System Technique

Theoretical No Explicit Evaluation Survey
Anecdotal Case Study Experiment
Usage Scenarios

Figure 2: The distribution of the 181 included papers categorized by paper types
and research strategy used to evaluate software visualization approaches.

collection methods (e.g., think-aloud, interview, questionnaire);
(ii) number of participants and their background, (iii) tasks,
(iv) subject system, (v) dependent variables, and (vi) statistical
tests.

4.5. Selected studies

We included in our study the 181 papers listed in Tables 1
and 2. The papers are identified by venue and evaluation strat-
egy.

5. Results

We report the characteristics of the extracted data and the
categories used to classify them for quantitative analysis. Fig-
ure 2 shows the distribution of the studies categorized by paper
type [28] and research strategy used to evaluate visualizations.
Table 3 presents our classification of the evaluation strategy
adopted by papers into one of three main categories: (i) theoret-
ical, (ii) no explicit evaluation, and (iii) empirical. For evalua-
tions that used an empirical strategy, we classified them into one
of five categories: (i) anecdotal evidence, (ii) usage scenarios,
(iii) survey, (iv) case study, and (v) experiment.

We report on characteristics of experiments such as data
collection methods, type of analysis, visual tasks, dependent
variables, statistical tests, and scope. The complete classifica-
tion of the 181 included studies is displayed in Tables 4, 5, 6, 7, 8,
and 9.

5.1. Data Collection Methods

In Table 4 we list the various methods that researchers used
to collect data from experiments. The most frequent were ques-
tionnaires, which are normally used to collect data of the back-
ground of participants at the beginning of experiments and fi-
nal observations at the end. Questionnaires are found across
all types of evaluation strategies (i.e., survey, experiment, case
study). Interviews are fairly frequent and found mostly in case
studies. We also found traditional observational methods (e.g.,
think-aloud), but also fairly new methods (e.g., eye tracking).

Table 1: The papers included in the study [S1-S107].
Id and Reference Venue Evaluation

[S1] Aesthetics of class diagrams, Eichelberger, H. V’02 Theorical
[S2] Specifying algorithm visualizations in terms of dat…, Francik, J. V’02 Usage Scenario
[S3] View definitions for language-independent multipl…, Sajaniemi, J. V’02 Usage Scenario
[S4] The CONCEPT project – applying source code analysis to…, Rilling, J. et al. V’02 –
[S5] UML collaboration diagram syntax: an empir…, Purchase, H.C. et al. V’02 Experiment
[S6] Runtime visualisation of object oriented soft…, Smith, M.P. et al. V’02 Usage Scenario
[S7] Reification of program points for visual execution , Diehl, S. et al. V’02 –
[S8] Metrics-based 3D visualization of large obj…, Lewerentz, C. et al. V’02 Usage Scenario
[S9] Analogical representations of programs, Ploix, D. V’02 Usage Scenario
[S10] Revision Towers, Taylor, C.M.B. et al. V’02 Usage Scenario
[S11] Self-Organizing Maps Applied in Visualising …, Brittle, J. et al. V’03 Experiment
[S12] KScope: A Modularized Tool for 3D Visualizati…, Davis, T.A. et al. V’03 Theorical
[S13] Visualization to Support Version Control Software…, Wu, X. et al. V’03 Experiment
[S14] Techniques for Reducing the Complexity o…, Hamou-Lhadj, A. et al. V’03 Usage Scenario
[S15] A topology-shape-metrics approach for the automa…, Eiglsperger, M. et al. S’03 –
[S16] A new approach for visualizing UML class diagrams, Gutwenger, C. et al. S’03 –
[S17] Visualizing model mappings in UML, Hausmann, J.H. et al. S’03 –
[S18] Visualizing software for telecommunication services…, Gansner, E.R. et al. S’03 –
[S19] Graph visualization for the analysis of the structure an…, Zhou, C. et al. S’03 –
[S20] Interactive locality optimization on NUMA architectures, Mu, T. et al. S’03 –
[S21] End-user software visualizations for fault …, Ruthruff, J. et al. S’03 Experiment
[S22] Interactive visual debugging with UML, Jacobs, T. et al. S’03 Usage Scenario
[S23] Designing effective program visualization too…, Tudoreanu, M.E. S’03 Experiment
[S24] Dancing hamsters and marble statue…, Huebscher-Younger, T. et al. S’03 Experiment
[S25] Algorithm visualization in CS education: com…, Grissom, S. et al. S’03 Experiment
[S26] A system for graph-based visualization of t…, Collberg, C. et al. S’03 Usage Scenario
[S27] Visualization of program-execution data for dep…, Orso, A. et al. S’03 Usage Scenario
[S28] Visualizing Java in action, Reiss, S.P. S’03 Usage Scenario
[S29] Plugging-in visualization: experiences integrating a …, Lintern, R. et al. S’03 –
[S30] EVolve: an open extensible software visualizatio…, Wang, Q. et al. S’03 Usage Scenario
[S31] 3D representations for software visualization…, Marcus, A. et al. S’03 Usage Scenario
[S32] Growing squares: animated visualization of …, Elmqvist, N. et al. S’03 Experiment
[S33] Program animation based on the roles of va…, Sajaniemi, J. et al. S’03 Experiment
[S34] Visualizing Feature Interaction in 3-D, Greevy, O. et al. V’05 Usage Scenario
[S35] Identifying Structural Features of Java Prog…, Smith, M.P. et al. V’05 Usage Scenario
[S36] Support for Static Concept Location with sv3D, Xie, X. et al. V’05 Usage Scenario
[S37] Interactive Exploration of Semantic Clusters, Lungu, M. et al. V’05 Usage Scenario
[S38] Exploring Relations within Software Systems …, Balzer, M. et al. V’05 Usage Scenario
[S39] The Dominance Tree in Visualizing Software Dep…, Falke, R. et al. V’05 Usage Scenario
[S40] User Perspectives on a Visual Aid to Program Com…, Cox, A. et al. V’05 Experiment
[S41] Interactive Visual Mechanisms for Exploring So…, Telea, A. et al. V’05 Usage Scenario
[S42] Fractal Figures: Visualizing Development Ef…, D’Ambros, M. et al. V’05 Usage Scenario
[S43] White Coats: Web-Visualization of Evolving S…, Mesnage, C. et al. V’05 Usage Scenario
[S44] Multi-level Method Understanding Using Microprints , Ducasse, S. et al. V’05 –
[S45] Visual Realism for the Visualization of Softwa…, Holten, D. et al. V’05 Usage Scenario
[S46] Visual Exploration of Combined Architectural and Met…, Termeer, M. et al. V’05 –
[S47] Evaluating UML Class Diagram Layout base…, Andriyevska, O. et al. V’05 Experiment
[S48] Interactive Exploration of UML Sequence Diagra…, Sharp, R. et al. V’05 Usage Scenario
[S49] SAB – The Software Architecture Browser, Erben, N. et al. V’05 –
[S50] Towards understanding programs through wear-b…, DeLine, R. et al. S’05 Experiment
[S51] Online-configuration of software visualizations with Vi…, Panas, T. et al. S’05 –
[S52] Visualization of mobile object environments…, Frishman, Y. et al. S’05 Case Study
[S53] Visualizing structural properties of irregular par…, Blochinger, W. et al. S’05 –
[S54] Jove: java as it happens, Reiss, S.P. et al. S’05 –
[S55] Methodology and architecture of JIVE, Gestwicki, P. et al. S’05 Anecdotal
[S56] Visual specification and analysis of use cas…, Kholkar, D. et al. S’05 Case Study
[S57] Visualizing multiple evolution metrics, Pinzger, M. et al. S’05 Usage Scenario
[S58] The war room command console: shared visual…, O’Reilly, C. et al. S’05 Case Study
[S59] CVSscan: visualization of code evolution, Voinea, L. et al. S’05 Case Study
[S60] Visual data mining in software archives, Burch, M. et al. S’05 Usage Scenario
[S61] Algorithm visualization using concept keyboa…, Baloian, N. et al. S’05 Experiment
[S62] Mondrian: an agile information visualization f…, Meyer, M. et al. S’06 Usage Scenario
[S63] Multiscale and multivariate visualizations of …, Voinea, L. et al. S’06 Usage Scenario
[S64] Visualization of areas of interest in softwar…, Byelas, H. et al. S’06 Case Study
[S65] Visual exploration of function call graphs for feature…, Bohnet, J. et al. S’06 –
[S66] Using social agents to visualize software…, Alspaugh, T.A. et al. S’06 Experiment
[S67] Transparency, holophrasting, and automatic layout appl…, Gauvin, S. et al. S’06 –
[S68] A data-driven graphical toolkit for softwa…, Demetrescu, C. et al. S’06 Usage Scenario
[S69] Visualizing live software systems in 3D, Greevy, O. et al. S’06 Usage Scenario
[S70] Execution patterns for visualizing web servic…, de Pauw, W. et al. S’06 Anecdotal
[S71] Experimental evaluation of animated-verifying o…, Jain, J. et al. S’06 Experiment
[S72] Narrative algorithm visualization, Blumenkrants, M. et al. S’06 Experiment
[S73] The Clack graphical router: visualizing net…, Wendlandt, D. et al. S’06 Anecdotal
[S74] A Visualization for Software Project Awaren…, Ripley, R.M. et al. V’07 Usage Scenario
[S75] YARN: Animating Software Evolution, Hindle, A. et al. V’07 Usage Scenario
[S76] DiffArchViz: A Tool to Visualize Correspondence …, Sawant, A.P. V’07 Usage Scenario
[S77] A Bug’s Life” Visualizing a Bug Database””A…, D’Ambros, M. et al. V’07 Usage Scenario
[S78] Task-specific source code dependency investig…, Holmes, R. et al. V’07 Experiment
[S79] Visualizing Software Systems as Cities, Wettel, R. et al. V’07 –
[S80] Onion Graphs for Focus+Context Views of UML Cl…, Kagdi, H. et al. V’07 Usage Scenario
[S81] CocoViz: Towards Cognitive Software Visuali…, Boccuzzo, S. et al. V’07 Usage Scenario
[S82] Distributable Features View: Visualizing the…, Cosma, D.C. et al. V’07 Usage Scenario
[S83] Trace Visualization Using Hierarchical Edge B…, Holten, D. et al. V’07 Usage Scenario
[S84] Visualization of Dynamic Program Aspects, Deelen, P. et al. V’07 Usage Scenario
[S85] Visualizing Dynamic Memory Allocations, Moreta, S. et al. V’07 Usage Scenario
[S86] Applying visualisation techniques in software…, Nestor, D. et al. S’08 Usage Scenario
[S87] Stacked-widget visualization of scheduling-…, Bernardin, T. et al. S’08 Usage Scenario
[S88] Visually localizing design problems with dish…, Wettel, R. et al. S’08 Usage Scenario
[S89] Visualizing inter-dependencies between scenarios, Harel, D. et al. S’08 –
[S90] Software visualization for end-user pr…, Subrahmaniyan, N. et al. S’08 Case Study
[S91] Streamsight: a visualization tool for large-s…, de Pauw, W. et al. S’08 Anecdotal
[S92] Improving an interactive visualization of transition …, Ploeger, B. et al. S’08 –
[S93] Automatic layout of UML use case diagrams, Eichelberger, H. S’08 –
[S94] Gef3D: a framework for two-, two-and-a-h…, von Pilgrim, J. et al. S’08 Usage Scenario
[S95] A catalogue of lightweight visualizations to …, Parnin, C. et al. S’08 Usage Scenario
[S96] An interactive reverse engineering environment…, Telea, A. et al. S’08 Experiment
[S97] Representing unit test data for large scale …, Cottam, J.A. et al. S’08 Anecdotal
[S98] HDPV: interactive, faithful, in-vivo run…, Sundararaman, J. et al. S’08 Usage Scenario
[S99] Analyzing the reliability of communication be…, Zeckzer, D. et al. S’08 Usage Scenario
[S100] Visualization of exception handling constructs…, Shah, H. et al. S’08 Experiment
[S101] Assessing the benefits of synchronization-adorn…, Xie, S. et al. S’08 Experiment
[S102] Extraction and visualization of call dependen…, Telea, A. et al. V’09 Usage Scenario
[S103] Visualizing the Java heap to detect memory proble…, Reiss, S.P. V’09 Anecdotal
[S104] Case study: Visual analytics in software prod…, Telea, A. et al. V’09 Usage Scenario
[S105] Visualizing massively pruned execution trace…, Bohnet, J. et al. V’09 Case Study
[S106] Evaluation of software visualization tool…, Sensalire, M. et al. V’09 Experiment
[S107] The effect of layout on the comprehension of…, Sharif, B. et al. V’09 Experiment

Table 2: The papers included in the study [S108-S181].
Id and Reference Venue Evaluation

[S108] Beyond pretty pictures: Examining the benef…, Yunrim Park et al. V’09 Experiment
[S109] Representing development history in s…, Steinbrueckner, F. et al. S’10 Usage Scenario
[S110] Visual comparison of software architectures, Beck, F. et al. S’10 Usage Scenario
[S111] An automatic layout algorithm for BPEL processes, Albrecht, B. et al. S’10 –
[S112] Off-screen visualization techniques for clas…, Frisch, M. et al. S’10 Experiment
[S113] Jype – a program visualization and programm…, Helminen, J. et al. S’10 Survey
[S114] Zinsight: a visual and analytic environment…, de Pauw, W. et al. S’10 Case Study
[S115] Understanding complex multithreaded softwa…, Truemper, J. et al. S’10 Case Study
[S116] Visualizing windows system traces, Wu, Y. et al. S’10 Usage Scenario
[S117] Embedding spatial software visualization in th…, Kuhn, A. et al. S’10 Experiment
[S118] Towards anomaly comprehension: using structural…, Lin, S. et al. S’10 Experiment
[S119] Dependence cluster visualization, Islam, S.S. et al. S’10 Usage Scenario
[S120] Exploring the inventor’s paradox: applying jig…, Ruan, H. et al. S’10 Usage Scenario
[S121] Trevis: a context tree visualization & anal…, Adamoli, A. et al. S’10 Usage Scenario
[S122] Heapviz: interactive heap visualizati…, Aftandilian, E.E. et al. S’10 Usage Scenario
[S123] AllocRay: memory allocation visualizati…, Robertson, G.G. et al. S’10 Experiment
[S124] Software evolution storylines, Ogawa, M. et al. S’10 –
[S125] User evaluation of polymetric views using a …, Anslow, C. et al. S’10 Experiment
[S126] An interactive ambient visualization fo…, Murphy-Hill, E. et al. S’10 Experiment
[S127] Follow that sketch: Lifecycles of diagrams an…, Walny, J. et al. V’11 Experiment
[S128] Visual support for porting large code base…, Broeksema, B. et al. V’11 Usage Scenario
[S129] A visual analysis and design tool for planning…, Beck, M. et al. V’11 Case Study
[S130] Visually exploring multi-dimensional code coup…, Beck, F. et al. V’11 Usage Scenario
[S131] Constellation visualization: Augmenting progra…, Deng, F. et al. V’11 Experiment
[S132] 3D Hierarchical Edge bundles to visualize relations …, Caserta, P. et al. V’11 –
[S133] Abstract visualization of runtime m…, Choudhury, A.N.M.I. et al. V’11 Usage Scenario
[S134] Telling stories about GNOME with Complicity, Neu, S. et al. V’11 Usage Scenario
[S135] E-Quality: A graph based object oriented so…, Erdemir, U. et al. V’11 Experiment
[S136] Automatic categorization and visualization o…, Reiss, S.P. et al. V’13 Usage Scenario
[S137] Using HTML5 visualizations in software faul…, Gouveia, C. et al. V’13 Experiment
[S138] Visualizing jobs with shared resources in di…, de Pauw, W. et al. V’13 Usage Scenario
[S139] SYNCTRACE: Visual thread-interplay analysis, Karran, B. et al. V’13 Usage Scenario
[S140] Finding structures in multi-type code c…, Abuthawabeh, A. et al. V’13 Experiment
[S141] SourceVis: Collaborative software visualizat…, Anslow, C. et al. V’13 Experiment
[S142] Visualizing software dynamicities with heat…, Benomar, O. et al. V’13 Usage Scenario
[S143] Performance evolution blueprint: Underst…, Sandoval, J.P. et al. V’13 Usage Scenario
[S144] An empirical study assessing the effect of s…, Sharif, B. et al. V’13 Experiment
[S145] Visualizing Developer Interactions, Minelli, R. et al. V’14 Usage Scenario
[S146] AniMatrix: A Matrix-Based Visualization of …, Rufiange, S. et al. V’14 Usage Scenario
[S147] Visualizing the Evolution of Systems and The…, Kula, R.G. et al. V’14 Usage Scenario
[S148] ChronoTwigger: A Visual Analytics Tool for Unde…, Ens, B. et al. V’14 Experiment
[S149] Lightweight Structured Visualization of Asse…, Toprak, S. et al. V’14 Experiment
[S150] How Developers Visualize Compiler Messages: A…, Barik, T. et al. V’14 Experiment
[S151] Feature Relations Graphs: A Visualisation …, Martinez, J. et al. V’14 Case Study
[S152] Search Space Pruning Constraints Visualizati…, Haugen, B. et al. V’14 Usage Scenario
[S153] Integrating Anomaly Diagnosis Techniques int…, Kulesz, D. et al. V’14 Experiment
[S154] Combining Tiled and Textual Views of Code, Homer, M. et al. V’14 Experiment
[S155] Visualizing Work Processes in Software Engine…, Burch, M. et al. V’15 Usage Scenario
[S156] Blended, Not Stirred: Multi-concern Visua…, Dal Sasso, T. et al. V’15 Usage Scenario
[S157] CodeSurveyor: Mapping Large-Scale Software to…, Hawes, N. et al. V’15 Experiment
[S158] Revealing Runtime Features and Constituent…, Palepu, V.K. et al. V’15 Usage Scenario
[S159] A Visual Support for Decomposing Complex Featu…, Urli, S. et al. V’15 Usage Scenario
[S160] Visualising Software as a Particle System, Scarle, S. et al. V’15 Usage Scenario
[S161] Interactive Tag Cloud Visualization of Sof…, Greene, G.J. et al. V’15 Usage Scenario
[S162] Hierarchical Software Landscape Visualizati…, Fittkau, F. et al. V’15 Experiment
[S163] Vestige: A Visualization Framework for Eng…, Schneider, T. et al. V’15 Usage Scenario
[S164] Visual Analytics of Software Structure and Met…, Khan, T. et al. V’15 Experiment
[S165] Stable Voronoi-Based Visualizations for Sof…, Van Hees, R. et al. V’15 Usage Scenario
[S166] Visualizing the Evolution of Working Sets, Minelli, R. et al. V’16 Experiment
[S167] Walls, Pillars and Beams: A 3D Decompositio…, Tymchuk, Y. et al. V’16 Case Study
[S168] CuboidMatrix: Exploring Dynamic Structura…, Schneider, T. et al. V’16 Experiment
[S169] A Tool for Visualizing Patterns of Spread…, Middleton, J. et al. V’16 Experiment
[S170] Jsvee & Kelmu: Creating and Tailoring Program Ani…, Sirkiae, T. V’16 Usage Scenario
[S171] Visualizing Project Evolution through Abstr…, Feist, M.D. et al. V’16 Usage Scenario
[S172] Merge-Tree: Visualizing the Integration of Com…, Wilde, E. et al. V’16 Usage Scenario
[S173] A Scalable Visualization for Dynamic Data in …, Burch, M. et al. V’17 Experiment
[S174] An Empirical Study on the Readability of R…, Hollmann, N. et al. V’17 Experiment
[S175] Concept-Driven Generation of Intuitive Explana…, Reza, M. et al. V’17 Usage Scenario
[S176] Visual Exploration of Memory Traces and Call …, Gralka, P. et al. V’17 Usage Scenario
[S177] Code Park: A New 3D Code Visualization Tool…, Khaloo, P. et al. V’17 Experiment
[S178] Using High-Rising Cities to Visualize Perform…, Ogami, K. et al. V’17 Usage Scenario
[S179] iTraceVis: Visualizing Eye Movement Data With…, Clark, B. et al. V’17 Experiment
[S180] On the Impact of the Medium in the Effective…, Merino, L. et al. V’17 Experiment
[S181] Method Execution Reports: Generating Text and …, Beck, F. et al. V’17 Experiment

5.2. Evaluation Strategies

In twenty-four (i.e., 13%) studies we did not find an explicit
evaluation that presents evidence for supporting the claim of
effectiveness of software visualization approaches. These stud-
ies indicate that the evaluation of the proposed visualization is
planned as future work. In the remaining studies, we found
that several strategies were used to evaluate software visualiza-
tion approaches. We observed that only two studies (i.e., 1%)
used theoretical references to support the claim of the effec-
tiveness of software visualizations. One technique paper [S1]
that proposes aesthetic criteria for class diagrams, considered
their proposed criteria effective since it was derived from the
UML specification, and one design study paper [S12] evalu-
ated the visualization based on previously proposed criteria for
visualizing software in virtual reality [47]. Both studies planned

Table 3: Research strategies used to evaluate software visualization approaches.

Category Strategy Reference #

Theoretical S1, S12 2
No Explicit
Evaluation S4, S7, S15, S16, S17, S18, S19, S20, S29, S44, S46, S49, S51, S53, S54, S65, S67, S79,

S89, S92, S93, S111, S124, S132
24

Empirical

Survey S13, S71, S100, S113 4
Anecdotal
Evidence S55, S70, S73, S91, S97, S103 6

Case Study S52, S56, S58, S59, S64, S90, S105, S114, S115, S129, S151, S167 12
Experiment S5, S11, S13, S21, S23, S24, S25, S32, S33, S40, S47, S50, S61, S66, S71, S72, S78, S96,

S100, S101, S106, S107, S108, S112, S117, S118, S123, S125, S126, S127, S131, S135,
S137, S140, S141, S144, S148, S149, S150, S153, S154, S157, S162, S164, S166, S168,
S169, S173, S174, S177, S179, S180, S181

Example S57, S60, S62, S63, S68, S69, S74, S75, S76, S77, S80, S81, S82, S83, S84, S85, S86, S87,
S88, S94, S95, S98, S99, S102, S104, S109, S110, S116, S119, S120, S121, S122, S128,
S130, S133, S134, S136, S138, S139, S142, S143, S145, S146, S147, S152, S155, S156,
S158, S159, S160, S161, S163, S165, S170, S171, S172, S175, S176, S178

Table 4: Data collection methods used to evaluate software visualization approaches.

Method Reference #

Questionnaire S11, S13, S25, S32, S40, S47, S50, S61, S66, S72, S90, S100, S106, S107, S108, S112, S125, S126,
S127, S135, S137, S140, S141, S144, S149, S150, S153, S154, S157, S162, S164, S168, S173, S177,
S179, S180, S181

Think-Aloud S40, S50, S100, S112, S117, S118, S123, S125, S126, S135, S141, S148, S150, S169, S173, S179,
S180

Interview S33, S71, S78, S90, S100, S106, S123, S127, S153, S174, S177, S180 12
Video Recording S33, S50, S117, S125, S127, S140, S141, S144, S180 9
Sketch Drawing S117, S127, S180 3
Others Eye Tracking (S144), Log Analysis (S166), Feelings Cards (S180) 3

as future work to conduct an experimental evaluation. The re-
maining 155 studies (i.e., 86%) adopted an empirical strategy to
evaluate software visualization approaches. Amongst them, we
found that multiple strategies were used. We investigated the
evidence of the effectiveness of visualization approaches pro-
vided by those strategies.

Figure 3 shows the relation between the data collection meth-
ods used in evaluation strategiesWe observe that most case stud-
ies do not describe the methods used to collect data; however,
we presume they are observational ones, such as one [S90]
that reported to have conducted interviews. The few surveys in
the analysis collected data using interviews and questionnaires.
One survey [S113] did not describe the method to collect data.
Experiments use multiple methods to collect data. They mainly
use questionnaires, interviews, and the think-aloud protocol.
Recent experiments have used video recording, and other meth-
ods such as sketch drawing, eye tracking, log analysis, and emo-
tion cards.

5.2.1. Anecdotal Evidence
We found six studies (i.e., 3%) that support the claim of

effectiveness of visualizations on anecdotal evidence of tool

adoption. Two papers [S55,S73] proposed a visualization to
support the student audience and reported that tools were suc-
cessfully used in software engineering courses. The remaining
four studies [S70,S91,S97,S103] that focused on the developer
audience reported that visualizations were used intensively and
obtained positive feedback.

5.2.2. Usage Scenarios
Eighty-three studies (i.e., 46%) evaluated software visual-

izations via usage scenarios. In this type of evaluation, authors
posed envisioned scenarios and elaborated on how the visual-
ization was expected to be used. Usually, they selected an open-
source software system as the subject of the visualization. The
most popular systems that we found were written in (i) Java,
such as ArgoUML (4×), Ant (4×), JHotDraw (3×), Java SDK
(2×), and Weka (2×); (ii) C++, such as Mozilla (7×), VTK (2×),
and GNOME (2×); and, (iii) Smalltalk Pharo (4×). We found
that several names were used among the studies to describe this
strategy. We observed that sixty-seven studies (i.e., 37%) la-
beled evaluations as case studies, while twenty-six (i.e., 14%)
presented them as use cases. In the rest of the cases, authors
used titles such as: “application examples”, “usage examples”,

Figure 3: Sankey diagram showing the data collection methods (right) em-
ployed in evaluation strategies (left) adopted in empirical evaluations.

“application scenarios”, “analysis example”, “example of use”,
“usage scenarios”, “application scenarios”, and “usage exam-
ple”.

5.2.3. Survey
Only four studies (i.e., 2%) performed a survey, which is

consistent with the findings of related work [19, 38]. Three
of them [S13,S71,S100] surveyed developers to identify com-
plex problems and collect requirements to design a proposed vi-
sualization approach: one focused on supporting development
teams who use version control systems [S13], another asked
former students of a course what they considered the most dif-
ficult subject in the lecture [S71], and another was concerned
with understanding exception-handling constructs [S100]. In
one study [S113] students who used a visualization approach
were surveyed to collect anecdotal evidence of its usefulness.
Two surveys [S71,S113] were conducted for visualization ap-
proaches that target the student audience in a software engi-
neering course, while the remaining two [S13,S100] target the
developer audience.

We found that surveys are used to identify frequent and
complex problems that affect developers; such problems are
then interpreted as requirements for a new visualization ap-
proach. We conjecture whether the low number of surveys has
an effect on the disconnect between the proposed software visu-
alization approaches and the needs of developers that we found
in the past [23].

5.2.4. Case Study
We classified twelve papers (i.e., 7%) in the case study cat-

egory. Usually, case studies are conducted to evaluate visual-
ization approaches that target professional developers working
on real-world projects in an industrial setting. The case of the
study describes the context of the project in which difficulties
arise, and shows how a visualization approach provides devel-
opers support for tackling them. We observed that in three stud-
ies [S56,S90,S114] some or all authors of the study come from
industry, while in the rest there seems to be a strong relation of
authors with industrial companies. In all of them, the evaluation
involved professional developers.

5.2.5. Experiment
Fifty-three studies (i.e., 29%) evaluated software visualiza-

tion via experiments. Although the level of detail varies, we
identified a number of characteristics such as (i) data collec-
tion methods; (ii) type of analysis; (iii) participants; (iv) tasks;
(v) dependent variables; and (vi) statistical tests. In the follow-
ing we describe the results of the extracted data.

i) Participants. We observed a high variance in the number of
participants in experiments (shown in Figure 4). The high-
est number of participants is found in a study [S25] that
included 157 students. The minimum number corresponds
to a study [S100] that involved three participants (graduate
students with experience in industry). The median was 13
participants. A similar analysis of participants in the eval-
uation of information visualization approaches [12] shows
similar results. Most evaluations of information visualiza-
tion approaches involve 1–5 participants (excluding eval-
uations that do not report on the number of participants).
The second most popular group includes 11–20 partici-
pants, and the group that includes 6–10 is the third most
popular. Overall the median is 9 participants. Although
many evaluations in software visualization included a num-
ber of participants in that ranges, the most popular ones
are 6–10 and 11–20, followed by 21–30. One reason that
might explain the difference could be that in our analy-
sis we only included full papers that might present more
thorough evaluations including a higher number of partici-
pants.
We noticed that experiments to evaluate software visual-
ization approaches for teaching software engineering (e.g.,
algorithms and data structures) include a high number of
participants since they usually involve a whole course and
sometimes several of them. This type of experiment typi-
cally evaluates the effect of introducing visualization tools
as a means for helping students to learn the subject of the

12 12
10

4
2

0
2
4
6
8
10
12
14

1-5 6-10 11-20 21-30 31-40 41-50 51-100 >100

Figure 4: Histogram of the number of participants reported in evaluation.

Table 5: Type of analysis adopted in experiments.

Type of
Analysis

References #

Quantitative S21, S23, S24, S25, S71, S78, S101,
S107, S137, S150, S154, S164, S174

Qualitative S11, S13, S33, S61, S66, S96, S100,
S106, S112, S117, S123, S127,
S135, S140, S141, S148, S149,
S153, S157, S166, S169, S181

Quantitative /
Qualitative S5, S32, S40, S47, S50, S72, S108,

S118, S125, S126, S131, S144,
S162, S168, S173, S177, S179, S180

course. All of them found that visualizations do help stu-
dents. However, they do not provide insights into whether
the particular visualization technique tested in the experi-
ment is the most suitable one. All experiments include par-
ticipants selected from a convenience sample. Normally,
they are students and academics at various levels with little
experience working in industry.

ii) Type of Analysis. Table 5 presents our classification of the
type of analysis adopted in experiments. We categorized
the type of analysis into one of two categories: quantita-
tive and qualitative. We found thirteen studies that adopted
a quantitative analysis, while twenty-two used a qualita-
tive one. In eighteen studies there was both a quantitative
and qualitative analysis. Common examples of quantita-
tive analyses in experiments include the measure of quan-
titative variables such as time and correctness
Typically, experiments were described as being formative
or exploratory, and adopted a qualitative analysis of results
(i.e., 75%). Several experiments also used a quantitative
analysis to report evidence that supports the effectiveness
of software visualization approaches. Although reporting
on early results of preliminary evaluations has contributed
important knowledge to the software visualization field, we
believe that for software visualization approaches to be-
come an actionable choice for developers, they have to
present sound evidence of their effectiveness via surveys,
controlled experiments, and case studies.

iii) Dependent Variables. Table 7 lists the dependent variables
that were measured in experiments. We adopted the clas-
sification proposed by Lam et al. [15] and classified the

dependent variables based on two of the proposed scenar-
ios for evaluation of the understanding of visualizations:
user performance and user experience. We found 35 (i.e.,
66%) studies that evaluated user performance, 8 (i.e., 15%)
evaluated user experience, and 10 (i.e., 19%) that evaluated
variables of both. To evaluate performance most experi-
ments defined as dependent variables correctness and time,
some others specified that the experiment aimed at evalu-
ating effectiveness without presenting details, and a few
described multiple variables such as recollection, visual ef-
fort, scalability, and efficiency. To evaluate user experience
researchers asked participants their perception of various
variables such as usability, engagement, understandability,
and emotions.

iv) Statistical Tests. Table 8 summarizes the statistical tests
used in experiments for the quantitative analysis of data.
We observed that the choice of the test is governed primar-
ily by the number of dependent variables, their treatment
and the type of the collected data (i.e., categorical, ordinal,
interval). For instance, a questionnaire that uses a 5-step
Likert scale to ask participants how suitable they find par-
ticular characteristics of a software visualization approach
for a certain task would be ordinal. In that case, there
would be one dependent variable, with five levels of ordinal
data, for which the Kruskal-Wallis test would be a suitable
match. Also, ANOVA is a common choice to test hypothe-
ses. However, we observed that in some cases researchers
found that parametric assumptions do not hold. Although
there are alternative tests for non-parametric data, we ob-
serve that for data that do not follow a normal distribution,
they can perform an Aligned Rank Transform [43] [S177].

v) Task. In table 9 the column Task summarizes exemplary
tasks that we extracted from the design of each experiment.
In almost half of the experiments (i.e., 26) we found ex-
plicit tasks that we identify with a check mark X. The
remaining tasks that we list correspond to rationales that
we inferred from analyzing the goals of experiments.
We observed that in several studies participants were asked
to use a visualization to lookup some aspects of the system.
Although in some cases a database query might be a more
effective tool than a visualization, we observed that these
tasks are often used as a stepping stone towards complex
tasks, in which developers certainly benefit from visualiz-
ing the context. For instance, participants used a visualiza-
tion to answer questions where they had to:

a) count elements such as “how many packages are in
the Java API?” [S125], “what is the number of pack-
ages?” [S164], “determine the total number of pack-
ages this system has” [S180], “how many methods does
the largest class have (in terms of LOC)?” [S144], and

b) find outliers such as “find the process with the longest
duration.” [S32], “who are the top three most active
code contributors?” [S108], “what are the two largest
classes?” [S141], “name three applications that have
a high fan-in” [S162], “find the three classes with the
highest NOA” [S180].

We also observe that most studies build on these answers
and ask participants to complete tasks that require them
to explore. We believe that visualizations inherently excel
in such tasks in contrast to text-based approaches. For in-
stance, participants used visualizations to answer questions
that involve:

a) Feature location such as “which method contains the
logic to increase the speed?” [S50], “locate the feature
that implements the logic: users are reminded that their
accounts will be deleted if they do not log in after a
certain number of months” [S117],

b) Change impact analysis such as “which classes of the
package dependency will be directly affected by this
change?” [S108], “analyze the impact of adding items
to a playlist” [S78],

c) Analyze the rationale of an artifact such as “find the
purpose of the given application” [S117], “what is the
purpose of the application” [S162], and

d) Pattern detection such as “can you identify some inter-
actions that are identical, along time, between groups
of classes?” [S168], “find the most symmetric subtree
in the tree” [S169], “locate the best candidate for the
god class smell” [S180].

Moreover, we classify these tasks according to the taxon-
omy proposed by Munzner [29]. In it, she proposed that
the task that motivates a visualization be classified using
the following dimensions:

a) Analyze. The goal of a visualization can be to consume,
that is, to discover new knowledge, present already dis-
covered knowledge, and enjoy it; or it can be to cre-
ate new material, which could be to annotate elements
in the visualization, record visualization elements, and
derive data elements from the existing ones.

b) Search. All analyses require users to search. How-
ever, the type of search can differ depending on whether
the target of the search and the location of that target
are known. When both the target and its location are
known, it is called lookup. When the target is known
but not its location, it is called locate. When the tar-
get is unknown but its location is known, it is called
browse. Finally, when both target and its location are
unknown, it is called explore.

c) Query. Once the searched targets are found, users query
them. In tasks that involve a single target, the type of
query is referred to as to identify. In tasks that involve
two targets, it is referred to as to compare. Finally, in
tasks that involve more than two targets, it is referred as
to summarize.

We classify all tasks collected from the studies into the dis-
covery category. The results of the classification in the re-
maining two dimensions is presented in Table 6. We ob-
served that most of the tasks were designed to explore and
summarize, that is, participants have to summarize many
targets that they neither know, nor for which they know the
location in the visualization. Almost half of the twenty-

Table 6: Classification of tasks used in experiments according to Munzner [29]

Query
Search

Identify Compare Summarize

Lookup — S5, S125 S108
Locate S123,

S131,
S137,
S153,
S177,
S180

S168 S21, S71,
S100, S112,
S126, S149,
S179

Explore S11,
S173

S72 S13, S23,
S24, S25, S32,
S33, S40, S50,
S61, S78, S96,
S106, S117,
S118, S127,
S135, S140,
S144, S148,
S150, S154,
S157, S162,
S166, S169,
S174, S181

Browse S66,
S101

S47 S107, S141,
S164

seven tasks in this category were explicitly described in the
studies, while for the other half we only found a rationale.
Tasks in this category tackle:

a) Comprehension [S23], [S24], [S25], [S32], [S33], [S40],
[S61], [S96], [S106], [S148], [S154], [S174];

b) Change impact analysis [S50], [S78], [S118];
c) Debugging [S144], [S150], [S181];
d) Code Structure [S140], [S157];
e) Project Management [S166], [S169];
f) Rationale [S13], [S117], [S127], [S162]; and
g) Refactoring [S135].

We found seven other studies with tasks in which partic-
ipants were asked to summarize targets but in which the
targets were known, and therefore we classified them in
the locate category. Studies in this category involve tasks
that deal with:

a) Comprehension [126];
b) Debugging [S21], [S71];
c) Dependencies [100], [149];
d) Code structure [112]; and
e) Project Management [S179].

Only five studies involved tasks that asked participants to
compare two targets. All of these tasks related to compre-
hension. Finally, the tasks of ten studies involved iden-
tifying a single target. These tasks deal with:

a) Comprehension [S11], [S101], [S173], [S180];
b) Change impact analysis [S177]; and
c) Debugging [S66], [S123], [S131], [S137], [S153].

6. Discussion

We now revisit our research questions. Firstly, we discuss
the main characteristics that we found amongst the analyzed
evaluations. Secondly, we discuss whether the conducted eval-
uations are appropriate considering their scope. Finally, we dis-
cuss the threats to the validity of our investigation.

RQ1.) What are the characteristics of evaluations that validate
the effectiveness of software visualization approaches?

Beyond traditional data collection methods. The methods
used to collect data during the evaluation have to facilitate the
subsequent analysis. Consequently, in a formative experiment
researchers interview participants to freely explore aspects of
complex phenomena. In a case study researchers can inter-
view developers in their work environment, which can help
researchers to formulate hypotheses that can be tested in ex-
periments. Questionnaires can be used in surveys for explo-
ration, reaching a higher number of participants who can pro-
vide researchers feedback of past experiences. We observed
that several studies record sessions with participants. After-
wards, these records are used to dissect a user’s performance
(e.g., correctness of answers and their completion time) and ex-
perience (e.g., level of engagement of participants with a tool).
We observed that few non-traditional methods are used: (i) eye
tracking to capture how participants see the elements in visual-
izations; (ii) log analysis to investigate how participants navi-
gate visualizations; and (iii) emotion cards to help participants
to report their feelings in a measurable fashion. Finally, we
believe that the capabilities of recent devices used to display vi-
sualizations [21] ( e.g., mobile phones, tablets, head-mounted
displays [22]) can complement the standard computer screen,
and provide researchers with useful data for investigating both
user performance and user experience.

Thorough reports of anecdotal evidence and usage scenar-
ios. Tool adoption can be considered the strongest evidence of
the usability of an application [1]. However, we observe a lack
of rigor amongst studies that reported anecdotal evidence. Nor-
mally, these studies report that tools were used, but often they
do not specify the context, for instance, whether the tools are
freely adopted or enforced as a requirement in a software en-
gineering teaching course. Moreover, they describe subjective
feedback from users using expressions such as “the tool was
used with much success” [S55], “feedback was positive” [S97]
We propose that also reporting objective evidence, for instance
number of downloads, would help them in making a stronger
case to support the effectiveness of visualizations.

We also observed that one third of studies employed usage
scenarios to demonstrate the effectiveness of the software visu-
alization approaches. Typically they describe how the approach
can answer questions about a software system. Normally, us-
age scenarios are carried out by the researchers themselves. Al-
though researchers in the software visualization field are fre-
quently both experts in software visualization and also expe-
rienced software developers, we believe they are affected by

construction bias to perform the evaluation. Usage scenarios
can help researchers to illustrate the applicability of a visual-
ization approach. In fact, use cases that drive usage scenarios
can reveal insights into the applicability of an visualization ap-
proach in an early stage [10]. Nonetheless, we believe they
must involve external developers of the target audience who
can produce a less biased evaluation, though related work [11]
found that software engineering students can be used instead of
professional software developers under certain conditions. We
found multiple subject systems in usage scenarios, of which
the most popular are open source. We reflect that open source
software systems provide researchers an important resource for
evaluating their proposed visualization approaches. They allow
researchers to replicate evaluations in systems of various char-
acteristics (e.g., size, complexity, architecture, language, do-
main). They also ease the reproducibility of studies. However,
we think that defining a set of software systems to be used in
benchmarks would facilitate comparison across software visu-
alization evaluation [18, 21].

The value of visualizations beyond time and correctness. We
believe that it is necessary to identify the requirements of de-
velopers and evaluate whether the functionality offered by a
visualization tool is appropriate to the problem. Indeed, past
research has found a large gap between the desired aspects and
the features of current software visualization tools [3]. A later
study [36] analyzed desirable features of software visualization
tools for corrective maintenance. A subsequent study [13] an-
alyzed the requirements of visualization tools for reverse engi-
neering. We observed, however, little adoption of the proposed
requirements. Usability is amongst them the most adopted one.
Scalability was adopted only in one study [S32]. Others such as
interoperability, customizability, adoptability, integration, and
query support were not found amongst the variables measured
in experiments (see Table 7). We observed that even though
none of the studies proposed that users of software visualiza-
tions should find answers quickly (i.e., time) and accurately
(i.e., correctness), there are many evaluations that only consid-
ered these two variables.

We observed that evaluations in most studies aimed at prov-
ing the effectiveness of software visualization approaches. How-
ever, some studies do not specify how the effectiveness of the
visualization is defined. Since something effective has “the power
of acting upon the thing designated”,5 we reflect that effective
visualization should fulfill its designated requirements. Then
we ask what are the requirements of software visualization? We
extract requirements from the dependent variables analyzed in
experiments. We observed that the two main categories are user
performance and user experience. Indeed, practitioners who
adopt a visualization approach expect to find not only correct
answers to software concerns, they expect that the visualiza-
tion approach is also efficient (i.e., uses a minimal amount of
resources), and helps them to find answers in a short amount
of time [42]. However, they also aim at obtaining a good ex-

5“effective, adj. and n.” OED Online. Oxford University Press, June 2017.
Accessed October 27, 2017.

Table 7: A summary of the dependent variables found in experiments.

Dependent Variable References #

User
Performance

Not Explicit S96, S108 2
Time S5, S11, S32, S40, S71, S107, S125, S137, S144, S162, S164, S173, S174, S177,

S180
15

Correctness S5, S11, S13, S21, S24, S25, S32, S33, S40, S47, S71, S72, S78, S101, S106, S107,
S108, S118, S123, S125, S126, S137, S144, S150, S162, S164, S168, S173, S179,
S180

Effectiveness S13, S21, S50, S66, S72, S78, S100, S101, S112, S127, S131, S141, S148, S157,
S162, S164, S166

Completion S50,S164 2
Recollection S150,S180 2
Others Visual Effort (S144), Scalability (S32), Efficiency (S32) 3

User
Experience

Not Explicit S96, S126, S49 3
Usability S11, S13, S32, S40, S61, S117, S137, S140, S49, S153, S164, S169, S177, S181 14
Engagement S154, S177 2
Understandability S118, S181 2
Feeling Enjoyment (S32), Intuitive (S137), Satisfaction (S164), Confidence (S107, S126) 5
Others Acceptability (S164), Learnability (S164), Difficulty (S180) 3

Table 8: Statistical tests used to analyze data from experiments.

Id. Test Reference #

T1 ANOVA S25, S32, S40, S107, S144,
S164, S174, S177, S180

T2 Pearson S25, S40, S50, S107, S108,
S150

T3 Cohen S107, S150 2
T4 Wilcoxon S101, S107, S126, S150, S164 5
T5 Student T S5, S72, S101, S137, S162 5
T6 Shapiro-

Wilk
S107, S162, S177, S180 4

T7 Kruskal-
Wallis

S25, S108, S180 3

T8 Mann-
Whitney

S25, S107, S168 3

T9 Descriptive
Statistics
and Charts

S24, S78, S118, S125, S131,
S141, S154, S173, S179

T10 Levene S162, S180 2
T11-
T18

Tukey (S180), Mauchly (S174),
Greenhouse-Geisser (S174),
Friedman (S21), Hotelling
(S71), Kolmogorov-Smirnov
(S72), Spearman (S25), Regres-
sion Analysis (S24)

perience in terms of (i) engagement when the target audience is
composed of students of a software engineering course; (ii) rec-
ollection when the audience involves developers understanding
legacy code [5]; and (iii) positive emotions in general.

We believe that effective software visualization approaches
must combine various complementary variables, which depend

on the objective of the visualization. That is, variables used
to explicitly define effectiveness relate to the domain problem
and the tasks required by a particular target audience. We think
that a deeper understanding of the mapping between users’ de-
sired variables to usage scenarios of visualization can bring in-
sights for defining quality metrics [4] in the software visualiza-
tion field.

The case in case studies. We classified twelve papers into the
case study category. In these papers, we identified a case that is
neither hypothetical nor a toy example, but a concrete context
that involves a real world system in which developers adopted
a visualization approach to support answering complex ques-
tions. In only one paper [S90] did we find a thorough evalua-
tion that describes the use of various research methods to col-
lect data such as questionnaires and interviews. In contrast, in
others we found less detail and no explicit description of the
methods employed to collect data. In particular, in three pa-
pers [S52,S114,S151] a reference was given to a paper that con-
tains more details. We observed that in studies in which authors
come from industry [S56,S90,S114] there are many details pro-
vided as part of the evaluation. In all of them, (i) users who
evaluated the proposed visualization approach were senior de-
velopers from industry, and (ii) the evaluation adopted a quali-
tative analysis. Case studies are often accused of lack of rigor
since biased views of participants can influence the direction
of the findings and conclusions [46]. Moreover, since they fo-
cus on a small number of subjects, they provide little basis for
generalization.

In summary, we reflect on the need for conducting more
case studies that can deliver insights into the benefits of soft-
ware visualization approaches, and highlight the compulsion of
identifying a concrete real-world case.

The scope of experiments in software visualization. Table 9
summarizes our extension to the framework proposed by Wohlin
et al. [45] to include key characteristics of software visualiza-
tions. We believe that the extended framework can serve as
a starting point for researchers who are planning to evaluate a
software visualization approach. Each row in the table can be
read as follows:

“Analyze [Object of study] executing in a [Environment] to
support the [Task] using a [Technique] displayed on a
[Medium] for the purpose of [Purpose] with respect to

[Quality Focus] from the point of view of [Perspective] in
the context of [Context].”

We used the framework to describe the scope of a recent
experiment of 3D visualization in immersive augmented real-
ity [20].

RQ2.) How appropriate are the evaluations that are conducted
to validate the effectiveness of software visualization?

Explicit goal of evaluations. We observed that studies often do
not explicitly specify the goal of the evaluation. They formulate
sentences such as “To evaluate our visualization, we conducted
interviews …” [S100]. We investigate what aspects of the vi-
sualization are evaluated. We reflect that a clear and explicit
formulation of the goal of the evaluation would help develop-
ers to assess if the evaluation provides them enough evidence
that support the claimed benefits of a proposed visualization
approach. Although in most studies we infer that the goal is
to evaluate the effectiveness of a visualization, in only a few
studies is there a definition of effectiveness. For instance, one
study [S131] defines effectiveness of a visualization in terms
of the number of statements that need to be read before identi-
fying the location of an error; however, we believe this defini-
tion suits better the definition of efficiency. Indeed, practitioners
will benefit from effective and efficient software visualization.
Nonetheless, we believe the game-changing attribute of a vi-
sualization resides in the user experience, for which multiple
variables should be included in evaluations (e.g., usability, en-
gagement, emotions).

Experiments’ tasks must be in-line with evaluations’ goal.
Software visualizations are proposed to support developers in
tasks dealing with multiple development concerns. A problem
thus arises for developers willing to adopt a visualization but
who need to match a suitable visualization approach to their
particular task at hand [24]. We investigate how suitable a vi-
sualization approach is for the tasks used in evaluations. We
reflect that proving a software visualization approach to be ef-
fective for tasks for which there exist other more appropriate
tools (but not included in the evaluation) can lead to misleading
conclusions. Since many evaluations included in our analysis
do not state an explicit goal, and some of the remaining ones
refer to rather generic terms (e.g., effectiveness, usability) with-
out providing a definition, understanding whether the tasks used

in experiments are in-line with the goals of evaluations is still
uncertain.

Beyond usage scenarios. Related work concluded that describ-
ing a case study is the most common strategy used to evaluate
software visualization approaches. Indeed, we found many pa-
pers that contain a section entitled case study; however, we ob-
served that most of them correspond to usage scenarios used
to demonstrate how the proposed visualization approach is ex-
pected to be useful. In all of them, the authors (who usually are
also developers) select a subject system and show how visual-
izations support a number of use cases. For example, one study
[S158] describes the presence of independent judges, but with-
out providing much detail about them. In the past, such a self-
evaluation, known as an assertion [48], has been used in many
studies, and is not considered an accepted research method for
evaluation [44]. Instead, we prefer to refer to them as usage sce-
narios (as they are called in many studies). This name has also
been adopted in the information visualization community [12],
and therefore its adoption in software visualization will ease
comparison across the two communities. Nonetheless, usage
scenarios do not represent solid evidence of the benefits of pro-
posed software visualization, and should be used only as a start-
ing point to adjust requirements, and improve an approach.

Surveys to collect software visualization requirements. We
observed that surveys are adequate to identifying requirements
for software visualizations. Through a survey, the problems that
arise in the development tasks carried out by a target audience
that involve a particular data set can be collected as assessed
as potential candidates for visualization. Then, researchers can
propose an approach that defines the use of a visualization tech-
nique displayed in a medium. We observed that a main threat
in software visualization is the disconnect between the develop-
ment concerns that are the focus of visualization, and the most
complex and frequent problems that arise during real-life de-
velopment.

Report on thorough experiments. Although formative eval-
uations can be useful at an early stage, evidence of the user
performance and user experience of a software visualization
approach should be collected via thorough experiments (when
variables included in the evaluation can be controlled). Exper-
iments should include participants of a random sample of the
target audience and real-world software systems. Experiments
should aim at reproducibility, for which open source software
projects are suitable. Moreover, open source projects boost
replicability of evaluations across systems of various charac-
teristics. The tasks used in experiments should be realistic, and
as already discussed, consistent with the goal of the evaluation,
otherwise conclusions can be misleading. Finally, we observed
that standardizing evaluations via benchmarks would promote
their comparison.

In summary, we observed that the main obstacles that pre-
vent researchers from doing more appropriate evaluations are
(i) the lack of a ready-to-use evaluation infrastructure, e.g., vi-
sualization tools to compare with; (ii) the lack of benchmarks

Table
9:T

he
evaluation

scope
ofexperim

entsin
softw

are
visualizations(left-to-right):reference,objectofstudy,task

(check
m

ark
X

identifiestasksthatw
ere

found
explicitin

evaluations),environm
ent,visualization

technique,m
edium

(i.e.,standard
com

puterscreens
SC

S,im
m

ersive
3D

environm
ents

I3D
,and

m
ultitouch

tables
M

T
T

),purpose,quality
focus,perspective,context,statisticaltest(acronym

s
show

n
in

Table
8).

R
ef.

O
bjectofStudy

Task
E

nv.
Technique

M
ed.

Purpose
Q

uality
Focus

Pers.
C

ontext
StatisticalTest

S5
U

M
L

diagram
notation

Identify
ifan

U
M

L
diagram

correspond
to

a
specification

–
U

M
L

SC
S

To
evaluate

w
hethera

specification
m

atches
a

diagram
C

orrectness,Tim
e

A
ll

35
C

S
students

T
5

S11
G

enisom
Search

forinform
ation

held
w

ithin
the

self-organizing
m

ap.
–

C
ity

SC
S

To
characterize

the
capability

ofusers
to

extractinform
ation

from
a

visual
C

orrectness,Tim
e,U

sability
A

ll
114

C
S

students
–

S13
X

ia
W

hy
a

particularfile
changed

–
N

ode-link
SC

S
To

testthe
initialrequirem

ents
E

ffectiveness,U
sability

A
ll

5
C

S
students

–
S21

Spreadsheets
L

ocalization
offaulty

cells
–

A
ug.source

code
SC

S
To

gain
insights

on
faulty

cells
in

spreadsheets
E

ffectiveness,R
obustness

N
ovice

87
C

S
students

T
14

S23
R

educing
C

ognitive
E

ffort
Tasks

related
to

distributed
com

putations
–

N
ode-link;Iconic

SC
S

To
evaluate

cognitive
econom

y
C

orrectness
N

ovice
20

C
S

students
(5

fem
ale)

–
S24

D
ancingH

am
sters;M

arbleSt.
Tasks

related
to

algorithm
analysis

–
A

nim
.N

ode-link
SC

S
To

evaluate
the

im
pactofvisualization

in
learning

C
orrectness

N
ovice

12
C

S
students;43

C
S

students
T

18
S25

A
lgorithm

visualization
Tasks

related
to

the
sorting

algorithm
s

–
A

ug.source
code

SC
S

To
evaluate

the
im

pactofvisualization
in

learning
C

orrectness
N

ovice
157

C
S

students
T

1,T
7,T

8,T
17

S32
G

row
ingSquares

Is
process

x
causally

related
tim

e
to

process
y?

X
–

N
ode-link;H

asse
SC

S
To

evaluate
the

im
pactofa

technique
C

orrectness,E
ffi

ciency,T.,…
A

ll
12

participants
(4

fem
ale)

T
1

S33
PlanA

ni
Tasks

related
to

sorting
algorithm

s
V

arious
A

ug.source
code

SC
S

To
gain

insights
on

supporting
teaching

program
m

ing
in

C
S

C
orrectness

N
ovice

91
C

S
students

–
S40

Variable
dependency

C
om

plete
an

unfinished
function

–
U

M
L

SC
S

To
evaluate

the
im

pactofofintra-proceduralvariable
dependencies

C
orrectness,Tim

e,U
sefulness

A
ll

38
C

S
students

(3
fem

ale)
T

1,T
2

S47
U

M
L

class
diagram

layout
M

atch
the

role
ofa

particularclass
–

U
M

L
SC

S
To

evaluate
the

im
pactofstereotype-based

architecturalU
M

L
layout

C
orrectness

A
ll

20
C

S
students

–
S50

W
ear-based

filtering
C

hange
the

program
to

obtain
an

expected
behavior

X
–

U
M

L
SC

S
To

evaluate
the

im
pactofusing

w
ear-based

filtering
C

om
pletion

A
ll

7
m

ale
developers

T
2

S61
A

lgorithm
visualization

Tasks
related

to
algorithm

analysis
–

N
ode-link

SC
S

To
evaluate

the
im

pactofusing
conceptkeyboards

Interactivity,U
sefulness

N
ovice

17
C

S
students;18

C
S

students
–

S66
Socialagents

W
hatfaults

did
you

find,and
w

hen
did

you
find

each
one?

–
Iconic

SC
S

To
evaluate

the
im

pactofthe
tool

E
ffectiveness

N
ovice

22
C

S
students

–
S71

jG
rasp

Find
and

correctallthe
non-syntacticalerrors

V
arious

A
ug.source

code
SC

S
To

gain
insights

on
supporting

teaching
program

m
ing

in
C

S
C

orrectness,Tim
e

N
ovice

–
T

15
S72

A
lgorithm

visualization
W

hatis
the

m
ain

difference
betw

een
Prim

and
D

ijkstra
algorithm

s?
X

–
N

ode-link
SC

S
To

evaluate
the

im
pactofthe

narrative
visualization

approach
C

orrectness
N

ovice
34

C
S

students
T

5,T
16

S78
G

illigan
A

nalyze
the

im
pactofchanging

a
program

.
X

–
A

ug.source
code

SC
S

To
evaluate

the
im

pactofa
tool

C
orrectness

A
ll

6
participants

T
9

S96
SolidFX

Tasks
related

to
reverse-engineering

open-source
code

W
indow

s
H

E
B

;Pixel
SC

S
To

gain
insights

on
architecture,m

etrics
and

dependencies
Perform

ance,U
serE

xperience
A

ll
8

participants
(ind.&

acad.)
–

S100
E

nhance
Find

dependencies
betw

een
structuralelem

ents
–

N
ode-link

SC
S

To
gain

insights
on

how
devs.understand

exception-handling
constructs

E
ffectiveness

N
ovice

3
C

S
students

–
S101

saU
M

L
Selectthe

candidate
thatbestdescribes

the
depicted

behavior
–

U
M

L
SC

S
To

evaluate
the

benefits
ofsynchronization-adorned

sequence
diagram

s
C

orrectness
N

ovice
24

C
S

students
T

4,T
5

S106
V

arious
Tasks

related
to

program
com

prehension
and

m
aintenance

–
V

arious
SC

S
To

evaluate
the

im
pactofa

tool
C

orrectness
A

ll
90

participants
(ind.&

acad.
–

S107
U

M
L

C
lass

diagram
Identify

classes
to

be
changed

to
add

a
requirem

ent
–

U
M

L
SC

S
To

evaluate
the

im
pactofthe

layout
C

onfidence,C
orrectness,Tim

e
A

ll
45

C
S

students
T

1-T
4,T

6
S108

V
ersion

Tree
vs

A
ugur

W
hich

classes
w

illbe
directly

affected
by

this
change?

X
–

N
ode-link

SC
S

To
gain

insights
on

the
benefits

ofvis.foropen
source

new
com

ers
C

orrectness
N

ovice
27

C
S

students
(9

fem
ales)

T
2,T

7
S112

U
M

L
C

lass
diagram

C
ountabstractclasses

to
see

ifproxies
are

distinguished
–

U
M

L
SC

S
To

testthe
initialrequirem

ents
E

ffectiveness
A

ll
8

C
S

stud.&
staff

(2
fem

ale)
–

S117
C

odeM
ap

Find
the

purpose
ofthe

given
application

X
–

Island
SC

S
To

gain
insighton

how
devs.interactw

ith
vis.thatare

em
bedded

in
the

ID
E

U
sability

A
ll

7
participants

(ind.&
acad.)

–
S118

ProfV
is

H
ow

the
program

can
be

m
odified

to
im

prove
its

perform
ance

X
Java

N
ode-link

SC
S

To
analyze

execution
traces

ofJava
program

s
C

orrectness,U
nderstanding

A
ll

4
participants

T
9

S123
A

llocR
ay

Find
the

location
ofa

m
em

ory
leak

–
Pixel

SC
S

To
evaluate

a
visualization

ofallocation
patterns

and
m

em
ory

problem
s

C
orrectness

A
ll

4
developers

–
S125

System
H

otspotsV
iew

H
ow

m
uch

biggeris
the

C
om

ponentclass
than

the
W

indow
class?

X
–

Polym
etric

view
s

SC
S

To
evaluate

visualization
rendered

on
a

w
alldisplay

C
orrectness,Tim

e
A

ll
11

par.(3
fem

.ind.&
acad.)

T
9

S126
StenchB

lossom
Identify

code
sm

ells
E

clipse
A

ug.source
code

SC
S

To
gain

insights
on

supporting
softw

are
quality

based
on

code
sm

ells
C

onfidence,C
orrectness

A
ll

12
participants

(ind.&
acad.

T
4

S127
Softw

are
dev.lifecycle

A
nalyze

the
context,and

roles
ofinvolved

people
in

projects
–

N
ode-link

SC
S

To
gain

insights
on

how
devs.draw

sketches
and

diagram
s

ofsoft.lifecycle
E

ffectiveness
A

ll
8

par.(C
S

stud.&
resear.)

–
S131

C
onstellation

Identify
the

location
in

the
code

ofa
fault

–
N

ode-link
SC

S
To

evaluate
a

technique
forsoftw

are
understanding

and
pattern

recognition
E

ffectiveness
A

ll
30

C
S

students
T

9
S135

E
-Q

uality
Selectthe

m
ostsignificantrefactoring

candidates
ofa

program
–

N
ode-link;Iconic

SC
S

To
gain

insights
on

visualization
ofdesign

flaw
s

and
refact.opportunities

Intuitiveness
A

ll
16

developers
–

S137
G

zoltar
Identify

the
location

in
the

code
ofa

fault
Java;E

cli.
Icicle;Treem

ap
SC

S
To

gain
insights

on
faultlocalization

fordebugging
Java

progs.
C

orrectness,Intuit.,Tim
e,…

A
ll

40
C

S
students

T
5

S140
PN

LV
;IM

M
V

W
hatinteresting

visualstructures
do

you
find

in
the

vis.?
X

–
N

ode-link
SC

S
To

gain
insights

on
visualization

forunderstanding
an

unknow
n

system
U

sefulness
A

ll
8

participants
(ind.&

acad.)
–

S141
SourceV

is
H

ow
m

any
interfaces

does
this

class
depend

on?
X

–
Polym

etric
view

s
M

M
T

To
gain

insights
on

vis.on
m

ultitouch
tab.forco-located

collab.in
unders

E
ffectiveness

A
ll

6
par.(C

S
stud.&

resear.)
T

9
S144

SeeIT
3D

Identify
w

hy
the

program
produce

a
poorprintquality

X
E

clipse
C

ity
SC

S
To

gain
insights

on
visualization

forarchitecture
ofJava

system
s

C
orr.,Tim

e,V
isualE

ffort
A

ll
97

C
S

students
T

1,T
18

S148
C

hronoTw
igger

Investigate
the

softw
are

w
hile

thinking
outloud

–
N

ode-link
I3D

To
evaluate

visualization
ofthe

developm
entprocess

and
testing

E
ffectiveness

A
ll

3
developers

(1
fem

ale)
–

S149
regV

IS
Track

ofthe
O

verallC
ontrolFlow

X
W

indow
s

V
isuallanguage

SC
S

To
gain

insights
on

supporting
assem

blercontrol-flow
ofregularexpr.

U
sability

A
ll

10
par.(C

S
stud.&

resear.)
–

S150
C

om
piler

M
essages

Identify
the

cause
ofan

errorby
analyzing

highlighted
elem

ents
X

–
A

ug.source
code

SC
S

To
evaluate

a
technique

to
aid

devs.on
com

prehending
errorm

essages
C

orrectness,R
ecollection

N
ovice

28
C

S
students

T
2-T

4
S153

SIFE
I

Find
a

failure
and

specify
a

testscenario
foritX

E
xcel

V
isuallanguage

SC
S

To
testspreadsheets

form
ulas

U
sability,U

sefulness
A

ll
9

participants
(ind.&

acad.)
–

S154
TiledG

race
D

escribe
the

behaviorofa
program

X
W

eb
V

isuallanguage
SC

S
To

gain
insights

on
supporting

program
m

ing
in

the
G

race
language

E
ngagem

ent,U
sefulness

N
ovice

33
C

S
students

T
9

S157
C

odeSurveyor
R

ank
the

code
m

aps
thatbestrepresentthe

codebase
X

–
Island

SC
S

To
evaluate

the
supportofcode

m
aps

in
learning

and
navigating

system
s

E
ffectiveness

A
ll

5
developers

(1
fem

ale)
–

S162
E

xplorV
iz

W
hatis

the
purpose

ofthe
W

W
W

PR
IN

T
application

in
youropinion?

X
W

eb
C

ity
I3D

To
evaluate

an
architecture

based
on

m
etric

analysis
C

orrectness,Tim
e

A
ll

25
C

S
students

T
5,T

6,T
10

S164
V

IM
E

T
R

IK
W

hatis
the

num
berofcom

pilation
units

in
the

Tom
catsystem

?
X

–
C

ity;M
atrix

SC
S

To
evaluate

m
etrics

and
vis.ofa

softw
are

system
according

to
requirem

ents
C

orrectness,Intuit.,Tim
e,…

A
ll

21
C

S
students

T
1,T

4,T
9

S166
W

orking
Sets

A
nalyze

the
developeractivity

on
entities

ofthe
w

orking
sets

–
N

ode-link
SC

S
To

gain
insights

on
visualization

ofthe
evolution

ofw
orking

sets
E

ffectiveness
A

ll
14

developers
–

S168
C

uboidM
atrix

Identify
identicalinteractions,along

tim
e,betw

een
classes?

X
–

Space-tim
e

cube
SC

S
To

evaluate
the

im
pactofthe

toolin
softw

are
com

prehension
C

orrectness
A

ll
8

par.(C
S

stud.&
resear.)

T
8

S169
Perquim

ans
W

hich
sheets

contained
the

m
ostdangerous

form
ula

practice
X

–
N

ode-link
SC

S
To

gain
insights

ofvis.tool’s
supportofexploration,and

quantification
A

pplicability
A

ll
4

C
S

students
–

S173
Indented

H
ierarchy

Find
the

m
ostsym

m
etric

subtree
in

the
tree.

X
–

Pixel
SC

S
To

evaluate
the

im
pactofa

technique
com

pared
w

ith
node-link

diagram
s

C
orrectness,R

eadability,Tim
e

A
ll

18
vis.experts

(3
fem

ale)
T

9
S174

R
egex

textualvs
graphical

Is
A

B
C

in
the

language
defined

by
a

regularexpression?
X

–
A

ug.source
code

SC
S

To
testthe

im
pactofa

graphicalnotation
on

the
readability

ofa
regex

C
orrectness,R

eadability,Tim
e

N
ovice

22
par.(C

S
stud.and

staff)
T

1,T
12,T

13
S177

C
ode

Park
Identify

w
here

in
the

code
add

the
logic

to
supporta

feature
X

–
C

ity
I3D

To
evaluate

the
im

pactofthe
toolon

usability
and

engagem
ent

E
ase-to-use,E

ngagem
ent,Tim

e
N

ovice
28

C
S

students
(6

fem
ale)

T
1,T

6
S179

iTraceV
is

W
here

did
the

dev.notlook
atw

ith
respectto

the
sam

e
m

ethod?
X

E
clipse

H
eatm

ap
SC

S
To

gain
insights

on
analyzing

eye
m

ovem
entdata

ofcode
reading

C
orrectness

A
ll

10
C

S
students

T
9

S180
C

ityV
R

L
ocate

the
bestcandidate

forthe
god

class
sm

ellX
Pharo;U

.
C

ity
SC

S
To

gain
insights

on
visualization

forarchitecture
based

on
m

etrics
in

O
O

P
C

orrectness,R
ecollection,T.,…

A
ll

21
participants

T
1,T

6,T
7,T

10,T
11

S181
M

ethodE
xecutionR

eports
Tasks

related
to

execution
reportforprofiling

and
debugging

X
Java

C
harts

SC
S

To
gain

insights
on

supporting
sum

m
arization

ofm
ethods

execution
U

nderstandability,U
sefulness

A
ll

11
participants

(ind.&
acad.

–

http://www.cs.uef.fi/~saja/var_roles/planani/index.html

http://www.jgrasp.org/

http://www.solidsourceit.com/products/SolidFX-static-code-analysis.html

http://ftaiani.ouvaton.org/7-software/profvis.html

https://github.com/DeveloperLiberationFront/refactoring-tools/tree/master/installables/update_sites/stench_blossom

http://www.gzoltar.com

https://github.com/davidmr/seeit3d

http://www.sts.tu-harburg.de/projects/regvis/regvis.html

https://github.com/kuleszdl/SIFEI

http://homepages.ecs.vuw.ac.nz/~mwh/

https://www.explorviz.net

https://github.com/SERESLab/iTrace-Archive

http://scg.unibe.ch/research/cityvr

https://github.com/fabian-beck/Method-Execution-Reports

that ease comparison across tools, e.g., quality metrics; (iii) the
tradeoff between the effort of conducting comprehensive eval-
uations and little added value to paper acceptance; and (iv) the
difficulties to involve industrial partners willing to share re-
sources, e.g., include participants of the target audience.

6.1. Threats to Validity

Construct validity. Our research questions may not provide
complete coverage of software visualization evaluation. We
mitigated this threat by including questions that focus on the
two main aspects that we found in related work: (1) characteri-
zation of the state-of-the-art, and (2) appropriateness of adopted
evaluations.

Internal validity. We included papers from only two venues,
and may have missed papers published in other venues that re-
quire more thorough evaluations. We mitigated this threat by
identifying relevant software visualization papers that ensure
an unbiased paper selection process. Therefore, we selected
papers from the most frequently cited venue dedicated to soft-
ware visualization: SOFTVIS/VISSOFT. We argue that even
if we would have included papers from other venues the trend
of the results would be similar. Indeed, related work did not
find important differences when comparing software visualiza-
tion evaluation in papers published in SOFTVIS/VISSOFT to
papers published in other venues [19, 38]. Moreover, our re-
sults are in line with the conclusions of related work that have
included papers from multiple venues [16, 30, 39]. We also
mitigated the paper selection bias by selecting peer-reviewed
full papers. We assessed the quality of these papers by exclud-
ing model papers (i.e., commentary, formalism, taxonomy) that
are less likely to include an evaluation. However, since soft-
ware visualization papers do not specify their types, we may
have missed some. We mitigated this threat by defining a cross-
checking procedure and criteria for paper type classification.

External validity. We selected software visualization papers
published between 2002 to 2017 in SOFTVIS/VISSOFT. The
excluded papers from other venues or published before 2002
may affect the generalizability of our results.

Conclusion validity. Bias in the data collection procedure
could obstruct reproducibility of our study. We mitigated this
threat by establishing a protocol to extract the data of each pa-
per equally, and by maintaining a spreadsheet to keep records,
normalize terms, and identify anomalies.

7. Conclusion

We reviewed 181 full papers of the 387 that were published
to date in the SOFTVIS/VISSOFT conferences. We extracted
evaluation strategies, data collection methods and other vari-
ous aspects of evaluations. We found that 62% (i.e., 113) of
the proposed software visualization approaches do not include
a strong evaluation. We identified several pitfalls that must
be avoided in the future of software visualization: (i) evalu-
ations with fuzzy goals (or without explicit goals), for which
the results are hard to interpret; (ii) evaluations that pursue ef-
fectiveness without defining it, or that limit the assessment to

time, correctness (user performance) and usability (user expe-
rience) while disregarding many other variables that can con-
tribute to effectiveness (e.g., recollection, engagement, emo-
tions); (iii) experiment tasks that are inconsistent with the stated
goal of the evaluation; (iv) lack of surveys to collect require-
ments that explain the disconnect between the problem domains
on which software visualization have focused and the domains
that get the most attention from practitioners; and (v) lack of
rigor when designing, conducting, and reporting on evaluation.

We call researchers in the field to collect evidence of the
effectiveness of software visualization approaches by means of
(1) case studies (when there is a case that must be studied in
situ), and (2) experiments (when variables can be controlled)
including participants of a random sample of the target audi-
ence and real-world open source software systems that promote
reproducibility and replicability.

We believe that our study will help (a) researchers to re-
flect on the design of appropriate evaluations for software vi-
sualization, and (b) developers to be aware of the evidence that
supports the claims of benefit of the proposed software visu-
alization approaches. We plan in the future to encapsulate the
characterization and insights from this study in a software vi-
sualization ontology that will allow developers to find suitable
visualizations for development concerns as well as researchers
to reflect on the domain.

Acknowledgments

We gratefully acknowledge the financial support of the Swiss
National Science Foundation for the project “Agile Software
Analysis” (SNSF project No. 200020-162352, Jan 1, 2016 –
Dec. 30, 2018). Merino has been partially funded by CONI-
CYT BCH/Doctorado Extranjero 72140330.

References

[1] Alves, V., Niu, N., Alves, C., Valença, G., 2010. Requirements engineer-
ing for software product lines: A systematic literature review. Information
and Software Technology 52 (8), 806–820.

[2] Amar, R., Stasko, J., 2004. A knowledge task-based framework for de-
sign and evaluation of information visualizations. In: Proc. of INFOVIS.
IEEE, pp. 143–150.

[3] Bassil, S., Keller, R., 2001. Software visualization tools: survey and anal-
ysis. In: Proc. of IWPC. pp. 7 –17.

[4] Bertini, E., Tatu, A., Keim, D., 2011. Quality metrics in high-dimensional
data visualization: An overview and systematization. Transactions on Vi-
sualization and Computer Graphics 17 (12), 2203–2212.

[5] Bloom, B. S., et al., 1956. Taxonomy of educational objectives. vol. 1:
Cognitive domain. New York: McKay, 20–24.

[6] Elmqvist, N., Yi, J. S., 2015. Patterns for visualization evaluation. Proc.
of INFOVIS 14 (3), 250–269.

[7] Fink, A., 2003. The survey handbook. Vol. 1. Sage.
[8] Forsell, C., 2010. A guide to scientific evaluation in information visual-

ization. In: Proc. of IV. IEEE, pp. 162–169.
[9] Greene, G. J., Esterhuizen, M., Fischer, B., 2017. Visualizing and explor-

ing software version control repositories using interactive tag clouds over
formal concept lattices. Information and Software Technology 87, 223–
241.

[10] Hornbæk, K., Høegh, R. T., Pedersen, M. B., Stage, J., 2007. Use case
evaluation (UCE): A method for early usability evaluation in software
development. In: Proc. of IFIP. Springer, pp. 578–591.

[11] Höst, M., Regnell, B., Wohlin, C., 2000. Using students as subjects —
a comparative study of students and professionals in lead-time impact
assessment. Empirical Software Engineering 5, 201–214.

[12] Isenberg, T., Isenberg, P., Chen, J., Sedlmair, M., Möller, T., 2013. A
systematic review on the practice of evaluating visualization. Transactions
on Visualization and Computer Graphics 19 (12), 2818–2827.

[13] Kienle, H. M., Müller, H. A., 2010. The tools perspective on software
reverse engineering: requirements, construction, and evaluation. In: Ad-
vances in Computers. Vol. 79. Elsevier, pp. 189–290.

[14] Kitchenham, B. A., Pfleeger, S. L., Pickard, L. M., Jones, P. W., Hoaglin,
D. C., Emam, K. E., Rosenberg, J., 2002. Preliminary guidelines for em-
pirical research in software engineering. IEEE Trans. Softw. Eng. 22 (8),
721–734.

[15] Lam, H., Bertini, E., Isenberg, P., Plaisant, C., Carpendale, S., 2012. Em-
pirical studies in information visualization: Seven scenarios. Transactions
on Visualization and Computer Graphics 18 (9), 1520–1536.

[16] Lopez-Herrejon, R. E., Illescas, S., Egyed, A., 2018. A systematic map-
ping study of information visualization for software product line engi-
neering. Journal of Software: Evolution and Process 30 (2).

[17] Mackinlay, J., 1986. Automating the design of graphical presentations of
relational information. Transactions On Graphics 5 (2), 110–141.

[18] Maletic, J. I., Marcus, A., 2003. CFB: A call for benchmarks-for software
visualization. In: Proc. of VISSOFT. Citeseer, pp. 113–116.

[19] Mattila, A.-L., Ihantola, P., Kilamo, T., Luoto, A., Nurminen, M., Väätäjä,
H., 2016. Software visualization today: systematic literature review. In:
Proc. of International Academic Mindtrek Conference. ACM, pp. 262–
271.

[20] Merino, L., Bergel, A., Nierstrasz, O., 2018. Overcoming issues of 3D
software visualization through immersive augmented reality. In: Proc. of
VISSOFT. IEEE, p. in review.

[21] Merino, L., Fuchs, J., Blumenschein, M., Anslow, C., Ghafari, M., Nier-
strasz, O., Behrisch, M., Keim, D., 2017. On the impact of the medium
in the effectiveness of 3D software visualization. In: Proc. of VISSOFT.
IEEE, pp. 11–21.
URL http://scg.unibe.ch/archive/papers/Meri17b

[22] Merino, L., Ghafari, M., Anslow, C., Nierstrasz, O., 2017. CityVR:
Gameful software visualization. In: Proc. of ICSME. IEEE, pp. 633–637.
URL http://scg.unibe.ch/archive/papers/Meri17c

[23] Merino, L., Ghafari, M., Nierstrasz, O., 2016. Towards actionable visual-
isation in software development. In: Proc. of VISSOFT. IEEE.
URL http://scg.unibe.ch/archive/papers/Meri16a

[24] Merino, L., Ghafari, M., Nierstrasz, O., Bergel, A., Kubelka, J., 2016.
MetaVis: Exploring actionable visualization. In: Proc. of VISSOFT.
IEEE.
URL http://scg.unibe.ch/archive/papers/Meri16c

[25] Merino, L., Lungu, M., Nierstrasz, O., 2015. Explora: A visualisation tool
for metric analysis of software corpora. In: Proc. of VISSOFT. IEEE, pp.
195–199.
URL http://scg.unibe.ch/archive/papers/Meri15b

[26] Merino, L., Seliner, D., Ghafari, M., Nierstrasz, O., 2016. Community-
Explorer: A framework for visualizing collaboration networks. In: Proc.
of IWST. pp. 2:1–2:9.
URL http://scg.unibe.ch/archive/papers/Meri16b

[27] Müller, R., Kovacs, P., Schilbach, J., Eisenecker, U. W., Zeckzer, D.,
Scheuermann, G., 2014. A structured approach for conducting a series
of controlled experiments in software visualization. In: Proc. of IVAPP.
IEEE, pp. 204–209.

[28] Munzner, T., 2008. Process and pitfalls in writing information visualiza-
tion research papers. In: Information visualization. Springer, pp. 134–
153.

[29] Munzner, T., 2014. Visualization analysis and design. CRC press.
[30] Novais, R. L., Torres, A., Mendes, T. S., Mendonça, M., Zazworka, N.,

2013. Software evolution visualization: A systematic mapping study. In-
formation and Software Technology 55 (11), 1860–1883.

[31] Panas, T., Epperly, T., Quinlan, D., Sæbjørnsen, A., Vuduc, R., 2016.
Comprehending software architecture using a unified single-view visual-
ization. In: Antonakos, J. L. (Ed.), Data Structure and Software Engineer-
ing: Challenges and Improvements. CRC Press, Ch. 2, pp. “22–41”.

[32] Razali, N. M., Wah, Y. B., et al., 2011. Power comparisons of shapiro-
wilk, kolmogorov-smirnov, lilliefors and anderson-darling tests. Journal
of Statistical Modeling and Analytics 2 (1), 21–33.

[33] Runeson, P., Höst, M., 2009. Guidelines for conducting and reporting case
study research in software engineering. Empirical Software Engineering
14 (2), 131.

[34] Schots, M., Vasconcelos, R., Werner, C., 2014. A quasi-systematic review
on software visualization approaches for software reuse. Technical report.

[35] Schots, M., Werner, C., 2014. Using a task-oriented framework to charac-
terize visualization approaches. In: Proc. of VISSOFT. IEEE, pp. 70–74.

[36] Sensalire, M., Ogao, P., Telea, A., 2008. Classifying desirable features
of software visualization tools for corrective maintenance. In: Proc. of
SOFTVIS. ACM, pp. 87–90.

[37] Sensalire, M., Ogao, P., Telea, A., 2009. Evaluation of software visual-
ization tools: Lessons learned. In: Proc. of VISSOFT. IEEE, pp. 19–26.

[38] Seriai, A., Benomar, O., Cerat, B., Sahraoui, H., Sep. 2014. Validation
of software visualization tools: A systematic mapping study. In: Proc. of
VISSOFT. pp. 60–69.

[39] Shahin, M., Liang, P., Babar, M. A., 2014. A systematic review of soft-
ware architecture visualization techniques. Journal of Systems and Soft-
ware 94, 161–185.

[40] Sjøberg, D. I., Hannay, J. E., Hansen, O., Kampenes, V. B., Kara-
hasanovic, A., Liborg, N.-K., Rekdal, A. C., 2005. A survey of controlled
experiments in software engineering. Transactions on Software Engineer-
ing 31 (9), 733–753.

[41] Storey, M.-A. D., Čubranić, D., German, D. M., 2005. On the use
of visualization to support awareness of human activities in software
development: a survey and a framework. In: Proc. of SOFTVIS. ACM
Press, pp. 193–202.
URL http://portal.acm.org/citation.cfm?id=1056018.
1056045

[42] Van Wijk, J. J., 2006. Views on visualization. Transactions on Visualiza-
tion and Computer Graphics 12 (4), 421–432.

[43] Wobbrock, J. O., Findlater, L., Gergle, D., Higgins, J. J., 2011. The
aligned rank transform for nonparametric factorial analyses using only
anova procedures. In: Proc. of SIGCHI. ACM, pp. 143–146.

[44] Wohlin, C., Runeson, P., Höst, M., Ohlsson, M. C., Regnell, B., Wesslén,
A., 2000. Experimentation in Software Engineering. Kluwer Academic
Publishers.

[45] Wohlin, C., Runeson, P., Höst, M., Ohlsson, M. C., Regnell, B., Wesslén,
A., 2012. Experimentation in software engineering. Springer Science &
Business Media.

[46] Yin, R. K., 2013. Case study research: Design and methods. Sage publi-
cations.

[47] Young, P., Munro, M., 1998. Visualising software in virtual reality. In:
Proc. of IWPC. IEEE, pp. 19–26.

[48] Zelkowitz, M. V., Wallace, D. R., 1998. Experimental models for validat-
ing technology. Computer 31 (5), 23–31.

http://scg.unibe.ch/archive/papers/Meri17b

http://scg.unibe.ch/archive/papers/Meri17c

http://scg.unibe.ch/archive/papers/Meri16a

http://scg.unibe.ch/archive/papers/Meri16c

http://scg.unibe.ch/archive/papers/Meri15b

http://scg.unibe.ch/archive/papers/Meri16b

http://portal.acm.org/citation.cfm?id=1056018.1056045

Turn in your highest-quality paper
Get a qualified writer to help you with

“ SLR Paper – Analyzing & Visualizing Data ”

Get high-quality paper

NEW! AI matching with writer

Hire a Writer

Client Reviews

4.9

Sitejabber

4.6

Trustpilot

4.8

Our Guarantees

100% Confidentiality

Information about customers is confidential and never disclosed to third parties.

Original Writing

We complete all papers from scratch. You can get a plagiarism report.

Timely Delivery

No missed deadlines – 97% of assignments are completed in time.

Money Back

If you're confident that a writer didn't follow your order details, ask for a refund.

New to Your Trusted Assignment Help Service? Sign up & Save

Calculate the price of your order

Type of paper needed:

Pages:

You will get a personal manager and a discount.

Academic level:

We'll send you the first draft for approval by at

Total price:

$0.00

Power up Your Academic Success with the
Team of Professionals. We’ve Got Your Back.

Power up Your Study Success with Experts We’ve Got Your Back.

Order Now Order Now