Home Uncategorized write a 2-3 page (that means a minimum of 2 full pages and no longer than 3 full pages) paper (in APA format)

Uncategorized

write a 2-3 page (that means a minimum of 2 full pages and no longer than 3 full pages) paper (in APA format)

2. Find ONE peer-reviewed article in EACH of the above three article types about your topic (so three articles total- one study article, one literature review article, one meta analysis article).

3. For each article, describe the following:

Purpose of the article
Format of the article (i.e. length, subheadings, etc.)
What did you learn about your topic from this kind of article?

4. Finally, address the following:

Which article seems the most informative about your topic overall?
Which article was easiest to read? Which was the most difficult?

I attach the three articles and more instruction on the assignment in the attachment box.

Full Terms & Conditions of access and use can be found at
https://www.tandfonline.com/action/journalInformation?journalCode=imif20

Informatics for Health and Social Care

ISSN: (Print) (Online) Journal homepage: https://www.tandfonline.com/loi/imif20

Patient-centered care via health information
technology: a qualitative study with experts from
Israel and the U.S.

Maxim Topaz , Ofrit Bar-Bachar , Hanna Admi , Yaron Denekamp & Eyal
Zimlichman

To cite this article: Maxim Topaz , Ofrit Bar-Bachar , Hanna Admi , Yaron Denekamp & Eyal
Zimlichman (2020) Patient-centered care via health information technology: a qualitative study
with experts from Israel and the U.S., Informatics for Health and Social Care, 45:3, 217-228, DOI:
10.1080/17538157.2019.1582055

To link to this article: https://doi.org/10.1080/17538157.2019.1582055

Published online: 27 Mar 2019.

Submit your article to this journal

Article views: 127

View related articles

View Crossmark data

Citing articles: 1 View citing articles

https://www.tandfonline.com/action/journalInformation?journalCode=imif20

https://www.tandfonline.com/loi/imif20

https://www.tandfonline.com/action/showCitFormats?doi=10.1080/17538157.2019.1582055

https://doi.org/10.1080/17538157.2019.1582055

https://www.tandfonline.com/action/authorSubmission?journalCode=imif20&show=instructions

https://www.tandfonline.com/doi/mlt/10.1080/17538157.2019.1582055

http://crossmark.crossref.org/dialog/?doi=10.1080/17538157.2019.1582055&domain=pdf&date_stamp=2019-03-27

https://www.tandfonline.com/doi/citedby/10.1080/17538157.2019.1582055#tabModule

Patient-centered care via health information technology:
a qualitative study with experts from Israel and the U.S.
Maxim Topaz*a,b, Ofrit Bar-Bachara, Hanna Admic, Yaron Denekampd, and Eyal Zimlichmane

aThe Cheryl Spencer Department of Nursing, Faculty of Social Welfare and Health Science, University of Haifa, Haifa,
Israel; bGeneral Medicine, Harvard Medical School & Brigham and Women’s Hospital, Boston, MA, USA; cGeneral
Medicine, Rambam Health Care Campus, Haifa, Israel; dHealth Information Technology, Clalit Health Services, Tel
Aviv, Israel; eSheba Medical Center, Ramat Gan, Israel

ABSTRACT
Although patient-centered care (PCC) is one of the cornerstones of modern
healthcare, the role that health information technology (HIT) plays in sup-
porting PCC remains unclear. In this qualitative study, we interviewed
academic and clinical experts from the US and Israel to understand to
what extent current HIT systems are supportive of PCC and how PCC should
be supported by HIT in the future. A maximum variation sampling approach
was used to identify nine experts in both HIT and PCC from clinical and
academic settings in Israel and the US. A qualitative descriptive method was
used to analyze the interviews and identify major themes. Experts sug-
gested that patient ownership of their disease is a core component of
PCC. The majority of the experts agreed that in both Israel and the US,
the current situation of PCC implementation is relatively poor. However, HIT
should play an important role in making patients owners of their health and
treatment and helping providers in delivering better PCC. Central domains
of PCC via HIT were providing clear information and support for patients
and promoting care that is based on patient values and preferences.

KEYWORDS
Patient-centered care;
quality of care; health
informatics; electronic
health record; qualitative
descriptive study

Introduction

Over the past few decades, there were several fundamental changes that shaped the way healthcare is
viewed and practiced internationally. On one hand, healthcare has become more patient-centered
with ever-increasing attention to patient attitudes, values, and preferences as an integral part of
health services delivery. Another major change is the massive introduction of health information
technology used today by many health-care providers in their everyday work, for example the
electronic health records.

Patient-centered care

Patient-centered care (PCC) is defined by the Institute of Medicine as “providing care that is
respectful of and responsive to individual patient preferences, needs, and values, and ensuring that
patient values guide all clinical decisions”.1 In the United States (US), PCC was included as one of
the six domains of quality by the groundbreaking Institute of Medicine report titled “Crossing the
Quality Chasm: A New Health System for the 21st Century”.1 In Israel, providing PCC was also
prioritized at the national level.2 Several key principles guiding the provision of PCC were developed

CONTACT Maxim Topaz mtopaz80@gmail.com The Cheryl Spencer Department of Nursing, Faculty of Social Welfare and
Health Science, University of Haifa, Abba Khoushy Ave 199, Haifa 3498838, Israel.

*The corresponding author has since changed his affiliation to: Columbia University School of Nursing, 560 W 168th St, New York,
NY 10032, USA Phone: +972-53-919-3777
© 2019 Taylor & Francis Group, LLC

INFORMATICS FOR HEALTH & SOCIAL CARE
2020, VOL. 45, NO. 3, 217–228
https://doi.org/10.1080/17538157.2019.1582055

https://crossmark.crossref.org/dialog/?doi=10.1080/17538157.2019.1582055&domain=pdf&date_stamp=2020-07-30

internationally during the past decades, for example the eight principles of PCC introduced by the
Picker Institute and The Commonwealth Fund.3

Over the years PCC has become one of the cornerstones of modern healthcare in many countries.
Multiple studies have shown that the practice of PCC leads to better outcomes, for example
decreasing the rates of depression remission,4 achieving better blood glucose levels among diabetes
patients,5 increasing patients’ levels of physical activity,6 among many others.7 PCC is recognized as
an important aspect of care in both inpatient and outpatient settings.8

Even though awareness for PCC and its benefits is growing, the integration of PCC into daily
clinical practice sometimes remains less than satisfactory. For example, our team has conducted
a series of studies that examined the extent to which patient preferences are taken into consideration
by doctors and nurses in four different countries (US, Denmark, United Kingdom, and Israel). We
found that health providers are rarely eliciting patient preferences and infrequently act according to
patient expectations to improve satisfaction with care.9,10

Health information technology and patient-centered care

Health information technology (HIT) has become increasingly prevalent in modern healthcare. For
example, the adoption of electronic health records (EHRs) has peaked over the last decade. In the
US, the Meaningful Use legislative initiative required that all health-care settings use EHRs by 2016,
leading to an unprecedented spike in hospital EHR adoption rates from 15% in 2010 to 96% in
2015.11 In contrast in some other countries, like Israel, EHRs have been implemented in many
hospitals and outpatient settings for a longer period of time of about two decades.12

HIT is envisioned to enable effective PCC13–16, however in practice, we found few studies that
examined the role of technology in PCC.17,18 Moreover, an emerging body of evidence suggests that
some HIT functionalities might hinder PCC.19–23 For example, in our recent survey, nurse informa-
ticians from 45 countries reported low satisfaction with EHRs and identified many ways in which
EHRs serve as a barrier to providing high-quality PCC.24 One-fourth of the nurses reported that
patient information cannot be shared between the EHR systems (which is a core requirement for
PCC), while one-third of the nurses complained on poor system usability (making extraction of
patient information very hard). Similar concerns were reported about limited EHRs’ ability to
facilitate care transitions,20 extensive documentation time hindering patient-provider contact,21,22

and limited EHR usability that prevents providers from getting patient information they need.23

Today, average health-care providers spend a staggering 20–40% of their clinical time on EHRs
use21,22 yet very little is known about the extent to which this had impacted PCC.

Unanswered questions about PCC and EHRs

Although PCC is one of the cornerstones of modern healthcare, the role that HIT plays in
supporting PCC remains unclear. In this qualitative study, we interviewed academic and clinical
experts from the US and Israel to understand to what extent current HIT systems are supportive of
PCC. In our interviews, we also explored the key domains of PCC that should be supported by HIT
in the near future. We have chosen the two countries because we wanted to examine PCC in two
different health-care systems. In Israel, the healthcare system is universal and it is centralized around
the major payer for health services – the Ministry of Health. The Ministry of Health distributes funds
to four health maintenance organizations who in turn provide health services to the Israeli popula-
tion. In Israel, several HIT systems, like electronic health records and regional/national health
information exchanges, were introduced into practice more than two decades ago.25 In comparison,
the US healthcare system is decentralized and there is no universal payer for health services. Instead,
about half of healthcare spending comes from private funds (households or private businesses), the
federal government contributes about 30% and the rest of healthcare is paid for by the state and local
governments.26 Most US healthcare is delivered privately, even if it is publicly financed. Although

218 M. TOPAZ ET AL.

several hospitals in the US were pioneers in HIT, most of the US HIT was introduced into practice at
the national scale during the past decade. For example, government incentives increased the
adoption rate of electronic health records by the US hospitals from less than 25% in 2008 to more
than 96% adoption by 2016.11 Examining PCC in the context of HIT in two different health-care
systems has the potential to uncover important aspects that might have been missed if we were to
limit the study to any one particular country. For the purposes of this study, we broadly defined HIT
as any technology that is used by health-care providers, patients, and their families during engage-
ment in health-care services.

Methods

Conceptual framework

Our conceptual framework is based on the Picker Principles of Patient-Centered Care first intro-
duced in the mid-1990s.27 Using a wide range of focus groups (with recently discharged patients,
family members, physicians, and non-physician hospital staff) combined with a review of recent
literature, researchers from the Harvard Medical School, on behalf of Picker Institute and The
Commonwealth Fund, defined eight primary dimensions of PCC. These dimensions included: (1)
Respect for patient’s values, preferences and expressed needs; (2) Coordination and integration of
care; (3) Information, communication and education; (4) Physical comfort; (5) Emotional support
and alleviation of fear and anxiety; (6) Involvement of family and friends; (7) Transition and
continuity; and (8) Access to care. Picker Principles of Patient-Centered Care have directed us in
constructing this study’s interview guide and identifying codes for thematic analysis of the study
findings.

Study sample

In our sample size estimation, we used an approach aimed at maximizing the study’s “information
power”.28 In general, the larger the information power of the sample, the lower the number of
participants needed. Factors affecting information include study aim, sample specificity, use of
established theory, quality of dialogue, and analysis strategy. In our case, the least amount of
participants was needed given the following considerations: the study’s aim was specific; the combina-
tion of participants was highly specific for the study aim; the analysis was supported by an established
conceptual framework; the interview dialogue was strong (there was a clear communication between
the researcher and study participants); and the data analytics approach was explicit (thematic analy-
sis) .28

We used a maximum variation sampling approach to identify experts in both HIT and PCC from
clinical and academic settings in Israel and the US. Study experts were identified via a combination
of literature search and clinical position seniority in major clinical institutions in both countries. The
final sample of nine experts was well balanced in terms of representation of clinical and academic
settings as well as representation from both countries (see Table 1). This study received an
Institutional Review Board (IRB) approval from the University of Haifa, Israel (#355/17).

Data collection

The interview guide was developed by the study team based on the study goals and the conceptual
framework (Picker Principles of Patient-Centered Care). Our first set of interview questions was focused
on exploring the experts’ definition of PCC and their view of PCC in general and in the context of HIT.
Next, we asked the experts about the current state of PCC in the setting they are familiar with, and
whether and how the principles of PCC are implemented in HIT systems. Lastly, experts were asked
about their “ideal world” vision for HIT to support PCC. All the interviews were conducted by the

INFORMATICS FOR HEALTH AND SOCIAL CARE 219

primary investigator on this study (MT), who is an experienced qualitative researcher. Five interviews
were conducted in person and four interviews were conducted on the phone.

Each of the study experts provided consent to participate in the study and completed a semi-
structured interview with the study team based on the interview guide. All interviews were
audio recorded and transcribed verbatim. The interviews lasted between one and a half to two
hours.

Data analysis

To analyze the expert interviews in this study, we used thematic analysis, a qualitative descriptive
approach for identifying, analyzing, and reporting themes within data.29 Analysis phases were:

(1) Familiarizing with data: The study team has familiarized themselves with the interviews by
listening to the recordings, reading the transcribed data and noting down initial ideas.

(2) Generating initial codes: An initial coding scheme was then generated based on the study’s
conceptual framework, the Picker Institute’s Eight Principles of Patient-Centered Care.

(3) Data coding: All coding was done using an RDQA software (R package for Qualitative Data
Analysis).30 Two researchers trained in qualitative analysis (MT and OB) independently
coded the transcribed interviews. The codes were assigned to informative features of the
data (phrases, sentences or paragraphs).

(4) Inter-coder reliability was assured by dual coding of the first three interviews; after each
interview, the coding was reviewed for similarities and variations by comparing the level of
agreement between the two coders. Discrepancies were discussed. The inter-coder reliability
was determined to be greater than 95% agreement and each researcher continued and coded
the remaining six interviews separately. When new codes emerged, they were discussed
between the two coders and added to the coding scheme, when necessary. The two coders
agreed on all final coding and these results were shared with the study team.

(5) Searching for themes: Codes were collated into potential themes, which helped organize the
data relevant to each potential theme. Themes were evaluated to estimate their fit in relation
to the coded extracts and the entire data set.

(6) Defining and naming themes: Themes that emerged were defined and further refined. For
each theme, we identifying the “essence” of what each theme is about, and determined what

Table 1. Expert sample characteristics.

Category N (%)

Professional background
Physician 2 (22%)
Nurse 2 (22%)
Human factors analysis/human computer interaction experts 3 (33%)
Other (socio-technical domain & social work) 2 (22%)

Highest degree
PhD 7 (88%)
MA 2 (22%)

Country
US 3 (33%)
Israel 3 (33%)
Both* 3 (33%)

Affiliation
Clinical setting (hospitals or health maintenance organizations) 3 (33%)
Academic setting (universities) 3 (33%)
Both academic and clinical setting** 3 (33%)

* Experts who spent more than 5 years of their professional career in both US and
Israel.

** Experts who are currently affiliated with universities and engaged in clinical work.

220 M. TOPAZ ET AL.

aspect of the data each theme captured. This helped to generate clear definitions and names
for each theme.

(7) Producing the report: A final report was produced with a selection of vivid, compelling
interview quotes. The study themes were also related back to the research question and
literature review, producing a report of the analysis presented here.

Methodological rigor of the thematic data analyses was maintained through an audit trail and
periodic debriefing with the study team. Reliability was measured via consistency of interpretation
and coding of the qualitative data.31 An audit trail of process and analytic memos and coding books
was maintained, supporting the credibility of the results.

Results

Five themes emerged from the analysis, including: “Expanding PCC definitions to include patients’
ownership of their health”, “Lack of PCC in clinical practice today”, “HIT provides only partial
support for PCC in today’s clinical practice”, “Ideal world: HIT should help seeing the whole patient
as owner of their health”, “Key areas for HIT support of PCC”. The next sections describe each of the
themes in detail.

Expanding PCC definitions to include patients’ ownership of their health

Overall, most of the experts in this sample described PCC as care that is based on patient’s values,
beliefs, and goals, for example “… patient centered care is care that is appropriate to the patients’
needs and in a personalized way, that takes into consideration the culture of the patient and the
views, the specific aims of life or goals of life.” In a similar vein, another expert said that PCC is “…
being respectful of and responsive to individual patient preferences, needs and values, and ensuring
that patient values guide our clinical decision.”

One prevalent and relatively new HIT-related theme of patient ownership of their health
emerged. For example, in defining PCC experts suggested that “…patient is the owner of his disease
management…” and that providers’ goal is to “…engage the patient in all phases and processes of
care…”. Technology is envisioned to support collaborative processes of health improvement and to
allow patients to take an active part in all care decisions and activities, for example “…[HIT should
allow] patient to be in charge of his own health, to allow the patient to make the decisions together
with the practitioners…” and “ …[HIT should enable] to achieve shared information, shared
deliberation, and shared mind.”

Lack of PCC in clinical practice today

Although there are a few positive examples of PCC in practice, most of the experts suggested that
PCC implementation remains poor in both the US and Israel and there is much room for improve-
ment in order to achieve care that is driven by patient values.

Lack of PCC organizational culture was viewed as one of the largest barriers for implementing
PCC in practice. For example, experts suggested that “… there are still gaps [in PCC implementa-
tion], and many organizations are struggling to create this culture of patient centered care…” or “…
We don’t do it enough [PCC]. We are patronizing. I think we know what’s best for our patient, and
sometimes it’s because we don’t have the time.” One expert deliberated “There are a lot of people
making decisions throughout any episode of care. And they are not aligned with some central focus;
which is the patient, and what the patient wants and what the patient states as preferences and needs
and goals… it really all falls apart at that point.”

Another common barrier was lack of adequate health providers’ training to be able to implement
PCC. For example, “… many of them [clinicians] believe that they don’t have the appropriate

INFORMATICS FOR HEALTH AND SOCIAL CARE 221

education to deal with that [PCC]…” and “… even in the medical school or nursing school we’re not
really taught now to be very much interested in what the patient wants …”. Another expert
identified lack of time and resources allocated for PCC training in practice “ … although they
very much agree with the idea of doing patient goal elicitation [homecare agency’s management],
they don’t have the time to train nurses on it.”

HIT provides only partial support for PCC in today’s clinical practice

There were a few examples of HIT support for PCC implemented in clinical practice today. For
example, experts from both countries described that patient portals are being used in inpatient and
outpatient settings to give patients access to their information. Another example was using mobile
devices to collect patient-reported outcomes in inpatient settings.

On the other hand, most of the experts suggested that there is still a large gap in HIT
implementation to promote PCC. For example, “…clinicians refrain from using the electronic health
record, especially while talking to the patient because it’s an interruption. They feel like, either
I focus on the computer, or I focus on the patient, I can’t do both… “, and “…I did not hear or see
how the patient goal information was connected to the plan of care in any way. It seemed
disconnected…”. Even in settings where information exchange is common (mostly Israel), practi-
tioners reported using it infrequently for PCC-related aspects of care because of poor usability and
other issues “… [about using the national information exchange system in Israel] it is time
consuming, and I think it’s focused on medical information, but not information about the person
in the center.”

In addition, several experts suggested that HIT in outpatient settings is more supportive of the
PCC. In inpatient settings, HIT systems’ design was often found to be disruptive of the PCC
workflows, for example “… But the design is so bad that the patient won’t even be able to see
anything [from the EHR], especially in inpatient care, because he is lying down…” or “… and I’ve
seen the [HIT] systems in Israel and I have seen the system here [the U.S.]. And it seems that, the
design does not support the patient narrative or the clinician work-flow.”

Ideal world: HIT should help seeing the whole patient as owner of their health

When asked about the role of HIT in PCC in an “ideal world”, experts suggested that informa-
tion technology should help providers see the whole patient, and provide tools for patients to
become owners of their health. For example, “… information technology would allow us to bring
them all into one [patients and providers], to talking the same language, into one platform …”.
Several experts pointed out that patients see healthcare as an integral part of their life, while
clinicians build systems that are often setting specific, for example inpatient or outpatient HIT
systems “… artificial focus on just [one] episode and encounter … takes away from that
perspective of the whole patient… the EHR often just drives the user into a very focused and
short-term view of the patient”. PCC requires systems that are much more integrative and
include diverse sources of health data. At the very least, HIT systems were envisioned to
summarize, collect and present PCC-related health data, like patient preferences and goals, for
example “… natural language processing that can identify which information to pull out, instead
of having someone search go through them“

In addition, HIT was suggested as one of the tools to guide providers in implementing different
steps of PCC in practice. For example, HIT can serve as an “… intelligent guide…” to trigger goal
elicitation conversations “… So to kind of trigger that the conversation should happen, here’s
information, and resources and evidence that can be shared with the patient…”, or HIT can be
used to connect patient goals, problems, and interventions, for example “…Ultimately, I would see
that, when [using HIT] the nurse does the plan of care, that the patient goals are documented, and
then the interventions and the education are configured to reflect those goals”.

222 M. TOPAZ ET AL.

Finally, experts suggested a few domains of care where HIT can help facilitate providing high-
quality PCC in the near future. The most common domain was using HIT to provide clear,
comprehensible information and support for patients, for example “… providing them [patients]
with additional information regarding their health condition, referring them to external recourses,
providing them hardcopy of any customized information …”. In addition, experts suggested that
HIT should be used to promote care that is based on patient values and preferences. For example, “
[HIT can help in] not losing actual patient voice… it will probably be a powerful thing to really make
a landing page for part of that summary for the care team to see; what is it that the patient’s saying,
or what they want.”

Key areas for HIT support of PCC

We found support for each of the eight Picker Institute’s Principles of Patient-Centered Care in
experts’ interviews. Table 2 provides summary descriptions on the potential role of technology for
each of the PCC principles and presents exemplary quotes for each principle.

Discussion

Although experts in this study sample defined PCC similarly to the most prevalent definitions
identified in the literature, one significant and relatively new theme of patient ownership of disease
emerged. In the context of this HIT-focused study, patient ownership of disease meant creating an
environment in which patients own, or at least have access to, their health information. In a scenario
where patients would be able to access their health information, clinicians’ role was described as
actively seeking to achieve shared deliberation and shared mind with the patients. HIT was seen as
a key tool to help achieve these goals.

This shift to patient’s ownership of their disease through better HIT is supported by the current
literature. For example, a recent article published in the Journal of American Medical Association
suggests that healthcare is undergoing a paradigm shift, moving from an approach in which “the
doctor will see you now” to “the patient will see the doctor now”.32 Patients are seen as active
participants in their care rather than passive recipients of health services. For this paradigm shift to
happen, patients or their authorized representatives need to be able to control and access their data.
Other studies are beginning to show that patient ownership of their data benefits patients and
clinicians. For example, access to physician notes after medical encounters was shown to improve
patients’ symptom management and medication adherence.33 Similarly, direct release of lab results
to patients was found to increases patient engagement and utilization of care.34 Our results support
this emerging evidence and highlight the critical role of HIT in achieving PCC.

Most of the experts from both countries felt that in the current clinical settings, there is much
room for improvement in order to achieve care that is driven by patient values. Some of the central
barriers to PCC implementation were the lack of PCC organizational culture and lack of adequate
clinician PCC training and education. These results are in tune with findings from other interna-
tional studies. For example, results from recent US study of health-care professionals’ conceptualiza-
tion of PCC show that many clinicians charged with PCC implementation lacked basic knowledge
about PCC.35 An international study that included Israel and US has found that nurses and
physicians are not actively asking patients about their care expectations and satisfaction.36,37

Researchers from Sweden,38 other European countries39 and Australia40 have found similar issues
with lack of clinician PCC understanding and rigid organizational culture prohibitive of PCC.

We identified a few examples where HIT was used to support PCC in the current clinical practice.
For example, two experts from major hospitals in Israel and the US described that they are currently
experimenting with patient portals in inpatient setting where important information is presented to
patients in several hospital departments. On the other hand, the majority of experts suggested that
there is still a large gap in HIT implementation to support PCC. HIT was reported to be interruptive

INFORMATICS FOR HEALTH AND SOCIAL CARE 223

Table 2. Description of specific domains where health information technology should support patient-centered care.

Sub-theme* Sub-theme description Quote/s

Information,
communication,
and education

Sharing information between patient
and healthcare providers is crucial to
support PCC. Information shared with
the patient should be patient-tailored
(in terms of health literacy and specific
data shared) to improve shared
decision making.

“… [HIT should help] to communicate visually, or in some way
that is cognitively understandable by laypersons.”
“… [HIT should help in] giving patients information and resources
where they can read it on their own, and digest it as they make
the decision.”
“Transparency is really important, so whether that’s in providing, the
different options for treatment, being really transparent what those
options are and what the risks and what are the potential benefits…”

Respect for
patients’ values,
preferences,
and expressed
needs

HIT can help providers in starting PCC
conversations (e.g., goal elicitation),
storing PCC-related information (e.g.,
patient preferences) and connecting
patient goals, health problems and
interventions.

“ … [HIT is] an instrument that would augment nurse training, so
that there would be a guide for helping the nurse phrase the goal
elicitation questions…”
“… [HIT should help in] capturing that conversation happened, and
why something was chosen, is important. Because later on, there can
be other clinicians that aren’t aware of that conversation and why
certain decisions were made. It can be essential and informative to
a treatment or a care plan moving forward…”
“…Ultimately, I would see that, when [using HIT] the nurse does the
plan of care, that the patient goals are documented, and then the
interventions and the education are configured to reflect those goals”.

Coordination and
integration of
care/transition
and
continuity**

HIT systems should enable providers to
see the whole patient rather than
a series of disconnected encounters.
Patient generated data should be
gathered and made accessible to the
care team across health care settings.

“… Interoperability helps encourage and provide patient-centered
care, because providers share the complete set of data, they know
exactly what happened to the patient, and obviously, care would
be better. We have a lot of evidence that quality of care has
improved, that waste is reduced and healthcare costs are
reduced.”
“… And so when you’re really delivering patient-centered care, to
have this artificial focus on just this episode and encounter, I think
takes away from that perspective of the whole patient. And so,
the EHR is often just drives the user into a very focused and short-
term view of the patient.”
“… But so, in designing our systems, we really need to figure out
that information flow, and that the patient entered data is highly
important and it shouldn’t be considered secondary. We need to
really centralize it, display it, and make sure that the whole care
team is aware of it.”

Involvement of
family and
friends

HIT can help to actively involve
patients’ significant others (e.g., family
or friends) in the care processes.
Sharing patient information should be
done in a safe manner where the
patient decides on the extent of access
to data.

“… Basically, these [electronic] tools create new opportunities for
patients and family members, a new potential to participate
actively in their care.”
“… So one of the things that acute care patient portals have yet
to fully succeed on, is showing the schedule of the day, and this is
probably one of the, at least I’ve seen, most frequently requested
things from patients and families. Particularly families, because
they’re working! Or they’re there, but they want to be able to go
grab a cup of coffee, and not miss when many doctors coming.
I mean families have schedules they have to keep too – but we
don’t really respect it in health care.”
“… proxy access means you are giving somebody access to your
whole record. You don’t have any control of the granularity of it…
a patient should be able to have some control over what that
looks like…”

Emotional support
and alleviation
of fear and
anxiety

HIT can support provider-to-patient
and patient-to-patient conversations to
alleviate fear and anxiety and provide
emotional support.

“… [providing relevant information] allows them [patients] to be
more comfortable in terms of understanding what’s going on
around them, giving them more information to reduce anxiety,
reduce uncertainty…”
“… HIT platforms that enable patients to communicate with
patients with similar conditions and empower them…”

Access to care HIT can support patients’ access to
care and enable providers to
understand patients’ access to care.

“… [HIT can support understanding] physical environment, does
somebody have access to grocery stores? If you are asking
someone to create a meal plan with certain characteristics while
there’s no grocery stores nearby that sell that type of food”

Physical comfort HIT systems should be designed to
support PCC (e.g., computer screens
that can be shown to patients who are
lying down).

“… But the design is so bad that the patient won’t even be able
to see anything [from their electronic health record], especially in
inpatient care, because he’s lying down. And it is not designed in
a way that is supposed to be seen by a patient.”

*Sub-themes of theme number 5 “Key areas for HIT support of PCC” are based on Picker Principles of Patient-Centered Care and
are sorted by the frequency of appearance in the interviews (from the most discussed to the least discussed).

**Principles “Coordination and integration of care” and “Transition and continuity” were merged since they referred to similar
ideas in expert interviews.

224 M. TOPAZ ET AL.

of PCC oriented workflows and lacking PCC related functionality. For example, clinicians refrained
from using the EHR while talking to the patient because it was considered an interruption and the
EHR did not allow clinicians to connect documented patient goals to interventions. These results are
supported by a growing body of literature. For example, in a recent systematic review of 41 studies
focused on patient-physician communication, researchers have found that EHR use interrupts
nonverbal engagement between patients and physicians and interferes with capture of psychosocial
and emotional information.41 In another study, it was found that multiple health practices experi-
enced common challenges with their EHRs’ capabilities to document and track relevant behavioral
health and physical health information.42

Experts in our study have also reported that clinicians are facing challenges with access to patient
data across the care continuum. Surprisingly, these challenges were persistent even in Israel where
a national health data exchange enables providers to access inpatient and outpatient information for
a significant proportion of Israeli population.43 Some of the major reasons for not using the available
information were the lack of EHR usability and insufficient PCC related information within the
EHR. In multiple other studies, similar issues with EHR use have also been identified. For example,
in a series of recent studies from the US, it was indicated that EHR did allow adequate documenta-
tion and tracking of relevant behavioral health and physical health information such as mental health
diagnoses or behavioral health visit notes.42,44 Other studies indicated that some PCC related
information is available in free text clinical notes within the EHR (e.g., information about poor
social support within discharge summaries),45 however EHRs lack built-in tools to easily search for
this information. Our results coupled with the recent studies suggest that more efforts are needed to
improve the current functionality of HIT to support PCC. This might require a complete redesign of
EHRs, in a fashion that would improve user functionality as well.46

When asked about the role of HIT in PCC in the ideal world, experts in our sample expressed
several common ideas. First, most of the experts suggested that HIT should help health providers see
the whole patient. Currently, information about healthcare is very fragmented and it includes
a mosaic of clinically focused episodes of care. In order to achieve PCC, a shift is needed from
this fragmented view to a holistic picture where patients’ values, preferences, and goals are in the
center. For example, one expert in our sample envisioned a patient “landing” page within the EHR
where the most important information about a patient is presented, including patients’ goals and
values. Another common idea was envisioning HIT as a tool for patients to become owners of their
health, which should lead to increased patient engagement and other positive outcomes. HIT was
described as a platform that should enable health providers and patients “… achieve shared
information, shared deliberation, and shared mind.”

Finally, our study pointed to several domains where HIT can help facilitate providing high-quality
PCC in the near future. The most common domains were using HIT to provide clear, comprehen-
sible information and support for patients and to advance care that is based on patient values and
preferences. Our results showed that HIT can help in promoting each of the eight Picker Institute’s
Principles of PCC in practice.

This study has several important limitations. First, this was a thematic analysis of interviews with
nine experts and our results’ generalizability is limited. Also, the study experts were from major
academic institutions and leading clinical centers, which also impacts the generalizability of results.
In addition, our study experts mostly represented medicine and nursing (except for two participants
with socio-technical and social work backgrounds) and incorporating experts from other profes-
sional backgrounds might have resulted in different results.

Conclusions

This qualitative descriptive study with nine experts in PCC and HIT from Israel and the US aimed to
explore the existing state of HIT for better PCC and identify venues for further development. In
addition to common PCC definitions in the literature, experts suggested that patient ownership of

INFORMATICS FOR HEALTH AND SOCIAL CARE 225

disease is a core HIT-related component of PCC. A majority of the experts agreed that in both Israel
and the US, the current situation of PCC implementation is relatively poor due to providers’ lack of
training and rigid organizational cultures. In the current clinical practice, HIT was not seen as
a critical tool for PCC implementation. However, HIT can and should play a major role in making
patients owners of their health and treatment, and helping providers in delivering better PCC. Some
of the central domains where HIT can be used in support of PCC were providing clear, compre-
hensible information and support for patients, and promoting care that is based on patients’ values
and preferences.

Ethics approval and consent to participate

This research has been performed in accordance with the Declaration of Helsinki. The Institutional Review Board of
the University of Haifa – Faculty of Social Welfare and Health Sciences grated approval for this study and research on
human subjects on October 19, 2017, under approval number 355/17. All participants in this study have given written
consent to the inclusion of material pertaining to themselves. They acknowledge that they cannot be identified via the
paper and have been fully anonymized.

Availability of data and materials

The data that support the findings of this study are available on request from the corresponding author, MT. The data
are not publicly available due to privacy restrictions.

Disclosure statement

No potential conflict of interest was reported by the authors.

Funding

This work was supported by the Israel National Institute for Health Policy Research (NIHPR) .[ר/2017/232]

References

1. Institute of Medicine (U.S.). Committee on Quality of Health Care in America. 2001. Crossing the quality
chasm : A new health system for the 21st Century. Washington, DC: National Academy Press.

2. Zimlichman E, Rozenblum R, Millenson ML. The road to patient experience of care measurement: lessons from
the United States. Isr J Health Policy Res. 2013;2(1):35. doi:10.1186/2045-4015-2-35.

3. Picker Institute and The Commonwealth Fund. The 8 principles of patient-centered care. Published 2000.
[Accessed 2017 Mar 21]. http://www.oneviewhealthcare.com/the-eight-principles-of-patient-centered-care/.

4. Rossom RC, Solberg LI, Vazquez-Benitez G, Crain AL, Beck A, Whitebird R, Glasgow RE. The effects of
patient-centered depression care on patient satisfaction and depression remission. Fam Pract. 2016;33
(6):649–55. doi:10.1093/fampra/cmw068.

5. Titchener J. A patient-centred clinical approach to diabetes care assists long-term reduction in HbA1c.
J Prim Health Care. 2014;6(3):195–202. [Accessed Mar 21 2017.]. http://www.ncbi.nlm.nih.gov/pubmed/
25194246.

6. Pirhonen L, Olofsson EH, Fors A, Ekman I, Bolin K. Effects of person-centred care on health outcomes –
a randomized controlled trial in patients with acute coronary syndrome. Health Policy. 2017;121(2):169–79.
doi:10.1016/j.healthpol.2016.12.003.

7. Rathert C, Wyrwich MD, Boren SA. Patient-centered care and outcomes: a systematic review of the literature.
Med Care Res Rev. 2013;70(4):351–79. doi:10.1177/1077558712465774.

8. Santana MJ, Manalili K, Jolley RJ, Zelinsky S, Quan H, Lu M. How to practice person-centred care: a conceptual
framework. Health Expect. 2018;21(2):429–40. doi:10.1111/hex.12640.

9. Topaz M, Lisby M, Morrison CR,Levtzion-Korach O, Hockey PM, Salzberg CA, Efrati N, Lipsitz S,Bates DW,
Rozenblum R. Nurses’ perspectives on patient satisfaction and expectations: an International cross-sectional
multicenter study with implications for evidence-based practice. Worldviews Evidence-Based Nurs. 2016;13
(3):185–96. doi:10.1111/wvn.12143.

226 M. TOPAZ ET AL.

https://doi.org/10.1186/2045-4015-2-35

http://www.oneviewhealthcare.com/the-eight-principles-of-patient-centered-care/

https://doi.org/10.1093/fampra/cmw068

http://www.ncbi.nlm.nih.gov/pubmed/25194246

https://doi.org/10.1016/j.healthpol.2016.12.003

https://doi.org/10.1177/1077558712465774

https://doi.org/10.1111/hex.12640

https://doi.org/10.1111/wvn.12143

10. Rozenblum R, Lisby M, Hockey PM, Levtizion-Korach O, Salzberg CA, Lipsitz S, Bates DW. Uncovering the
blind spot of patient satisfaction: an international survey. BMJ Qual Saf. 2011;20(11):959–65. doi:10.1136/
bmjqs-2011-000306.

11. Office of National Coordinator for Information Technology. Adoption of electronic health record systems
among U.S. Non-federal acute care hospitals: 2008-2015. Published 2016. [Accessed 2017 Mar 21]. https://
dashboard.healthit.gov/evaluations/data-briefs/non-federal-acute-care-hospital-ehr-adoption-2008-2015.
php.

12. Topaz M, Ash N. Overview of the US policies for health information technology and lessons learned for Israel.
Harefuah. 2013;152(5):262–66. 310, 309

13. Cliff B. Using technology to enhance patient-centered care. J Healthc Manag. 57(5):301–03. [Accessed 2018 Dec
19]. http://www.ncbi.nlm.nih.gov/pubmed/23087992.

14. Fisher CA, Feigenbaum K. Harnessing technology to promote patient-centered care. Nurs Manag. 2015;46
(1):14–15. doi:10.1097/01.NUMA.0000459044.24554.af.

15. Forss A, Ceci C. Technology, health care and person centeredness: beyond utopia and dystopia. thinking the
future. Nurs Philos. 2017;18(1):e12162. doi:10.1111/nup.12162.

16. Epstein RM, Fiscella K, Lesser CS, Stange KC. Why the nation needs a policy push on patient-centered health
care. Health Aff. 2010;29(8):1489–95. doi:10.1377/hlthaff.2009.0888.

17. Street RL. Mediated consumer-provider communication in cancer care: the empowering potential of new
technologies. Patient Educ Couns. 2003;50(1):99–104. [Accessed 2018 Dec 19]. http://www.ncbi.nlm.nih.gov/
pubmed/12767594.

18. Ovretveit J, Keller C, Hvitfeldt Forsberg H, Essén A, Lindblad S, Brommels M. Continuous innovation:
developing and using a clinical database with new technology for patient-centred care—the case of the
Swedish quality register for arthritis. Int J Qual Heal Care. 2013;25(2):118–24. doi:10.1093/intqhc/mzt002.

19. Topaz M, Ronquillo C, Peltonen L-M, Pruinelli L, Sarmiento RF, Badger MK, Ali S, Lewis A, Georgsson M,
Jeon E, et al. Nurse informaticians report low satisfaction and multi-level concerns with electronic health
records: results from an International survey. AMIA. Annu Symp proceedings AMIA Symp. 2016;2016:2016–25,
Washington, DC. [Accessed 2017 Mar 21]. http://www.ncbi.nlm.nih.gov/pubmed/28269961.

20. O’Malley AS. Tapping the unmet potential of health information technology. N Engl J Med. 2011;364
(12):1090–91. doi:10.1056/NEJMp1011227.

21. Hendrich A, Chow MP, Skierczynski BA, Lu Z. A 36-hospital time and motion study: how do medical-surgical
nurses spend their time? Perm J. 2008;12(3):25–34. [Accessed 2017 Mar 21]. http://www.ncbi.nlm.nih.gov/
pubmed/21331207.

22. Sinsky C, Colligan L, Li L, Prgomet M, Reynolds S, Goeders L, Westbrook J, Tutty M, Blike G. Allocation of
physician time in ambulatory practice: A time and motion study in 4 specialties. Ann Intern Med. 2016;165
(11):753–60. doi:10.7326/M16-0961.

23. Roman LC, Ancker JS, Johnson SB, Senathirajah Y. Navigation in the electronic health record: a review of the
safety and usability literature. J Biomed Inform. 2017;67:69–79. doi:10.1016/j.jbi.2017.01.005.

24. Topaz M,Ronquillo C, Peltonen LM, Pruinelli L, Sarmiento RF, Badger MK,Ali S, Lewis A, Georgsson M, Jeon
E, Tayaben JL, Kuo CH, IslamT, Sommer J, Jung H, Eler GJ, Alhuwail D, Lee YL. Nurse informaticians report
low satisfaction and multi-level concerns with electronic health records: results from an international survey.
AMIA. Annu Symp proceedings AMIA Symp. 2016;2016, Washington, DC.

25. Topaz M, Ash N. [Overview of the US policies for health information technology and lessons learned for
Israel]. Harefuah. 2013; 152(5):262–266, 310, 309. [Accessed 2016 Oct 24]. http://www.ncbi.nlm.nih.gov/
pubmed/23885448.

26. Martin AB, Hartman M, Washington B, Catlin A. National health care spending In 2017: growth slows to post–
great recession rates; Share of GDP stabilizes. Health Aff. 2018 December. doi:10.1377/hlthaff.2018.05085.

27. Gerteis M, Edgman-Levitan DJ, Delbanco TL, Picker/Commonwealth program for patient-centered care.
Through the Patient’s Eyes : Understanding and Promoting Patient-Centered Care. [Accessed 2018 Apr 21]
https://psnet.ahrq.gov/resources/resource/1573/through-the-patient-s-eyes-understanding-and-promoting-
patient-centered-care.

28. Malterud K, Siersma VD, Guassora AD. Sample size in qualitative interview studies: guided by information
power. Qual Health Res. 2015;26(13):1753–60. doi:10.1177/1049732315617444.

29. Braun V, Clarke V. Using thematic analysis in psychology. Qual Res Psychol. 2006;3(2):77–101. doi:10.1191/
1478088706qp063oa.

30. R Core Team. R: A language and environment for statistical computing. 2013. [Accessed 2018 Apr 13] http://
www.r-project.org/.

31. Byrne M. Data analysis strategies for qualitative research. Aorn J. 2001;74(6):904–05. doi:10.1016/S0001-
2092(06)61511-1.

32. Mikk KA, Sleeper HA, Topol EJ. The pathway to patient data ownership and better health. JAMA. 2017;318
(15):1433. doi:10.1001/jama.2017.12145.

INFORMATICS FOR HEALTH AND SOCIAL CARE 227

https://doi.org/10.1136/bmjqs-2011-000306

https://dashboard.healthit.gov/evaluations/data-briefs/non-federal-acute-care-hospital-ehr-adoption-2008-2015.php

http://www.ncbi.nlm.nih.gov/pubmed/23087992

https://doi.org/10.1097/01.NUMA.0000459044.24554.af

https://doi.org/10.1111/nup.12162

https://doi.org/10.1377/hlthaff.2009.0888

http://www.ncbi.nlm.nih.gov/pubmed/12767594

https://doi.org/10.1093/intqhc/mzt002

http://www.ncbi.nlm.nih.gov/pubmed/28269961

https://doi.org/10.1056/NEJMp1011227

http://www.ncbi.nlm.nih.gov/pubmed/21331207

https://doi.org/10.7326/M16-0961

https://doi.org/10.1016/j.jbi.2017.01.005

http://www.ncbi.nlm.nih.gov/pubmed/23885448

https://doi.org/10.1377/hlthaff.2018.05085

https://psnet.ahrq.gov/resources/resource/1573/through-the-patient-s-eyes-understanding-and-promoting-patient-centered-care

https://doi.org/10.1177/1049732315617444

https://doi.org/10.1191/1478088706qp063oa

http://www.r-project.org/

https://doi.org/10.1016/S0001-2092(06)61511-1

https://doi.org/10.1001/jama.2017.12145

33. Delbanco T, Walker J, Bell SK, Darer JD, Elmore JG, Farag N, Feldman HJ, Mejilla R, Ngo L, Ralston JD, et al.
Inviting patients to read their doctors’ notes: a Quasi-experimental study and a look ahead. Ann Intern Med.
2012;157(7):461. doi:10.7326/0003-4819-157-7-201210020-00002.

34. Pillemer F, Price RA, Paone S, Martich GD, Albert S, Haidari L, Updike G, Rudin R, Liu D, Mehrotra A, et al.
Direct release of test results to patients increases patient engagement and utilization of care. Kuivaniemi H, ed.
PLoS One. 2016;11(6):e0154743. doi:10.1371/journal.pone.0154743.

35. Fix GM, VanDeusen Lukas C, Bolton RE, Hill JN, Mueller N, LaVela SL, Bokhour BG. Patient-centred care is
a way of doing things: how healthcare employees conceptualize patient-centred care. Heal Expect. 2018;21
(1):300–07. doi:10.1111/hex.12615.

36. Rozenblum R, Lisby M, Hockey PM, Levtzion-Korach O, Salzberg CA, Efrati N, Lipsitz S, Bates DW. The
patient satisfaction chasm: the gap between hospital management and frontline clinicians. BMJ Qual Saf.
2013;22(3):242–50. doi:10.1136/bmjqs-2012-001045.

37. Topaz M, Lisby M, Morrison CR, Levtzion-Korach O, Hockey PM, Salzberg CA, Efrati N, Lipsitz S, Bates DW,
Rozenblum R. Nurses’ perspectives on patient satisfaction and expectations: an International cross-sectional
multicenter study with implications for evidence-based practice. Worldviews Evidence-Based Nurs. 2016.
doi:10.1111/wvn.12143.

38. Moore L, Britten N, Lydahl D, Naldemirci Ö, Elam M, Wolf A. Barriers and facilitators to the implementation
of person-centred care in different healthcare contexts. Scand J Caring Sci. 2017;31(4):662–73. doi:10.1111/
scs.12376.

39. Taylor A, Groene O. European hospital managers’ perceptions of patient-centred care. J Health Organ Manag.
2015;29(6):711–28. doi:10.1108/JHOM-11-2013-0261.

40. Bolster D, Manias E. Person-centred interactions between nurses and patients during medication activities in an
acute hospital setting: qualitative observation and interview study. Int J Nurs Stud. 2010;47(2):154–65.
doi:10.1016/j.ijnurstu.2009.05.021.

41. Rathert C, Mittler JN, Banerjee S, McDaniel J. Patient-centered communication in the era of electronic health
records: what does the evidence say? Patient Educ Couns. 2017;100(1):50–64. doi:10.1016/j.pec.2016.07.031.

42. Cifuentes M, Davis M, Fernald D, Gunn R, Dickinson P, Cohen DJ. Electronic health record challenges,
workarounds, and solutions observed in practices integrating behavioral health and primary care. J Am Board
Fam Med. 2015;28(Supplement 1):S63–S72. doi:10.3122/jabfm.2015.S1.150133.

43. Ministry of Health S of I. Health information exchange, ministry of health. Published 2018. [Accessed 2018 May
17] https://www.health.gov.il/English/About/projects/shared_medical_info/Pages/default.aspx.

44. Madden JM, Lakoma MD, Rusinak D, Lu CY, Soumerai SB. Missing clinical and behavioral health data in
a large electronic health record (EHR) system. J Am Med Informatics Assoc. 2016;23(6):1143–49. doi:10.1093/
jamia/ocw021.

45. Navathe AS, Zhong F, Lei VJ, Chang FY, Sordo M, Topaz M, Navathe SB, Rocha RA, Zhou L. Hospital
readmission and social risk factors identified from physician notes. Health Serv Res. 2018;53(2):1110–36.
doi:10.1111/1475-6773.12670.

46. Choi K, Gitelman Y, Asch DA. Subscribing to your patients—reimagining the future of electronic health
records. N Engl J Med. 2018;378(21):1960–62. doi:10.1056/NEJMp1800874.

228 M. TOPAZ ET AL.

https://doi.org/10.7326/0003-4819-157-7-201210020-00002

https://doi.org/10.1371/journal.pone.0154743

https://doi.org/10.1111/hex.12615

https://doi.org/10.1136/bmjqs-2012-001045

https://doi.org/10.1111/wvn.12143

https://doi.org/10.1111/scs.12376

https://doi.org/10.1108/JHOM-11-2013-0261

https://doi.org/10.1016/j.ijnurstu.2009.05.021

https://doi.org/10.1016/j.pec.2016.07.031

https://doi.org/10.3122/jabfm.2015.S1.150133

https://www.health.gov.il/English/About/projects/shared_medical_info/Pages/default.aspx

https://doi.org/10.1093/jamia/ocw021

https://doi.org/10.1111/1475-6773.12670

https://doi.org/10.1056/NEJMp1800874

Abstract

Introduction
Patient-centered care
Health information technology and patient-centered care
Unanswered questions about PCC and EHRs
Methods
Conceptual framework
Study sample
Data collection
Data analysis
Results
Expanding PCC definitions to include patients’ ownership of their health
Lack of PCC in clinical practice today
HIT provides only partial support for PCC in today’s clinical practice
Ideal world: HIT should help seeing the whole patient as owner of their health
Key areas for HIT support of PCC
Discussion
Conclusions
Ethics approval and consent to participate
Availability of data and materials
Disclosure statement
Funding
References

How technology is empowering patients? A
literature review

Jorge Calvillo PhD,*§ Isabel Rom�an PhD*†§ and Laura M. Roa PhD†‡§
*IT Health Scientist in Biomedical Engineering Group, †Professor, ‡Head of Biomedical Engineering Group, University of

Seville, Seville and §CIBER de Bioingenierı́a, Biomateriales y Nanomedicina (CIBER-BBN), Seville,

Spain

Correspondence
Jorge Calvillo

Escuela T�ecnica Superior de

Ingenierı́a

C. de los Descubrimientos

s/n 41092 Sevilla

Spain

E-mail: jorgecalvilloarbizu@gmail.com

Accepted for publication

1 May 2013

Keywords: health information
systems, medical informatics, patient

education, patient empowerment,

patient participation

Abstract

Background The term ‘Patient Empowerment’ (PE) is a growing

concept – so in popularity as in application – covering situations
where citizens are

encouraged to take an active role in the man-

agement of their own health. This concept is serving as engine

power for increasing the quality of health systems, but a question

is still unanswered, ‘how PE will be effectively achieved?’ Beyond

psychological implications, empowerment of patients in daily prac-

tice relies on technology and the way it is used. Unfortunately, the

heterogeneity of approaches and technologies makes difficult to

have a global vision of how PE is being performed.

Objective To clarify how technology is being applied for enhanc-

ing patient empowerment as well as to identify current (and

future) trends and milestones in this issue.

Search strategy Searches for relevant English language articl

using Medline, Scopus, ACM Digital Library, Springer Link, EB-

SCO host and ScienceDirect databases from the year 2000 until

October 2012 were conducted. Among others, a selection criterion

was to review articles including terms ‘patient’ and ‘empowerment’

in title, abstract or as keywords.

Main results and conclusions Results state that practical

approaches to empower patients vary in scope, aim and technol-

ogy. Health literacy of patients, remote access to health services,

and self-care mechanisms are the most valued ways to accom-

plish PE. Current technology already allows establishing the first

steps in the road ahead, but a change of attitude by all

stakeholders (i.e. professionals, patients, policy makers, etc.) is

required.

Introduction

As the appearance of the first Internet-based

applications supporting new methods of health-

care delivery, the potential of Information and

Communication Technologies (ICT) for chang-

ing the role of users has been well-known.
1,2

Experts predicted a range of benefits from the

efficient adoption of ICT in the health-

care domain.
3
For example, the possibility of

providing citizens with mechanisms for access-

ing information and knowledge required to

643

doi: 10.1111/hex.12089

understand their health status and to make

informed decisions.
4
ICT could also bring new

ways of connectivity among patients, users and

health providers to establish virtual communi-

ties in which end users adopt the role of

information providers for their peers.
5,6

Furthermore, technology would ease the devel-

opment of tools and solutions for maintenance

of healthy habits, education in health, self-man-

agement of chronic diseases and deployment

and use of Personal Health Records (PHR)

controlled by patients.
1

Finally, technology

promised to reduce economic costs and pro-

mote a more sustainable health care.
7

Patient Empowerment (PE) is a growing

concept – so in popularity as in application –
that covers situations where citizens are

encouraged to take an active role in the man-

agement of their own health, transforming the

traditional patient–doctor relationship and
providing citizens with real management capa-

bilities.
1

Gibson, in a review about PE in

health, redefined empowerment as a process of

helping people to assert control over the fac-

tors which affect their health.
8
Another litera-

ture review defines PE as a continuous process

through which patients (and patient groups)

work in partnership with their health-care sys-

tem. The objective of this collaboration is to

enable patients to become more responsible for

and involved in their treatment and health

care.
9

Despite differences among definitions, the

central idea is shared: an attempt for patients

to take charge of their own health. Some com-

mon keys are as follows:

1. Health-care professionals should be the first

promoters of PE.
1

2. An empowered patient should be educated

to think critically, make informed decisions

and then adjust to prescribed care plans (i.e.

become a health literate).
11,12

3. Dependency on people should be partially

transformed to dependency on systems.
6

4. Technology for PE may also bring problems

due to digital divide existing in society

between people with and without technology

skills (what is called ‘digital literacy’).
13–16

At organizational level, it is assumed that

PE is a cornerstone for the transformation and

evolution of the health-care domain, becoming

a philosophy inspiring policies and services.
17,18

As an example, the European Commission, the

European Council and the World Health Orga-

nization (WHO) – Europe – are supporting
actively the development of PE solutions by

acting in several points such as bringing access

to information and trust advice to people, pro-

moting health literacy of patients or supporting

new models of chronic care.
19

Thus, the con-

cept PE is serving as engine power for increas-

ing the quality of health systems by policy

makers.

Beyond psychological implications, empow-

erment of patients in daily practice relies on

technology and the way it is used.
16

As the

application of ICT in the health-care domain

has been performed in an uncontrolled way

and with no formalization or guidelines, now

empowerment of patients (as will be shown in

this paper) is shared by a wide spectrum of dif-

ferent and separate research fields such as

end-user applications, homecare, information

systems and communications. This multidisci-

plinary divergence increases the complexity of

the matter because it often requires a knowl-

edge translation among areas. Moreover, PE

(even the term) is performed through many dif-

ferent ways depending on the specific research

field addressing it. These facts lead to a frag-

mented application of heterogeneous technolo-

gies for empowering patients.

To our knowledge, several literature reviews

focus on PE, but none has both a general pur-

pose and an emphasis on technology. The

scope of these reviews varies in different ways.

Aujoulat et al.
21

examined how the term

‘empowerment’ has been used in relation to

care and education of patients with chronic

conditions over an 11-year period (1995–2006).
This review bases on the theory of patient edu-

cation as a mechanism for empowerment in

chronic scenarios and its benefits for patients.

Laugharne et al.
22

had mental health as

domain for their review of trust, choice and PE

from 1980 to 2005. This review stated that PE

How technology is empowering patients? J Calvillo, I Rom�an and L M Roa644

was accomplished at organizational level (e.g.

policies of patient involvement in health care),

but there was no evidence of real performance

of PE in daily practice of mental health care.

Technology applications were not reported in

this paper. Macq et al.
23

reviewed literature to

extract findings of PE in tuberculosis control.

They identified trends and barriers of empow-

ering of tuberculosis patients but did not

include any reference to technology applica-

tions. Another review focused on methodolo-

gies for PE was published by Virtanen et al.
24
.

This paper made a literature review to describe

the nature of empowering discourses between

patient and nurse. The main result was a heter-

ogeneous range of discourse methodologies to

empower patients, but no technology was iden-

tified to do it. Finally, Lober et al.
25

published

a review strongly oriented to find synergies

between health care and ICT. Although its

health focus was oncology nursing, it presented

papers of general purpose. This review mainly

placed emphasis on new models of interaction

between patients, doctors and others thanks to

the Internet and social media as well as known

areas such as monitoring and administration

systems. This was a broad review but with a

research methodology covering rather than for-

mal published papers (i.e. web references, tools

and technical reports).

Through a literature review, our paper aims

at clarifying how technology is being applied

to PE as well as future trends and milestones.

The review focuses on identifying current

trends of technology applied to PE, in particu-

lar, which technologies are being used, how

they are applied and how empowerment is pro-

posed and accomplished. Because approaches

of technology applied to PE have recently

appeared, this paper makes a review from the

year 2000 to October 2012 considering that

prior works are theoretical or related to social

interaction. Finally, although there exist other

concepts related to empowerment for nurses,

doctors and others,
3
considering them is out of

the scope of our review.

Methods

We searched for relevant English language arti-

cles using Medline, Scopus, ACM Digital

Library, Springer Link, EBSCO host and

ScienceDirect databases from the year 2000

until October 2012 (papers in press included).

The first step was collecting articles that

included ‘patient’ and ‘empowerment’ in title,

abstract or as keywords. For the sake of thor-

oughness, several searches were performed by

substituting ‘patient’ by ‘citizen’, ‘user’, ‘con-

sumer’, ‘human’ and ‘subject of care’ but

always with focus on the health domain.

Another set of searches was performed to col-

lect publications with the terms ‘empowering

patients’, ‘empower patients’, ‘empower peo-

ple’, ‘health informatics’, ‘health information

systems’, ‘patient-centred’ or ‘patient-centric’ in

title, abstract or keywords. Opinion papers, let-

ters and reviews were excluded. Figure 1 shows

the selection process for this article review.

Identified abstracts and contents were

screened by two peers in parallel to determine

eligibility for further review. The following eli-

gibility considerations were made:

1. The development, validation or assessment

of technology for PE should be presented in

the article. Those papers expressing opinions

or conducting reviews were excluded;

2. The empowerment action should be per-

formed for the direct benefit of patients.

Technology applied to empowerment of

medical staff, relatives or others were not

considered, even although this had indirect

benefits for patients;

3. Many technology applications in health lead

to more efficient care but not to empowering

individuals. For example, remote monitor-

ing with no intervention by patients, or

EHRs only for health professional use. This

kind of applications means better health for

patients but not increasing the trust, auton-

omy or safe sense of patients. Articles where

patient empowerment is not properly justi-

fied or stated were excluded.

How technology is empowering patients?, J Calvillo, I Rom�an and L M Roa 645

From the eligible papers, the reviewers

extracted data such as publication year, if they

were focused on particular medical specialties,

technology used or how PE was claimed to be

achieved.

Results

Searching the online databases resulted in

12 389 articles. After combining results and

excluding duplication, 9540 articles left. Later,

reviews of abstract and content were per-

formed. 9087 and 187 articles were excluded,

respectively, according to eligibility criteria pre-

viously defined. Thus, only 266 papers satisfied

all the requirements of the process. The set of

references included in the review can be con-

sulted via web.
26

As first point of this review, it

is worthy to mention that the number of

articles published about technology for PE has

increased notably. However, the number of

papers does not follow an exponential ten-

dency, but it presents an increasing variability

between consecutive years (Fig. 2).

The articles selected have been classified

according to the medical specialty or disease

which the technology developed is planned to.

Figure 3 shows the results. Technology in

45.9% (123) of articles is applied to all knowl-

edge areas of health as in the case of medical

information management or patient educa

tion.

Both examples may cover the whole range of

health issues and diseases. 11.5% (31) of arti-

cles focus on medical specialties (e.g. dermatol-

ogy and paediatrics), and 13.6% (36) present

solutions applied to some specific disease (e.g.

HIV and depression). Diabetes (10.8%, 29),

oncology (8.2%, 22) and chronic diseases

Scopus
Search strategy

(n = 4.186)

Medline
Search strategy

(n = 6.285)

ScienceDirect
Search strategy

(n = 595)

Combine results
(n = 12.389)

Abstract review
(n = 9.540)

Exclude
duplicates
(n = 2.849)

Content review
(n = 453)

Excluded in
abstract review

(n = 9.087)

Articles included
(n = 266)

Excluded in
content review

(n = 187)

ACM Digital
Library

Search strategy
(n = 213)

Springer Link
Search strategy

(n = 613)

EBSCO host
Search strategy

(n = 497)

Figure 1 Flow of article selection in the literature review.

0
5

15
20
25

2000 2001 2002 2003 2004 2005 2006 2007 2008 2009 2010 2011 2012

N
um

be
r

of
a

rt
ic

le
s

Publication year (until October 2012)

Figure 2 Number of articles published on technology for patient empowerment over the years.

How technology is empowering patients? J Calvillo, I Rom�an and L M Roa646

(5.2%, 14) have been classified separately due

to the relevant number of articles on these top-

ics. Finally, technology in 4.8% (13) of

reviewed papers is centred on other areas such

as Pharmacy and Lactation, which cannot be

grouped easily by the previous categories.

Once determined the health issue in which

PE is approached, the next point is to classify

what technology is used to accomplish it

(Fig. 4). Web services and communication net-

works are the most used technologies (74 and

51 articles, respectively). Both technologies are

applied to different scenarios and purposes but

always easing remote communication and

access to health information and services.

Besides them, both PHR and Electronic Health

Record (EHR) approaches share outstanding

positions. The reason could be that they have

been considered cornerstones for the techno-

logical revolution of the health-care domain,

and there are many efforts developing them.

EHR and PHR are similar each other although

address PE differently. EHR provides the

patient with access and knowledge of his/her

health information; meanwhile, PHR grants a

patient with administration privileges too.
27,28

Another relevant set of approaches focuses on

translating the methodology of patient support

groups to virtual world using social media and

online communities. In these scenarios, the

patient receives advice from peers and he/she

can be an information provider for others.

Finally, other relevant technologies are as fol-

lows: Internet as source of information, soft-

ware and mobile apps, security mechanisms,

devices and communication media (such as tra-

ditional and IP telephony or e-mail).

The last point of this review categorizes how

the different approaches empower patients. The

same technology may deploy two different

General
45.9%

Medical
specialties

11.5%
Diabetes

10.8%

Chronic
diseases

5.2%

Oncology
8.2%

Other
4.8%

Specific
disease
13.6%

Figure 3 Classification of selected articles according to the

health area where they empower patients.

0
10
20
30
40
50

N
u

m
b

er
o

f
ar

ti
cl

Technology applied

Figure 4 Number of articles categorized by technology they use to empower patients.

How technology is empowering patients?, J Calvillo, I Rom�an and L M Roa 647

approaches for PE. For example, e-mail com-

munication could be used for strengthening

doctor–patient relation or alerting patient of
modification of his/her health information

record. Figure 5 summarizes the findings. Note

that most articles adopt two or more mecha-

nisms for PE. The most popular way for

empowerment is patient education, shared for

40% of reviewed articles. It is widely argued

that an educated patient can make more

informed decisions, improve compliance,

reduce anxiety levels and participate actively in

the treatment of his/her diseases. This fact is

more relevant in chronic scenarios where the

patient must modify his/her life and adapt to

permanent conditions. If healthy scenarios

were considered, benefits of patient education

could be translated to the maintenance of

health and prevention tasks through citizen

education. Enhancing commodity of patients is

another important approach for

empowerment.

It is accomplished by reducing the complexity

of daily tasks such as the patient–doctor com-
munication (e.g. e-mail or instant messaging),

online access to administrative services and tel-

ediagnosis. Self-care is a benefit of patient edu-

cation with a significant number of articles

proposing mechanisms to accomplish it. The

access to own health information and to reli-

able advices improves awareness of patients in

their condition and adherence to treatment

plans. Another relevant paradigm to empower

patients is to turn them into providers of sup-

port and advice for peers. As providers,

patients feel useful, and as receivers, they

obtain support and comprehension of peers

that suffer (or suffered) similar conditions.

Security also counts as a driving force for

PE in different ways. Control of distribution

and disclosure of personal information are the

most relevant PE mechanisms followed by

control over its edition, and privacy and

confidentiality of communications. Due to the

confidential content of health information,

patients are very concerned with security

requirements. Many reviewed articles consider

scenarios with no mechanism to protect data

and communications, but security is an essen-

tial requirement of technology applications in

the health-care domain. Other PE ways are to

strengthen the doctor–patient relation, to
access general or personalized information and

to promote behaviour modification, etc.

Discussion

In the introduction section, the objective of the

review was stated: to identify how PE is being

approached through technology and which

milestones would be required to accomplish a

real empowerment of patients. We selected and

described the results of 266 papers on technol-

ogy applications for PE. The number of such

approaches shows a strong increase in recent

years (Fig. 2). Two facts could be responsible

of that tendency: the advancements in ICT

(with the consequent wider application to

health) and the currently rising awareness of

0
20
40
60
80

100
120

N
um
be
r
of
a
rt
ic
le
s

Paradigms for empowering patients

Figure 5 Number of articles categorized by how they empower patients.

How technology is empowering patients? J Calvillo, I Rom�an and L M Roa648

providing patients with an active role and

capabilities of management.

In a broad sense, the term ‘patient empower-

ment’ is often used in the literature and health

policy to characterize an essential key for

involving patients in their health care. In most

approaches, PE is addressed from a psycholog-

ical perspective (i.e. studies of how health-care

professionals could make patients more confi-

dent and involved in their health-care pro-

cesses), but also in some cases, technology is

applied for empowering patients. Thus, impli-

cations on daily practice are restricted to mod-

ify attitudes of patients (what we classify as

Milestone 1 of empowerment, as it will be fur-

ther explained in the following) and not to

involve them actively in processes.

From the results of the review, some interest-

ing points can be extracted for discussion.

First, there is a wide spectrum of technologies

empowering patients. There are initiatives that

use promising technologies (such as games and

virtual worlds
29

or textile monitoring
30
), and

others reuse common (and sometimes in disuse)

technologies (e.g. audio call
31

or video record-

ing
32
). According to current and future tech-

nology trends for empowering patients, it can

be stated that approaches in this domain fol-

low the general tendencies of research and

development in ICT. As revolutionary technol-

ogy in every sector of society, Web services is

also one of the most used technologies for

empowering patients. Its application covers

several approaches such as information web

pages, interactive portals, infrastructure of dis-

tributed services and remote data access. Its

versatility, popularity and development in

other domains make it the first choice for

developers of solutions in health domain.

Booming technologies in other sectors (e.g.

social media and mobile apps) are being stea-

dily applied to empower patients. Forums,

blogs and social networks are suitable vehicles

to translate support groups from real life to

electronic world, ease the communication

among patients and professionals and

strengthen the continuity of care beyond

physical appointments. In addition, the wide

adoption of smartphones in daily life brings

many potential trends for patient empower-

ment such as ubiquitous access to health infor-

mation for patients and professionals or

smartphone applications for monitoring

chronic conditions, disease prevention tasks

and promotion of healthy habits.

Finally, an interesting result is the slightly

higher use of PHR (12.7% of articles) for

empowering patients than of EHR (10%).

According to public opinion, PHR is a natural

evolution of EHR towards a real data manage-

ment by patients. Thus, the common assump-

tion a priori is that PHR would be a much

more significant trend for empowering patients

than EHR. But that is not right because

although EHR does not delegate management

capabilities to patients, it contributes to

empower them through the access to their

information. Often, the scenario of patients

suffering burdens for accessing their own

health data is underestimated or not consid-

ered. But our review reveals that is still a nec-

essary action line without which other

approaches (such as data management in

PHR) cannot be addressed.

From the literature review, we can conclude

that different levels of empowerment exist (as

has been stated in previous works
33
). All the

reviewed approaches have the same objective

(i.e. to empower patients), but the grade of

autonomy or involvement that the subject

obtains varies from one solution to another.

These levels of empowerment may serve as

overview of milestones in the road of patient

empowerment.

1. Milestone 1: Patient is aware of his/her

health condition and properly informed by

doctors. There is a first change of attitude

from passive and ignorant to active and par-

ticipant to face diseases. In this stage,

patients are well informed about prognosis

and treatment options, and this makes them

to be more likely to make decisions, adhere

to their treatment plan and have better out-

comes.

2. Milestone 2: Individual (not necessarily

patient) active not only in the treatment of

How technology is empowering patients?, J Calvillo, I Rom�an and L M Roa 649

diseases but also in the maintenance of his/

her health and prevention tasks. Now citi-

zen is willing to perform healthy activities,

monitor food and hygiene habits, modify

damaging behaviours, etc., in order to pre-

vent diseases or enhance his/her wellness.

Doctors play the role of encouraging these

attitudes which result in a healthier popula-

tion.

3. Milestone 3: Citizens are educated (and not

simply informed) in health. A democratiza-

tion of knowledge is accomplished, that is,

citizens can proactively access information,

knowledge and advices not guarded by

health professionals. Health knowledge must

be understandable, reliable and accessible

for any citizen according to different skills.

Now doctor and patient are collaborators at

the same level, and there is a mutual trust.

Figure 5 shows how education of patients is

the first concern (with 39.4% of selected

articles). From their point of view, doctors

plead for educated patients that adopt effec-

tively their care plans and willing to know

more about their health conditions.

4. Milestone 4: Citizens are not only health

information sinks but also sources. Technol-

ogy allows citizens to act as health informa-

tion and advice providers for peers

worldwide with little or no supervision by

doctors. Citizens embrace a new dimension

in empowerment when their health experi-

ences can help others and when they are

counselled by people who speak their own

language.

Finally, there may be some obstacles against

patient empowerment. First, citizens must able

to trust in technology empowering them in

order to play their proactive roles. Thus, con-

sidering carefully, the application of technology

to empower patients is required. For example,

Web pages as information sources are promis-

ing for rising health literacy of patients, but

they can lead to misunderstandings and wrong

decisions without a precise assessment by trust-

worthy third party; or privacy and confidential-

ity of information flows from/to EHR and

PHR should not be a secondary feature of sys-

tems. Indeed, security is an essential require-

ment on communication and health

information systems, but only 6% of articles

focus on (or include as complement) security

mechanism.

Another major obstacle is reluctance of doc-

tors to lose their power. Health professionals

encourage citizens to be informed and adhere

to their treatment plans, but sometimes more

knowledge is not desirable. This fact is conse-

quence of people using Internet as first source

of health information without considering the

harm that unreliable and understandable infor-

mation can make. But a well-educated patient

through precisely assessed information sources

by health professionals is a win–win scenario.
Obviously, also citizens may desire to hold the

status quo, that is, maintain a passive role on

their health, as some studies pointed out previ-

ously.
34

Thus, involved actors’ attitudes

towards PE will determine the real swift of

health-care delivery models and the role of

each actor.

Conclusion

In conclusion, practical approaches to

empower patients vary in scope, aim and tech-

nology. The set of areas where empowerment

may be accomplished is so wide (as was

showed in Fig. 5) that almost any current ini-

tiative of ICT applied to health covers mecha-

nisms for empowering patients. As has been

reviewed, there exist different (in scope and

autonomy grade of citizens) levels of empower-

ment that may be mapped to specific mile-

stones. Current technology already allows

establishing the first steps in the road ahead,

but a change of attitude by all stakeholders

(i.e. professionals, patients and policy makers)

is required. Furthermore, despite motivation,

PE strongly depends on accessibility of solu-

tions and interfaces. For a real empowerment

of patients, all citizens must be capable of

accessing systems empowering them, no matter

their digital literacy, economic level, education

or disabilities.
35

Therefore, if obstacles and

How technology is empowering patients? J Calvillo, I Rom�an and L M Roa650

gaps are successfully addressed, at medium-

term technology will ease the emergence of a

new patient fully equipped for the health-care

challenging scenarios of the 21st century.

Source of funding

This work has been partially supported by the

CIBER-BBN (inside project PERSONA), the

Biomedical Engineering Group at University

of Seville, an Excellence Project of the Anda-

lusian Council (TIC-6214) and a grant from

the Fondo de Investigaci�on Sanitaria inside

project PI082023. CIBER-BBN is an initiative

funded by the VI National R&D&i Plan

2008–2011, Iniciativa Ingenio 2010, Consolider
Program, CIBER Actions and financed by the

Instituto de Salud Carlos III with assistance

from the European Regional Development

Fund.

Conflicts of interest

No conflicts of interest occurred.

References

1 Hasman A. Medical informatics: quo vadis?

International Journal of Medical Informatics, 2002;

66: 113–120.
2 Gage EA, Panagakis C. The devil you know:

parents seeking information online for paediatric

cancer. Sociology of Health and Illness, 2012; 34:

444–458.
3 Haux R. Aims and tasks of medical informatics.

International Journal of Medical Informatics, 1997;

44: 9–20.
4 Haux R, Ammenwerth E, Herzog W, Knaup P.

Health care in the information society. A prognosis

for the year 2013. International Journal of Medical

Informatics, 2002; 66: 3–21.
5 Frank SR. Digital health care – The convergence of
health care and the internet. Journal of Ambulatory

Care Management, 2000; 23: 8–17.
6 Bos L, Marsh A, Carroll D, Gupta S, Rees M.

Patient 2.0 empowerment. Proceedings of the 2008

International Conference on Semantic Web and

Web Services, SWWS 2008; 164-168.

7 Solovy A. The big payback: 2001 Survey shows a

healthy return on investment for info tech. Hospitals

and Health Networks, 2001; 75: 40–50.

8 Gibson CH. A concept analysis of empowerment.

Journal of Advanced Nursing, 1991; 16: 354–361.
9 Loukanova SN, Bridges JFP. Empowerment in

medicine: an analysis of publication trends 1980–
2005. Central European Journal of Medicine, 2008; 3:

105–110.
10 Anderson RM, Funnell MM. Patient empowerment:

myths and misconceptions. Patient Education and

Counselling, 2010; 79: 277–282.
11 Health literacy. National network of libraries of

medicine. Available at: http://nnlm.gov/outreach/

consumer/ hlthlit.html, accessed 17 May 2013.

12 Schulz PJ, Nakamoto K. Health literacy and patient

empowerment in health communication: the

importance of separating conjoined twins. Patient

Education and Counseling, 2013; 90: 4–11.
13 Parent F, Coppieters Y, Parent M. Information

technologies, health, and “globalization”: anyone

excluded? Journal of Medical Internet Research,

2001; 3: 134–139.
14 Brodie M, Flournoy RE, Altman DE, Blendon RJ,

Benson JM, Rosenbaum MD. Health information,

the internet, and the digital divide. Health Affairs,

2000; 19: 255–265.
15 Buckingham D. Defining digital literacy – What do

young people need to know about digital media?

Nordic Journal of Digital Literacy, 2006; 4: 263–276.
16 D’Alessandro DM, Dosa NP. Empowering children

and families with information technology. Archives

of Pediatrics and Adolescent Medicine, 2001; 155:

1131–1136.
17 Lenert L. Transforming healthcare through patient

empowerment. Studies in Health Technology and

Informatics, 2010; 153: 159–175.
18 Colombo C, Moja L, Gonzalez-Lorenzo M, Liberati

A, Mosconi P. Patient empowerment as a

component of health system reforms: rights, benefits

and vested interests. Internal and Emergency

Medicine, 2012; 7: 183–187.
19 Monteagudo JL, Moreno O. e-Health for patient

empowerment in Europe. Report, 2007.

20 Chatzimarkakis J. Why patients should be more

empowered: a European perspective on lessons

learned in the management of diabetes. Journal of

diabetes science and technology, 2010; 4: 1570–1573.
21 Aujoulat I, d’Hoore W, Deccache A. Patient

empowerment in theory and practice: polysemy or

cacophony? Patient Education and Counselling, 2007;

66: 13–20.
22 Laugharne R, Priebe S. Trust, choice and power in

mental health: a literature review. Social Psychiatry

and Psychiatric Epidemiology, 2006; 41: 843–852.
23 Macq J, Torfoss T, Getahun H. Patient

empowerment in tuberculosis control: reflecting on

past documented experiences. Tropical Medicine and

International Health, 2007; 12: 873–885.

How technology is empowering patients?, J Calvillo, I Rom�an and L M Roa 651

24 Virtanen H, Leino-Kilpi H, Salanter€a S.

Empowering discourse in patient education. Patient

Education and Counselling, 2007; 66: 140–146.
25 Lober WB, Flowers JL. Consumer empowerment in

health care amid the internet and social media.

Seminars in Oncology Nursing, 2011; 27: 169–182.
26 References of the review. Available at: http://

gibserv.us.es/wiki/doku.php?id=tools:patient-

empowerment, accessed 17 May 2013.

27 Wiljer D, Leonard KJ, Urowitz S et al. The anxious

wait: assessing the impact of patient accessible

EHRs for breast cancer patients. BMC Medical

Informatics and Decision Making, 2010; 10: 46.

28 Uckert F, G€orz M, Ataian M, Tessmann S,

Prokosch H. Empowerment of patients and

communication with health care professionals

through an electronic health record. International

Journal of Medical Informatics, 2003; 70: 99–108.
29 Kharrazi H, Faiola A, Defazio J. Healthcare game

design: behavioral modeling of serious gaming

design for children with chronic diseases. Lecture

Notes in Computer Science, 2009; 5613: 335–344.

30 Layzell B, Manning B, Benton S. The elderly

demographic time bomb – Sharing the load with the
active ageing: can eHealth technologies help defuse

it? Studies in Health Technology and Informatics

Series, 2009; 146: 166–170.
31 O’Cathain A, Goode J, Luff D, Strangleman T,

Hanlon G, Greatbatch D. Does NHS Direct

empower patients? Social Science and Medicine,

2005; 61: 1761–1771.
32 Rich M. Health literacy via media literacy: video

intervention/prevention assessment. American

Behavioral Scientist, 2004; 48: 165–188.
33 Falc~ao-Reis F. Patient empowerment. Information

and privacy mechanisms for the 21st century.

Master thesis, 2011.

34 Salmon P, Hall GM. Patient empowerment or the

emperor’s new clothes. Journal of the Royal Society

of Medicine, 2004; 97: 53–56.
35 Angelmar R, Berman PC. Patient empowerment

and efficient health outcomes. Financing sustainable

healthcare in Europe: new approaches for new

outcomes 2007.

How technology is empowering patients? J Calvillo, I Rom�an and L M Roa652

Copyright of Health Expectations is the property of Wiley-Blackwell and its content may not
be copied or emailed to multiple sites or posted to a listserv without the copyright holder’s
express written permission. However, users may print, download, or email articles for
individual use.

Instruction read carefully

here are essentially three types of peer-reviewed journal articles:

1. A study (can be
qualitative
or
quantitative, and primary or secondary
)- where the authors of the article actually carried out a study with a stated methodology, collected data, and analyzed the data to reach a conclusion about a specific study question

2. A literature review- where the authors of the article search the existing journal articles and studies about a specific topic and write a comprehensive review about the topic- but do not actually carry out a study and/or collect data

3. A meta-analysis- where the authors of the article combine the results of several high-quality articles that used similar methods to collect and analyze data into one summary statistic- starts with a literature review, but goes a step further by doing data analysis of the data and outcomes from other studies

In past semesters, students have stated that they sometimes don’t feel truly comfortable with the differences between these three types of articles. They all have different formats, purposes, and types of information, so it’s important that we can establish the core differences early in the semester so you know what to look for moving forward. In chapter 22, you will learn about numbers 2 and 3 (for more information, you can check out this article:

Systematic Literature Reviews and Meta-Analyses (Links to an external site.)

For this assignment, you must write a 2-3 page (that means a minimum of 2 full pages and no longer than 3 full pages) paper (in APA format) with the following:

1. Pick ONE broad topic within healthcare (examples: cancer, Medicare, anxiety, pregnancy, health information technology, nutrition, physical therapy, etc.).

2. Find ONE peer-reviewed article in EACH of the above three article types about your topic (so three articles total- one study article, one literature review article, one meta analysis article).

3. For each article, describe the following:

· Purpose of the article

· Format of the article (i.e. length, subheadings, etc.)

· What did you learn about your topic from this kind of article?

4. Finally, address the following:

· Which article seems the most informative about your topic overall?

· Which article was easiest to read? Which was the most difficult?

How will you know which article is which type? Studies usually have specific sections for methodology (i.e. where they explain what they did in detail) and data collection/analysis. Literature review articles usually do not have a methodology or, if they do, their methodology consists only of the guidelines they followed to find the articles they used. Meta analysis articles do have methodology and data analysis sections, but they describe which existing articles they used (kind of like a literature review). An even better indicator: most literature review and meta analysis articles will have the terms “literature review” or “meta-analysis” somewhere in the title or abstract- read carefully!

RESEARCH ARTICLE Open Access

Using routine health information data for
research in low- and middle-income
countries: a systematic review
Yuen W. Hung1, Klesta Hoxha1, Bridget R. Irwin2, Michael R. Law3 and Karen A. Grépin4*

Abstract

Background

: Routine health information systems (RHISs) support resource allocation and management decisions at
all levels of the health system, as well as strategy development and policy-making in many low- and middle-
income countries (LMICs). Although RHIS data represent a rich source of information, such data are currently
underused for research purposes, largely due to concerns over data quality. Given that substantial investments have
been made in strengthening RHISs in LMICs in recent years, and that there is a growing demand for more real-time
data from researchers, this systematic review builds upon the existing literature to summarize the extent to which
RHIS data have been used in peer-reviewed research publications.

Methods

: Using terms ‘routine health information system’, ‘health information system’, or ‘health management
information system’ and a list of LMICs, four electronic peer-review literature databases were searched from
inception to February 202,019: PubMed, Scopus, EMBASE, and EconLit. Articles were assessed for inclusion based on
pre-determined eligibility criteria and study characteristics were extracted from included articles using a piloted
data extraction form.

Results

: We identified 132 studies that met our inclusion criteria, originating in 37 different countries. Overall, the
majority of the studies identified were from Sub-Saharan Africa and were published within the last 5 years. Malaria
and maternal health were the most commonly studied health conditions, although a number of other health
conditions and health services were also explored.

Conclusions

: Our study identified an increasing use of RHIS data for research purposes, with many studies applying
rigorous study designs and analytic methods to advance program evaluation, monitoring and assessing services,
and epidemiological studies in LMICs. RHIS data represent an underused source of data and should be made more
available and further embraced by the research community in LMIC health systems.

Keywords: Routine health information systems, Low- and middle-income countries, Systematic review

Background
Routine health information systems (RHISs) collect and
provide information at regular intervals on services and
activities delivered in health facilities [1]. RHISs have
been implemented in many low and middle-income

country (LMIC) health systems to support resource allo-
cation and day-to-day management decisions at facility,
district, provincial, and national levels, as well as to fa-
cilitate strategy development and policy-making [2, 3].
Despite the fact that RHISs are being implemented at
scale in many LMICs, and that they have been widely
recognized as an important component of health sys-
tems strengthening [4, 5], prior studies have suggested

© The Author(s). 2020 Open Access This article is licensed under a Creative Commons Attribution 4.0 International License,
which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give
appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if
changes were made. The images or other third party material in this article are included in the article’s Creative Commons
licence, unless indicated otherwise in a credit line to the material. If material is not included in the article’s Creative Commons
licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain
permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/.
The Creative Commons Public Domain Dedication waiver (http://creativecommons.org/publicdomain/zero/1.0/) applies to the
data made available in this article, unless otherwise stated in a credit line to the data.

* Correspondence: kgrepin@hku.hk
4School of Public Health, Hong Kong University, Pok Fu Lam, Hong Kong
Full list of author information is available at the end of the article

Hung et al. BMC Health Services Research (2020) 20:790
https://doi.org/10.1186/s12913-020-05660-1

http://crossmark.crossref.org/dialog/?doi=10.1186/s12913-020-05660-1&domain=pdf

http://creativecommons.org/licenses/by/4.0/

http://creativecommons.org/publicdomain/zero/1.0/

mailto:kgrepin@hku.hk

that researchers continue to prefer using intermittent
cross-sectional population-based surveys rather than
RHISs data to conduct studies, including the monitoring
of health programs and policy evaluations [6–8].
In order to improve health system performance, reli-

able, timely, and transparent data on health services are
crucial [9, 10]. RHISs collect such data and thus could
provide important insights into health system perform-
ance [4]. Substantial investments have been made in the
development and strengthening of RHISs in many LMICs
over the past two decades [5, 11], and interventions tar-
geting data collection, processing, analysis, and dissemin-
ation have increased the accessibility of RHIS data [5,
12]. While early RHISs were established using paper-
based health facility reports, newer web-based systems
have been adopted in many LMICs over the last decade
[13, 14]. The most common of these is the District
Health Information System 2 (DHIS 2) platform, which is
used as the foundation for the national health manage-
ment information systems (HMIS) in at least 46 countries
and has been piloted in at least another 21 countries [15].
Studies have shown that the implementation of newer in-
formation and communication technology systems, along
with supportive feedback mechanisms to encourage their
use in routine practice, can lead to substantial improve-
ments in RHIS reporting and data quality [5, 13, 16, 17].
Despite the documented improvements in data quality,

recent studies have shown a persistent underuse of RHIS
data for research purposes in LMICs [8, 18]. A number
of factors may contribute to the underuse of RHIS data.
Numerous studies and commentators have questioned
the usefulness of RHIS-sourced data to monitor and
evaluate health services due to data quality concerns,
such as incompleteness and inaccuracy [19–23]. Add-
itionally, RHIS data are often not publicly available for
secondary analyses, which further limits their use [24].
Due to these concerns, the research community has
shown a persistent preference to use data sourced from
intermittent cross-sectional population-based surveys ra-
ther than data sourced from RHISs to conduct research
on health service utilization and policy evaluation in
LMICs [8, 18, 25, 26]. However, population-based sur-
veys also have drawbacks, including the fact that they
may be costly [26] and are often unable to generate suf-
ficient data at the district or other subnational-levels
[27]. In addition, reliance on such data may encourage
the use of potentially weak evaluation designs [8] and
may make establishing an appropriate baseline challen-
ging when trying to evaluate specific programs [28].
Intermittent cross-sectional population-based surveys
themselves also suffer from a number of quality
concerns and thus should not be considered the gold
standard for estimating service coverage or other
population-based estimates [29].

Given the potential of RHISs to play a greater role in
the evaluation of health policy and programs and to
monitor the performance of health systems, it is import-
ant to better understand the extent to which such data
are currently being used in research studies. To date,
there have been no systematic reviews of the use of
RHIS data for research purposes beyond studies that
were specific to malaria control [18], a gap this paper
seeks to address. Specifically, we systematically reviewed
the published literature to identify and describe the dif-
ferent ways in which RHIS data have been used in peer-
reviewed research, including the types of health
conditions studied. We also summarized the different
methodologies that have been used to analyze RHIS data
in research and the types of strategies that were applied
to circumvent common RHIS data issues, such as in-
complete or inaccurate data. It is our goal to provide
guidance to other researchers who may be interested in
using such data for research purposes by helping them
to gain a better understanding on how such data have
been successfully used in other contexts.

Methods
This systematic literature review followed the Preferred
Reporting Items for Systematic Reviews and Meta-
Analyses (PRISMA) guidelines. Peer-reviewed published
studies that used data from RHISs in LMICs were in-
cluded in this study, where RHISs were defined as data
systems designed to collect and generate information on
services provided by health facilities at regular intervals
of a year or less [1]. This included data systems that col-
lect information on clinical service delivery, pharmaceu-
ticals, or diagnostic service provision, as well as routine
administrative management. Conversely, systems that
collect individual-level data for clinical decision-making
purposes and pilot systems to test the implementation of
a new data collection component or method were not
considered to be RHISs.

Search strategy
Relevant studies were identified through an electronic
search of four databases of peer-reviewed literature:
PubMed, Scopus, EMBASE, and EconLit — from incep-
tion through February 20, 2019, the date we launched
the search. For each database, we identified studies that
contained any of the following free text terms in their ti-
tles or abstracts: ‘routine health information system’,
‘health information system’, or ‘health management in-
formation system’, and any LMIC, as defined by the
World Bank’s 2019 classifications (Appendix 1). Articles
were included in the study if they met the following cri-
teria: a) full-text article available in English, b) original
research, and c) used data from a RHIS in at least one
LMIC for research purposes. In order to be considered

Hung et al. BMC Health Services Research (2020) 20:790 Page 2 of 15

as having used data from a RHIS for research purposes,
studies had to involve an analysis, either descriptive or
analytical, of RHIS data, or applied RHIS data to inform
their study design. We excluded studies that: a) only de-
scribed RHISs, b) only described the administrative
decision-making process, c) only focused on RHIS data
collection issues, or d) only assessed RHIS data quality.

Selection of studies
Figure 1 shows the number of articles identified and
retained at each stage of the systematic review process.
After removing duplicates from the various database
searches, we identified 1459 potential articles. Two re-
viewers independently screened the search results by
title and abstract for inclusion eligibility. When there
was insufficient information to determine eligibility at
the title and abstract screening stage, the article was in-
cluded for full-text screening. Full texts of the potentially
eligible articles were then obtained and further screened

for inclusion eligibility. At both stages, the reasons for
excluding individual articles were recorded. The full-
texts for all but one article were found. Disagreements at
each stage were resolved through discussion. Where an
agreement could not be reached, a third reviewer made
the final determination.

Data extraction and analysis
Two authors extracted data from all included studies
using a piloted data extraction form. For each included
article, data were extracted on study design, study ob-
jective, disease or health condition categories, study
sample, description of RHIS data used, use of other data
sources, analytic methods of RHIS data, strategies ap-
plied to circumvent data quality issues, and study find-
ings. Due to the heterogeneity of the studies in terms of
study design, study purpose, health conditions, and ana-
lysis methods, we thematically analyzed the studies ac-
cording to research purpose, types of diseases studied,

Fig. 1 PRISMA flowchart of study identification and screening process of publications use RHIS data

Hung et al. BMC Health Services Research (2020) 20:790 Page 3 of 15

analytic methods applied, impact factor of journals in
which the articles were published, and types of strategies
used to circumvent RHIS data quality issues.

Results
Of the 1459 unique articles retrieved from the database
search, 132 studies met the inclusion criteria after full-
text screening and were thus included in the review. The
characteristics of these studies are presented in Table 1.
Our review identified studies from 37 different countries.
Three quarters of the studies were from Sub-Saharan
African countries (74%), followed by South Asia (11%).
The vast majority of the studies were published in the
last decade, and more than half were published after
2014 (55%), suggesting an increase in the use of RHIS
data for research purposes over time. Most of the studies
included an analysis of RHIS data (97%), and a few used
RHIS data to inform the study but did not describe ana-
lysis of RHIS data. One study, for example, used infor-
mation from RHISs to justify for the selection of the
indicators to be used at the individual-level in their
study. Among the studies that analyzed RHIS data, most
utilized an ecological study design (79%). Of those, more
than half included statistical inferences (61%), while the
remaining studies only used RHIS data for descriptive
purposes (39%). Nearly a fifth of the studies were mixed
methods or case studies (18%), a third of which included
statistical analyses of RHIS data (33%). A quarter of
articles included a description of how they managed
missing data (25%), while only a small number of
studies described how they detected and dealt with
extreme values (14%).

Types of disease and research purpose
Figure 2 shows the different research purposes for which
RHIS data were used, along with the health topics inves-
tigated. The most common purpose of the studies was
program evaluation (51%). RHIS data have been used to
evaluate a wide range of interventions, ranging from
programs that targeted specific diseases to interventions
or policies that affected multiple types of diseases or
health services. These included: the effect of malaria
control strategies [30–36], user fee exemption policies
[37–40], health financing schemes [41–44], interventions
on health governance [45–53], the administration of
new vaccines and vaccination campaigns [54–56], as well
as community-level interventions such as approaches to
enhance community participation and improve referrals
from traditional birth attendants in increasing the de-
mand for maternal and child care [57–59].
Additionally, RHIS data were used to monitor or as-

sess service provision (23%) and to describe disease epi-
demiology (17%). Similar to the program evaluation
studies, these studies also investigated a diverse set of

health services and the allocation of healthcare re-
sources. Some of these studies found large discrepancies
between RHIS data and an estimated disease burden in
populations or highlighted the lack of service provision.
A few studies also used RHIS data to describe specific
programs [60–64], conduct impact evaluations (non-
programmatic) [65–68], and estimate costs [69, 70].
Most of the studies investigated a communicable dis-
ease (95%), of which malaria was most studied health
condition (24%). A few studies focused on mental
health (2%), diabetes mellitus (1%), and permanent
tooth extraction (1%). Only two studies used RHIS
data to research the health workforce or the equity of
funding allocations [71, 72].

Analytic methods using RHIS data
Among articles that conducted statistical analyses using
RHIS data (n = 68), time series analyses to test or ac-
count for trends were most commonly performed (25%),
followed by geostatistical analyses (16%), pre-post com-
parisons (15%), interrupted time series (ITS) (10%), and
difference-in-difference analyses (7%). Other longitudinal
analyses (13%), other cross-sectional analyses (12%), and
scenario analysis on cost effectiveness (2%) were also
conducted. Table 2 presents the range of methodologies
identified across studies using RHIS data, as well as the
corresponding articles.

Time series analysis

Time series analysis using RHIS data was most often ap-
plied to evaluate programs and identify disease epidemi-
ology, with one study assessing the impact of an
infectious disease outbreak on primary health service
utilization [82]. Studies analyzed indicators using large
quantities of monthly or yearly data to estimate change
(range of time units: 5–168). For instance, two-thirds of
the studies analyzed three or more years of monthly
data. Many of the studies utilized the highly disaggre-
gated nature of the data by using either facility or district
level data, with the exception of two studies which mod-
elled national trends [33, 116]. Studies commonly ap-
plied strategies to account for temporal autocorrelation
and the correlation between geographical units, includ-
ing generalized linear models [58], multi-level analysis
[77, 78], and ordinary least-squares regression with ad-
justment for seasonality and lag [34, 37, 117]. Among
studies that modelled multiple facilities or administrative
regions, random effects were commonly applied to ac-
count for heterogeneity.
In addition to RHIS data, a number of included studies

incorporated data from external sources in their models
based on geographical location such as district or region.
Studies of malaria, for example, commonly included cli-
mate data from satellites in their models to control for

Hung et al. BMC Health Services Research (2020) 20:790 Page 4 of 15

important temporal factors, for example precipitation,
humidity, and temperature [73, 117]. Other studies in-
corporated information from other national community
surveys, health facility surveys, and program data as co-
variates [34, 77]. While most studies controlled for po-
tential confounders by including covariates in analytic
models, one study on maternal health service applied
propensity score matching to further remove biases from
differences in covariate distribution [37].

Geostatistical analysis

Geostatistical analyses using RHIS data were predomin-
antly conducted for epidemiological purposes and the
monitoring and assessment of service provision by
exploiting geospatial information included in the RHIS
at the facility or district level. Three of the studies that
applied geostatistical analysis were cross-sectional, while
the remainder were spatial-temporal. About half of the
studies focused on malaria, of which three compared
and illustrated various kriging methods to provide a reli-
able estimate of malaria burden amid missing reporting
[105–107], and one study applied geostatistical modeling
to select the most relevant health facility indicators for
severe malaria outcomes [108]. Studies on other topics
investigated the spatial or spatial-temporal dynamics of
malaria in pregnancy [100], childhood diarrhea [101],
clustering of malaria and HIV [102], and meningitis
[118]. About half of the studies did not include data
from external sources, and others triangulated data
sourced from satellite data, Demographic and Health
Surveys, national Malaria Indicator Surveys, and Service
Delivery Indicator Surveys in their analyses. Studies that
included covariates in the geostatistical analysis applied
Bayesian hierarchical Poisson models or Bayesian geosta-
tistical negative binomial models [103, 108, 110].

Pre-post comparison analysis

Pre-post comparison was commonly applied among
studies that used RHIS data for program evaluation, and
several studies used simple descriptive statistics to

Table 1 Characteristics of research studies that used RHIS data

n Percent

Geographical region

East Asia and Pacific 8 6.1

Latin America and the Caribbean 9 6.8

Middle East and North Africa 2 1.5

South Asia 15 11.4

Sub-Saharan Africa 98 74.2

Year of publication

< 2000 3 2.3

2000–2004 7 5.3

2005–2009 10 7.6

2010–2014 40 30.3

2015–2019 72 54.5

RHIS data as source or to inform study

Data source 128 97.0

Inform study 4 3.0

Types of study design

Ecological study – cross-sectional 13 9.8

Ecological study – longitudinal 51 38.6

Ecological study – descriptive 41 31.1

Case study 11 8.3

Mixed methods study 13 9.8

Cross-sectional study 1 0.8

Pre- and post-intervention study 1 0.8

Nested clustered randomized controlled trial 1 0.8

Data use purpose

Program evaluation 67 50.8

Epidemiology 23 17.4

Monitoring and assessment of service provisions 30 22.7

Program description 6 4.5

Impact evaluation 4 3.0

Cost estimation 2 1.5

Health conditions/service type

General (multiple aspects) 21 15.9

Secondary health utilization 2 1.5

General causes of death 1 0.8

Maternal and Child health/healthcare 12 9.1

Maternal health/healthcare 24 18.2

Child health/healthcare 11 8.3

Vaccine prevented childhood illnesses 10 7.6

Malaria 30 22.7

Malaria & HIV/AIDS 1 0.8

Malaria & other parasitic diseases 1 0.8

HIV and related diseases 8 6.1

Mental health/healthcare 3 2.3

Table 1 Characteristics of research studies that used RHIS data
(Continued)

n Percent

Other diseases 5 3.8

Healthcare workforce and other resources 2 1.5

Data issue of RHIS: missingness

Described how missing data was managed 33 25.0

No description of how missing data was managed 99 75.0

Data issue of RHIS: outlier

Described how outlier was detected 19 14.4

No description of how outlier was detected 113 85.6

Hung et al. BMC Health Services Research (2020) 20:790 Page 5 of 15

compare the periods before and after interventions.
As pre-post comparison is subject to the limitation of
temporal confounders and secular trends, two of the
studies included contextual factors in regression mod-
elling [35, 119].

Interrupted time series analysis

Most of the studies that conducted ITS analysis used it
to evaluate interventions, and one assessed the impact of
an infectious disease outbreak on maternal and child
health service use [68]. The studies used large quantities
of monthly data to model trend and level change (range
of time unit: 44–132). RHIS data were minimally aggre-
gated in these studies, which mostly analyzed facility or
district level data, and similar to studies using time series
analysis, accounted for autocorrelation through incorp-
orating autoregressive structures or clustered standard
errors in their modelling.
As ITS analyses are generally unaffected by confound-

ing variables that do not change over time by design

[120], baseline characteristics were typically not included
in these models. Nonetheless, ITS analyses can be af-
fected by time-varying confounders that rapidly change
and some models included contextual factors from other
data sources, such as climate and program data. To
strengthen the quasi-experimental design, two studies
also included a contrast group of time series to control
for contextual changes that occurred at the same time as
the interventions [38, 45].

Difference-in-difference analysis

Five studies applied difference-in-difference techniques
using a wide range of time periods (range of time units:
4–48) and levels of geographical units (facility, district,
provincial). Only one study included contextual charac-
teristics from other data sources in its analysis. Analytic
methods varied from descriptive comparison between
and within intervention and control groups [41, 59, 87,
88], to ordinary least square regression with propensity
score matching [42].

Fig. 2 Types of service and research purpose of RHIS data use (n = 132)

Hung et al. BMC Health Services Research (2020) 20:790 Page 6 of 15

Table 2 Types of analytic methods applied among studies that analyzed RHIS data

Data use
purpose

Type of disease/service
studied

Range
of data
(unit)

Level of
aggregation

Analytic methods Other information sources
included

Reference

Time series analysis

Epidemiology
Child health, malaria, tooth
extraction

15 (year)
– 120
(month)

Ward,
municipal,
district

Time series correlograms;
ordinary least-squares regres-
sions adjusted for seasonality
and lag; non-linear time series
correlation and regressions

GPS coordinates, Climate
Hazards Group Infrared
Precipitation with Station Data,
satellite data, meteorological
department data, program
data

[73–76]

Program
evaluation

General, maternal and
child health, maternal
health, vaccine prevented
childhood illnesses, malaria

5 (year)
– 168
(month)

Facility,
district,
region,
nation

Ordinary least squares
regression; negative binomial
generalized linear model;
random effects negative
binomial regressions; switching
regression methods weighted
by propensity scores

Program data, program reports,
data from Bureau of Statistics
and Ministry of Health, Malaria
Indicator Survey, Demographic
Health Survey, Health Facility
Survey, community survey,
satellite data, sentinel site case-
investigations/surveillance, ab-
straction from hospital
registries

[33, 34,
37, 40,
54, 55,
58, 77–
81]

Impact
evaluation
(non-
program)

General 84
months
(month)

Facility Linear mixed-effect time-series
analysis with a segmented re-
gression parameterization

None [82]

Interrupted time series analysis
Program
evaluation

General, maternal and
child health, maternal
health, malaria

53
(month)
– 132
(month)

Facility,
intervention
vs. control
groups,
district

Generalized least square model
with autoregressive structure;
generalized least square model
with controls, with
autoregressive process and
moving average process;
segmented linear regression

Meteorology Department data,
program data, facility survey

[38, 45,
83–86]

Impact
evaluation
(non-
program)

Maternal and child health 44
(month)

District Segmented linear regression
with district fixed effect and
clustered standard error at
district level

Demographic Health Survey [68]

Difference-in-difference analysis
Program
evaluation

General, child health,
maternal health

4 (year)
– 48
(month)

Facility,
district,
province

Ordinary least squares
regression with and without
propensity score matching;
Wilcoxon rank-sum test on me-
dian difference-in-differences
between facilities; descriptive
comparison of means

Verified data from
Performance-Based Financing
system

[41, 42,
59, 87,
88]

Pre-post comparison analysis
Program
evaluation

Child health, maternal
health, maternal and child
health, vaccine prevented
childhood illnesses,
malaria, HIV or related
diseases

2 (year)
– 48
(month)

Facility,
district

Chi-square test; Pearson
correlation; Wilcoxon signed-
rank test; paired sample t-test;
linear regressions; Poisson re-
gression; negative binomial re-
gression; logistic regression

Bureau of Statistics data,
program reports,
Meteorological Department
data, entomological sentinel
surveys, Demographic and
Health Survey, UN Interagency
Group for Childhood Mortality
Estimation(CME Info) database,
abstraction from facility
registers, community surveys,
vital registry, provincial
maternal death notification
register

[35, 39,
48, 57,
89–93]

Impact
evaluation
(non-
program)

Child health 26
(month)

District Pearson chi-square test District hospital registers, Safe
and dignified burials for all
deaths database

[67]

Other longitudinal analysis

Hung et al. BMC Health Services Research (2020) 20:790 Page 7 of 15

Impact of research using RHIS data
Most of the studies that conducted statistical analyses
using RHIS data were published in journals with impact
factors (88%, Fig. 3), two-thirds of which were two or
higher, and more than a fifth of which were greater than
three. Among those studies published in journals with

the highest impact factors, most of them focused on
program evaluation (53%), followed by monitoring
and assessment of service provision (20%), epidemi-
ology (20%) and impact evaluation (7%). These studies
encompassed a range of health topics commonly stud-
ied using RHIS data.

Table 2 Types of analytic methods applied among studies that analyzed RHIS data (Continued)

Data use
purpose
Type of disease/service
studied
Range
of data
(unit)
Level of
aggregation
Analytic methods Other information sources
included
Reference

Epidemiology
Maternal health, malaria 12 (year)

– 16
(year)

District Chi-square test; negative
binomial regression

Review of hospital death
records

[94, 95]

Monitoring
and
assessment
of service
provision

HIV or related diseases 3 (year) District Descriptive comparison over
time

Surveys with health facility
managers

[96]

Program
evaluation

Genera, child health,
malaria, malaria and other
parasitic diseases

3 (year)
– 24
(month)

Facility,
district,
nation

Poisson regression to explore
association between
intervention coverage and
disease burden; Mann–Whitney
U Test to compare prevalence
in intervention and non-
intervention area; linear regres-
sion model; student t-test

Sentinel surveillance data,
program reports, national
facility and community survey,
Bureau of Statistics data,
program data

[47, 52,
66, 97–
99]

Geostatistical analysis

Epidemiology
Child health, malaria,
malaria and HIV/AIDS,
meningococcal meningitis

1 (year)
– 520
(week)

District Cluster analysis; cross-
correlations of different spatial
scales between time series of
cases; Bayesian hierarchical
Poisson model and smoothed
model estimates plotted on dis-
trict maps

Malaria Indicator Survey,
Demographic Health Survey,
program data

[100–104]

Monitoring
and
assessment
of service
provision

Malaria, maternal health 1 (year)
– 57
(month)

Facility,
district

Kriging (ordinary kriging, space-
time ordinary kriging, local
space-time ordinary kriging);
Bayesian geostatistical negative
binomial model

Service Delivery Indicator
Survey

[105–109]

Program
evaluation

Malaria 36
(month)

District Bayesian geostatistical models
and Bayesian generalized linear
models

Malaria Indicator Survey,
malaria control program data,
satellite data, Demographic
Health Survey, ACTWatch
household surveys

[110]

Other cross-sectional analysis

Epidemiology
Maternal health Median

of 24
months

Province Linear regression model None [111]

Monitoring
and
assessment
of service
provision

General, child health,
maternal health, mental
health

1 (year) Facility,
district,
municipality,
state

Descriptive statistics, Tobit
regression model, bivariate and
multivariate linear regression
models,

Nutrition Service Delivery
Assessment, abstraction from
Integrated Nutrition Register,
structured questionnaire with
district health officers, District-
level household and facility
surveys, National Register of
Health Service Providers, data
from Institute of Geography
and Statistics

[112–115]

Program
evaluation

HIV and related diseases 1 (year) District Mixed-methods Register reviews and a series of
patient folder (health record)
reviews

[51]

Hung et al. BMC Health Services Research (2020) 20:790 Page 8 of 15

Strategies to circumvent RHIS data quality issues
Data quality is commonly cited as a barrier to using
RHIS data in research, and slightly more than a quarter
of the included studies described the strategies that they
used to handle missing data and/or identify extreme
values (Table 3). These strategies consisted of exclusion,
imputation, interpolation, verification, and accounting
for missing data in modeling. Exclusion of missing data
was the most common practice, and among studies that
used this technique, they excluded facilities from the
analytic samples [38, 41, 45, 52, 65, 79, 83, 84, 87, 94, 96,
121], restricted the study period based on explicit cri-
teria [54, 122], or applied sensitivity analysis to compare
various exclusion criteria [41, 89, 90]. Imputation
methods varied from assigning specific values to the
missing observation [42, 87, 118, 123–125], to various
modeling strategies such as conditional autoregressive
model [110], generalized linear regression [124], and it-
erative singular value decomposition [124]. A sensitivity
analysis was also conducted to select a specific imput-
ation strategy [124]. Interpolation involves predicting
values at unsampled locations. Methods described in-
cluded the use of space-time kriging [105–107], and the
adjustment of results by calibrating with other relevant
information [52, 53, 55]. Some studies assumed data
were missing at random, which was accounted for in
specific modeling methods such as mixed-effect models
[65, 124]. When the source of data could be reached,
some studies also described verifying the missing infor-
mation using registries where the original data were re-
corded [39, 73, 97, 111, 122].
Slightly fewer articles described methods to identify

and handle extreme values in the RHIS data, of which
three types of strategies emerged: setting specific

thresholds, visual inspection, and analytic assessment.
Thresholds were set based on the distribution of the
data, such as proportions or standard deviations from
univariate regression. Several studies used visual inspec-
tion of outliers [38, 107], while the use of jackknifing
analysis and the identification of influential points
through Cook’s distance statistics were also applied [112,
126]. Upon identification of extreme values, several
strategies were utilized: exclusion, replacement with the
average value, replacement with the missing value, verifi-
cation with a data source, or discounting the observation
in statistical estimation. However, studies that replaced
the extreme value with an explicit value potentially in-
troduced bias into their estimates. A few studies also de-
scribed the strategies applied to assess the reliability of
the RHIS data, some of which were routine processes
administered in the health systems [39, 97].

Discussion

In recent years, there have been increased investments
made to improve the quality of RHIS data in many
LMICs. Over the same time period, we found an in-
crease in published research using RHIS sourced data,
especially over the past 5 years, likely due to the in-
creased availability, accessibility, and quality of RHIS
data [18]. While these studies have made contributions
to the literature, we also found that the total number of
studies conducted (n = 132) remains a small part of the
overall literature base on health system evaluation and
performance in LMICs.
Malaria and maternal health conditions were the most

commonly studied health conditions, despite the fact
that RHISs collect data on a wide range of other diseases
and conditions. In particular, the use of RHIS data for
non-communicable diseases (NCDs) research was very
limited. As LMICs are undergoing an epidemiologic
transition and the importance of NCDs is increasing
[127], LMIC health systems face the increasing chal-
lenges of addressing the dual burden of communicable
and non-communicable diseases [128, 129]. In spite of
the limited implementation of non-communicable disea-
seinterventions [129], the few studies that used RHIS
data for non-communicable disease research mainly ana-
lyzed the gap in service provision and estimated disease
burden, highlighting the large unmet need for health
care in affected populations. A couple of the studies de-
scribed how their research was limited by data availabil-
ity and quality, such as the lack of diagnostic categories
of the investigated health conditions in the RHIS. Future
research should investigate how RHIS data on non-
communicable diseases could better help to provide
insights on its epidemiology and service provision to ad-
dress these health conditions.

Fig. 3 Distribution of impact factor of journals that published
research studies that conducted statistical analysis of RHIS
data (n = 68)

Hung et al. BMC Health Services Research (2020) 20:790 Page 9 of 15

Our systematic review found that many of the studies
took advantage of some of the features of RHIS data, in
particular by exploiting the high frequency nature of
these data at the level of health facilities, as well as com-
bining external information to enhance estimations and
enable assessing new research questions. The triangula-
tion of populational health characteristics,

environmental factors, and service coverage strengthens
the analysis and the understanding of their influence
[130]. In addition, the overlay of different information in
analyses of RHIS data allows for the advancement of re-
search methods. For instance, a recent study demon-
strated how to assess the effects of facility readiness on
severe malaria outcomes through constructing a

Table 3 Strategies applied in research articles to counter issues of RHIS data

Type of strategy Description of strategy

Missing data

Exclusion Exclude facility data if a certain threshold was reached (e.g. more than two-thirds of months in
a year; more than a sixth of baseline data; facilities with any missing data)

Restrict analysis to a period with a low level of missing data

Sensitivity analysis to compare analysis of restricted period and full period

Imputation Assign missing observations with mean-value for the year

Assign missing observations with the average of precedent and subsequent data

Imputation using conditional autoregressive model

Missing value was replaced as positive (binary form) to prevent exaggeration of the
fade-out effect

Sensitivity analysis of imputation strategies: 1) single imputation using means, trimmed means,
and median, 2) Poisson generalized linear modeling, 3) iterative singular value decomposition
method

Interpolation Interpolation using space-time kriging

Adjust results by dividing each indicator by the percentage of reports submitted

Adjust the data by calibrating to the total population using proportion reported in a household
survey to have occurred in health facilities

Verification
Account in the modeling method

Manual verification of the missing data with register at the health facility

Missing data was assumed missing at random and accounted for in the mixed-effect models
using standard maximum likelihood estimation

Identifying extreme values

Specific threshold Establishing a lower and upper limit based on proportion of the annual average or feasible value

Univariate regression on individual facility-level to identify deviation from the mean time trend
(e.g. if exceed 8 standard deviations)

Visual Visual inspection of outliers

Analytic assessment Jackknifing analysis to assess influence

Student residual higher than an absolute value of 2 and influence on the estimated coefficients
determined by high Cook’s distance statistics

Handling of extreme values

Exclusion Extreme values were excluded from analyses

Replacing extreme value with average Extreme values were assigned the average value of the year; with exceptions of low average values

Replacing extreme value with missing Outliers set to missing

Verification with data source Any drastic change in monthly data reported electronically were manually verified with register at
the health facility. Discrepancies were replaced with data in the register

Discount observation in estimation Outliers were allocated a dummy coding to discount the observation in the calculation of
coefficients

Assess reliability

Data validation process Randomly selected 10% of the total sample to check accuracy and reliability of data with reports
and registers

Verify data with another source (e.g. payroll)

Established routine data validation process by health information and records officer
(e.g. monthly data review meetings)

Hung et al. BMC Health Services Research (2020) 20:790 Page 10 of 15

composite facility readiness index based on health facil-
ity characteristics and spatial data, and using RHIS data
as the outcome variable [108]. The detailed routine
nature of RHIS data and the ability to link with other
geographically based information, including data on
population, environmental, health behavior, and facility
characteristics, can generate high impact research and
advance our understanding of disease epidemiology and
health improvement efforts in LMICs.
Despite the increasing use of RHIS data for research

purposes, the quality of these data remains imperfect
and such issues should be identified and addressed in
order to limit estimation error and bias. RHIS data qual-
ity issues remain a particular concern in some settings
[131–133], however, other studies have shown that strat-
egies that have been implemented to improve RHIS data
across different international contexts can be successful
[5, 134]. Multiple strategies were discussed in the articles
we reviewed in our paper, including strategies to address
common data quality issues such as missingness and
data validity, for example the simple exclusion of miss-
ing data and various imputation and interpolation
methods. However, the majority of the studies that used
RHIS data did not describe the extent of the quality is-
sues or the steps they took to overcome them. The use
of sensitivity analyses in assessing the effect of specific
cut-offs or methods was scarce. Explicit descriptions of
the extent of the data quality issues and the reasons for
selecting a particular approach should be encouraged in
future research.
While our review used major databases and systematic

methods, it nevertheless has some limitations that are
worth noting. First, we included only peer-reviewed
studies that were published in English, and therefore
may have overlooked potentially relevant studies
published in the grey literature or written in other
languages. Additionally, given our focus on original re-
search, we did not search the broader body of literature
for books, reports, or grey literature. Our literature
search also identified phrases that described health infor-
mation systems in title and abstracts only, possibly
resulting in the exclusion of studies that only mentioned
RHIS data use in the full text. Finally, additional variants
on these search terms may have generated more articles
or a slightly different set of articles.

Conclusions
In this systematic review we summarized the use of data
collected from RHISs in LMICs. Overall, we found that
researchers are increasingly using data sourced from
RHISs to conduct health system planning and evaluation
studies in LMIC health systems, however these data
likely remain underutilized by the broader research com-
munity. As many of the studies included in this review

were published in prominent journals and were able to
use strong quasi-experimental or geo-spatial methods,
we believe this makes the case for greater use of these
data for research purposes in the future, which will likely
happen as RHIS data become more openly available to
the research community. However, there is a need to
help build the case to use these data for a broader range
of health conditions and to develop more of a consensus
on methods to deal with data imperfections, given that
our findings underlined the limited use and comparison
of these methods. That said, our review clearly demon-
strates the feasibility of use RHIS data in conjunction
with rigorous study designs and analytic methods in
LMICs. We suggest that future program evaluations
should consider their use more broadly, to assess an in-
creased variety of health conditions in conjunction with,
or as a replacement for, household or facility survey
methods.

Supplementary information

Supplementary information accompanies this paper at https://doi.org/10.
1186/s12913-020-05660-1.

Additional file 1.

Abbreviations

DHIS 2: District health information system 2; HMIS: Health management
information system; ITS: Interrupted time series; LMIC: Low- and middle-
income country; NCD: Non-communicable disease; PRISMA: Preferred
reporting items for systematic reviews and meta-analyses; RHIS: Routine
health information system

Acknowledgements

We acknowledge that this work was conducted on the Haldimand Tract,
traditional territory of the Neutral, Anishinaabe and Haudenosaunee peoples.

Authors’ contributions

YWH, KH, and KAG identified the search strategy. YWH and KH conducted
the literature screening, and KAG was consulted when required. YWH, KH,
and BRI extracted data from the included studies. YWH wrote the
manuscript. KAG and MRL provided feedback and edits on the manuscript.
The authors read and approved the final manuscript.

Funding

This work was supported by AXA Research Fund. The foundation had no
role in the design and implementation of the study, writing of the findings,
or decision regarding any of these aspects of the study.

Availability of data and materials

Not applicable.

Ethics approval and consent to participate

Not applicable.

Consent for publication

Not applicable.

Competing interests

The authors declare that they have no competing interests.

Author details

1University of Waterloo, School of Public Health and Health Systems,
Waterloo, Canada. 2Department of Health Sciences, Wilfrid Laurier University,
Waterloo, Canada. 3Centre for Health Services and Policy Research, The

Hung et al. BMC Health Services Research (2020) 20:790 Page 11 of 15

https://doi.org/10.1186/s12913-020-05660-1

University of British Columbia, Vancouver, Canada. 4School of Public Health,
Hong Kong University, Pok Fu Lam, Hong Kong.

Received: 18 April 2020 Accepted: 16 August 2020

References

1. Hotchkiss DR, Diana ML, Foreit KGF. How can routine health information

systems improve health systems functioning in lowand middle-income
countries? Assessing the evidence base. Adv Health Care Manag. 2012.
https://doi.org/10.1108/S1474-8231(2012)0000012006.

2. AbouZahr C, Boerma T. Health information systems: the foundations of
public health. Bull World Health Organ. 2005;83:578–83.

3. World Health Organization. Everybody’s business–strengthening health
systems to improve health outcomes: WHO’s framework for action; 2007.

4. Lippeveld T, Sauerborn R, Bodart C, Organization WH. Design and
implementation of health information systems; 2000.

5. Mutale W, Chintu N, Amoroso C, et al. Improving health information
systems for decision making across five sub-Saharan African countries:
implementation strategies from the African health initiative. BMC Health
Serv Res. 2013;13(Suppl 2):1–12.

6. Hogan DR, Stevens GA, Hosseinpoor AR, Boerma T. Monitoring universal
health coverage within the sustainable development goals: development
and baseline data for an index of essential health services. Lancet Glob
Health. 2018;6(2):e152–68. https://doi.org/10.1016/S2214-109X(17)30472-2.

7. Cutts FT, Claquin P, Danovaro-Holliday MC, Rhoda DA. Monitoring
vaccination coverage: defining the role of surveys. Vaccine. 2016;34(35):
4103–9. https://doi.org/10.1016/j.vaccine.2016.06.053.

8. Wagenaar BH, Sherr K, Fernandes Q, Wagenaar AC. Using routine health
information systems for well-designed health evaluations in low- and
middle-income countries. Health Policy Plan. 2016;31(1):129–35. https://doi.
org/10.1093/heapol/czv029.

9. World Health Organization. Monitoring the building blocks of health
systems: a handbook of indicators and their measurement strategies.
Geneva; 2010. http://www.who.int/healthinfo/systems/WHO_MBHSS_2010_
full_web ?ua=1.

10. World Health Organization. Framework and standards for country health
information systems (second edition). Geneva; 2012. http://www.who.int/
healthinfo/country_monitoring_evaluation/who-hmn-framework-standards-
chi .

11. Warren AE, Wyss K, Shakarishvili G, Atun R, de Savigny D. Global health
initiative investments and health systems strengthening: a content analysis
of global fund investments. Glob Health. 2013;9(1):30. https://doi.org/10.
1186/1744-8603-9-30.

12. Gimbel S, Mwanza M, Nisingizwe MP, et al. Improving data quality across 3
sub-Saharan African countries using the consolidated framework for
implementation research (CFIR): results from the African health initiative.
BMC Health Serv Res. 2017. https://doi.org/10.1186/s12913-017-2660-y.

13. Kiberu VM, Matovu JKB, Makumbi F, Kyozira C, Mukooyo E, Wanyenze RK.
Strengthening district-based health reporting through the district health
management information software system: the Ugandan experience. BMC
Med Inform Decis Mak. 2014;14:40. https://doi.org/10.1186/1472-6947-14-40.

14. Hazel E, Wilson E, Anifalaje A, Sawadogo-Lewis T, Heidkamp R. Building
integrated data systems for health and nutrition program evaluations:
lessons learned from a multi-country implementation of a DHIS 2-based
system. J Glob Health. 2018;8(2):20307. https://doi.org/10.7189/jogh.08.
020307.

15. Dehnavieh R, Haghdoost A, Khosravi A, et al. The district health information
system (DHIS2): a literature review and meta-synthesis of its strengths and
operational challenges based on the experiences of 11 countries. Heal Inf
Manag J. June 2018:1833358318777713. https://doi.org/10.1177/
1833358318777713.

16. Githinji S, Oyando R, Malinga J, et al. Completeness of malaria indicator data
reporting via the district health information software 2 in Kenya, 2011–2015.
Malar J. 2017;16:344. https://doi.org/10.1186/s12936-017-1973-y.

17. MEASURE Evaluation. Lessons in health information system strengthening:
what worked in the Democratic Republic of the Congo. Chapel Hill; 2019.
https://www.measureevaluation.org/resources/publications/wp-19-229.

18. Ashton RA, Bennett A, Yukich J, Bhattarai A, Keating J, Eisele TP.
Methodological considerations for use of routine health information
system data to evaluate malaria program impact in an era of declining

malaria transmission. Am J Trop Med Hyg. 2017. https://doi.org/10.4269/
ajtmh.16-0734.

19. Rowe AK, Kachur SP, Yoon SS, Lynch M, Slutsker L, Steketee RW. Caution is
required when using health facility-based data to evaluate the health
impact of malaria control efforts in Africa. Malar J. 2009;8(1):209.

20. Hazel E, Requejo J, David J, Bryce J. Measuring coverage in MNCH:
evaluation of community-based treatment of childhood illnesses
through household surveys. PLoS Med. 2013. https://doi.org/10.1371/
journal.pmed.1001384.

21. Lim SS, Stein DB, Charrow A, Murray CJL. Tracking progress towards
universal childhood immunisation and the impact of global initiatives: a
systematic analysis of three-dose diphtheria, tetanus, and pertussis
immunisation coverage. Lancet. 2008;372(9655):2031–46. https://doi.org/10.
1016/S0140-6736(08)61869-3.

22. Cutts FT, Izurieta HS, Rhoda DA. Measuring coverage in MNCH: design,
implementation, and interpretation challenges associated with tracking
vaccination coverage using household surveys. PLoS Med. 2013;10(5):
e1001404. https://doi.org/10.1371/journal.pmed.1001404.

23. Sychareun V, Hansana V, Phengsavanh A, Chaleunvong K, Eunyoung K,
Durham J. Data verification at health centers and district health offices in
Xiengkhouang and Houaphanh provinces, Lao PDR. BMC Health Serv Res.
2014;14(1):255. https://doi.org/10.1186/1472-6963-14-255.

24. Gloyd S, Wagenaar BH, Woelk GB, Kalibala S. Opportunities and challenges
in conducting secondary analysis of HIV programmes using data from
routine health information systems and personal health information. J Int
AIDS Soc. 2016. https://doi.org/10.7448/IAS.19.5.20847.

25. Aqil A, Lippeveld T, Hozumi D. PRISM framework: a paradigm shift for
designing, strengthening and evaluating routine health information
systems. Health Policy Plan. 2009;24(3):217–28.

26. Short Fabic M, Choi Y, Bird S. A systematic review of demographic and
health surveys: data availability and utilization for research. Bull World
Health Organ. 2012. https://doi.org/10.2471/BLT.11.095513.

27. Fernandes QF, Wagenaar BH, Anselmi L, Pfeiffer J, Gloyd S, Sherr K. Effects of
health-system strengthening on under-5, infant, and neonatal mortality: 11-
year provincial-level time-series analyses in Mozambique. Lancet Glob
Health. 2014. https://doi.org/10.1016/S2214-109X(14)70276-1.

28. Ratnayake R, Ratto J, Hardy C, et al. The effects of an integrated community
case management strategy on the appropriate treatment of children and
child mortality in Kono District, Sierra Leone: a program evaluation. Am J
Trop Med Hyg. 2017. https://doi.org/10.4269/ajtmh.17-0040.

29. Bishai DM. Global initiatives in universal childhood immunisation. Lancet.
2008;372(9655):2004–5. https://doi.org/10.1016/S0140-6736(08)61853-X.

30. Seitio-Kgokgwe O, Mashalla Y, Seloilwe E, et al. Developing block wise
composite health index in Yavatmal district, Maharashtra state, India: an
analysis of available data. Malar J. 2017;16(1 PG):35. https://doi.org/10.4103/
2008-7802.160056.

31. Pagnoni F, Convelbo N, Tiendrebeogo J, Cousens S, Esposito F. A
community-based programme to provide prompt and adequate treatment
of presumptive malaria in children. Trans R Soc Trop Med Hyg NS. 1997;91(5
PG-512-7):512–7.

32. Rehman A, Mann A, Schwabe C, et al. Five years of malaria control in the
continental region, Equatorial Guinea. Malar J. 2013;12(PG-154):154. https://
doi.org/10.1186/1475-2875-12-154.

33. Nyarango P, Gebremeskel T, Mebrahtu G, et al. A steep decline of malaria
morbidity and mortality trends in Eritrea between 2000 and 2004: the effect
of combination of control methods. Malar J. 2006;5(PG-33):33. https://doi.
org/10.1186/1475-2875-5-33.

34. Over M, Bakote’e B, Velayudhan R, Wilikai P, Graves PM. Impregnated nets
cannot fully substitute for DDT: the Field effectiveness of alternative methods
of malaria prevention in Solomon Islands, 1993–99. World Bank, Policy Res
Work Pap Ser 3044, 2003. 2003;(PG) https://libproxy.wlu.ca/login?url=https://
search.proquest.com/docview/56217804?accountid=15090NS-.

35. Okullo AE, Matovu JKB, Ario AR, et al. Malaria incidence among children less
than 5 years during and after cessation of indoor residual spraying in
northern Uganda. Malar J. 2017;16(1 PG-319):319. https://doi.org/10.1186/
s12936-017-1966-x.

36. Sande S, Zimba M, Mberikunashe J, Tangwena A, Chimusoro A. Progress
towards malaria elimination in Zimbabwe with special reference to the
period 2003-2015. Malar J. 2017. https://doi.org/10.1186/s12936-017-1939-0.

37. Manthalu G. User fee exemption and maternal health care utilisation at
mission health facilities in Malawi: an application of disequilibrium

Hung et al. BMC Health Services Research (2020) 20:790 Page 12 of 15

https://doi.org/10.1108/S1474-8231(2012)0000012006

https://doi.org/10.1016/S2214-109X(17)30472-2

https://doi.org/10.1016/j.vaccine.2016.06.053

https://doi.org/10.1093/heapol/czv029

http://www.who.int/healthinfo/systems/WHO_MBHSS_2010_full_web ?ua=1

http://www.who.int/healthinfo/country_monitoring_evaluation/who-hmn-framework-standards-chi

https://doi.org/10.1186/1744-8603-9-30

https://doi.org/10.1186/s12913-017-2660-y

https://doi.org/10.1186/1472-6947-14-40

https://doi.org/10.7189/jogh.08.020307

https://doi.org/10.1177/1833358318777713

https://doi.org/10.1186/s12936-017-1973-y

https://www.measureevaluation.org/resources/publications/wp-19-229

https://doi.org/10.4269/ajtmh.16-0734

https://doi.org/10.1371/journal.pmed.1001384

https://doi.org/10.1016/S0140-6736(08)61869-3

https://doi.org/10.1371/journal.pmed.1001404

https://doi.org/10.1186/1472-6963-14-255

https://doi.org/10.7448/IAS.19.5.20847

https://doi.org/10.2471/BLT.11.095513

https://doi.org/10.1016/S2214-109X(14)70276-1

https://doi.org/10.4269/ajtmh.17-0040

https://doi.org/10.1016/S0140-6736(08)61853-X

https://doi.org/10.4103/2008-7802.160056

https://doi.org/10.1186/1475-2875-12-154

https://doi.org/10.1186/1475-2875-5-33

https://libproxy.wlu.ca/login?url=https://search.proquest.com/docview/56217804?accountid=15090NS-

https://doi.org/10.1186/s12936-017-1966-x

https://doi.org/10.1186/s12936-017-1939-0

theory of demand and supply. Health Econ. 2019;(PG). https://doi.org/
10.1002/hec.3856.

38. Nguyen H, Zombré D, Ridde V, De Allegri M. The impact of reducing and
eliminating user fees on facility-based delivery: a controlled interrupted time
series in Burkina Faso. Health Policy Plan. 2018;33(8 PG-948-956):948–56.
https://doi.org/10.1093/heapol/czy077.

39. Njuguna J, Kamau N, Muruka C. Impact of free delivery policy on utilization
of maternal health services in county referral hospitals in Kenya. BMC Health
Serv Res. 2017. https://doi.org/10.1186/s12913-017-2376-z.

40. Ridde V, Richard F, Bicaba A, Queuille L, Conombo G. The national subsidy
for deliveries and emergency obstetric care in Burkina Faso. Health Policy
Plan. 2011. https://doi.org/10.1093/heapol/czr060.

41. Steenland M, Robyn P, Compaore P, et al. Performance-based financing to
increase utilization of maternal health services: evidence from Burkina Faso.
SSM – Popul Heal. 2017;3(PG-179-184):179–84. https://doi.org/10.1016/j.
ssmph.2017.01.001.

42. Falisse JB, Ndayishimiye J, Kamenyero V, Bossuyt M. Performance-based
financing in the context of selective free health-care: an evaluation of
its effects on the use of primary health-care services in Burundi using
routine data. Health Policy Plan. 2015. https://doi.org/10.1093/heapol/
czu132.

43. Witter S, Zulfiqur T, Javeed S, Khan A, Bari A. Paying health workers for
performance in Battagram district, Pakistan. Hum Resour Health. 2011.
https://doi.org/10.1186/1478-4491-9-23.

44. Ir P, Horemans D, Souk N, Van Damme W. Using targeted vouchers and
health equity funds to improve access to skilled birth attendants for poor
women: a case study in three rural health districts in Cambodia. BMC
Pregnancy Childbirth. 2010. https://doi.org/10.1186/1471-2393-10-1.

45. Iyer HS, Hirschhorn LR, Nisingizwe MP, et al. Impact of a district-wide health
center strengthening intervention on healthcare utilization in rural Rwanda:
use of interrupted time series analysis. PLoS One. 2017;12(8 PG-e0182418):
e0182418. https://doi.org/10.1371/journal.pone.0182418.

46. de Lima RT, Fernandes TG, da Balieiro AA, et al. Primary health Care in Brazil
and the Mais Médicos (more doctors) program: an analysis of production
indicators. Cien Saude Colet. 2016. https://doi.org/10.1590/1413-
81232015219.15412016.

47. Mikkelsen-Lopez I, Cowley P, Kasale H, Mbuya C, Reid G, de Savigny D.
Essential medicines in Tanzania: does the new delivery system improve
supply and accountability? Heal Syst. 2014. https://doi.org/10.1057/hs.2013.
14.

48. Mphatswe W, Mate K, Bennett B, et al. Improving public health information:
a data quality intervention in KwaZulu-Natal, South Africa. Bull World Health
Organ. 2012. https://doi.org/10.2471/blt.11.092759.

49. Newbrander W, Ickx P, Feroz F, Stanekzai H. Afghanistan’s basic package of
health services: its development and effects on rebuilding the health
system. Glob Public Health. 2014;9(SUPPL.1 PG-S6-S28):S6-S28. https://doi.
org/10.1080/17441692.2014.916735.

50. Santos LMP, Oliveira A, Trindade JS, et al. Implementation research: towards
universal health coverage with more doctors in Brazil. Bull World Health
Organ. 2017. https://doi.org/10.2471/BLT.16.178236.

51. Scott VE, Sanders D. Evaluation of how integrated HIV and TB programs are
implemented in South Africa and the implications for rural-urban equity.
Rural Remote Health. 2013;13(2):2165.

52. Vindigni SM, Riley PL, Kimani F, et al. Kenya’s emergency-hire nursing
programme: a pilot evaluation of health service delivery in two districts.
Hum Resour Health. 2014;12(1 PG). https://doi.org/10.1186/1478-4491-12-16.

53. Field E, Usurup J, Nathan S, Rosewell A. Contextual factors and health
service performance from the perspective of the provincial health
administrators in Papua New Guinea. Rural Remote Health. 2018. https://doi.
org/10.22605/RRH4484.

54. Inchauste L, Patzi M, Halvorsen K, Solano S, Montesano R, Iñiguez V. Impact
of rotavirus vaccination on child mortality, morbidity, and rotavirus-related
hospitalizations in Bolivia. Int J Infect Dis. 2017. https://doi.org/10.1016/j.ijid.
2017.06.006.

55. Nanyunja M, Lewis RF, Makumbi I, et al. Impact of mass measles campaigns
among children less than 5 years old in Uganda. J Infect Dis. 2003. https://
doi.org/10.1086/368026.

56. Tafesse B, Tekle E, Wondwossen L, et al. Effects of polio eradication activities
on routine immunization: lessons from the 2013 outbreak response in
Somali region of Ethiopia. Pan Afr Med J. 2017;27(Suppl 2 PG-4):4. https://
doi.org/10.11604/pamj.supp.2017.27.2.10569.

57. Mochache V, Irungu E, El-Busaidy H, Temmerman M, Gichangi P. “Our voices
matter”: a before-after assessment of the effect of a community-
participatory intervention to promote uptake of maternal and child health
services in Kwale, Kenya 11 medical and health sciences 1117 public health
and health services. BMC Health Serv Res. 2018;18(1 PG). https://doi.org/10.
1186/s12913-018-3739-9.

58. Kitui J, Dutton V, Bester D, Ndirangu R, Wangai S, Ngugi S. Traditional birth
attendant reorientation and Motherpacks incentive’s effect on health facility
delivery uptake in Narok County, Kenya: an impact analysis. BMC Pregnancy
Childbirth. 2017;17(1 PG-125):125. https://doi.org/10.1186/s12884-017-1307-7.

59. Paudel D, Shrestha I, Siebeck M, Rehfuess E. Impact of the community-
based newborn care package in Nepal: a quasi-experimental evaluation.
BMJ Open. 2017;7(10 PG-e015285):e015285. https://doi.org/10.1136/
bmjopen-2016-015285.

60. Khurmi M, Sayinzoga F, Berhe A, et al. Newborn survival case study in
Rwanda – bottleneck analysis and projections in key maternal and child
mortality rates using lives saved tool (LiST). Int J MCH AIDS. 2017;6(2 PG-93-
108):93–108. https://doi.org/10.21106/ijma.214.

61. Lindgren TG, Deutsch K, Schell E, et al. Using mobile clinics to deliver HIV testing
and other basic health services in rural Malawi. Rural Remote Health. 2011;11(2):
1682.

62. Mapira P, Morgan C. The contribution of church health services to maternal
health care provision in Papua New Guinea. P N G Med J. 2011;54(3-4):139–146.

63. Robertson LJ, Szabo CP. Community mental health services in southern
Gauteng: an audit using Gauteng district health information systems data.
South African J Psychiatry. 2017;23(1 PG-a1055):a1055. https://doi.org/10.
4102/sajpsychiatry.v23i0.1055.

64. Yellapurkar MV. Health management information system in leprosy control
programme. Indian J Lepr. 1984;56(1 PG-86-97):86–97 https://www.scopus.
com/inward/record.uri?eid=2-s2.0-0021634802&partnerID=40&md5=9c9d755
790bfa4154b2372c12bdeeb37NS-.

65. Wagenaar BH, Augusto O, Beste J, et al. The 2014–2015 Ebola virus disease
outbreak and primary healthcare delivery in Liberia: time-series analyses for
2010–2016. PLoS Med. 2018;15(2):e1002508. https://doi.org/10.1371/journal.
pmed.1002508.

66. Quaglio G, Pizzol D, Bome D, et al. Maintaining maternal and child health
services during the Ebola outbreak: experience from Pujehun, Sierra Leone.
PLoS Curr. 2016;8. https://doi.org/10.1371/CURRENTS.OUTBREAKS.
D67AEA257F572201F835772D7F188BA5.

67. Sesay T, Denisiuk O, Shringarpure KK, et al. Paediatric care in relation to the
2014–2015 Ebola outbreak and general reporting of deaths in Sierra Leone.
Public Heal Action. 2017. https://doi.org/10.5588/pha.16.0088.

68. Sochas L, Channon AA, Nam S. Counting indirect crisis-related deaths in the
context of a low-resilience health system: the case of maternal and
neonatal health during the Ebola epidemic in Sierra Leone. Health Policy
Plan. 2017;32(suppl_3):iii32–9. https://doi.org/10.1093/heapol/czx108.

69. Cassim N, Coetzee LM, Schnippel K, Glencross DK. Estimating the cost-per-
result of a national reflexed Cryptococcal antigenaemia screening program:
forecasting the impact of potential HIV guideline changes and treatment
goals. PLoS One. 2017;12(8 PG-e0182154):e0182154. https://doi.org/10.1371/
journal.pone.0182154.

70. Novaes H, Itria A, Silva G, Sartori A, Rama C, Soárez P. Annual national direct
and indirect cost estimates of the prevention and treatment of cervical
cancer in Brazil. Clinics (Sao Paulo). 2015;70(4 PG-289-95):289–95. https://doi.
org/10.6061/clinics/2015(04)12.

71. Namaganda G, Oketcho V, Maniple E, Viadro C. Making the transition to
workload-based staffing: using the workload indicators of staffing need
method in Uganda. Hum Resour Health. 2015;13(PG-89):89. https://doi.org/
10.1186/s12960-015-0066-7.

72. Kusemererwa D, Alban A, Obua OT, Trap B. An exploratory study on equity
in funding allocation for essential medicines and health supplies in
Uganda’s public sector. BMC Health Serv Res. 2016;16:453((Obua) Ministry of
Health, Pharmacy Division, Lourdel Road, Wandegeya, Kampala, Uganda PG-
453). https://doi.org/10.1186/s12913-016-1698-6.

73. Hajison PL, Mwakikunga BW, Mathanga DP, Feresu SA. Seasonal variation of
malaria cases in children aged less than 5 years old following weather
change in Zomba district, Malawi. Malar J. 2017. https://doi.org/10.1186/
s12936-017-1913-x.

74. Wright J, Vazé P, Russell G, et al. Seasonal aspects of weight-for-age in
young children in Zimbabwe. Public Health Nutr. 2001. https://doi.org/10.
1079/phn2000100.

Hung et al. BMC Health Services Research (2020) 20:790 Page 13 of 15

https://doi.org/10.1002/hec.3856

https://doi.org/10.1093/heapol/czy077

https://doi.org/10.1186/s12913-017-2376-z

https://doi.org/10.1093/heapol/czr060

https://doi.org/10.1016/j.ssmph.2017.01.001

https://doi.org/10.1093/heapol/czu132

https://doi.org/10.1186/1478-4491-9-23

https://doi.org/10.1186/1471-2393-10-1

https://doi.org/10.1371/journal.pone.0182418

https://doi.org/10.1590/1413-81232015219.15412016

https://doi.org/10.1057/hs.2013.14

https://doi.org/10.2471/blt.11.092759

https://doi.org/10.1080/17441692.2014.916735

https://doi.org/10.2471/BLT.16.178236

https://doi.org/10.1186/1478-4491-12-16

https://doi.org/10.22605/RRH4484

https://doi.org/10.1016/j.ijid.2017.06.006

https://doi.org/10.1086/368026

https://doi.org/10.11604/pamj.supp.2017.27.2.10569

https://doi.org/10.1186/s12913-018-3739-9

https://doi.org/10.1186/s12884-017-1307-7

https://doi.org/10.1136/bmjopen-2016-015285

https://doi.org/10.21106/ijma.214

https://doi.org/10.4102/sajpsychiatry.v23i0.1055

https://www.scopus.com/inward/record.uri?eid=2-s2.0-0021634802&partnerID=40&md5=9c9d755790bfa4154b2372c12bdeeb37NS-

https://doi.org/10.1371/journal.pmed.1002508

https://doi.org/10.1371/CURRENTS.OUTBREAKS.D67AEA257F572201F835772D7F188BA5

https://doi.org/10.5588/pha.16.0088

https://doi.org/10.1093/heapol/czx108

https://doi.org/10.1371/journal.pone.0182154

https://doi.org/10.6061/clinics/2015(04)12

https://doi.org/10.1186/s12960-015-0066-7

https://doi.org/10.1186/s12913-016-1698-6

https://doi.org/10.1186/s12936-017-1913-x

https://doi.org/10.1079/phn2000100

75. Gunda R, Chimbari M, Shamu S, Sartorius B, Mukaratirwa S. Malaria
incidence trends and their association with climatic variables in rural
Gwanda, Zimbabwe, 2005–2015. Malar J. 2017;16(1 PG-393):393. https://doi.
org/10.1186/s12936-017-2036-0.

76. De Melo Cunha MAG, Lino PA, Dos Santos TR, Vasconcelos M, Lucas SD, De
Abreu MHNG. A 15-year time-series study of tooth extraction in Brazil. Med
(United States). 2015;94(47 PG-e1924):e1924. https://doi.org/10.1097/MD.
0000000000001924.

77. Luboga SA, Stover B, Lim T, et al. Did PEPFAR investments result in health
system strengthening? A retrospective longitudinal study measuring non-
HIV health service utilization at the district level. Health Policy Plan. 2016;
31(7 PG-897-909):897–909. https://doi.org/10.1093/heapol/czw009.

78. Wagenaar B, Gimbel S, Hoek R, et al. Effects of a health information system
data quality intervention on concordance in Mozambique: time-series
analyses from 2009–2012. Popul Health Metr. 2015;13(PG-9):9. https://doi.
org/10.1186/s12963-015-0043-3.

79. Sibomana H, Rugambwa C, Mwenda JM, et al. Impact of routine rotavirus
vaccination on all-cause and rotavirus hospitalizations during the first four
years following vaccine introduction in Rwanda. Vaccine. 2018. https://doi.
org/10.1016/j.vaccine.2018.01.072.

80. Larsen DA, Winters A, Cheelo S, et al. Shifting the burden or expanding
access to care? Assessing malaria trends following scale-up of community
health worker malaria case management and reactive case detection. Malar
J. 2017;16(1 PG-441):441. https://doi.org/10.1186/s12936-017-2088-1.

81. Ngabo F, Tate J, Gatera M, et al. Effect of pentavalent rotavirus vaccine
introduction on hospital admissions for diarrhoea and rotavirus in children
in Rwanda: a time-series analysis. Lancet Glob Health. 2016;4(2 PG-e129–36):
e129–36. https://doi.org/10.1016/S2214-109X(15)00270-3.

82. Wagenaar BH, Augusto O, Beste J, et al. The 2014–2015 Ebola virus disease
outbreak and primary healthcare delivery in Liberia: time-series analyses for
2010–2016. PLoS Med. 2018;15(2 PG-e1002508):e1002508. https://doi.org/10.
1371/journal.pmed.1002508.

83. Ruton H, Musabyimana A, Gaju E, et al. The impact of an mHealth
monitoring system on health care utilization by mothers and children: an
evaluation using routine health information in Rwanda. Health Policy Plan.
2018. https://doi.org/10.1093/heapol/czy066.

84. Gebrehiwot TG, Sebastian MS, Edin K, Goicolea I. The health extension
program and its association with change in utilization of selected maternal
health services in Tigray region, Ethiopia: a segmented linear regression
analysis. PLoS One. 2015. https://doi.org/10.1371/journal.pone.0131195.

85. Ir P, Korachais C, Chheng K, Horemans D, Van Damme W, Meessen B.
Boosting facility deliveries with results-based financing: a mixed-methods
evaluation of the government midwifery incentive scheme in Cambodia.
BMC Pregnancy Childbirth. 2015. https://doi.org/10.1186/s12884-015-0589-x.

86. Landoh E, Tchamdja P, Saka B, et al. Morbidity and mortality due to malaria
in Est mono district, Togo, from 2005 to 2010: a times series analysis. Malar
J. 2012;11(PG-389):389. https://doi.org/10.1186/1475-2875-11-389.

87. Assegaai T, Reagon G, Schneider H. Evaluating the effect of ward-based
outreach teams on primary healthcare performance in north West Province,
South Africa: a plausibility design using routine data. S Afr Med J. 2018;
108(4 PG-329-335):329–35. https://doi.org/10.7196/SAMJ.2017.v108i4.12755.

88. Anwari Z, Shukla M, Maseed B, et al. Implementing people-centred health
systems governance in 3 provinces and 11 districts of Afghanistan: a case
study. Confl Health. 2015;9(PG-2):2. https://doi.org/10.1186/1752-1505-9-2.

89. Mugeni C, Levine AC, Munyaneza RM, et al. Nationwide implementation of
integrated community case management of childhood illness in Rwanda.
Glob Heal Sci Pract. 2014. https://doi.org/10.9745/ghsp-d-14-00080.

90. phane VS, Jassat W, Bertram MY, et al. Supplementary immunization
activities (SIAs) in South Africa: comprehensive economic evaluation of an
integrated child health delivery platform. Glob Health Action. 2013. https://
doi.org/10.3402/gha.v6i0.20056.

91. Björkman A, Shakely D, Ali AS, et al. From high to low malaria transmission
in Zanzibar – challenges and opportunities to achieve elimination. BMC
Med. 2019. https://doi.org/10.1186/s12916-018-1243-z.

92. Oguttu DW, Matovu JKB, Okumu DC, et al. Rapid reduction of malaria
following introduction of vector control interventions in Tororo District,
Uganda: a descriptive study. Malar J. 2017. https://doi.org/10.1186/s12936-
017-1871-3.

93. Schoon MG. Impact of inter-facility transport on maternal mortality in the
free state province. South African Med J. 2013;103(8 PG-534-537):534–7.
https://doi.org/10.7196/SAMJ.6828.

94. Buchmann EJ, Mnyani CN, Frank KA, Chersich MF, McIntyre JA. Declining
maternal mortality in the face of persistently high HIV prevalence in a
middle-income country. BJOG An Int J Obstet Gynaecol. 2015. https://doi.
org/10.1111/1471-0528.13064.

95. Kakchapati S, Ardkaew J. Modeling of malaria incidence in Nepal. J Res
Health Sci. 2011;1(1 PG-7-13):7–13 https://www.scopus.com/inward/record.
uri?eid=2-s2.0-79959717379&partnerID=40&md5=259481ca13eda8f1ef5
9d22356034ffd NS.

96. Walsh A, Ndubani P, Simbaya J, Dicker P, Brugha R. Task sharing in Zambia:
HIV service scale-up compounds the human resource crisis. BMC Health
Serv Res. 2010. https://doi.org/10.1186/1472-6963-10-272.

97. Njuguna J. Effect of eliminating open defecation on diarrhoeal morbidity:
an ecological study of Nyando and Nambale sub-counties, Kenya. BMC
Public Health. 2016. https://doi.org/10.1186/s12889-016-3421-2.

98. Mufunda J, Nyarango P, Usman A, et al. Roll back malaria – an African
success story in Eritrea. South African Med J. 2007;97(1):46–50.

99. Kamuliwo M, Chanda E, Haque U, et al. The changing burden of malaria
and association with vector control interventions in Zambia using district-
level surveillance data, 2006-2011. Malar J. 2013. https://doi.org/10.1186/
1475-2875-12-437.

100. Kamuliwo M, Kirk K, Chanda E, et al. Spatial patterns and determinants of
malaria infection during pregnancy in Zambia. Trans R Soc Trop Med Hyg.
2015;109(8 PG-514-21):514–21. https://doi.org/10.1093/trstmh/trv049.

101. Beyene H, Deressa W, Kumie A, Grace D. Spatial, temporal, and
spatiotemporal analysis of under-five diarrhea in southern Ethiopia. Trop
Med Health. 2018;46(PG-18):18. https://doi.org/10.1186/s41182-018-0101-1.

102. Gwitira I, Murwira A, Mberikunashe J, Masocha M. Spatial overlaps in the
distribution of HIV/AIDS and malaria in Zimbabwe. BMC Infect Dis. 2018;18(1
PG-598):598. https://doi.org/10.1186/s12879-018-3513-y.

103. Sartorius B, Cohen C, Chirwa T, Ntshoe G, Puren A, Hofman K. Identifying
high-risk areas for sporadic measles outbreaks: lessons from South Africa.
Bull World Health Organ. 2013. https://doi.org/10.2471/BLT.12.110726.

104. Philippon S, Broutin H, Constantin de Magny G, et al. Meningococcal
meningitis in Mali: a long-term study of persistence and spread. Int J Infect
Dis. 2009. https://doi.org/10.1016/j.ijid.2008.05.1223.

105. Gething PW, Noor AM, Goodman CA, et al. Information for decision making from
imperfect national data: tracking major changes in health care use in Kenya using
geostatistics. BMC Med. 2007. https://doi.org/10.1186/1741-7015-5-37.

106. Gething PW, Noor AM, Gikandi PW, et al. Improving imperfect data from
health management information systems in Africa using space-time
geostatistics. PLoS Med. 2006;3(6 PG-0825-0831):825–31. https://doi.org/10.
1371/journal.pmed.0030271.

107. Gething P, Atkinson P, Noor A, Gikandi P, Hay S, Nixon M. A local space-
time kriging approach applied to a national outpatient malaria dataset.
Comput Geosci. 2007;33(10 PG-1337–1350):1337–50. https://doi.org/10.1016/
j.cageo.2007.05.006.

108. Ssempiira J, Kasirye I, Kissa J, et al. Measuring health facility readiness and its
effects on severe malaria outcomes in Uganda. Sci Rep. 2018;8(1 PG-17928):
17928. https://doi.org/10.1038/s41598-018-36249-8.

109. Bosomprah S, Dotse-Gborgbortsi W, Aboagye P, Matthews Z. Use of a
spatial scan statistic to identify clusters of births occurring outside Ghanaian
health facilities for targeted intervention. Int J Gynecol Obstet. 2016. https://
doi.org/10.1016/j.ijgo.2016.04.016.

110. Bennett A, Yukich J, Miller JM, et al. A methodological framework for the
improved use of routine health system data to evaluate national malaria
control programs: evidence from Zambia. Popul Health Metr. 2014. https://
doi.org/10.1186/s12963-014-0030-0.

111. Pires G, Rosa M, Zangarote M, Chicumbe S. Determinants of stillbirths
occurred in health facilities in Zambezia province, Mozambique (2013–
2014). Rev Bras Saude Matern Infant. 2016;16(4 PG-415-420):415–20. https://
doi.org/10.1590/1806-93042016000400004.

112. Sebastian MS, Lemma H. Efficiency of the health extension programme in
Tigray, Ethiopia: a data envelopment analysis. BMC Int Health Hum Rights.
2010. https://doi.org/10.1186/1472-698X-10-16.

113. Himanshu M, Källestål C. Regional inequity in complete antenatal services
and public emergency obstetric care is associated with greater burden of
maternal deaths: analysis from consecutive district level facility survey of
Karnataka, India. Int J Equity Health. 2017. https://doi.org/10.1186/s12939-
017-0573-3.

114. Souza ML. Registro de distúrbios mentais no Sistema de Informação da
Atenção Básica do Brasil, 2014. Epidemiol e Serv saude Rev do Sist Unico

Hung et al. BMC Health Services Research (2020) 20:790 Page 14 of 15

https://doi.org/10.1186/s12936-017-2036-0

https://doi.org/10.1097/MD.0000000000001924

https://doi.org/10.1093/heapol/czw009

https://doi.org/10.1186/s12963-015-0043-3

https://doi.org/10.1016/j.vaccine.2018.01.072

https://doi.org/10.1186/s12936-017-2088-1

https://doi.org/10.1016/S2214-109X(15)00270-3

https://doi.org/10.1371/journal.pmed.1002508

https://doi.org/10.1093/heapol/czy066

https://doi.org/10.1371/journal.pone.0131195

https://doi.org/10.1186/s12884-015-0589-x

https://doi.org/10.1186/1475-2875-11-389

https://doi.org/10.7196/SAMJ.2017.v108i4.12755

https://doi.org/10.1186/1752-1505-9-2

https://doi.org/10.9745/ghsp-d-14-00080

https://doi.org/10.3402/gha.v6i0.20056

https://doi.org/10.1186/s12916-018-1243-z

https://doi.org/10.1186/s12936-017-1871-3

https://doi.org/10.7196/SAMJ.6828

https://doi.org/10.1111/1471-0528.13064

https://www.scopus.com/inward/record.uri?eid=2-s2.0-79959717379&partnerID=40&md5=259481ca13eda8f1ef59d22356034ffd

https://doi.org/10.1186/1472-6963-10-272

https://doi.org/10.1186/s12889-016-3421-2

https://doi.org/10.1186/1475-2875-12-437

https://doi.org/10.1093/trstmh/trv049

https://doi.org/10.1186/s41182-018-0101-1

https://doi.org/10.1186/s12879-018-3513-y

https://doi.org/10.2471/BLT.12.110726

https://doi.org/10.1016/j.ijid.2008.05.1223

https://doi.org/10.1186/1741-7015-5-37

https://doi.org/10.1371/journal.pmed.0030271

https://doi.org/10.1016/j.cageo.2007.05.006

https://doi.org/10.1038/s41598-018-36249-8

https://doi.org/10.1016/j.ijgo.2016.04.016

https://doi.org/10.1186/s12963-014-0030-0

https://doi.org/10.1590/1806-93042016000400004

https://doi.org/10.1186/1472-698X-10-16

https://doi.org/10.1186/s12939-017-0573-3

Saude do Bras. 2016;25(2 PG-405-410):405–10. https://doi.org/10.5123/
S1679-49742016000200018.

115. Wanzira H, Muyinda R, Lochoro P, et al. Quality of care for children with
acute malnutrition at health center level in Uganda: a cross sectional study
in West Nile region during the refugee crisis. BMC Health Serv Res. 2018.
https://doi.org/10.1186/s12913-018-3366-5.

116. Nanyunja M, Lewis RF, Makumbi I, et al. Impact of mass measles campaigns
among children less than 5 years old in Uganda. J Infect Dis. 2003;
187(SUPPL. 1 PG-S63-S68):S63–8. https://doi.org/10.1086/368026.

117. Gunda R, Chimbari M, Mukaratirwa S. Assessment of burden of malaria in
Gwanda District, Zimbabwe, using the disability adjusted life years. Int J
Environ Res Public Health. 2016;13(2 PG-244):244. https://doi.org/10.3390/
ijerph13020244.

118. Philippon S, Broutin H, Constantin de Magny G, et al. Meningococcal
meningitis in Mali: a long-term study of persistence and spread. Int J Infect
Dis. 2009;13(1 PG-103-9):103–9. https://doi.org/10.1016/j.ijid.2008.05.1223.

119. Njuguna J. Effect of eliminating open defecation on diarrhoeal morbidity:
an ecological study of Nyando and Nambale sub-counties, Kenya. BMC
Public Health. 2016;16(1 PG). https://doi.org/10.1186/s12889-016-3421-2.

120. Bernal JL, Cummins S, Gasparrini A. Interrupted time series regression for
the evaluation of public health interventions: a tutorial. Int J Epidemiol.
2017. https://doi.org/10.1093/ije/dyw098.

121. Heymann DL, Chen L, Takemi K, et al. Global health security: the wider
lessons from the west African Ebola virus disease epidemic. Lancet. 2015.
https://doi.org/10.1016/S0140-6736(15)60858-3.

122. Luboga SA, Stover B, Lim TW, et al. Did PEPFAR investments result in health
system strengthening? A retrospective longitudinal study measuring non-
HIV health service utilization at the district level. Health Policy Plan. 2016.
https://doi.org/10.1093/heapol/czw009.

123. Amouzou A, Kachaka W, Banda B, Chimzimu M, Hill K, Bryce J. Monitoring
child survival in “real time” using routine health facility records: results from
Malawi. Trop Med Int Health. 2013. https://doi.org/10.1111/tmi.12167.

124. Inguane CA, Gloyd S, Manuel JL, et al. Assessment of linkages from HIV
testing to enrolment and retention in HIV care in Central Mozambique. J Int
AIDS Soc. 2016. https://doi.org/10.7448/IAS.19.5.20846.

125. Wagenaar BH, Gimbel S, Hoek R, et al. Effects of a health information system
data quality intervention on concordance in Mozambique: time-series
analyses from 2009-2012. Popul Health Metr. 2015. https://doi.org/10.1186/
s12963-015-0043-3.

126. Jat TR, Sebastian MS. Technical efficiency of public district hospitals in
Madhya Pradesh, India: a data envelopment analysis. Glob Health Action.
2013. https://doi.org/10.3402/gha.v6i0.21742.

127. Stanaway JD, Afshin A, Gakidou E, et al. Global, regional, and national
comparative risk assessment of 84 behavioural, environmental and
occupational, and metabolic risks or clusters of risks for 195 countries and
territories, 1990-2017: a systematic analysis for the global burden of disease
Stu. Lancet. 2018. https://doi.org/10.1016/S0140-6736(18)32225-6.

128. Remais JV, Zeng G, Li G, Tian L, Engelgau MM. Convergence of non-
communicable and infectious diseases in low- and middle-income
countries. Int J Epidemiol. 2013. https://doi.org/10.1093/ije/dys135.

129. Heller O, Somerville C, Suggs LS, et al. The process of prioritization of non-
communicable diseases in the global health policy arena. Health Policy
Plan. 2019. https://doi.org/10.1093/heapol/czz043.

130. Victora CG, Black RE, Boerma JT, Bryce J. Measuring impact in the millennium
development goal era and beyond: a new approach to large-scale effectiveness
evaluations. Lancet. 2011. https://doi.org/10.1016/S0140-6736(10)60810-0.

131. Endriyas M, Alano A, Mekonnen E, et al. Understanding performance data:
health management information system data accuracy in southern nations
nationalities and People’s region, Ethiopia. BMC Health Serv Res. 2019.
https://doi.org/10.1186/s12913-019-3991-7.

132. Ward K, Mugenyi K, Benke A, et al. Enhancing workforce capacity to
improve vaccination data quality, Uganda. Emerg Infect Dis. 2017. https://doi.
org/10.3201/eid2313.170627.

133. Maïga A, Jiwani SS, Mutua MK, et al. Generating statistics from health facility
data: the state of routine health information systems in eastern and
southern Africa. BMJ Glob Health. 2019;4(5):e001849. https://doi.org/10.1136/
bmjgh-2019-001849.

134. Necochea E, Tripathi V, Kim YM, et al. Implementation of the standards-
based management and recognition approach to quality improvement in
maternal, newborn, and child health programs in low-resource countries. Int
J Gynecol Obstet. 2015. https://doi.org/10.1016/j.ijgo.2015.04.003.

Publisher’s Note

Springer Nature remains neutral with regard to jurisdictional claims in
published maps and institutional affiliations.

Hung et al. BMC Health Services Research (2020) 20:790 Page 15 of 15

https://doi.org/10.5123/S1679-49742016000200018

https://doi.org/10.1186/s12913-018-3366-5

https://doi.org/10.1086/368026

https://doi.org/10.3390/ijerph13020244

https://doi.org/10.1016/j.ijid.2008.05.1223

https://doi.org/10.1186/s12889-016-3421-2

https://doi.org/10.1093/ije/dyw098

https://doi.org/10.1016/S0140-6736(15)60858-3

https://doi.org/10.1093/heapol/czw009

https://doi.org/10.1111/tmi.12167

https://doi.org/10.7448/IAS.19.5.20846

https://doi.org/10.1186/s12963-015-0043-3

https://doi.org/10.3402/gha.v6i0.21742

https://doi.org/10.1016/S0140-6736(18)32225-6

https://doi.org/10.1093/ije/dys135

https://doi.org/10.1093/heapol/czz043

https://doi.org/10.1016/S0140-6736(10)60810-0

https://doi.org/10.1186/s12913-019-3991-7

https://doi.org/10.3201/eid2313.170627

https://doi.org/10.1136/bmjgh-2019-001849

https://doi.org/10.1016/j.ijgo.2015.04.003

BioMed Central publishes under the Creative Commons Attribution License (CCAL). Under
the CCAL, authors retain copyright to the article but users are allowed to download, reprint,
distribute and /or copy articles in BioMed Central journals, as long as the original work is
properly cited.

Turn in your highest-quality paper
Get a qualified writer to help you with

“ write a 2-3 page (that means a minimum of 2 full pages and no longer than 3 full pages) paper (in APA format) ”

Get high-quality paper

NEW! AI matching with writer

Hire a Writer

Client Reviews

4.9

Sitejabber

4.6

Trustpilot

4.8

Our Guarantees

100% Confidentiality

Information about customers is confidential and never disclosed to third parties.

Original Writing

We complete all papers from scratch. You can get a plagiarism report.

Timely Delivery

No missed deadlines – 97% of assignments are completed in time.

Money Back

If you're confident that a writer didn't follow your order details, ask for a refund.

New to Your Trusted Assignment Help Service? Sign up & Save

Calculate the price of your order

Type of paper needed:

Pages:

You will get a personal manager and a discount.

Academic level:

We'll send you the first draft for approval by at

Total price:

$0.00

Power up Your Academic Success with the
Team of Professionals. We’ve Got Your Back.

Power up Your Study Success with Experts We’ve Got Your Back.

Order Now Order Now