Module 5 discussion

Based upon Module 5 readings, summarize and assess the evolving nature and impact of performance measurement on public service program creation and implementation regarding a specific public service program and/or service that you are familiar with. 

Your initial response to this discussion should be at least 500 words in length. Be careful to cite sources appropriately.

271

C H A P T E R 1 7

ASSESSMENT AND EVALUATION
Keneca Boyce

Every nonprofit should adopt practices that support and promote assessment and
evaluation. Implementing and enhancing systems and protocols that foster assess-
ment and evaluation impact everyone: constituents and communities, nonprofit
staff and leadership, board members, and funders. Assessment and evaluation are
ongoing processes that can be beneficial at all stages of a program. Evaluation is a
well-thought-out systematic approach to assess the performance, quality, and ben-
efit of a program, service, or the overall functioning of the organization. It is the
means by which we are able to establish consistency in exploring, collecting, track-
ing, and examining the work we “say” we do. It involves a set of predetermined
protocols to assure data are captured in a way that accurately represents the services
delivered to determine if the outcomes have been achieved.

Assessment and evaluation provide an opportunity for nonprofits and funders
to discuss outcomes and program improvement concretely rather than abstractly.
The current funding climate requires nonprofits to embrace evaluation as a required
means for achieving sustainability. Although some nonprofits have made great
strides in establishing the infrastructure and competencies required to effectively
and efficiently assess and evaluate outcomes, it is, in fact, a multifaceted under-
taking. This chapter provides an overview of key steps to support assessment and
evaluation in addition to building the capacity and competency required to do so
successfully.

WHY EVALUATE?

The question should be, why not evaluate? When we consider the scope and the
depth of contributions from the nonprofit sector to society, at its core, we find the
collective persistence to prioritize the preservation and well-being of human capi-
tal. Many nonprofits tout their services, resources, and supports in line with some
aspect of social justice and social good, often reflected through their mission. An
organizational culture defined by accountability, innovation, and learning, driven
by systems and protocols associated with assessment and evaluation, is essential for
nonprofit sustainability and social change.

There was a time when “stories of success” were sufficient for the nonprofit
sector and funders (Miles, 2006). Although the concept of outcomes and account-
ability has been around for some time, there was an evident surge in the demand for

Congress_27372_PTR_17_271-286_07-25-16.indd 271Congress_27372_PTR_17_271-286_07-25-16.indd 271 7/26/2016 4:43:46 PM7/26/2016 4:43:46 PM

C
o
p
y
r
i
g
h
t

2
0
1
7
.

S
p
r
i
n
g
e
r

P
u
b
l
i
s
h
i
n
g

C
o
m
p
a
n
y
.

A
l
l

r
i
g
h
t
s

r
e
s
e
r
v
e
d
.

M
a
y

n
o
t

b
e

r
e
p
r
o
d
u
c
e
d

i
n

a
n
y

f
o
r
m

w
i
t
h
o
u
t

p
e
r
m
i
s
s
i
o
n

f
r
o
m

t
h
e

p
u
b
l
i
s
h
e
r
,

e
x
c
e
p
t

f
a
i
r

u
s
e
s

p
e
r
m
i
t
t
e
d

u
n
d
e
r

U
.
S
.

o
r

a
p
p
l
i
c
a
b
l
e

c
o
p
y
r
i
g
h
t

l
a
w
.

EBSCO Publishing : eBook Academic Collection (EBSCOhost) – printed on 6/16/2020 12:06 PM via GEORGIA ONMYLINE
AN: 1336884 ; Congress, Elaine Piller, Luks, Allan, Petit, Francis.; Nonprofit Management : A Social Justice Approach
Account: ecor.main.ehost

272 NONPROFIT MANAGEMENT

outcomes before, during, and certainly after the economic downturn in 2008. The
ongoing demand from funders for nonprofits to demonstrate evidence to justify a
return on investment created a shift in the culture of nonprofit funding (Morley,
Vinson, & Hatry, 2001). The nonprofit leaders and organizations that were attuned
to the changing tides in the external environment were much more poised and
prepared to respond to the increasing demands for evidence and outcomes from
funders. However, many nonprofits were ill equipped, struggling to demonstrate
evidence of impact, and, in extreme cases, ultimately phased out. The approach to
meeting these demands left many nonprofits floundering to find the appropriate
staff, interventions, outcomes, and data tracking systems to remain competitive.
This harried approach to collecting and reporting outcomes left many nonprofits
scrambling to desperately demonstrate and convince funders that their outcomes
were worthy of investment. Assessment and evaluation should be perceived as
direct benefits to the organization and not solely to comply with funding require-
ments. The following examples highlight the benefits of evaluation to nonprofit
organizations.

Example 1: Citizen BUILD is a nonprofit organization with a youth-mentoring
workforce development program, designed to increase job placement and reten-
tion for formerly incarcerated youth. Findings from the evaluation helped the
organization to identify the most effective methods of communication between
mentors and mentees, the most effective mentor–mentee engagement trainings,
and barriers that compromised mentee job retention.

Example 2: Renewed Reformed Reentry (R3) conducted an evaluation of the alter-
native to incarceration treatment program to see if they achieved their goal
to reduce recidivism and hospitalization rates. Findings from the evaluation
helped program administrators identify the ideal length of time in treatment
that resulted in fewer triggers for recidivism. It also revealed the most effective
community treatment partnerships and collaborations.

Example 3: Urban Coders is a nonprofit organization that partners with General
Education Development (GED) programs to engage youth through technol-
ogy to reignite their passion to learn and pursue a college education and major
in computer science. Although youth study and prepare for the GED, they
receive intensive training in coding, web design, app development, and so forth.
Findings from the evaluation revealed a 60%-college engagement and placement
rate; however, 15% of students indicated an entrepreneurial career goal. The
organization found this information extremely helpful for future fundraising
goals in addition to innovative expansion to recruit entrepreneurs as mentors.

BUILDING CAPACITY AND COMPETENCY

Data-Driven Culture

A data-driven agency uses evidence to drive practice and decision making. In order
to become “data driven,” “outcomes driven,” or promote a culture of “inquiry” or
“continuous improvement,” it is essential that leadership understands assessment

Congress_27372_PTR_17_271-286_07-25-16.indd 272Congress_27372_PTR_17_271-286_07-25-16.indd 272 7/26/2016 4:43:47 PM7/26/2016 4:43:47 PM

EBSCOhost – printed on 6/16/2020 12:06 PM via GEORGIA ONMYLINE. All use subject to https://www.ebsco.com/terms-of-use

17: ASSESSMENT AND EVALUATION 273

FIGURE 17.1 Building a culture of evaluation.

and evaluation as both a process and an outcome (Miles & Woodruff-Bolte, 2013).
Transitioning or enhancing an organization’s data-driven culture takes time to learn,
refine, and master. Figure 17.1, developed by Community Solutions (2001), is an illus-
tration of 30 ideas with which to build a culture of evaluation within organizations.

Congress_27372_PTR_17_271-286_07-25-16.indd 273Congress_27372_PTR_17_271-286_07-25-16.indd 273 7/26/2016 4:43:47 PM7/26/2016 4:43:47 PM

EBSCOhost – printed on 6/16/2020 12:06 PM via GEORGIA ONMYLINE. All use subject to https://www.ebsco.com/terms-of-use

274 NONPROFIT MANAGEMENT

Nonprofit Leadership

Nonprofit leadership must prioritize assessment and evaluation, as they would any
key human resource, budgeting, or fundraising matter. Promoting a data-driven cul-
ture requires leadership to be consistent with messaging and communication that rein-
forces the value and priority for all staff to adopt and adhere to evaluation systems
and protocols (Hernandez & Visher, 2001). At the onset, the leadership and/or the
designated evaluation team or staff are tasked with the responsibility of cultivating an
outlook of enthusiasm and optimism about assessment and evaluation. Formal and
informal meetings with staff should explore and address any discomfort, taboos, or
misunderstandings associated with assessment and evaluation. Leadership will need
to continually assess and address the impact of increased responsibility of a data-driven
culture on the quality of service delivery. This style of open and ongoing communica-
tion sets the tone for a data-driven culture of partnership where everyone plays a role
in maintaining the integrity of the organization’s assessment and evaluation efforts.

Leadership must remain engaged yet make it clear that the appointed staff,
division, or consultant will take the lead in facilitating the organization’s ongoing
efforts to be data driven. In the absence of dedicated and qualified personnel with
the support of the leadership, the path to become data driven will be more compli-
cated and challenging for all (Miles, 2006). Promoting a data-driven culture should
be an experience defined by collective learning among leadership, middle manage-
ment, and staff.

Characteristics of a Data-Driven Organization

1. Data-informed board, executives, program leadership, and staff
2. Accessible and visible evaluation staff and resources
3. A functional Theory of Change (ToC)
4. Comprehensive evaluation plan
5. Data collection tools align with program strategies and indicators
6. Data collection is integrated and prioritized with existing program practices
7. Effective and efficient data management systems and protocols
8. Collect, store, and analyze intentional data
9. Utilize data to inform quality improvement

10. Data-driven organizational decision making
11. Annual budget allocation to develop and sustain data management system
12. Accessible evaluation and data management staff and resources
13. Prioritize data integrity
14. Effective communication among leadership, management, and staff
15. Functional data reports
16. Formal and informal structures to review and discuss data
17. Quality assurance and evaluation protocols align to monitor compliance, fidel-

ity, and data integrity
18. In-service professional development training on assessment and evaluation
19. Leadership, management, and staff understand their role in assessment and

evaluation

Congress_27372_PTR_17_271-286_07-25-16.indd 274Congress_27372_PTR_17_271-286_07-25-16.indd 274 7/26/2016 4:43:48 PM7/26/2016 4:43:48 PM

EBSCOhost – printed on 6/16/2020 12:06 PM via GEORGIA ONMYLINE. All use subject to https://www.ebsco.com/terms-of-use

17: ASSESSMENT AND EVALUATION 275

20. Employee performance evaluation aligns with compliance to data management
protocols

THEORY OF CHANGE

Value of a ToC

An often overlooked, yet critical step in assessment and evaluation involves the
development of a ToC as a precursor to evaluation. Every nonprofit should have
a functional ToC as a blueprint to provide common knowledge for how organiza-
tions assess performance, outcomes, and impact (Hunter, 2013). A functional ToC
is designed to (a) foster communication, accountability, strategic planning, and
decision making and (b) describe how and why a set of interventions and a set of
strategies are expected to lead to outcomes. Stakeholders and funders value a ToC
because it provides a commonly understood vision of long-term goals, how they
will be achieved, and strategies to measure progress along the way (Mackinnon,
Ammot, & McGarvey, 2006).

Irrespective of staff roles, discipline, or expertise, a ToC helps all staff under-
stand how his or her individual and collective work with the target population
will lead to the desired outcomes. It is a visual representation that provides deeper
insight to the multidimensional construction of social issues. A ToC illustrates the
added value and appreciation for interdisciplinary teams with expertise to address
complex social issues. For example, a nonprofit organization worked for years
fulfilling its mission to fight hunger by providing meals and building capacity in
under-resourced communities. During a strategic planning meeting to review and
update their ToC, they ultimately realized the need to rethink their approach to
community capacity building. They decided to expand the scope of their services
with a nutrition division to educate communities and their existing partners. They
determined that more informed communities would be a key strategy to advance
their mission to fight hunger. This expansion of a nutrition division deepened the
interdisciplinary team expertise and strategy to address hunger as a multidimen-
sional social issue.

ToC Development, Implementation, and Utilization

Developing a ToC involves a series of critical thinking steps to produce a compre-
hensive picture about the nature of a social problem, and how specific actions will
lead to solutions (Hunter, 2013). It is a collaborative process to achieve internal
alignment following the exploration of several programmatic building blocks.

Some organizations will have the staff expertise to facilitate the process for
developing a ToC. Others will need to hire staff or consultants who are skilled and
able to support the organization in developing a ToC and the systems and protocols
to support ongoing assessment and evaluation. At the onset, representatives from
all levels of staffing (i.e., executive leadership, program manager, case planner, psy-
chiatrist, peers, board members, volunteers) should be identified and engaged in
the ToC development process. Often, as a result of stretched resources and capacity,

Congress_27372_PTR_17_271-286_07-25-16.indd 275Congress_27372_PTR_17_271-286_07-25-16.indd 275 7/26/2016 4:43:48 PM7/26/2016 4:43:48 PM

EBSCOhost – printed on 6/16/2020 12:06 PM via GEORGIA ONMYLINE. All use subject to https://www.ebsco.com/terms-of-use

276 NONPROFIT MANAGEMENT

leadership and management take the lead in developing a ToC. Nevertheless, before
a ToC is finalized, organizations should engage representatives of various levels of
staff for their input and feedback.

Once finalized, organizations should provide training on the implementation
and utilization of a ToC. A fundamental component of training should provide all
stakeholders with a deeper understanding of how the organization will meet the
mission through the contributions of their roles, expertise, and available resources.
ToC training should be ongoing and a frequent point of discussion among staff and
departments. Training is particularly essential for direct service delivery profes-
sionals who spend a great deal of time with program participants. Leadership and
management must help them see the connection in delivering and documenting
services as meaningful to their roles and the clients they serve (Miles, 2006). Shifting
to a data-driven culture requires continuous communication regarding the added
benefits of documenting services with accuracy and consistency. The ToC provides
a tangible way for all levels of staff to internalize and actualize the mission. A data-
driven organization assures that the ToC is visible and accessible to staff. The ToC

THEORY OF CHANGE

SOCIAL IMPACT

OUTCOMES

INDICATORS

OUTPUTS

STRATEGIES

INPUTS

HIGH PRIORITY
SERVICES

TARGET GROUP

SOCIAL ISSUE

FIGURE 17.2 Components of a Theory of Change.

Congress_27372_PTR_17_271-286_07-25-16.indd 276Congress_27372_PTR_17_271-286_07-25-16.indd 276 7/26/2016 4:43:48 PM7/26/2016 4:43:48 PM

EBSCOhost – printed on 6/16/2020 12:06 PM via GEORGIA ONMYLINE. All use subject to https://www.ebsco.com/terms-of-use

17: ASSESSMENT AND EVALUATION 277

planning team should convene frequently, to review and determine if the ToC needs
to be updated.

If developed, implemented, and used effectively, a ToC allows nonprofits to
streamline efforts to assure they collect “intentional data” rather than “dead data.”
Intentional data have justification and alignment with the mission and serves the
needs of the organization and funders. Although there may be several data points
of interest, the goal is to collect data that advances organizational development,
organizational impact, and sustainability. The overall goal of the evaluation will
determine if nonprofits should develop a programmatic ToC or an organizational
ToC. An organizational ToC assesses the quality, cost, and alignment (goodness of
fit) among all programs and the mission. A programmatic ToC assesses alignment
of the mission, program strategies, and outcomes. A ToC (Figure  17.2) should be
carefully developed as an invaluable organizational tool.

Theory of Change

Social Impact: Broad statement about the expected benefi ts or cumulative effect on the general
population. Example: More youth will be career or college ready with technical knowledge and
skills.

Outcomes: Changes in knowledge, skill, attitude, behavior, condition, or status as a result
of program strategies. Example: Greater knowledge of career planning, improved academic
performance, effective responses to confl ict, greater fi nancial stability.

• Short-term—knowledge, skills, attitude, motivation, awareness
• Mid-term—behaviors, practices, policies, procedures
• Long-term—environmental, social, economical, political conditions

Indicators: Specifi c and measurable changes that represent achievement of an outcome.
Example: A nonprofi t college prep and placement program for high school seniors would
have the following indicators: 80% of high school seniors will apply to a minimum of three
universities; 75% of high school students will identify a major; 90% of youth will develop a
career plan.

Outputs: Units of service or the amount of work a nonprofi t does. Outputs produce the desired
outcomes for program participants. Example: The number of workshops, meals provided,
resource packets distributed, participants served, and so forth.

Strategies: What a program does with the inputs. The activities and interventions to achieve
the outcomes. Strategies produce outputs. Example: Educate the public about signs of mental
illness and depression in youth, provide adult mentors for youth, feed homeless families, and
so forth.

Inputs: Resources an organization or program uses to run the program. Example: Staff,
volunteers, facilities, clients, equipment, curricula, money, partnerships, and so forth.

High Priority Services: Essential areas of practice supported by literature, research, trends, and
practice experience, which drive service delivery.

Target Group: Customers, participants, communities, or population to be served by the
program.

Social Issue: Social problem or point of focus identifi ed by the mission.

Congress_27372_PTR_17_271-286_07-25-16.indd 277Congress_27372_PTR_17_271-286_07-25-16.indd 277 7/26/2016 4:43:48 PM7/26/2016 4:43:48 PM

EBSCOhost – printed on 6/16/2020 12:06 PM via GEORGIA ONMYLINE. All use subject to https://www.ebsco.com/terms-of-use

278 NONPROFIT MANAGEMENT

Utilizing a ToC to Guide Assessment and Evaluation

Assessment: Provide evidence of target population needs and trends; identify new areas for
program support.

Data Collection: Provide a framework for when, how, and what to measure; identify staff
responsible for documentation and time frames for documentation.

Management: A centralized guide for defi ning work tasks, roles, and program deliverables;
inform hiring, orientation, and professional development training needs; identify areas of
emphasis for supervision.

Organizational Functioning: Identify targeted and general fundraising goals; proactive
evidence-informed response to new and existing funder request; update the organizational
chart.

Quality Improvement: Determine which program and services need to be improved to
strengthen outcomes; reveal organizational and programmatic strengths and gaps in services;
monitor program fi delity.

Strategic Planning: Identify services and practice models that can be replicated; determine
new benchmarks, explore new interventions; reveal critical areas for collaborations and
partnerships.

DATA MANAGEMENT

Data Collection

Once a functional ToC is developed, it will inform the evaluation plan to determine
the data collection measures, processes, and systems. Data collection requires atten-
tion to detail and adherence to strict and predetermined protocols. There are many
factors that can compromise the integrity of data, so nonprofits should develop
a well-thought-out evaluation plan. Identifying the data collection tools, persons
responsible for data collection, and timing for data collection are essential to the
quality and integrity of findings. The quality of the data collected will be reflected
in the quality of the data reported.

There are several options available for data collection. Nonprofits may decide to
administer a survey, have participants complete a questionnaire, extract data from
client records, or hold a focus group. An evaluation plan may include a combination
of methods. The selected technique will depend on the goal of the evaluation, stage
of the program, characteristics of the target population, and available budget and
resources to successfully complete the evaluation (W.K. Kellogg Foundation, 2004).

The designated person(s) responsible for collecting data play(s) a critical role
in the success of the evaluation. Key staff should be trained on all data collection
tools and the data management protocols. Many nonprofits will select a survey or
questionnaire and will designate staff with the responsibility to solicit information
from participants. All of the staff responsible for interviewing participants should
receive training on the role of the interviewer to assure they do not unintentionally
compromise the integrity of the data.

Congress_27372_PTR_17_271-286_07-25-16.indd 278Congress_27372_PTR_17_271-286_07-25-16.indd 278 7/26/2016 4:43:48 PM7/26/2016 4:43:48 PM

EBSCOhost – printed on 6/16/2020 12:06 PM via GEORGIA ONMYLINE. All use subject to https://www.ebsco.com/terms-of-use

17: ASSESSMENT AND EVALUATION 279

The following tips on interviewing skills should be addressed when training
staff.

1. Emphasize overall purpose of the evaluation
2. Discuss confidentiality
3. Ask participants to answer questions as honestly as possible
4. Review response categories clearly and check to make sure the respondent

understands
5. Review response categories (to remind participants) before beginning a new

section
6. Maintain a neutral attitude
7. Refrain from behaviors that could influence how the respondent answers
8. Avoid sharing your personal opinions
9. Never suggest an answer

10. Speak clearly and slowly

It is essential to maintain communication with staff about their experiences
collecting data from participants and compliance with the evaluation systems and
protocols. These communications will often reveal gaps or challenges in the evalu-
ation plan that may not be as transparent to leadership and data management staff.
Establishing organizational compliance is fundamental, so systems and protocols
must be transparent, effective, and efficient.

Selecting the appropriate data collection measure is as important as the method
in which the data is collected. Data collection tools should accurately and efficiently
measure the indicators to determine if the outcomes were achieved. More impor-
tantly, data collection measures should align with the strategies and services deliv-
ered. For example, a nonprofit organization recently received funding to develop
and evaluate a new youth entrepreneurship training prevention program. They
hired a new program director to develop and implement the program and eval-
uation. The program was designed to teach and train youth aged 15 to 17 years
entrepreneurship skills and support them in the development and pilot of a small
business. On completion of the training program, youth graduated to peer trainers
with the responsibility to mentor the next cohort of youth trainees. As a data-driven
organization, they identified key indicators and developed a survey designed to
measure the outcomes. During the data analysis phase, the data management team
notified the program leadership about a gap in the findings when compared with
the projected outcomes to the funder. None of the data analyzed measured the par-
ticipant’s leadership skills. Although they intended to address leadership skills
through the training program, they realized this was an underdeveloped compo-
nent of the training curriculum. This evaluation proved to be a valuable lesson to
the organization. In retrospect, they acknowledged proceeding with a great deal
of haste throughout the program planning and evaluation phase. They also real-
ized that they should have allotted more time to assess the new program direc-
tor’s evaluation competencies. Three essential lessons emerged from the evaluation:
(a) assess program development and evaluation expertise during the interview and
hire phase; (b) provide ToC training and supervision for new leadership and staff;

Congress_27372_PTR_17_271-286_07-25-16.indd 279Congress_27372_PTR_17_271-286_07-25-16.indd 279 7/26/2016 4:43:49 PM7/26/2016 4:43:49 PM

EBSCOhost – printed on 6/16/2020 12:06 PM via GEORGIA ONMYLINE. All use subject to https://www.ebsco.com/terms-of-use

280 NONPROFIT MANAGEMENT

and (c) confirm alignment throughout the program planning and evaluation plan-
ning phase. They agreed to apply these lessons to all future program initiatives and
evaluations to avoid this disconnect in services delivered, data collection tools, and
outcomes.

Evaluation Planning

1. What are the program indicators?
2. What do you want to learn from the evaluation?
3. What data will you collect?
4. What methods will be used for data collection?
5. Will data be collected from the entire target population?
6. Who will collect the data?
7. How often will data be collected?
8. Who will analyze the data?
9. How frequently will data reports be generated for review?

10. How will the data be used?

Reporting Data

Establishing clarity at the onset regarding the purpose of assessment and evalu-
ation will be helpful to inform the organization on how it can leverage the data.
Initial decisions about the usage of data will evolve as the organization evolves or as
the program continues to demonstrate success or experience new trends and chal-
lenges. A nonprofit organization responsible for recruiting and training foster par-
ents initially produced evaluation reports that were useful for the board, managers,
and funders. As they continued to review and apply the data to strategic decision
making for the organization, they determined that the data would be instrumental
in local advocacy efforts. It was also determined that the data would be used to
inform updates and revisions to the foster parent training curriculum.

The capacity of any nonprofit organization to generate reports will depend
on the data management system and staffing capacity. Often times, limitations of
an efficient data management system result in a great deal of staff time to gener-
ate reports with accuracy. With these considerations in mind, nonprofits should
determine the feasibility, frequency, and consistency of generating and dissemi-
nating reports. For example, program managers at a nonprofit organization were
responsible for providing immigration services and support including English as
a second language and computer literacy. At the beginning of the funding cycle,
program managers received monthly reports highlighting key performance areas.
Throughout the entire third quarter of the funding cycle, program managers did not
receive monthly reports, despite their repeated requests. In the absence of current
and relevant data, program managers struggled to keep staff engaged in discus-
sions specific to assessment and evaluation. Monthly reports resumed during the
final quarter of the funding cycle. Program managers reported, “We feel like we’re
right where we started. We have to work just as hard as we did in the beginning
to reacquaint staff with data and twice as hard to get them engaged to discuss the

Congress_27372_PTR_17_271-286_07-25-16.indd 280Congress_27372_PTR_17_271-286_07-25-16.indd 280 7/26/2016 4:43:49 PM7/26/2016 4:43:49 PM

EBSCOhost – printed on 6/16/2020 12:06 PM via GEORGIA ONMYLINE. All use subject to https://www.ebsco.com/terms-of-use

17: ASSESSMENT AND EVALUATION 281

data.” A culture driven by continuous improvement is contingent on staff consis-
tently expecting to receive, review, and make decisions based on data.

The ToC can be instrumental in helping staff and leadership identify a dash-
board of performance indicators that will communicate progress at different points
throughout the year.

Reports should be developed in a way that will translate a clear story about the
population, the services, the outcomes, and the overall performance of a program.
The goal of disseminating reports is to tell a story in a way that makes it engaging
for the audience to connect, reflect, and respond to the data.

Characteristics of a Functional Report

• Features standard template to organize the data
• Includes easily interpreted data (graphs, charts, colors, etc.)
• Provides relevant data for the intended recipient (board, managers, staff, volun-

teer, clients, etc.)
• Highlights performance areas of success and areas of improvement
• Provides brief explanatory notes (as needed) for special considerations for inter-

preting the data
• Provides comparative or trend data to support interpretation of data and

performance

Review and Discussion of Data

The report has been prepared and disseminated . . . now what? Once reports are
disseminated, it can cause a great deal of anxiety for staff and program leaders.
Therefore, ongoing support and communication to normalize the experience of
discussing performance are key. Forums that are dedicated to the review and
discussion of data should be guided and aimed at celebrating the successes and
brainstorming a plan of action for areas of improvement. Program leadership
should create internal structures to convene key staff and support staff who
play a critical role in achieving outcomes. Everyone should be familiar with
the contents of the report and prepared to engage in a discussion about the
findings.

CASE EXAMPLE

One nonprofit leader decided that it was beneficial for his team to review a dashboard
of performance indicators every other month. He took the lead in disseminating reports,
convening the team, and leading the discussion. Six months later, he implemented several
changes, which proved to be instrumental in supporting staff to build a data-driven culture
that values assessment and evaluation. He implemented the following protocol:

• Disseminate reports electronically 1 week in advance of meetings
• Instruct program staff to bring one question, comment, or plan of action to the meeting
• Rotate meeting facilitators

Congress_27372_PTR_17_271-286_07-25-16.indd 281Congress_27372_PTR_17_271-286_07-25-16.indd 281 7/26/2016 4:43:49 PM7/26/2016 4:43:49 PM

EBSCOhost – printed on 6/16/2020 12:06 PM via GEORGIA ONMYLINE. All use subject to https://www.ebsco.com/terms-of-use

282 NONPROFIT MANAGEMENT

Following the general review of the report, staff shared their comment, question,
or plan of action. The team used both small and large group discussion formats. Each
meeting closed with a brief recap of lessons learned and a plan of action. This new format
generated a great deal of discussion and gave the team more ownership of the data. They

FIGURE 17.3 The Innovation Network.

Congress_27372_PTR_17_271-286_07-25-16.indd 282Congress_27372_PTR_17_271-286_07-25-16.indd 282 7/26/2016 4:43:49 PM7/26/2016 4:43:49 PM

EBSCOhost – printed on 6/16/2020 12:06 PM via GEORGIA ONMYLINE. All use subject to https://www.ebsco.com/terms-of-use

17: ASSESSMENT AND EVALUATION 283

were more focused on reviewing and using the data to determine the need for course-
corrective decisions. Overall, staff developed a deeper appreciation and understanding of
nonprofi t assessment and evaluation.

Figure 17.3 illustrates the various ways nonprofi ts use and report data, based on the
2011 research on a national sample of nonprofi ts conducted by the Innovation
Network Inc.

Generate Discussion When Reviewing Data

• Do the data reveal any new information about the target population that infl uence program
strategies and outcomes?

• Do the data reveal key areas of quality performance that lead to outcomes?
• Do the data reveal any challenges or obstacles to achieving outcomes?
• Do the data reveal an area of focus for program improvement?
• Do the data inform new services and supports for staff or participants?
• Should new indicators of success be explored or included in future reports?
• Are there any internal or external collaborations and partnerships to be explored?

Data Management System

Technology is a key component for building an infrastructure to support assess-
ment and evaluation. The ability to analyze data and maximize on lessons learned
will depend on the limitation or sophistication of the data management system.
A comprehensive assessment of the needs of the program or organization and the
intended purpose of the data management system is the first step to determine
the best technology solution. The path to identifying the right data management
system requires a great deal of resources and investment to implement and main-
tain. Nonprofits must consider these expenses and build it into their budget and
fundraising goals (Morariu, Athanasiades, & Emery, 2012).

Increased demands to provide quality services, while maintaining compliance
for internal and external purposes, often result in duplicative data entry. Many non-
profits are faced with the daunting task of managing several reports for several
funders. Reporting requirements vary, with different expectations for program out-
comes and how they should be measured and reported (Major, 2011). Several gov-
ernment funders require nonprofits to enter data into an online database. However,
the data are available for a limited time, so institutional data are lost once the case is
closed or when the contract ends. Often times, nonprofits cannot extract data from
these online databases to tell their story of success, to inform program improve-
ment, or to leverage data for fundraising efforts (Snibbe, 2006). This complicates
organizational efforts to motivate staff to enter data that ultimately will have little
impact on their day-to-day activities and interventions. The goal of entering data
is to use the data. Until these issues are resolved, nonprofits will need to identify
the appropriate data management system that will streamline data entry and pro-
vide direct access for storing, analyzing, and reporting data. Exploring mechanisms
and solutions to leverage data from these portals will ultimately strengthen the sec-
tor’s efforts to become more data driven, data informed, and avoid duplicative data
entry (Miles & Woodruff-Bolte, 2013).

Congress_27372_PTR_17_271-286_07-25-16.indd 283Congress_27372_PTR_17_271-286_07-25-16.indd 283 7/26/2016 4:43:50 PM7/26/2016 4:43:50 PM

EBSCOhost – printed on 6/16/2020 12:06 PM via GEORGIA ONMYLINE. All use subject to https://www.ebsco.com/terms-of-use

284 NONPROFIT MANAGEMENT

Characteristics of a Functional Data Management System

• Systematically store and analyze data
• Provide efficiency for staff and program leadership
• Generate reports to assess performance at any given time
• User friendly
• Easily accessible and updated
• Minimize duplicity across multiple reporting systems
• Interface with other systems in the public domain (i.e., Excel, Access)
• Customized to reflect relevant data points specific to the target population, strate-

gies, and indicators

THE ROLE OF FUNDERS

Although funders have increased their demand on nonprofit assessment and evalu-
ation with a focus on outcomes, funding for these efforts have not shared the same
momentum and fervor. Nonprofits would benefit from a group of funders (govern-
ment agencies, private foundations, corporate funders, and individual donors) who
partner and invest in the capacity building of data management software, systems,
training on assessment and evaluation, and technical assistance (Major, 2011). The
technology industry can play a more visible and critical role in building and upgrad-
ing the infrastructure of nonprofit data management systems. Capacity is increas-
ingly becoming an area of mention and concern across nonprofits. Many nonprofits
are taxed with various reporting requirements, frequent audits, new initiatives, and
policies. Everyone is required to do more with less, while maintaining quality.

Some funders have made advances by allocating funds for capacity develop-
ment, training, and technical assistance. They have redefined a collaborative and
supportive relationship with grantees to support mutually beneficial assessment
and evaluation (Major, 2011). More funders should broaden the scope of their role
in the investment for social justice by acquiring a deeper and realistic understand-
ing of the costs and efforts associated with achieving outcomes and creating change.
Often times, the approved budget and staffing plan do not account for the required
expertise of evaluation staff and the associated technology resources to effectively
and efficiently collect and report data. The absence of an effective infrastructure for
data management, data integrity, and reporting can compromise the ability of an
organization to compete for sustainable funding. It is both logical and timely that
funders invest in services and outcomes as well as data-driven capacity- building
resources to support assessment and evaluation. It is in the best interest for all
invested in social good to assure organizations have the tools to successfully exe-
cute their mission.

REFERENCES

Community Solutions. (2011). Building a culture of innovation: 30 ideas to apply to your organiza-
tion. Retrieved from http://communitysolutions.ca/web/

Congress_27372_PTR_17_271-286_07-25-16.indd 284Congress_27372_PTR_17_271-286_07-25-16.indd 284 7/26/2016 4:43:51 PM7/26/2016 4:43:51 PM

EBSCOhost – printed on 6/16/2020 12:06 PM via GEORGIA ONMYLINE. All use subject to https://www.ebsco.com/terms-of-use

17: ASSESSMENT AND EVALUATION 285

Hernandez, G., & Visher, M. (2001). Creating a culture of inquiry: Changing methods and minds
on the use of evaluation in nonprofit organizations (pp. 1–20). San Francisco, CA: The James
Irvine Foundation.

Hunter, D. (2013). Working hard—And working well: A practical guide to performance manage-
ment. Washington, DC: Venture Philanthropy Partners.

Mackinnon, A., Ammot, N., & McGarvey, C. (2006). Mapping change: Using a theory of change
to guide planning and evaluation (pp. 1–11). New York, NY: Grantcraft.

Major, D. (2011). Expanding the impact of grantees: How do we build the capacity of nonprofits
to evaluate, learn and improve? Grantmakers for Effective Organizations. Retrieved from
http://www.socialimpactexchange.org/sites/www.socialimpactexchange.org/files/
GEO_SWW_BuildCapacityToEvaluateLearnImprove

Miles, M. (2006). Good stories aren’t enough: Becoming outcomes-driven in workforce development.
Working Ventures, Public/Private Ventures. Retrieved from http://www.issuelab.org/
resource/good_stories_arent_enough_becoming_outcomes_driven_in_workforce_
development

Miles, M., & Woodruff-Bolte, S. (2013). Nurturing inquiry and innovation: Lessons from the work-
force benchmarking improvement collaborative (pp. 1–22). Ann Arbor, MI: Corporation for a
Skilled Workforce.

Morariu, J., Athanasiades, K., & Emery, A. (2012). State of evaluation 2012: Evaluation practice
and capacity in the nonprofit sector (pp. 1–20). Washington, DC: Innovation Network.

Morley, E., Vinson, E., & Hatry, H. (2001). Outcome measurement in nonprofit organizations:
Current practices and recommendations (pp. 3–10). Washington, DC: Independent Sector.

Snibbe, A. (Fall 2006). Drowning in data. Stanford Social Innovation Review, 4(3), 39–45.
W. K. Kellogg Foundation. (2004). Evaluation handbook. Battle Creek, MI: W. K. Kellogg

Foundation. Retrieved from https://www.wkkf.org/resource-directory/resource/2010/
w-k-kellogg-foundation-evaluation-handbook

Congress_27372_PTR_17_271-286_07-25-16.indd 285Congress_27372_PTR_17_271-286_07-25-16.indd 285 7/26/2016 4:43:51 PM7/26/2016 4:43:51 PM

EBSCOhost – printed on 6/16/2020 12:06 PM via GEORGIA ONMYLINE. All use subject to https://www.ebsco.com/terms-of-use

Congress_27372_PTR_17_271-286_07-25-16.indd 286Congress_27372_PTR_17_271-286_07-25-16.indd 286 7/26/2016 4:43:51 PM7/26/2016 4:43:51 PM

EBSCOhost – printed on 6/16/2020 12:06 PM via GEORGIA ONMYLINE. All use subject to https://www.ebsco.com/terms-of-use

79

6

Ci t i zen Par t i ci pat i on i n Per f or mance Measur ement
Alfred T. Ho

For the past two decades, many governments have paid growing attention to a tool that has been around for
almost a century—performance measurement. With the advancement of information technologies, the ease of
data analysis, and the popular concept of “results-oriented government,” performance measurement has
become more sophisticated and is now commonly used in today’s public management. While collection of
input, workload, and cost-efficiency measures was the focus for many decades and has remained an
indispensable part of the exercise, more effort is now placed on measuring outcomes and results and
exploring the links between performance measurement and management. Collecting and reporting data are no
longer sufficient. Government officials are now expected to use the information intelligently to align
performance goals and activities and demonstrate results and progress. Nonetheless, the major clientele
utilizing performance information has largely remained the same—program managers, budget analysts, and
the elected officials of the government. The public and external stakeholders are seldom involved in defining,
selecting, and using performance measures.

The purpose of this chapter is to explore the possible role of citizens in performance measurement and to
discuss why they should be involved more and how they should be engaged in the process. The chapter
argues that performance measurement may lose part of its potential relevancy and significance in the political
decision-making process if the public is not involved. Even though there are many technical and political
hurdles to engaging citizens in performance measurement, public managers also have a professional and
ethical duty to expand the scope of users of performance measurement so that the tool can indeed be used to
hold government more accountable to the public.

Per f or mance Measur ement i n Gover nment

Performance measurement refers to the usage of quantifiable indicators to measure the output, efficiency, and
results of public services. Even though the practice has caught the attention of many policy makers and
managers over the past two decades, it is hardly a recent innovation. As early as the turn of the twentieth
century, the New York Bureau of Municipal Research had already proposed tracking the cost and output of
public programs so that managers could conduct unit-cost analysis to improve efficiency and prevent fraud
and corruption (New York Bureau of Municipal Research, 1918; Ridley & Simon, 1938). The practice was
gradually adopted by state and local governments that were more progressive in reforming their managerial
practices, and later, by the U.S. federal government in the 1950s when some of the early reformers from the
New York Bureau of Municipal Research, such as Frederick Cleveland, went to the federal government to
help implement budgetary reforms (Kahn, 1997).

For the past few decades, the practice of performance measurement has continued to evolve, becoming
broader in scope and more sophisticated. For example, the types of measures that government agencies keep
track of have expanded from cost-efficiency data to output, workload, intermediate outcome, outcome, and
explanatory data (Ho & Ni, 2005). Agencies that adopt performance measurement have also expanded, from
more technical departments such as public works and police, to human-service-oriented departments, such as
education, welfare, and community development. Many reforms have also been introduced to integrate
performance measurement into public decision-making, particularly the budgetary process. For example, the
“planning-programming-budgeting system” (PPBS) in the late 1960s attempted to introduce the measurement
of program output, efficiency, and effectiveness to guide policy making and program budgeting. Later
reforms, such as “zero-base budgeting” (ZBB) in the 1970s and “management-by-objectives” (MBO) in the

C
o
p
y
r
i
g
h
t

2
0
0
7
.

R
o
u
t
l
e
d
g
e
.

A
l
l

r
i
g
h
t
s

r
e
s
e
r
v
e
d
.

M
a
y

n
o
t

b
e

r
e
p
r
o
d
u
c
e
d

i
n

a
n
y

f
o
r
m

w
i
t
h
o
u
t

p
e
r
m
i
s
s
i
o
n

f
r
o
m

t
h
e

p
u
b
l
i
s
h
e
r
,

e
x
c
e
p
t

f
a
i
r

u
s
e
s

p
e
r
m
i
t
t
e
d

u
n
d
e
r

U
.
S
.

o
r

a
p
p
l
i
c
a
b
l
e

c
o
p
y
r
i
g
h
t

l
a
w
.

EBSCO Publishing : eBook Collection (EBSCOhost) – printed on 6/16/2020 12:04 PM via GEORGIA ONMYLINE
AN: 199747 ; Box, Richard C..; Democracy and Public Administration
Account: ecor.main.usg

80

•

•

•
•

1980s followed the same emphasis and tried to use performance information to rationalize budgetary and
program decisionmaking (U.S. GAO, 1997).

In the 1990s, the performance measurement movement reached another peak of development. Because of
the antigovernment movement and the tax revolts of the 1980s, many politicians looked for ways to change
the image of government bureaucracy and rebuild public trust in their capacity to deliver public services
efficiently and effectively (Nye, Zelikow, & King, 1997). It was in this context that Osborne and Gaebler
(1993) published their landmark work, , in which they pushed for new ways toReinventing Government
manage government operations, such as the idea of “competitive government” through contracting-out and
“mission-driven government,” which focuses more on goals, not rules. They also recommended
“results-oriented government,” in which government should measure and reward policy outcomes.

Traditional bureaucratic governments … focus on inputs, not outcomes. They fund schools based on how many
children enroll; welfare based on how many poor people are eligible; police departments based on police estimates
of manpower needed to fight crime. They pay little attention to outcomes—to results. … Entrepreneurial
governments seek to change these rewards and incentives. Public entrepreneurs know that when institutions are
funded according to inputs, they have little reason to strive for better performance. But when they are funded
according to outcomes, they become obsessive about performance. (Osborne & Gaebler, 1993, p. 139)

The idea of “results-oriented government” was quickly disseminated among federal, state, and local
governments. At the federal level, the Clinton administration introduced the National Performance Review
and Congress passed the Government Performance and Results Act to require agencies to establish goals and
measure outcomes. At the state and local level, many governments introduced their own versions of
performance measurement reforms that emphasized public accountability to taxpayers. More public officials
began to ask questions like the following:

Are my performance measures aligned with the goals and performance targets of programs?
How can the budget office use performance information to evaluate program results more effectively,
ensuring that tax money is put to the best use?
How can program managers use performance measures to motivate line staff to make continuous
improvements to program delivery?
How can policy makers and managers use performance information to evaluate the current status of
program delivery and establish strategic goals for programs?

Beyond Per f or mance Management

There is evidence to show that the new emphasis of performance measurement has made a difference in the
way government is managed. At the federal government, for example, program managers are found to pay
closer attention to program results and public accountability (U.S. GAO, 2004). Studies of state and local
government reforms have also confirmed similar effects and show that performance measurement can help
improve communication between the budget office and departments and between the executive branch and
the legislative branch, as well as strengthen the culture of public accountability (Ho, 2006a; Melkers &
Willoughby, 2001).

However, the impact to date seems to have been limited to the executive process of decision-making. Even
though performance information is advocated as a way to influence how legislators make policy and
budgetary decisions so that appropriation decisions can be rationalized to maximize program efficiency and
effectiveness, empirical evidence has generally shown that many legislators pay limited attention to
performance information (Jordan & Hackbart, 1999; Joyce, 1993). Special interests, partisan influence, and
political maneuvering seem to remain the primary driving force behind budgeting and other policy decisions.
Hence, until performance information has greater political weight—that is, when the information means
something to voters and major stakeholders who will use it to hold politicians accountable for results—the
political reality that performance information has limited influence in the legislative phase of government is
unlikely to change.

EBSCOhost – printed on 6/16/2020 12:04 PM via GEORGIA ONMYLINE. All use subject to https://www.ebsco.com/terms-of-use

81

•
•
•
•
•

For advocates of performance measurement, this has been a disappointing finding. Although many data
and reports are being generated each year by the government bureaucracy, the information has not been fully
used, which means time and resources have been wasted in the data collection and analysis process. Also,
one has to question whether managers have been measuring the “right” thing. If major stakeholders are not
interested in the information, which is supposed to show the “results” that matter, the whole purpose of
“results-oriented management” may be an empty promise.

The lack of citizen participation may also create implementation hurdles in performance-oriented reforms
even within the executive branch of government. Past studies have shown that if a government engages
citizens more in performance measurement, its officials are more likely to use performance information to
make managerial changes, including setting strategic goals, improving internal and external communication,
and reinforcing the customer focus of government (Ho, 2006a). They are also more likely to establish
performance targets for departments and discuss performance results in meetings to hold department officials
accountable for results (Ho, 2006b). Hence, insufficient effort to engage policy stakeholders and the public
not only limits the impact of performance measurement on the legislature, but also reduces the incentive for
managers to follow through and use the information to make a difference in program management.

Rol e of Ci t i zens i n Per f or mance Measur ement

Perhaps these are some of the reasons why in recent years several professional organizations have started to
advocate the role of citizens in performance measurement. For example, in the guidelines for reporting
“service effort and accomplishments” (SEA) released in 2003, the Governmental Accounting Standards
Board (GASB) recommends the following practices (GASB, 2003, pp. 6–8):

The [performance measurement] report should include a discussion of involvement of citizens, elected officials,
management, and employees in the process of establishing goals and objectives for the organization. … Citizen and
customer perceptions of the quality and results of major and critical programs and services should be reported when
appropriate.

Many local governments and community organizations have also started to involve citizens. For example,
the Jacksonville Community Council has worked with local officials and community leaders to produce an
annual quality-of-life report that evaluates the performance and needs of health and human services
according to a strategic vision and specific goals, and the experience has been highly constructive for the
community. The Fund for the City of New York has involved citizens in focus-group discussions and
monitoring and measurement efforts to deal with street-level problems, such as graffiti, garbage collection,
and potholes in roadways, and the effort has made some impact on how New York City manages its
neighborhood services. The Sustainable Seattle group has organized continuous dialogues among local
residents and officials to analyze community indicators and rethink neighborhood issues. Several Iowa
municipalities have also launched an initiative called “citizen-initiated performance assessment,” in which
citizens help elected officials and managers develop, select, and use performance indicators to improve the
quality of public services (Ho & Coates, 2004).

Reviewing some of these successful experiences of different communities that engage citizens in
performance measurement, Epstein, Coates, and Wray (2005) in their recent book, ,Results That Matter
summarize five roles that citizens may play:

Citizens as customers and stakeholders
Citizens as advocates
Citizens as issue framers
Citizens as collaborators
Citizens as evaluators

EBSCOhost – printed on 6/16/2020 12:04 PM via GEORGIA ONMYLINE. All use subject to https://www.ebsco.com/terms-of-use

82

•
•
•
•

Chal l enges i n Engagi ng Ci t i zens

One may agree normatively that citizens should be more involved. However, public administrators face many
practical challenges in their attempt to engage citizens meaningfully and effectively in the exercise of
performance measurement. Theories of public choice have long established that citizens are rational
decision-makers and have little incentive to participate in public decision-making when the benefits of
participation are spread across a community but the costs of participation, both monetary and non-monetary,
are individualized and can be very high. Political apathy is especially common in a well-governed
community in which citizens are satisfied and see no looming crisis that should prompt their immediate
participation in public affairs.

Citizen participation in performance measurement is even more difficult when compared to other forms of
public participation, such as voting, because of the following reasons:

Performance measurement involves technical details and data questions. Ordinary citizens may not
feel interested in understanding the methodological and technical questions involved.
Performance measurement is a routine exercise that tries to track data over time to monitor progress
and evaluate results. It is not a single event that has a clear beginning and end.
Performance measurement does not necessarily dictate policy outcomes. Performance measures are
simply information that allows more meaningful and informed dialogues about policy and program
decisions. How the information should be used and what policy options should be proposed and
chosen are often beyond the scope of performance measurement. Citizens who expect to use
performance measurement to dictate how elected officials should govern may feel disappointed and
may not be interested in participating.
Even if elected officials and managers are serious about performance measurement and are committed
to use public input and performance information to make a difference in policy making and program
management, citizens are unlikely to see concrete results from their input until years later. This again
may discourage citizen participants to commit their time and effort to the exercise.

How t o Engage Ci t i zen Par t i ci pant s: The Exper i ences of t he I owa Ci t i zen- I ni t i at ed
Per f or mance Assessment Pr oj ect

These inherent challenges are real and significant and can easily deter government officials and citizens from
public engagement in performance measurement. Overcoming these hurdles requires diligent and innovative
effort in rethinking and reorganizing the performance measurement routines. In 2001–4, a number of Iowa
cities experimented with “citizen-initiated performance assessment” (CIPA), in which citizens joined with
elected officials and managers to develop and use performance measures and help government evaluate
various municipal services, such as nuisance control, garbage disposal, snow removal, police and fire
protection, and transportation (Ho & Coates, 2004). Based on the three-year project experience, several
practical lessons about citizen engagement have emerged.

1. Traditional Mechanisms of Citizen Participation

Citizen committees, public hearings or town hall meetings, and focus-group discussions are still useful tools
to engage citizens in discussing performance of government programs and services. There is nothing more
effective than face-to-face interactions between citizens and public officials, helping break down stereotypes
and mistrust and showing each other they can be sincere and equal participants in making government more
effective in meeting the needs of a community.

However, these mechanisms have significant limitations. For example, they allow only a small number of
citizens to engage in in-depth dialogues and exchanges of ideas. The frequency and length of discussion are
also constrained by the physical location of the meeting place and the time schedules of the participants.

EBSCOhost – printed on 6/16/2020 12:04 PM via GEORGIA ONMYLINE. All use subject to https://www.ebsco.com/terms-of-use

83

Finally, citizens who volunteer to participate in these meetings tend to be community activists or citizens who
can afford the time. As a result, they may not be highly representative of the demographic profile of a
community.

2. Usage of Surveys and Response Cards

Citizen surveys are another viable mechanism to solicit public input about the quality and performance of
public programs. Many local governments have annual or biennial citizen surveys to evaluate citizen
satisfaction with and perception of community priorities. Many governments also have user response cards
for specific services. Instead of surveying a whole community where not many may be users of particular
services, user response cards are targeted surveys of particular user groups to evaluate specifically how users
perceive the quality of services. The tool is applicable for many local services, such as library, public works,
water and sewage services, and other customer services. In Iowa, for example, some communities use
response cards to evaluate the responsiveness and professionalism of fire and medical emergency staff. If
these citizen and user surveys are conducted with the appropriate sampling methodology, usually by random
phone or mail surveys of the community or the specific user group, the data can give managers and policy
makers valuable input about the performance of the government from a representative sample of citizens. If
the data are tracked consistently over time, they can also provide a trend analysis of program performance so
that policy makers can evaluate whether steady improvement has been accomplished.

Like the above mechanisms, citizen surveys and response cards also have their limitations. First, the
validity of the instruments depends on the sampling methodology. If the sampling frame is not representative,
possibly because of out-of-date addresses or phone numbers, non-listing of certain residents, or using only a
selected segment of the population, policy makers may get biased results that may misinform
decision-making. Second, the response rate can be a major concern. Many citizens are tired of answering
phone and mail surveys. The response rate problem can be worse in communities with many low-income
families and minority population, as these groups tend to have lower response rates to government surveys.
To compensate for these problems and to make the survey results more reliable and representative,
government officials usually have to invest more time and resources to circulate follow-up surveys and must
use different incentives to induce better responses. However, these mechanisms can be expensive, and not
many communities have the fiscal capacity or the willingness to invest in them.

Moreover, surveys suffer from the fact that the public feedback is constrained heavily by the structure and
wording of the survey instruments. Unlike committee work or focus-group discussion, in which citizens have
more freedom to express their opinions and concerns, survey respondents have to respond to specific
questions and select specific answers in multiple choice questions. If the questions and answers are framed in
a biased way or if certain questions are not asked to avoid potential political embarrassment, the true public
perception of public program performance may not be revealed.

Finally, there is a constraint about how frequently a survey can be conducted, and when a survey can be
sent out. Too many surveys will create survey fatigue and low response rates. Surveys sent at the wrong time,
such as before holidays, may also yield a low response rate. Also, because of the length and complexity
concerns, a government cannot give too much information prior to asking a question and can only ask a
limited number of questions in a survey. A mail survey is also more constrained in structure than a phone
survey and survey questions cannot be tailored easily by user responses. Hence, though surveys are good
instruments to get at public perception of program performance, they have many technical and cost
constraints of which managers should be aware.

3. Usage of the Internet

Information technologies and the World-Wide Web open up new possibilities to solicit public input and
evaluate the performance of public programs more conveniently and easily. Instead of coming to a physical
location or filling out a paper survey, citizens can now visit a specific website to file complaints or service
requests, report their satisfaction level with government services, and conduct synchronous or asynchronous
discussions with officials and other fellow citizens to find solutions to improve government performance.

EBSCOhost – printed on 6/16/2020 12:04 PM via GEORGIA ONMYLINE. All use subject to https://www.ebsco.com/terms-of-use

84

The advantages of the Internet over traditional means of public engagement are not only that it is more
convenient and less constrained by time and physical location, but also that the content can be flexibly
tailored to the needs and interests of the users and can allow many interactive features to help the public
make more informed decisions. For example, when a citizen is asked to evaluate the performance of the
police department online, he or she may be presented with some performance statistics, such as the crime
rates and the average response times to different types of crimes and in different neighborhoods of a
community, side by side with the survey. After the survey, the respondent can be asked to give open-ended
comments on how the department may improve its services, whether the respondent is interested in learning
more about various volunteering opportunities, and whether he or she is willing to participate in some of the
programs and connect with other citizen volunteers.

However, there are also significant limitations to using the Web. First, even though the Internet has
become more widely accessible and commonly used, there is still a concern about the “digital divide”
problem among racial minorities, the elderly, and the poor. Second, how to get residents to learn about the
city government website and visit it can be a major challenge. Third, because Internet surveys can be done so
easily and cheaply, ensuring that no one can easily “game” the system and try to provide multiple entries to
bias the results can be a technical challenge. These limitations suggest that while the Internet can offer a lot
to enhance public participation, it is still at a developing stage and so should be complemented with other
forms of participating channels to get a more balanced and representative view of government performance.

4. Usage of Administrative Data

Finally, a government may tap into its internal database and evaluation instruments to get objective public
input about program performance. For example, the number of program users or members and the number of
volunteers and donations that support a program may indirectly reflect the performance and user satisfaction
with the program. The number of service requests and complaints and types of service requests may show
what areas are poorly rated by the public and need improvement. Data such as response times and scientific
or standardized test results may also complement the survey-based data to facilitate evaluation of the
performance of public programs.

For the past few decades, many professional organizations and government agencies have invested
significant resources in developing and standardizing methodologies for generating and collecting various
kinds of administrative data. For example, crime statistics and response times for the police department and
many user statistics for library services are now commonly available in many local governments because of
these efforts by the federal government and professional organizations. These accomplishments should be
applauded and maintained. However, public managers should also recognize that these administrative data
have some limitations and cannot replace direct public input. First, collecting these data may require
significant investment of time and resources. Managers should strike a balance between the benefits and costs
of getting the data and should be aware of the opportunity cost implications for service delivery. Second,
some of the data, such as scientific data about water quality and usage, can be highly technical. How to
communicate the data and analytical results effectively to policy makers and the public is an important but
often overlooked step. Finally, managers need to remember that “perception is reality” in the political
decision-making process. One or two tragedies in a community, an unexpected event such as a natural
disaster, and changes in the political and economic atmosphere may completely change the political
significance and interpretation of these “objective” data. Even if the data themselves have not changed much
over time, policy makers and managers may still need to make policy and program changes to cater to the
changing expectations and demands of the public. Hence, the usage of administrative data should also be
complemented with other forms of public input to give a full picture of the public’s evaluation of government
performance.

EBSCOhost – printed on 6/16/2020 12:04 PM via GEORGIA ONMYLINE. All use subject to https://www.ebsco.com/terms-of-use

85

Concl usi on

For the past few decades, the practice of performance measurement in the public sector has become more
widespread and sophisticated. Many useful and detailed data are collected and reported internally by federal,
state, and local governments each year. As the data-collection effort matures, policy makers and managers
today are challenged to think more carefully about how to use and report the data more intelligently and
effectively. One of the responses to this challenge, which has been a major emphasis in recent public
administration reforms, is to focus on “performance management” or “results-oriented management” and
think about how to align performance measurement with strategic planning, program evaluation, budgeting,
and personnel decisions. Another response, which is equally important but has been overlooked by many
practitioners, is how to engage the public and policy stakeholders more to develop and use the performance
information so that the information becomes more relevant and significant in the political process.

The second response about public engagement is especially important in the current fiscal environment, in
which the federal government has a serious problem of structural deficits and must ask state and local
governments to take on additional responsibilities. This is occurring while many voters are not fully prepared
to think about the tax implications of federal devolution, yet expect state and local governments to do more
without paying more. Every state and local politician will eventually face this harsh reality and will have to
consider which programs and services should be cut or what taxes will have to be raised. To help make these
tough decisions, both voters and politicians are better off if they are more informed about the needs of the
community and the service accomplishments and efforts of the government so that they can make informed
and balanced decisions about revenue and spending choices. It is in this context that performance
measurement can contribute much to meeting the future challenges of public administration, but its potential
benefits can only be fully realized if it is used along with effective public engagement strategies.

Tremendous progress has been made in efforts to obtain performance measurement data for the past few
decades. However, performance measurement in the twenty-first century has to move beyond the data focus
and pay more attention to issues of performance management and governance—how different stakeholders
and users can be more effectively involved to use the data. As this happens, it is inevitable that performance
measurement may become less technically driven by professional managers, some of the measures may
become less objective and scientific, and political pressure to manipulate the collection and interpretation of
performance data may increase. These challenges, however, are some of the inherent social costs of
democracies, in which information is always vulnerable to distortion by different political, social, and
economic segments of society. Citizens should not be shielded from performance measurement and
performance politics. After all, they are the owners of a democratic government and have the right to define
the “results” and “performance” for which government managers should strive.

Ref erences

Epstein, P., Coates, P. M. & Wray, L. D. (2005). Results that matter: Improving communities by engaging citizens,
. San Francisco: Jossey-Bass.measuring performance, and getting things done

Governmental Accounting Standards Board [GASB], (2003). Reporting performance information: Suggested criteria for
. Norwalk, CT.effective communication

Ho, A. T.-K. (2006a). Accounting for the value of performance measurement from the perspective of city mayors.
, , 217–237.Journal of Public Administration Research & Theory 16

Ho, A. T.-K. (2006b, in press). Exploring the roles of citizens in performance measurement. International Journal of
.Public Administration

Ho, A. T.-K., & Coates, P. (2004). Citizen-initiated performance assessment: The initial Iowa experience. Public
, , 29–50.Performance & Management Review 27

Ho, A. T.-K., & Ni, A. (2005). Have cities shifted to outcome-oriented performance reporting? A content analysis of city
budgets. , , 61–83.Public Budgeting & Finance 25

Jordan, M., & Hackbart, M. (1999). Performance budgeting and performance funding in the states: A status assessment.

EBSCOhost – printed on 6/16/2020 12:04 PM via GEORGIA ONMYLINE. All use subject to https://www.ebsco.com/terms-of-use

86

, , 68–88.Public Budgeting & Finance 19
Joyce, P. G. (1993). Using performance measures for federal budgeting: Proposals and prospects. Public Budgeting &

, , 3–17.Finance 13
Kahn, J. (1997). . Ithaca, NY: CornellBudgeting democracy: State building and citizenship in America, 1890–1928

University Press.
Melkers, J. E. & Willoughby, K. G. (2001). Budgeters’ views of state performance budgeting systems: Distinctions

across branches. , 67,54–64.Public Administration Review
New York Bureau of Municipal Research. (1918). The citizen and the government— a statement of policy and method.

, 57, 1–4.Municipal Research
Nye, J. S. Jr., Zelikow, J. D. & King, D. C. (Eds.). 1997. . Cambridge, MA: HarvardWhy people don’t trust government

University Press.
Osborne, D., & Gaebler, T. (1993). Reinventing government: How the entrepreneurial spirit is transforming the public

. New York: Plume.sector
Ridley, C. E. & Simon, H. A. (1938). Measuring municipal activities: A survey of suggested criteria and reporting forms

. Chicago: International City Managers’ Associatio.for appraising administration
U.S. General Accounting Office [GAO]. (1997). Performance budgeting: Past initiatives offer insights for GPRA

. Washington, DC.implementation. GAO/AIMD-97–46
U.S. General Accounting Office [GAO]. (2004). Results-oriented government: GPRA has established a solid foundation

. Washington, DC.for achieving greater results. GAO-04–38

EBSCOhost – printed on 6/16/2020 12:04 PM via GEORGIA ONMYLINE. All use subject to https://www.ebsco.com/terms-of-use