Assessment Course
Assessment Course: First Year Experience Case Study.
PROMPT:
The institution has employed a first-year experience (FYE) pilot program for at-risk, first-year student populations. In addition to a specific introductory course sequence, students
are engaged with intrusive and proactive student success resources and interventions (e.g., academic advising, tutoring, collaborative learning). Faculty and student services have reported enhanced outcomes for these students compared to students who were not part of the FYE pilot and are considering scaling the pilot to a larger population or potentially the entire student body.
Given that information, please respond to the following prompts:
1. Which elements of your mental model resonate with the situation?
2. For your given assessment approach, how might you mitigate bias from you or others associated with this scenario?
COMPLETION CONSIDERATIONS:
The rubric below is provided based on related elements to the prompt, as well as common clarifying questions from the instructor based on student responses.
Regardless of quality, length of past submissions varied between 100 and 400 words. Know appropriate responses tended to be in the middle or closer to the top end of the range. This is not a hard word limit, so you can go over if needed.
Rubric
Module 5: FYE Case Study Rubric
| Module 5: FYE Case Study Rubric | ||||||
|
Criteria |
Ratings |
Pts |
||||
|
This criterion is linked to a Learning Outcome Individual Application |
5 pts Shares or applies their mental model from personal identities and/or professional experiences in a way that is relevant to the assignment and easy to understand. 3 pts 0 pts |
5 pts | ||||
|
This criterion is linked to a Learning Outcome Critical Lens |
5 pts 3 pts 0 pts |
|||||
|
This criterion is linked to a Learning Outcome Complete Response |
5 pts 3 pts 0 pts |
|||||
|
This criterion is linked to a Learning Outcome Reference to Materials |
5 pts 3 pts 0 pts |
|||||
|
This criterion is linked to a Learning Outcome Response Structure |
5 pts 3 pts 0 pts |
|||||
|
Total Points: 25 |
Articles and Youtube videos to use
https://www.presence.io/blog/how-you-can-use-assessment-to-strive-toward-equity-in-higher-education/
https://www.presence.io/blog/informing-relationships-with-personality-and-the-platinum-rule/
https://journals.canisius.edu/index.php/CSPANY/article/view/303/502
·
American College Personnel Association (ACPA). (2007). ASK standards: Assessment skills and knowledge content standards for student affairs practitioners and scholars. Washington, D. C.: American College Personnel Association (ACPA).
· Association for Institutional Research (AIR). (2013, May 2). Code of ethics and professional practice (CODE). Retrieved from Association for Institutional Research (AIR) website:
https://airweb.org/Membership/Pages/CodeOfEthics.aspx (Links to an external site.)
·
Bensimon, E. M. (2005). Closing the achievement gap in higher education: An organizational learning perspective. New Directions for Higher Education, 2005: 99–111. doi:10.1002/he.190
download
· Chávez, A. F. & Sanlo, R. (Eds.). (2013). Identity and leadership: Informing our lives, informing our practice. Washington, DC: NASPA Publications.
· Demeter, M. (2013).
“Assessing Student Learning Outcomes in Student Affairs: A Primer and Mixed-Methods Strategies.” (Links to an external site.)
CSPA-NYS Journal of Student Affairs, 13 (2).
· Heiser, C., Prince, K., & Levy, J. (2017).
Examining critical theory as a framework to advance equity through student affairs assessment (Links to an external site.)
. Journal of Student Affairs Inquiry, 2(1).
· Henning, G. W., Mitchell, A. A., & Maki, P. L. (2008). The assessment skills and knowledge standards: Professionalizing the work of assessing student learning and development. About Campus, 13(4), 11-17.
· Holland, J. H., Holyoak, K. J., Nisbett, R.E., & Thagard, P. R. (1986). Induction: Processes of inference, learning, and discovery. Cambridge, MA: MIT Press.
· Johnson-Laird, P. N. (1983). Mental models: Towards a cognitive science of language, inference, and consciousness. Cambridge, MA: Harvard University Press.
· Maki, P. L. (2010). Assessing for learning: Building a sustainable commitment across the institution (2nd ed.). Stylus Publishing LLC: Sterling, VA.
· Montenegro, E., & Jankowski, N. A. (2017, January).
Equity and assessment: Moving towards culturally responsive assessment (Occasional Paper No. 29) (Links to an external site.)
. Urbana, IL: University of Illinois and Indiana University, National Institute for Learning Outcomes Assessment (NILOA).
·
· NILOA has captured the many responses and extensions of the conversation from this paper on this
Equity in Assessment (Links to an external site.)
page.
· Schuh, J. H., Biddix, J. P., Dean, L. A., & Kinzie, J. (2016). Assessment in student affairs: A contemporary look (2nd ed.). San Francisco, CA: Jossey Bass.
· Schuh, J. H., & Upcraft, M. L. (1998). Facts and myths about assessment in student affairs. About Campus, 3(5), 2-8.
The author us
es
the theory and process of organizational
learning to make a case for how to understand and
address the cultural and structural barriers that preclud
e
colleges and universities from producing equitable
educational outcomes for students.
NEW DIRECTIONS FOR HIGHER EDUCATION, no. 131, Fall 2005 © Wiley Periodicals, Inc. 99
8
Closing the Achievement Gap in
Higher Education: An Organizational
Learning Perspective
Estela Mara Bensimon
In this chapter I address one of the most urgent and intractable problems in
higher education—inequality in educational outcomes for historically under-
served groups—from the perspective of organizational learning theory.
Historically, in the higher education research community, the study of
minority students has been primarily through the lens of student develop-
ment theories. (In this chapter, I use the terms minority and underrepresented
interchangeably to refer to racial and ethnic groups that are experiencing the
greatest achievement gaps as measured by traditional educational indicator
s
such as attainment of the bachelor’s degree: Puerto Ricans, Mexican
Americans, African Americans, Native Americans, Native Hawaiians, and
others.) I propose that the theory and processes of organizational learning
can help researchers and practitioners understand and address the structural
and cultural obstacles that prevent colleges and universities from producing
equitable educational outcomes. Organization learning, in both theory and
practice, is particularly effective in making the invisible visible and the undis-
cussable discussable, two conditions that aptly describe the status of race-
and ethnic-based unequal outcomes on most campuses.
Among the many factors that contribute to the invisibility of unequal
college outcomes for underrepresented minorities, an obvious one is that
The study on which this chapter is based, “Designing and Implementing a Diversit
y
Scorecard to Improve Institutional Effectiveness for Underserved Minority Students,” is
funded by the James Irvine Foundation. The findings and opinions here are solely those
of the author and do not reflect the position or priorities of the foundation.–– Bensimon
100 ORGANIZATIONAL LEARNING IN HIGHER EDUCATION
the disaggregation of student outcome data by race and ethnicity (and by
gender within racial and ethnic categories) is not an institutionalized prac-
tice. Institutional practices develop from and reflect the shared cognitive
frames of institutional participants. Cognitive frames, also known as men-
tal maps, represent “the rules or reasoning” that govern how individuals
interpret situations and how they design and implement their actions
(Argyris, 1991). Organizational learning theory can help us understand the
nature of cognitive frames and the ways in which some reveal patterns of
unequal outcomes, while others hide them. If patterns of inequality are
invisible, they will not be discussed, and if institutional participants do not
have a reason or opportunity to talk about unequal outcomes, the problem
will not be addressed directly.
I am concerned here with a particular kind of organizational learning
problem: the persistence of unequal educational outcomes for racial and eth-
nic groups with a history of past discrimination in postsecondary education.
I view inequality in educational outcomes as a learning problem of institu-
tional actors—faculty members, administrators, counselors, and others—
rather than as a learning problem of students, the more typical interpretation
(Garmoran and others, 2003). The problem of unequal outcomes resides
within individuals, in the cognitive frames that govern their attitudes, beliefs,
values, and actions. Similarly, the reduction of inequalities also lies within
individuals, specifically, in their capacity to develop equity as their cognitive
frame. That is, individuals whose institutional roles can influence whether
students are successful or not need to learn cognitive processes that enable
them to think about the situation of underrepresented students and their out-
comes through the lens of equity. To put it simply, faculty members, coun-
selors, and institutional leaders need to become equity minded. However,
even if they were to consider the educational status of underrepresented stu-
dents within their own institutions or departments (reflection on the educa-
tional outcomes of minorities is not a routine practice in most institutions of
higher education), institutional actors are more predisposed to do so from
the standpoint of diversity or deficit. Institutional actors are more likely to
view diversity as a generalized characteristic of institutions and be blind
to the particular circumstances of the racial and ethnic groups that constitute
diversity. Or if they are or become aware of the educational status of specific
racial/ethnic groups within their own campuses and departments, they are
more likely to make stereotypical attributions, such as associating deficit with
blacks and Hispanics and achievement with whites and Asians.
The Role of Individuals in Organizational Learning
The key concepts in regard to individuals are that (1) learning is done by
individuals who are members of an organizational entity such as a college
or university, an administrative division, an academic department, or a
research team; (2) individuals inquire into a problem collectively, on behalf
CLOSING THE ACHIEVEMENT GAP IN HIGHER EDUCATION 101
of an organizational entity (Huber, 1991); and (3) organizational culture
and structures can promote or inhibit individual learning (Argyris and
Schön, 1996; Kezar, Glenn, Lester, and Nakamoto, 2004).
Contrary to the dominant belief that the solution to unequal educa-
tional outcomes lies in a new program or technique, somewhere out there,
that has been validated as a “best practice,” I (along with my colleagues at
the Center for Urban Education) believe that institutional actors, as a con-
sequence of their beliefs, expectations, values, and practices, create or per-
petuate unequal outcomes and that the possibility for reversing inequalities
depends on individual learning that holds the potential for bringing about
self-change. That is, individuals—the ways in which they teach, think stu-
dents learn, and connect with students, and the assumptions they make
about students based on their race or ethnicity—can create the problem of
unequal outcomes. Such individuals, if placed in situations where they learn
the ways in which their own thinking creates or accentuates inequities, can
also learn new ways of thinking that are more equity minded. Individually
and collectively, campus members can be the creators of the conditions that
result in unequal or equitable outcomes.
What Is a Cognitive Frame? I use the concept of cognitive frame to
describe the interpretive frameworks through which individuals make sense
of phenomena. A cognitive frame is the way in which an individual under-
stands a situation. Cognitive frames represent conceptual maps and deter-
mine what questions may be asked, what information is collected, how
problems are defined, and what action should be taken (Bensimon, 1989;
Bensimon and Neumann, 1993; Neumann, 1989; Neumann and Bensimon,
1990). Understanding cognitive frames is important because at the same time
that frames make some things visible, they also function as cognitive blind-
ers in that whatever is out of frame may be imperceptible (Bensimon, 1990).
Over time, individuals develop cognitive frames that represent implicit
sense-making theories to help them interpret why things are as they are.
Cognitive frames are reflections of how individuals think; they represent
the cognitive “rules or reasoning” they use to design and implement their
actions” (Argyris, 1991). Cognitive frames are important because they help
us understand the ways in which individuals can manufacture inequality,
as well as reduce it.
The Cognitive Frames of Diversity, Deficit, and Equity. Briefly,
when individuals are guided by diversity as their cognitive frame (see
Bensimon, Hao, and Bustillos, forthcoming, for a more expanded discussion
of the three cognitive frames), they focus their attention on demographic
characteristics of the student body, and view diversity in terms of interra-
cial contact and human relations. Diversity is also viewed as an institutional
characteristic that promotes learning outcomes and better prepares students
for an increasingly diverse workforce and society. For example, the Supreme
Court’s ruling in favor of the University of Michigan’s consideration of race
as a criterion for admission to the law school is based on the premise that
universities have a “compelling interest in attaining a diverse student body”
because diversity yields educational benefits, promotes cross-racial under-
standing, and so forth (Grutter v. Bollinger, 2003).
Individuals with a deficit cognitive frame may value diversity and have
positive attitudes toward increasing minority student participation in higher
education, but they are inclined to attribute differences in educational out-
comes for black, Hispanic, and Native American students, such as lower
rates of retention or degree completion, to cultural stereotypes, inadequate
socialization, or lack of motivation and initiative on the part of the students.
The deficit cognitive frame is expressed in disapproving attributions such
as complaining that “minority students” do not take advantage of the tuto-
rial and academic support services the institution makes available. It can
also be conveyed in well-meaning but pessimistic attributions, such as con-
cluding that students cannot be expected to overcome the disadvantages of
poverty and undepreparation; therefore, unequal outcomes are to be
expected. Attibutions framed by a deficit perspective imply that the aca-
demic difficulties of minority students are either self-inflicted or a natural
outcome of socioeconomic and educational background. Essentially, from
a deficit perspective, unequal outcomes are a problem without a solution.
Diversity-minded individuals are attuned to demographic differences;
for example, they will comment on how diverse the student population is
or how it lacks diversity, but more likely than not, they will be blind to the
fact that the very students whose presence makes campus diversity possible
are themselves experiencing unequal educational outcomes. In contrast,
individuals whose beliefs and actions are guided by the deficit cognitive
frame may be cognizant that their student body is diverse, and they may also
be cognizant that there are racial disparities in educational outcomes, but
they are impervious to the fact that they attribute the problem to the stu-
dents and fail to take into account their own roles in the creation or solu-
tion of unequal outcomes. In sum, diversity-minded individuals may
embrace diversity but not take into account racial achievement patterns
(Pollock, 2001), and deficit-minded individuals take note of racial achieve-
ment patterns but treat them as “natural” in the light of the individuals’ cul-
tural, socioeconomic, and educational backgrounds.
Individuals who are guided by the equity cognitive frame focus inten-
tionally on the educational results or outcomes of black, Hispanic, and
Native American students. They are color conscious in an affirmative sense.
For example, they are more prone to notice and question patterns of edu-
cational outcomes, and they are also more likely to view inequalities in the
context of a history of exclusion, discrimination, and educational apartheid.
Most important, equity-minded individuals are far more likely to under-
stand that the beliefs, expectations, and actions of individuals influence
whether minority group students are construed as being capable or inca-
pable. Table 8.1 compares the three cognitive frames on four dimensions:
orientation, discourse, strategy, and guiding questions.
102 ORGANIZATIONAL LEARNING IN HIGHER EDUCATION
T
a
b
le
8
.1
.
D
iv
er
si
ty
,
D
efi
ci
t,
a
n
d
E
q
u
it
y
C
o
gn
it
iv
e
F
ra
m
es
C
o
m
p
a
re
d
o
n
F
o
u
r
D
im
en
si
o
n
s
D
iv
er
si
t
y
C
og
n
it
iv
e
F
ra
m
e
D
efi
ci
t
C
og
n
it
iv
e
F
ra
m
e
E
qu
it
y
C
og
n
it
iv
e
F
ra
m
e
O
ri
e
n
ta
ti
o
n
F
o
cu
s
o
n
th
e
re
p
re
se
n
ta
ti
o
n
o
f
d
if
fe
re
n
ce
s
(f
o
r
ex
am
p
le
,
ge
n
d
er
,
ra
ce
,
et
h
n
ic
it
y,
se
x
u
al
o
ri
en
ta
ti
o
n
,
re
li
gi
o
n
,
an
d
so
o
n
)
in
th
e
st
u
d
en
t
b
o
d
y
F
o
cu
s
o
n
st
er
eo
ty
p
ic
al
ch
ar
ac
te
ri
st
ic
s
as
so
ci
at
ed
w
it
h
th
e
cu
lt
u
re
o
f
d
is
ad
va
n
ta
ge
an
d
p
o
ve
rt
y
F
o
cu
s
o
n
in
st
it
u
ti
o
n
al
p
ra
ct
ic
es
an
d
th
e
p
ro
d
u
ct
io
n
o
f
u
n
eq
u
al
ed
u
ca
ti
o
n
al
o
u
tc
o
m
es
fo
r
m
in
o
ri
ty
gr
o
u
p
st
u
d
en
ts
D
is
co
u
rs
e
C
el
eb
ra
ti
n
g
d
iv
er
si
ty
,
cr
o
ss
ra
ci
al
re
la
ti
o
n
sh
ip
s,
an
d
co
lo
r-
b
li
n
d
n
es
s,
en
h
an
ci
n
g
ac
ce
ss
,
co
gn
it
iv
e
an
d
so
ci
al
b
en
efi
ts
o
f
h
av
in
g
a
d
iv
er
se
st
u
d
en
t
b
o
d
y
L
ac
k
o
f
p
re
p
ar
at
io
n
,
m
o
ti
va
ti
o
n
,
st
u
d
y
sk
il
ls
,
b
la
m
i
n
g
st
u
d
en
ts
an
d
/o
r
th
ei
r
b
ac
k
gr
o
u
n
d
s
In
st
it
u
ti
o
n
al
re
sp
o
n
si
b
il
it
y
fo
r
st
u
d
en
t
o
u
tc
o
m
es
,
th
e
m
an
if
es
ta
ti
o
n
o
f
in
st
it
u
ti
o
n
al
iz
ed
ra
ci
sm
,
co
lo
r-
co
n
sc
io
u
s,
aw
ar
en
es
s
o
f
ra
ci
al
iz
ed
p
ra
ct
ic
es
an
d
th
ei
r
d
if
fe
re
n
ti
a
l
co
n
se
q
u
en
ce
s,
aw
ar
en
es
s
o
f
w
h
it
e
p
ri
vi
le
ge
St
ra
te
gi
es
W
o
rk
sh
o
p
s,
se
n
si
ti
vi
ty
tr
ai
n
in
g,
ex
p
o
si
n
g
w
h
it
es
to
th
e
“o
th
er
,”
d
iv
er
si
fy
in
g
th
e
cu
rr
ic
u
lu
m
,
cr
ea
ti
n
g
in
te
r-
cu
lt
u
ra
l
ce
n
te
rs
C
o
m
p
en
sa
to
ry
ed
u
ca
ti
o
n
al
p
ro
gr
am
s,
re
m
ed
ia
l
co
u
rs
es
,
sp
ec
ia
l
p
ro
gr
am
s,
al
l
fo
cu
se
d
o
n
fi
x
in
g
th
e
st
u
d
en
t
C
h
an
gi
n
g
in
st
it
u
ti
o
n
s,
d
ev
el
o
p
in
g
in
st
it
u
ti
o
n
al
ac
co
u
n
ta
b
il
it
y
o
f
eq
u
it
ab
le
ed
u
ca
ti
o
n
al
o
u
tc
o
m
es
,
ch
an
gi
n
g
in
d
iv
id
u
al
s’
co
gn
it
iv
e
fr
am
es
In most institutions of higher education, the discourses of deficit and
diversity are more likely to be heard than the discourse of equity. But the
kinds of personal and institutional changes needed to eliminate the achieve-
ment gap are more likely to originate from equity thinking, which raises the
following questions: In what ways can equity thinking be encouraged? In
what ways might we shift individuals’ cognitive frames from deficit and
diversity toward equity? More to the point, what kinds of structures and
processes might produce individual and collective learning that brings about
equity thinking? In the section that follows, I offer ways of considering
these questions, but with a caveat. Given the intractability of the problem
of racial inequity in the United States, it would be foolhardy to claim a solu-
tion. Instead, what I offer is a way of thinking about the problem, one that
is grounded in the theory of organization learning.
Equity Thinking Requires Double-Loop Learning. Argyris and
Schön (1996) differentiate between two types of learning: single loop and
double loop. Single-loop learners are prone to externalize problems by
attributing them to forces and circumstances that are beyond their control
and to resort to compensatory strategies as the treatment for problems that
are perceived as dysfunctions. In single-loop learning, the focus is on
reestablishing stability and normality by enacting corrections and eliminat-
ing errors. Solutions that come from single-loop learning focus on the exter-
nal manifestations of the problem and leave internal values, norms, and
beliefs intact—hence, the label single loop.
For example, individuals who have a deficit cognitive frame turn the
focus of unequal outcomes away from their own attitudes, beliefs, and
behaviors to those of the students. They externalize the problem and by so
doing bring their “own learning to a grinding halt” (Argyris, 1991, p. 7). To
put it simply, they fail to see how changes in their own attitudes, beliefs,
and practices could reverse unequal outcomes.
Double-loop learning focuses attention on the root causes of a problem
and the changes that need to be made in the attitudes, values, beliefs, and
practices of individuals to bring about enduring results (Bauman, 2002).
Looking inward is the capacity to reflect on how practices (also beliefs and
expectations) at the individual and institutional levels produce racial
inequalities. In particular, according to Argyris (1991), individuals “must
learn how the very way they go about defining and solving problems can be
a source of the problems in its own right” (p. 2).
Simply put, the difference between single-loop and double-loop learn-
ing is that in the former, change is at a surface level, whereas in the latter,
the change is in underlying norms, beliefs, and principles (Coburn, 2003).
Thus, bringing about a cognitive shift from diversity to equity or from
deficit to equity involves double-loop learning.
The development of equity as a cognitive frame is a double-loop learn-
ing problem because it requires the willingness of individuals (1) to make
the disaggregating of data on student outcomes by race/ethnicity and gender
104 ORGANIZATIONAL LEARNING IN HIGHER EDUCATION
a routine and necessary practice to self-assess progress toward equity in edu-
cational outcomes; (2) identify equity in educational outcomes as an essen-
tial indicator of institutional performance and quality; and (3) assume
responsibility for the elimination of unequal results.
Inquiry as a Method of Developing New Cognitive
Frames
Bringing about a cognitive shift in individuals whose dominant frames are
diversity or deficit requires an approach that enables them to see, on their
own and as concretely as possible, racial and ethnic patterns in educational
outcomes. Over the past three years, researchers at the University of
Southern California’s Center for Urban Education have been experimenting
with such an approach. This approach, which is described in detail in other
publications (Bensimon, 2004; Bensimon, Polkinghorne, Bauman, and
Vallejo, 2004; www.usc.edu/dept/education/CUE), is designed to create or
intensify awareness of equity or inequity by organizing campus members,
such as professors, counselors, and deans, into inquiry teams that have been
dubbed evidence teams because their role is to collect data on student out-
comes disaggregated by race and ethnicity and analyze them. Their purpose
is to hold a mirror up to their institution that reflects clearly and unam-
biguously the status of underrepresented students with respect to basic edu-
cational outcomes. Through inquiry, it is expected that individuals will
learn of the nature of racial patterns in educational outcomes. By “learning,”
I mean noticing and seeing—that is, developing an awareness that racial and
ethnic patterns of inequalities exist. By “equity,” I mean that the outcomes
of minority group students should more closely reflect their representation
in the student body (for a more technical definition, see Bensimon, Hao, and
Bustillos, forthcoming). Some individuals lack complete awareness, while
others have a generalized sense of them; thus, for some individuals, there is
a need to develop initial awareness, and for others there is a need to inten-
sify their awareness. The challenge is how to develop or intensify equity-
oriented awareness.
The critical importance of learning new or intensified awareness is
exemplified by some of the initial reactions of individuals who were
appointed by their presidents to serve on campus evidence teams. For exam-
ple, a dean whose president had appointed him as the leader of the campus’s
evidence team told us on our first meeting, “We are 100 percent diverse.
The Equity Scorecard may be relevant for other institutions like yours
[meaning the University of Southern California], but we don’t need to do
that [disaggregate]; we know what it will look like . . . for us there are no
differences by ethnicity.” Clearly, this individual was aware of diversity as
an institutional characteristic and could not entertain the possibility that
within the diversity of the student body, some racial or ethnic groups may
have been experiencing more equitable educational outcomes than others.
CLOSING THE ACHIEVEMENT GAP IN HIGHER EDUCATION 105
106 ORGANIZATIONAL LEARNING IN HIGHER EDUCATION
However, it is possible that through a process of inquiry, a diversity-minded
individual such as this dean can learn to think from the perspective of
equity. As it happens, this individual’s cognitive frame evidenced a shift
toward equity. In addition, members of the evidence teams whose dominant
cognitive frame was diversity initially failed to see the need for disaggregat-
ing the data, a necessary condition for double-loop learning. Although dis-
aggregating of data is not a guarantee of double-loop learning or equity
thinking, it is a necessary step.
Other individuals were generally aware of unequal results, and the
inquiry process was a catalyst for intensifying it and giving the individual
the impetus to act more assertively to bring about change. For example, an
individual, after having seen data on outcomes disaggregated by race and
ethnicity, said, “I had always felt and had a pretty good sense of the situa-
tion of minority students, but then for the first time started looking at the
data, and it was just overwhelming. So, [seeing the data] has really had a
tremendous impact” (unpublished field notes, Center for Urban Education).
Although most institutions routinely disaggregate enrollment data, they
rarely disaggregate data on more finely grained indicators of outcomes.
When the evidence teams were asked to do this, these were some of the
reactions we heard:
“We track financial aid, but we don’t usually disaggregate it by ethnicity and
types of awards.”
“No one has ever asked us to disaggregate data by ethnicity and gender, and
by program and academic preparation.”
“I [chair of a humanities discipline] never asked [the institutional
researcher] to disaggregate the data for my department. . . . I didn’t have
a reason.”
In sum, disaggregated data serve as the medium through which indi-
viduals learn about unequal outcomes on behalf of their campuses. The way
in which data are displayed and discussed can intensify learning, confirm
or refute untested hypotheses, challenge preconceived ideas, motivate fur-
ther inquiry, and provide the impetus for change.
Becoming Equity Minded. For practitioners to realize the enormity
of the problem of unequal outcomes, they have to see hard evidence for
themselves. This is accomplished by scrutinizing the data, asking questions
that have suddenly come to mind, and discovering patterns of student con-
ditions that had been concealed before the data were examined. Thus, to
bring about new or intensified awareness of unequal results, evidence team
members are directly involved in collecting student data, talking about the
information, and using it to create equity measures and benchmarks to put
into an institutional self-assessment tool known as the Equity Scorecard.
The scorecard provides four concurrent perspectives on institutional per-
formance in terms of equity in educational outcomes: access, retention,
institutional receptivity, and excellence. The responsibility of the evidence
teams was to create indicators of equity for each of the four perspectives.
(The measures are available at http://www.usc.edu/dept/education/CUE/
projects/ds/diversityscorecard.html.)
Typically institutional researchers are responsible for gathering and
analyzing data, and their findings are disseminated primarily to adminis-
trators in written reports. In order to bring the members of the evidence
teams in close proximity to the problem of unequal outcomes, they are
assigned the role of researchers and have the responsibility for developing
and interpreting the needed equity indicators. This heightens their aware-
ness of the issues. Faculty members and others may be generally aware that
there are disparities in educational outcomes, but persuading individuals to
reflect on how their own practices may be contributing to the problem is
another matter. They must learn to look at the particulars of the problem
within their own context.
Shifting from Diversity and Deficit to an Equity Cognitive Frame.
To illustrate the process of individuals’ becoming more equity-minded, I
introduce two individuals, whom I refer to as Carter and Stone, both actual
members of evidence teams whose language during the course of the proj-
ect changed noticeably from diversity and deficit to equity. I focus on these
two individuals because their initial attitude toward the project was one of
skepticism and lack of enthusiasm and because it was clear that for both
of them, the concept of equity in educational outcomes was new and sus-
pect. I will describe their cognitive frames before they saw any data disag-
gregated by race and ethnicity and after their team began to examine and
talk about disaggregated data. These descriptions are based on field notes
that describe what these individuals said in the context of their participa-
tion in their campus evidence team.
Carter is the dean that I referred to earlier whose initial reaction to the
Equity Scorecard was that since the campus was so diverse, it would not
be very useful and that he doubted what could be learned from the process
of disaggregating data. Carter was a dean at a community college that was
predominantly Hispanic and also had a large number of immigrants of all
races and ethnicities from nations around the world. On our first meeting
with this team, Carter, despite not having seen any data, was quick to say,
“We are like the UN, so for us, there is not going to be any difference by
ethnicity. In fact, by the very nature of the student population, what we are
likely to find is that it is all bad” regardless of the students’ ethnicity or
racial background. The cognitive frames that are identifiable in this brief
excerpt are diversity (“we are like the UN”) and deficit (“the outcomes will
be bad for all”).
In subsequent meetings, when the team began to look at actual out-
comes data that unequivocally showed Hispanics and blacks faring much
worse than whites on just about every measure of educational outcomes,
Carter’s language began to change. Examining a printout showing grades
CLOSING THE ACHIEVEMENT GAP IN HIGHER EDUCATION 107
earned in math courses broken down by race and ethnicity and seeing dra-
matic differences, he said, “I just think that there’s going to be some
nonpedagogical explanation, a racist explanation for lack of a better term.”
On the same day as he looked at data on student performance in gateway
courses into the majors, he suddenly exclaimed, “Goddamit! Look at
Business. There is a much higher success rate for whites than for the other
groups. I bet that the reason for this is that some professors encourage par-
ticular students [high-achieving white immigrant ethnic groups] to take
their course sections and give them better grades.”
The point in this brief example is not whether this individual was right
or wrong in attributing the inequalities he was seeing for the first time to
racism. What matters is that Carter, on becoming aware of unequal out-
comes, began to see the problem in ways that he had not previously consid-
ered. Rather than talking about diversity or suggesting that the differences
in outcomes were a reflection of student deficits, he was considering the pos-
sibility that differences in outcomes might be attributable to individuals’
unconscious practices or to institutional practices that unintentionally cre-
ate circumstances that result in inequalities.
Like Carter, Stone is also in a college that is predominantly Hispanic
and black, except that it is a four-year college. Before seeing data disag-
gregated by race and ethnicity, Stone’s cognitive frame was clearly iden-
tifiable as diversity and deficit. At the outset of the project, he protested
that “the Equity Scorecard focuses on remediating wrongs instead of cel-
ebrating differences.” He said he would much rather “focus on how diver-
sity is encouraged, celebrated, and welcomed” (diversity cognitive frame).
At another meeting but before any data had been reviewed, he expressed
a concern about the “low enrollment of Asians and whites among the
first-time freshmen” and said that maybe they should be more concerned
“with the dynamic of white flight” rather than with equity in outcomes
(deficit cognitive frame). While this individual exhibited both diversity
and deficit thinking, it was clear that deficit was his dominant cognitive
frame. For example, on seeing data that Hispanics were graduating at a
higher rate than whites, he commented that this was an “atypical” find-
ing because it went against his expectation that Hispanics would do less
well than whites.
After several months, this team finally began to look at disaggregated
data, and once they did, Stone’s language changed noticeably. For example,
in looking at data that showed large gaps in the outcomes for African
American students in mathematics, he said to the others on the team, “I am
profoundly affected by the performance of African Americans.” Had this
statement been made by someone who had been identified as having an
equity cognitive frame, it would not have attracted our attention. However,
since up to this point Carter had been resistant to the equity-oriented
aspects of the project and on different occasions had made comments that
108 ORGANIZATIONAL LEARNING IN HIGHER EDUCATION
reflected a deficit perspective, being “profoundly” affected represented a
departure from his usual way of thinking. I am not suggesting that simply
because Stone admitted to being “profoundly affected by the performance
of African Americans,” he had experienced a sudden and dramatic shift in
cognitive frames. Rather, his statement hinted at a possible change that we
should watch for.
Indeed, subsequent statements demonstrated that he was undergoing
a cognitive shift. For example, when one of his colleagues on the team men-
tioned how much had been learned by disaggregating data by race and eth-
nicity, Stone experienced an Aha! moment. He suddenly realized that the
collaborative process of examining data served the purpose of “raising con-
sciousness about disparities among different groups.” “We almost do a dis-
service by not looking at equity as a focal point,” he said. At another
meeting, he spoke about the results of a faculty survey: “We conducted a
faculty survey, and one item that was rated very high was the potential of
our students.” “But in conversations with faculty,” it was disturbing for him
to discover that despite espousing a belief in the students’ potential, “they
disparage their academic quality.”
After this team began to examine data disaggregated by race and eth-
nicity and started discussing the clear-cut patterns of inequality that were
revealed, Stone’s language shifted from diversity and deficit toward equity.
The language of deficit that had been prevalent in the first year of the proj-
ect was gradually replaced by discourse that reflected a growing awareness
of racism and inconsistencies in what faculty espouse at an abstract level as
opposed to their actual perceptions when they speak about students from
particular groups.
Do these brief illustrations suggest that individuals who reflected
changes in their language and interpretations become equity minded? That
is, do these subtle changes in language indicate that these individuals had
changed and therefore were more likely to examine their own practices?
Were they now ready to spearhead change within their own institutions? At
this juncture in our work, it is premature to suggest that the learning evi-
denced in the shifts in interpretation will systematically translate into sig-
nificant and large-scale changes. In addition, I cannot rule out that Carter
and Stone will not revert to diversity or deficit thinking. Ultimately what is
important is whether individuals like Carter and Stone consistently act from
an equity frame of mind so that it spreads throughout the institution and
becomes a shared way of thinking and acting. It would be foolhardy for me
to assert that this goal has been achieved. Nevertheless, our work under-
scores that in order to move toward the reversal of unequal higher educa-
tional outcomes, individuals who occupy positions of power and authority,
like Carter and Stone, or like me and the other authors of this volume, we
all need to learn to think from the standpoint of equity. Unless that hap-
pens, we are not likely to even get started.
CLOSING THE ACHIEVEMENT GAP IN HIGHER EDUCATION 109
Conclusion
After four years of listening to and interpreting the conversations of the
individuals who form the teams in the Equity Scorecard project, I believe
that organizational learning, at the local level, by individuals who are clos-
est to the problem may have a greater impact in reversing inequality in
higher education than the numerous diversity-oriented interventions devel-
oped throughout the 1980s and 1990s. The illustrations I have shared pro-
vide a glimpse into the power of organizational learning to bring about
changes in the cognitive frames of individuals. In essence, “the knowledge
production itself may become the form of mobilization” that induces indi-
viduals to make the cognitive shift (Gaventa and Cornwall, 2001, p. 76) that
leads to change from within the self outward to the institution.
References
Argyris, C. “Teaching Smart People How to Learn.” Reflections, 1991, 4(2), 4 – 15.
Argyris, C. “Good Communication That Blocks Learning.” In C. Argyris (ed.), On
Organizational Learning. Boston: Harvard Business Review Press, 1994.
Argyris, C., and Schön, D. A. Organizational Learning II: Theory, Method, and Practice.
Reading, Mass.: Addison-Wesley, 1996.
Bauman, G. L. “Developing a Culture of Evidence: Using Institutional Data to Identify
Inequitable Educational Outcomes.” Unpublished doctoral dissertation, University of
Southern California, 2002.
Bensimon, E., and Neumann, A. Redesigning Collegiate Leadership. Baltimore, Md.: Johns
Hopkins University Press, 1993.
Bensimon, E. M. “The Meaning of ‘Good Presidential Leadership’: A Frame Analysis.”
Review of Higher Education, 1989, 12(2), 107 – 123.
Bensimon, E. M. “Viewing the Presidency: Perceptual Congruence Between Presidents
and Leaders on Their Campuses.” Leadership Quarterly, 1990, 1(2), 71 – 90.
Bensimon, E. M. “The Diversity Scorecard: A Learning Approach to Institutional
Change.” Change, 2004, 36(1), 45 – 52.
Bensimon, E. M. Equality in Fact, Equality in Results: A Matter of Institutional
Accountability. Washington D.C.: American Council on Education, 2005.
Bensimon, E. M., Hao, L., and Bustillos, L. T. “Measuring the State of Equity in Higher
Education.” In P. Gandara, G. Orfield, and C. Horn (eds.), Leveraging Promise and
Expanding Opportunity in Higher Education. Albany: State University of New York
Press, forthcoming.
Bensimon, E. M., Polkinghorne, D. E., Bauman, G. L., and Vallejo, E. “Research That
Makes a Difference.” Journal of Higher Education, 2004, 75(1), 104 – 126.
Coburn, C. E. “Rethinking Scale: Moving Beyond Numbers to Deep and Lasting
Change.” Educational Researcher, 2003, 32(6), 3 – 12.
Gaventa, J., and Cornwall, A. “Power and Knowledge.” In P. Reason and H. Bradbury
(eds.), Handbook of Action Research. Thousand Oaks, Calif.: Sage, 2001.
Grutter v. Bollinger, 539 U.S. 306, 2003.
Huber, G. P. “Organizational Learning: The Contributing Processes and the Literatures.”
Organization Science, 1991, 2(1), 88 – 115.
Kezar, A., Glenn, W., Lester, J., and Nakamoto, J. Institutional Contexts and Equitable
Educational Outcomes: Empowered to Learn. University of Southern California: Center
for Urban Education, 2004.
110 ORGANIZATIONAL LEARNING IN HIGHER EDUCATION
Kim, D. “The Link Between Individual and Organizational Learning.” Sloan Management
Review, 1993, 35(1), 37 – 50.
Neumann, A. “Strategic Leadership: The Changing Orientations of College Presidents.”
Review of Higher Education, 1989, 12(2), 137 – 151.
Neumann, A., and Bensimon, E. M. “Constructing the Presidency: College Presidents’
Images of Their Leadership Roles, a Comparative Study.” Journal of Higher Education,
1990, 61(6), 678 – 701.
Pollock, M. “How the Question We Ask Most About Race in Education Is the Very
Question We Most Suppress.” Educational Researcher, 2001, 30(9), 2 – 12.
Stanton-Salazar, R. Manufacturing Hope and Despair: The School and Kin Support Networks
of U.S.-Mexican Youth. New York: Teachers College Press, 2001.
ESTELA MARA BENSIMON is a professor of higher education and the director of
the Center for Urban Education in the Rossier School of Education at the
University of Southern California.
CLOSING THE ACHIEVEMENT GAP IN HIGHER EDUCATION 111
Vol. 3, Issue 1, 2017
Examining Critical
Theory as a Framework to
Advance Equity Through
Student Affairs
Assessment
Ciji A. Heiser Krista Prince and Joseph D. Levy
*
Apr 14, 2017
TTags:ags: critical theory, critical practitioner, equity, assessment cycle
Institution:Institution: University of North Carolina Chapel Hill, National Loius University
Department:Department: See “about the authors”
*
Abstract
Inquiry in student affairs plays a critical role in advancing equity
efforts since it is utilized for the improvement of programs and
services supporting student learning and experiences. Assessment
practice, when undergirded by a critical theoretical framework,
employs intentional approaches corresponding to each phase of
the assessment cycle. Critical practitioners begin by
acknowledging their own subjectivity and the ways their
positionality influences their practice. Further, they acknowledge
the agency of participants as knowers and collaborators in this
work. Additionally, practitioners employ methodological diversity
and center marginalized voices not only in evidence gathering, but
also in interpretation and when implementing change. Employing
such approaches enriches assessment practice and enables data
to be used in transformative ways in the pursuit of equity. This
article explores critical theory and its implications for assessment
practice. Examples and considerations are provided throughout
as well as questions posed for institutional and personal practice
reflection.
Examining Critical Theory as a Framework to Advance Equity Through Student Affairs Assessment
The Journal of Student Affairs Inquiry 1
Assessment is a practice deeply rooted in accountability related
to the costs of higher education. The rise of assessment for
accountability in the 1980s influenced traditional assessment goals
such as evaluating student learning, examining programs, and
determining institutional effectiveness (DeLuca Fernández, 2015;
Wall, Hursh, & Rodgers III, 2014). In an age where increasing
scrutiny is seen as the answer to higher education limitations,
assessment has served as a form of control (Wall et al., 2014).
More recently, external accountability is starting to be
complemented by internal curiosity about the impact of programs
on student learning (Kuh, Jankowski, Ikenberry, & Kinzie, 2014).
At its core, assessment in higher education is “designed to help
faculty and staff improve instruction, programs, and services, and
thus student learning, continuously” (Banta & Palomba, 2015, p.
3). Efforts to identify the impact of student affairs could be made
more comprehensive and inclusive through the incorporation of
an equity orientation.
Assessment in higher education is uniquely positioned to
transform inquiry into a more inclusive practice in pursuit of
equity because it draws “on a wealth of scholarly traditions in
order to critique the status quo, interrogate power, theorize
agency, and work toward social justice” (Pasque, Carducci, Kuntz,
& Gildersleeve, 2012 p. 17). Critical theory is grounded in notions
of justice and centering marginalized voices in order to promote
emancipation, liberation, and equity (Levinson, 2011). While an
axiology of accountability differs vastly from one motivated by
equity and justice, the latter motivations can strengthen
approaches guided by the former. Core components of assessment
practice outlined by the assessment cycle (Maki, 2010, p.7) include
evidence gathering, interpretation, and implementing change;
these components are vastly enhanced when supported by a
critical theoretical framework. Critical approaches have been
examined with regard to research, qualitative inquiry (Pasque et
al., 2012), the study of higher education (Martinez-Alemán,
Pusser, & Bensimon, 2015), and addressing achievement gaps in
higher education (Bensimon, 2005); but have only begun to be
examined in student affairs assessment practices (DeLuca
Fernández, 2015). Strengthened by critical theory, traditional
best-practice approaches to student affairs assessment become
Examining Critical Theory as a Framework to Advance Equity Through Student Affairs Assessment
The Journal of Student Affairs Inquiry 2
transformative for all students by considering the positionality of
the evaluator, recognizing agency of the participants, employing
methodological diversity, and extending analysis strategies.
Critical Social Theory and Assessment
Cultural theorists began their work at the Institute for Social
Research within the Frankfurt School in 1923 (Hanks, 2011, p.
81). Forerunners such as Karl Marx, analyzing capitalism as a
form of domination, brought to light the ways in which market
values left power in the hands of few (Levinson, Gross, Link, &
Hanks, 2011, p. 26). In the context of assessment, this critique
of capitalism is relevant today given how assessment has served
to answer calls for institutional accountability; to show that
institutions are creating workers and knowledge for economic
development (Wall et al., 2014). For example, reporting systems
often emphasize “graduation rates, job placement, and
debt-to-earnings ratios” (Banta & Palomba, 2015, p. 6) rather than
student learning. Drawing on Marx’s work, Max Horkheimer
named critical theory and described emancipation as its central
feature. In pursuit of a more just society, he and others sought
to better understand and expose the systems and institutions that
regulate behavior and perpetuate inequitable outcomes. He named
critical theory to highlight a change-oriented approach in contrast
to traditional theories that only sought understanding. While it
was originally concerned specifically with the effects of capitalism
and its structures on socioeconomic status, now “critical social
theories are those conceptual accounts of the social world that
attempt to understand and explain the causes of structural
domination and inequality in order to facilitate human
emancipation and equity” (Levinson, 2011, p. 2). Such theories
question common sense assumptions and taken for granted
norms. Critical inquiry’s multiple branches include critical race
theories, LatCrit, queer theory, critical feminist theories, critical
discourse analysis, and theories of power and marginalization.
Critical theory, in any of its many forms, centers lived experiences
in order to “identify and locate the ways in which societies
produce and preserve specific inequalities through social, cultural,
and economic systems” (Martinez-Alemán et al., 2015, p. 8). In
this way, critical approaches oriented towards equity differ from
those motived by economics and accountability.
Examining Critical Theory as a Framework to Advance Equity Through Student Affairs Assessment
The Journal of Student Affairs Inquiry 3
An emphasis on economic outcomes for higher education has
led to assessment for accountability, whereby evaluators employ
positivistic and detached approaches. Assessment may be pursued
using unreflective procedural notions emphasizing fair, neutral,
valid, rational, functional, normative, value-free, apolitical
approaches (Martinez-Alemán et al., 2015; McArthur, 2015; Wall
et al., 2014). These approaches leave considerations of power and
privilege largely unexamined if assessment does not interrogate
“by and for whom?” (McArthur, 2015; Wall et al., 2014; DeLuca
Fernández, 2015). Traditional approaches to assessment reinforce
notions of neutrality, sameness, and objectivity, which hinder
potential for transforming inequitable policies, procedures, and
outcomes. Critical practitioners attend to the differences between
groups and seek to remedy underlying systemic inequities that
produce differential outcomes. Critical assessment “expose[s] and
address[es] power, privilege, and structures; consider[s]
thoughtfully histories and contexts; make[s] explicit assumptions
and intentions; [and] eschew[s] colorblind and ideological neutral
claims” (DeLuca Fernández, 2015, p. 5). Thus, critical approaches
enable us to transgress the limitations of, and strengthen,
traditional assessment approaches. While it is not meant to be
prescriptive, practitioners can embody critical principles through
their approach to student affairs assessment work. In order for
assessment to be critical, practitioners must adopt an equity
orientation when approaching each phase of the assessment cycle
by considering positionality, agency, methodological diversity,
and analysis.
PrPractitioner Pactitioner Positionality and Subjectivityositionality and Subjectivity
Ethical standards in assessment and evaluation include
maintaining objectivity, limiting bias, avoiding conflicts of
interest, maintaining confidentiality, determining political risks
of data, and being aware of the impact of data on stakeholders
(American College Personnel Association, 2007; Association for
Institutional Research, 2013). A critical framework challenges the
ability of practitioners to be neutral and unbiased because the
practice of assessment is inextricably linked to the identities held
by the practitioner such that,
as individual leaders, we practice within norms, assumptions, values, beliefs,
Examining Critical Theory as a Framework to Advance Equity Through Student Affairs Assessment
The Journal of Student Affairs Inquiry 4
and behaviors originating in our multiple identities…In addition, identity
influences experiences and perceptions of power or lack thereof and affects
how we think about and practice within power structures of colleges and
universities. (Chávez & Sanlo, 2013, p. 9)
Attention to our identities and experiences is imperative because
“our positionalities -how we see ourselves, how we are perceived
by others, and our experiences- influence how we approach
knowledge, what we know, and what we believe to know” (Bettez,
2015, p. 934-935). In order to address the influence of one’s
subjectivity on their work, a critical practitioner continually
engages in self-reflexivity by interrogating “how [their]
experiences, knowledge, and social positions might impact each
aspect and moment” (Bettez, 2015, p.940) of the assessment cycle.
The influence of one’s positionalities is pervasive, reaching even
the most fundamental of assessment practices such as the notion
of asking the right questions. When designing instruments and
employing different methodologies, acknowledging the myriad
of intersecting identities that shape one’s own lens may lead to
the conclusion that this notion of asking the right questions is
influenced by one’s experiences and biases. Inviting additional
voices to discuss assessment processes such as determining what
to measure, which questions to ask, what methods to use, and
how to analyze and report findings can address positionality and
subjectivity as well as give agency to stakeholders. For example,
a white, cisgender, heterosexual, female assessment practitioner
does not have identities congruent with the assessment of a
program designed to serve men of color. This does not make such
a practitioner ineffective; however, a more effective approach to
assessment would include individuals with similar positionality.
Including students and staff who share the identities of the
population being assessed helps practitioners challenge power
dynamics, be more inclusive of diverse identities, address
assumptions, disrupt ideological neutral claims, and acknowledge
implicit biases throughout the assessment process. This is critical
given the ways positionality can unknowingly influence
practitioners responsible for the data collection and
interpretation.
Practitioners operating from primarily dominant identities may
Examining Critical Theory as a Framework to Advance Equity Through Student Affairs Assessment
The Journal of Student Affairs Inquiry 5
further amplify instances lacking perspective. Critical inquiry
encourages evaluators to account for implicit biases pertaining to
one’s identities. Implicit bias is “a descriptive term encompassing
thoughts and feelings that occur independently of conscious
intention, awareness, or control” (Nosek & Riskind, 2012, p. 115).
Thus, our exposure to societal messages and our experiences may
subconsciously influence our associations both about groups to
which we belong and those we do not. For example, when career
coaches evaluate resumes they may subconsciously associate either
positively or negatively with student name, perceived race/
ethnicity, education background, experience, or geographical
location; but a rubric may mitigate the effects these associations
could have on review and feedback. When utilizing rubrics,
recommended practices of calibration and norming activities help
ensure reliability and work to minimize subjectivity of the
evaluator. Having a well-designed rubric and conducting
calibration activities can norm evaluators with content and
scoring, ultimately aiming to account for existing subjectivity or
implicit biases. Beyond assisting the practitioner, rubrics support
students by clearly communicating examined content and how
scores are determined. Sharing rubrics with students ahead of an
intervention as in the example of reviewing a resume provides
transparency, while also enabling students to set themselves up
for success and familiarizes them with process prior to interacting
with a career coach. Critical approaches such as this work to
navigate positionality and subjectivity, while improving
traditional approaches to assessment, by empowering students
and honoring their agency as subjects in the assessment effort.
Agency of the PAgency of the Participantsarticipants
Rather than positioning the participant as the object of study,
critical practitioners acknowledge the agency of the human
“subject,” who is expert and authority on their own experiences
because “all critical inquiry is grounded in lived experiences, and
power relations and social justice are central concerns”
(Martinez-Alemán et al., 2015, p. 3; Steinberg & Cannella, 2012).
Facilitating collaborative processes by inviting stakeholders to
operate as partners in assessment work, rather than objects of
it, recognizes agency of participants and strengthens assessment
work. Collaboration can occur in multiple elements of assessment
Examining Critical Theory as a Framework to Advance Equity Through Student Affairs Assessment
The Journal of Student Affairs Inquiry 6
practice: mapping learning experiences and programs provided to
larger outcomes or competencies of the institution, writing and
approving learning outcomes, and identifying what is meaningful
and measurable. Of note is that time is a significant consideration
for many practitioners. Culp and Dungy (2012) assert that
institutional leaders should encourage their staff to block off time
on their calendars for assessment related activities such as analysis
and reporting. Incorporating collaborative approaches to
assessment work may be more time intensive than initially
planned, but such approaches build both assessment culture and
competence – which is strongly supported throughout assessment
literature as not only appropriate, but necessary.
Engaging in collaborative processes brings the voices of students,
staff, and faculty from across the institution to the assessment
table. Accreditation standards and criteria already expect students
to be consulted and engaged by institutions in decision making
and providing feedback on university goals and overall
governance processes, not to mention be actively engaged in
assessment (Commission on Institutions of Higher Education,
2016; Higher Learning Commission, 2014; Middle States
Commission on Higher Education, 2015; WASC Senior College
and University Commission, 2013). The responsibility rests with
the institution to execute and determine how to engage students
and ensure all student voices and needs are represented. Maki
(2010) reinforces this concept, stating, “assessment is not a task
for small groups of experts but a collaborative activity; its aim is
wider, better-informed attention to student learning by all parties
with a stake in its improvement” (p. 41). Inviting stakeholders
to operate as collaborative partners in assessment work honors
agency of the stakeholders by prioritizing how their experiences
inform data collection and provide meaningful insight during data
analysis.
One suggestion for considering the agency of the participant is
empowering students as content developers. Seeking perspectives
from minoritized populations for experiential feedback when
creating educational workshops related to race, diversity, or social
justice is one example. Frustrations, concerns, and fears, as well
as points of pride and praise, could also be coupled with theory
and existing needs or campus climate data to generate workshop
Examining Critical Theory as a Framework to Advance Equity Through Student Affairs Assessment
The Journal of Student Affairs Inquiry 7
content. These approaches establish minoritized students as
subject matter experts on the topic of their lived experiences,
enhances engagement, and may attract students to attend given
their role in program development. Student feedback and
positioning may also inform the methodological approaches taken
to collect relevant data.
Methodological DivMethodological Diversityersity
An emphasis on economic outcomes has led many practitioners
to employ positivistic and detached methodological approaches.
Critical approaches to methodology encourage practitioners to
consider what to measure and how, using multiple modalities
for triangulation, and questioning whether a tool measures the
intended topic for different groups. These practices, guided by
principles of critical theory, compliment the notion that learning
is complex and multifaceted; it needs methodological approaches
that work for students engaged in the learning process who are
equally complex and multifaceted (Maki, 2010).
Practitioners employing approaches to assessment grounded in
critical theory reflect thoroughly on the implications of what is
measured and how. In determining what to measure and how,
critical evaluators consider the effects of economic drivers and
which values are attached to what is measured (DeLuca
Fernández, 2015). For example, the outcome that students living
on campus will have higher average grade point averages than
those living off campus may be driven by the economic need to
boost occupancy, by the level of academic support provided to
students living in the residence halls, or both.
The different ways in which participants make meaning and
process information around their experiences influences how
their experience is measured. Approaches to measurement
undergirded by critical theory include exploring multiple
modalities and multiple methods of data collection. Because
learning can be multifaceted and non-linear, Maki (2010)
encourages “…employing a diverse array of methods, including
those that call for actual performance, using them over time so as
to reveal change, growth, and increasing degrees of integration”
(p. 40). Such methods cannot be grounded in normative
Examining Critical Theory as a Framework to Advance Equity Through Student Affairs Assessment
The Journal of Student Affairs Inquiry 8
assumptions or they fail to interrogate underlying disparity
(Martinez-Alemán et al., 2015). When selecting assessment
methods, practitioners operating from an equity orientation pose
questions such as: Will this method reinforce a power dynamic?
Does this method work for this population (e.g. survey or
storytelling)? What additional method would provide a more
comprehensive narrative around a program or service?
Employing multiple measures can open new possibilities and
resistance to a universal truth or interpretation of data. This is
important given traditional approaches to data interpretation of
single or isolated sources of data, as well as potential biases from
positionality or identities.
Not only are multiple measures encouraged, critical approaches
to assessment support the employment of a wider variety of
methods. The most common methodological approach to
assessment is surveying, with an increase in recent years in the
use of rubrics and portfolios (Kuh et al., 2014). Methodological
approaches such as rubrics, journaling, focus groups, interviews,
surveys, and portfolios could be complemented with approaches
such as ethnography, textual analysis, historiography, literary
analysis, aesthetic criticism, theatrical and dramatic ways of
observing (Steinberg & Cannella, 2012, p. 21). Including diverse
methods allows assessment practitioners to leverage the collection
of data as a tool for equity by creating the space for students to
share data around their learning and development in ways that are
as rich and complex as their learning processes and intersecting
identities.
Consider context and audience when determining methodological
approaches. Lacking direct contact with the intended populations
for assessment, a survey might be the best chance to capture data.
When a captive audience or engagement is possible, it affords
opportunities for populations to be engaged in focus groups,
interviews, observations, case studies, reflections, and perhaps
even pre- and post-tests; all direct measures as opposed to likely
indirect survey measures. Knowing specific demographic
information can further shape the approach, allowing or enabling
given populations to share their stories in meaningful and familiar
ways. Critical approaches challenge notions of correctness,
validity, and truth. When considering ideas of validity and truth,
Examining Critical Theory as a Framework to Advance Equity Through Student Affairs Assessment
The Journal of Student Affairs Inquiry 9
critical approaches question whether a specific tool measures the
intended topic across diverse groups. Matsuda et al. (1993)
recommend:
recognizing the experiential knowledge of people of color. Such recognition
is filtered through counterstorytelling, narrative, biographies, and life
histories. When the experiences and knowledges of people of color are
shared, the process allows for a more authentic and unique understanding
how they experience racist, oppressive structures. (p. 197)
Intentionally asking questions that resonate across groups, and
not just for the majority population, in ways that empower diverse
groups to respond with their truth provides richer,
contextualized, and valid data for practitioners. The medium and
method with which practitioners can collected data also provide
opportunities for sharing.
Data Analysis and ReportingData Analysis and Reporting
The transformation of data from a raw mass of material to easily
digestible information is a core component of assessment practice.
Making data easily understandable is fundamental for the usability
of the data in order to facilitate data-driven discussion and
decisions that influence students, staff, and other stakeholders.
Making meaning of the data through analysis and reporting makes
data actionable and closes the assessment loop. Employing critical
approaches to data analysis and reporting, assessment
practitioners begin to ask: how do one’s identities or lived
experiences influence data analysis? Do institutional values and
norms influence data processing? Who are the findings serving?
Critical theory can be used to strengthen core assessment practices
and advance equity efforts by centering the lived experiences of
populations typically left at the margins by examining how
meaning is assigned to data and employing collaborative
approaches to analysis and reporting.
A common practice for the analysis of quantitative data is
reporting the average or mean of the data. Generally, it is thought
that by aggregating individual measures, evaluators can find group
trends that guide decision-making about curriculum, policy,
services, and programs. The common practice of reporting
aggregated data has positive merits including the identification of
Examining Critical Theory as a Framework to Advance Equity Through Student Affairs Assessment
The Journal of Student Affairs Inquiry 10
patterns throughout variables and across time. A critical approach
must interrogate what knowledge such objective, apolitical,
neutral approaches to assessment might obscure, and consider
alternative methods of inquiry that attempt to address and remedy
systemic inequities. Aggregation may not tell the complete story.
While an aggregate story may be positive, differing narratives
from specific populations may emerge. Disaggregating data serves
as a tool for advancing equity because analyzing data by different
populations allows practitioners to identify if programs and
services are equitably meeting the needs of all students across
the institution. Data disaggregation can complement aggregated
findings by allowing professionals to identify if programs are
meeting established outcomes for all students
Disaggregating the data can center the lived experiences of
historically marginalized populations and creates space for such
voices to be heard. For example, when breaking down a large
data set by different subgroups, a practitioner may find that eight
students who self-identified as transgender responded to a survey
lower than the rest of the responding population. Regardless of
statistical significance, critical practitioners acknowledge that
significance and importance are not synonymous; they would
encourage dialogue around the data by sharing it with
stakeholders supporting this population.
Critical inquiry is grounded in lived experiences with power
relations and social justice as concerns, thus reciprocity between
the practitioner and participants is key. Therefore, DeLuca
Fernández (2015) advises practitioners to discuss how they attach
meaning to data. Practitioners’ intersecting identities frame their
worldview, perceptions, and how they make meaning from data.
Using uncritical, neutral, or objective approaches could lead
practitioners to reject underrepresented voices, over-privilege
existing ways of knowing, and reproduce systemic inequality
(DeLuca Fernández, 2015, p. 12). As previously discussed, one
way to balance practitioner positionality and power relations is to
invite additional perspectives to analyze and report on the data.
This approach can help moderate biases held by the practitioner,
while also working to address power dynamics or inequities of
programs or services identified in data collected. Critical
approaches to data analysis and reporting can serve as an
Examining Critical Theory as a Framework to Advance Equity Through Student Affairs Assessment
The Journal of Student Affairs Inquiry 11
important step toward redressing inequity by utilizing a
collaborative approach to discussions and interpretations of data,
When only looking at end-results or metrics according to external
entities, the importance and meaning can be lost on the student
experience. This includes factors influencing success and elements
contributing to a safe, encouraging, and inclusive learning
environment. Consequently, there becomes less incentive to
examine the interaction of identity or diversity elements with
institutional interventions. This can have dangerous
consequences such that “the implications of methodological
conservatism for individuals and communities who regularly
encounter individual, institutional, and/or societal oppression
include the preservation of discriminatory educational practices,
policies, and environments and perpetuation of the inequitable
status quo” (Pasque et al., 2012 p. ix). It becomes increasingly
important to treat data sets as part of an inclusive batch of
information rather than in a silo or vacuum. For example, while
student success rates point to particular courses as critical to
success, early interventions for students struggling in those
courses may need to be tailored to their identities or
circumstances. One student on scholarship may struggle in
chemistry because they are not studying enough or taking
advantage of tutoring resources available to them. Another
student may struggle in chemistry because their job, which
provides income necessary for them to make tuition payments,
prohibits them from making their lab section every other week.
Intervention for this latter student needs to be different from the
former, as there are additional circumstances for consideration in
which to offer guidance beyond coaching time management or
study skills.
Implications for Inquiry and Equity
Pasque et al. (2012) asserts that, “equity concerns are foundational
to students’ lives: marginalized identities, opportunity to learn,
access, persistence, attainment, pedagogy, and the social
stratification produced by participation in higher education” (p. 7).
The application of critical theory positions assessment practices
to expose inequalities. An integral component of assessment work
is sharing and using data to make decisions for improvement.
Examining Critical Theory as a Framework to Advance Equity Through Student Affairs Assessment
The Journal of Student Affairs Inquiry 12
Because issues of equity are central to students’ experiences in
higher education, and critical approaches to assessment facilitate
the exposure of inequities in programs and services offered,
practitioners taking a critical approach to assessment will be better
able to serve all students.
To that end, it is important to examine assessment practices,
processes, and resources for opportunities to integrate critical
approaches. As plans are put together, is consideration given to
involving appropriate stakeholders? When designing an
instrument, are demographics and identity-related components
stressed for inclusion? Could report templates have built-in
sections or prompts encouraging reflection of overall data? Are
findings disaggregated with respect to particular populations or
identities? Such questions should be posed by assessment
professionals, integrated in areas involved in assessment work,
and focused on populations that are often the subject of inquiry.
To best inform focus and approach, institutional needs should
be considered. As critical inquiry examines identity and
equity-related topics, professionals need to be knowledgeable
about the populations of students served and existing institutional
equity issues. Examining pain points, areas to improve, and
strengths surrounding these topics could give purposeful direction
when integrating new approaches for programs and services.
Knowing institutional priorities and trends could provide a
baseline or framework with which to direct initial efforts. This
may mean priorities themselves are challenged to evolve and serve
equity aims.
Finally, integrated reflection of practice and efficacy will be
crucial. Examining over time if professionals are truly taking a
critical approach or exemplifying needed inquiry. If not,
additional education or professional development may be needed.
Questions to help facilitate this reflection may include: Has
assessment effectiveness been impacted positively or negatively
after integrating critical inquiry? Is critical assessment yielding
actionable and meaningful data in relation to inquiry and equity
needs at the institution? Like any other assessment approach,
where problems, barriers, or opportunities for improvement
exist, , iterate for improvement.
Examining Critical Theory as a Framework to Advance Equity Through Student Affairs Assessment
The Journal of Student Affairs Inquiry 13
Conclusion
Traditional approaches to assessment characterized by
impartiality, validity, and objectivity may provide useful data in
the age of reporting and accountability based on economic
measures of success. However, such objective approaches to
assessment may obscure critical questions, methods, and data
interpretations that would enable us to uncover and respond to
systemic inequities that render differential outcomes in learning
or experience for students. Therefore, evaluators should ground
their assessment in critical theory, in order for assessment to
advance equity in programs and services at institutions of higher
education. Critical approaches can be applied and positively
influence every facet of assessment work. Grounding assessment
approaches in critical theory enables practitioners to examine
further learning and development experiences of all students and
collect evidence through a wider array of methods meaningful for
triangulation.
About the authors:
Ciji A. Heiser is the Assistant Director for Assessment and
Strategic Initiatives at the University of North Carolina at Chapel
Hill. Krista Prince is the Coordinator for Leadership Development
in the Department of Housing and Residential Education at The
University of North Carolina at Chapel Hill. She is also a doctoral
student in Educational Studies/Cultural Foundations
concurrently pursuing a certificate in Women’s and Gender
Studies at UNC-Greensboro. Joe Levy is the Director of
Assessment at National Louis University, with responsibility to
guide university level assessment, support academic program
assessment, and coordinate student affairs assessment. Joe is
passionate about data-informed decision making, accountability,
and promoting a student-centered approach inside and outside of
the classroom. Joe earned his MS in Student Affairs in Higher
Education from Colorado State University and his BA in English
from Baldwin-Wallace College.
References
ACPA-College Student Educators International. (2007). ASK
Examining Critical Theory as a Framework to Advance Equity Through Student Affairs Assessment
The Journal of Student Affairs Inquiry 14
standards: Assessment skills and knowledge content standards for
student affairs practitioners and scholars. Washington, D. C.:
American College Personnel Association (ACPA).
Association for Institutional Research (AIR). (2013, May 2). Code
of ethics and professional practice (CODE). Retrieved from
Association for Institutional Research (AIR) website: https://ww
w.airweb.org/Membership/Pages/CodeOfEthics.aspx
Banta, T., & Palomba, C. (2015). *Assessment essentials: Planning,
implementing, and *improving assessment in higher education
(2nd ed.). San Francisco: Jossey-Bass Publishers.
Bensimon, E. M. (2005). Closing the achievement gap in higher
education: An organizational learning perspective. New
Directions for Higher Education, 2005: 99–111.
Bettez, B.C. (2015). Navigating the complexity of qualitative
research in postmodern contexts: assemblage, critical reflexivity,
and communion as guides. International Journal of Qualitative
Studies in Education, 28(8), 932-954.
Chávez, A. F., & Sanlo, R. (Eds.). (2013). Identity and leadership:
Informing our lives,informing our practice. Washington, DC:
NASPA Publications.
Commission on Institutions of Higher Education (CIHE). (2016).
Standards for accreditation: Commission on institutions of higher
education. Burlington, MA: New England Association of Schools
and Colleges.
Culp, M.M., & Dungy, G. (Eds). (2012). Building a culture of
evidence in student affairs: A guide for leaders and practitioners.
Washington, DC: NASPA Publications.
DeLuca Fernández, S. (2015). Critical Assessment [PDF of
PowerPoint slides]. Retrieved from http://studentaffairsassessme
nt.org/structured-conversations
Hanks, C. (2011). The double-edge of reason: Jürgen Habermas
and the Frankfurt School. In B.A.U.Levinson (Ed.), Beyond
critique: Exploring critical social theories and education (pp.
Examining Critical Theory as a Framework to Advance Equity Through Student Affairs Assessment
The Journal of Student Affairs Inquiry 15
https://www.airweb.org/Membership/Pages/CodeOfEthics.aspx
https://www.airweb.org/Membership/Pages/CodeOfEthics.aspx
http://studentaffairsassessment.org/structured-conversations
http://studentaffairsassessment.org/structured-conversations
80-112). Boulder, CO: Paradigm Publishers.
Higher Learning Commission (2014). The Criteria for
accreditation and core components. Retrieved from: https://w
ww.hlcommission.org/Criteria-Eligibility-and-Candidacy/criteri
a-and-core-components.html
Kuh, G. D., Jankowski, N., Ikenberry, S. O., & Kinzie, J. (2014).
Knowing what students know and can do: The current state of
student learning outcomes assessment in US colleges and
universities. Urbana, IL: University of Illinois and Indiana
University, National Institute for Learning Outcomes Assessment
(NILOA).
Levinson, B.A.U. (2011). Introduction: Exploring critical social
theories in education. In B.A.U.
Levinson. (Ed.), Beyond critique: Exploring critical social theories
and education (pp. 1-24). Boulder, CO: Paradigm Publishers.
Levinson, B.A.U., Gross, J.P. K., Link, J., & Hanks, C. (2011).
Forerunners and foundation builders: Origins of western critical
social theory tradition. Beyond critique: Exploring critical social
theories and education (pp. 25-50). Boulder, CO: Paradigm
Publishers.
Maki, P. L. (2010). Assessing for learning: Building a sustainable
commitment across the institution (2nd ed.). Stylus Publishing
LLC: Sterling, VA.
Martinez-Alemán, A. M., Pusser, B., & Bensimon, E. M. (Eds.).
(2015). Critical approaches to the study of higher education: A
practical introduction. Baltimore, MD: Johns Hopkins University
Press.
Matsuda, M. J., Lawrence, C., Delgado, R., & Crenshaw, K. (Eds.).
(1993). Words that wound: Critical race theory, assaultive speech,
and the First Amendment. Boulder, CO: Westview Press.
McArthur, J. (2016). Assessment for social justice: The role of
assessment in achieving social justice. Assessment and Evaluation
in Higher Education, 41, 7, 967-981.
Examining Critical Theory as a Framework to Advance Equity Through Student Affairs Assessment
The Journal of Student Affairs Inquiry 16
https://www.hlcommission.org/Criteria-Eligibility-and-Candidacy/criteria-and-core-components.html
https://www.hlcommission.org/Criteria-Eligibility-and-Candidacy/criteria-and-core-components.html
https://www.hlcommission.org/Criteria-Eligibility-and-Candidacy/criteria-and-core-components.html
Middle States Commission on Higher Education. (2015).
Standards for accreditation and requirements of affiliation (13th
ed.). Philadelphia, PA: Middle States Commission on Higher
Education.
Nosek, B.A., & Riskind, R.G. (2012). Policy implications of
implicit social cognition. Social Issues and Policy Review, 6,
113-147.
Pasque, P., Carducci, R., Kuntz, A., & Gildersleeve, R. (2012).
Qualitative inquiry for equity in higher education: Methodological
innovations, implications, and interventions. ASHE Higher
Education Report, 37.
Steinberg, S., R. and Cannella, G. (Eds.). (2012). Critical
qualitative research. NY: Peter Lang Publishers.
Wall, A. F., Hursh, D., & Rodgers, J. W. (2014). Assessment for
whom: Repositioning higher education assessment as an ethical
and value-focused social practice. Research and Practice in
Assessment, 9, 5-17.
WASC Senior College and University Commission. (2013).
Standard 4: Creating an organization committed to quality,
assurance, institutional learning, and improvement. Retrieved
from: https://www.wascsenior.org/resources/handbook-accredit
ation-2013/part-ii-core-commitments-and-standards-accreditati
on/wasc-standards-accreditation-2013/standard-4-creating-orga
nization-committed-quality-assurance-institutional-learning-and
Examining Critical Theory as a Framework to Advance Equity Through Student Affairs Assessment
The Journal of Student Affairs Inquiry 17
https://www.wascsenior.org/resources/handbook-accreditation-2013/part-ii-core-commitments-and-standards-accreditation/wasc-standards-accreditation-2013/standard-4-creating-organization-committed-quality-assurance-institutional-learning-and
https://www.wascsenior.org/resources/handbook-accreditation-2013/part-ii-core-commitments-and-standards-accreditation/wasc-standards-accreditation-2013/standard-4-creating-organization-committed-quality-assurance-institutional-learning-and
https://www.wascsenior.org/resources/handbook-accreditation-2013/part-ii-core-commitments-and-standards-accreditation/wasc-standards-accreditation-2013/standard-4-creating-organization-committed-quality-assurance-institutional-learning-and
https://www.wascsenior.org/resources/handbook-accreditation-2013/part-ii-core-commitments-and-standards-accreditation/wasc-standards-accreditation-2013/standard-4-creating-organization-committed-quality-assurance-institutional-learning-and
- Abstract
Critical Social Theory and Assessment
Practitioner Positionality and Subjectivity
Agency of the Participants
Methodological Diversity
Data Analysis and Reporting
Implications for Inquiry and Equity
Conclusion
About the authors:
References
www.learningoutcomesassessment.org
Erick Montenegro
& Natasha A. Jankowski
A New Decade for Assessment:
Embedding Equity into
Assessment Praxis
N0 42
January 2020
National Institute for Learning Outcomes Assessment | 2
Contents
Abstract . . . . .3
A New Decade for Assessment: Embedding Equity into Assessment Praxis . . . . .4
What Equity-Minded Assessment Is and Is Not . . . . .6
Culturally Responsive Assessment . . . . .6
Socially Just Assessment . . . . .7
Critical Assessment . . . . .9
Bringing it All Together: Equity-Minded Assessment . . . . .9
1. Meaningful Student Involvement . . . . .10
2. Data Disaggregation, Exploration, and Action . . . . .11
3. Context-Specific Approaches and Responses . . . . .12
4. Embedded in All Things Assessment . . . . .13
The Barriers and Challenges to Equity in Assessment . . . . .14
Looking to the Decade Ahead . . . . .16
Professional Development: The Key to Unlocking the Potential of Equity-Minded
Assessment . . . . .16
Focusing on Equity and Assessment . . . . .18
Final Thoughts . . . . .18
References . . . . .20
About the Authors . . . . .24
About NILOA . . . . .25
NILOA Mission
The National Institute for
Learning Outcomes Assessment
(NILOA), established in 2008,
is a research and resource-
development organization
dedicated to documenting,
advocating, and facilitating
the systematic use of learning
outcomes assessment to
improve student learning.
Please Cite As:
Montenegro, E., & Jankowski, N. A. (2020, January). A new decade for assessment:
Embedding equity into assessment praxis (Occasional Paper No. 42). Urbana, IL: University
of Illinois and Indiana University, National Institute for Learning Outcomes Assessment
(NILOA).
National Institute for Learning Outcomes Assessment | 3
Abstract
Entering into a new decade with an even more diversified college student population will not only require
more assessment models involving students but also deeper professional development of institutional
representatives key to student learning. Reflecting upon the conversations over the last three years around
culturally responsive assessment and related equity and assessment discussions, this occasional paper
highlights questions, insights, and future directions for the decade ahead by exploring what equitable
assessment is and is not; the challenges and barriers to equitable assessment work; where the decade ahead
may lead; and next steps in the conversation on equity and assessment.
National Institute for Learning Outcomes Assessment | 4
A New Decade for Assessment:
Embedding Equity into
Assessment Praxis
Erick Montenegro & Natasha A. Jankowski
In addition to being the dawn of a new decade, January 2020 marks the three-year
anniversary of the National Institute for Learning Outcomes Assessment’s (NILOA)
equity conversation. Working to bring the field of assessment in alignment with
practices that support the success of diverse learners, NILOA’s equity work focuses on
being collaborative. The conversation on equity in assessment began with the launch of
Occasional Paper 29 titled Equity and Assessment: Moving Towards Culturally Responsive
Assessment. The goal of the paper was to open a dialogue on the relationship between
equity and assessment by presenting the concept of culturally responsive assessment.
Equity and Assessment implored the field of assessment to examine assessment processes in
order to be responsive to both issues of equity and the needs of diverse learners, focusing
on embedding culturally responsive assessment into processes and practices. Using
the paper as a space for dialogue on the assumptions from which assessment operates,
Montenegro & Jankowski (2017) sought to encourage assessment work to be reflective of
the students served and to ultimately use assessment data to address learning, persistence,
and attainment gaps.
Since 2017, the message of Equity and Assessment has spread beyond the confines of
those initial pages. While our original focus was upon exploring culturally responsive
assessment, what we did not expect was the interest in intersections of assessment and
equity related work. Through inviting responses to the paper introducing the concept of
culturally responsive assessment, practitioner authors brought multiple related elements
into the conversation, widening the dialogue space to explore the relationships between
equity and assessment. These practice and thought leaders introduced various elements to
consider—challenges to overcome, promising practices to move the needle, and supports
needed in advancing equity-minded assessment work. As became clear through the
conversations, an assessment process that is not mindful of equity can risk becoming a
tool that promotes inequities, whether intentional or otherwise, leading to a broadening
of the conversation from culturally responsive assessment to how assessment could address
equity in education (Zamani-Gallaher, 2017).
In over 15 published responses to Equity and Assessment from May 2017 to November
2019, respondents in the field set forth common themes in their exploration of equitable
assessment, specifically culturally responsive assessment. For one, respondents noted
the need for models or frameworks to inform this work and raise additional awareness
(Henning & Lundquist, 2018a; Laird & BrckaLorenz, 2017; Tullier, 2018; McArthur
2017; Rudnick, 2019). Questions regarding where to start, how to scale, and who
to involve arose (Fisler, 2017), with people doing the work offering answers to these
questions, such as beginning by disaggregating data to identify areas of need (Wright,
2017; Williams, R., 2018), and aligning equitable practices to larger institutional goals to
promote sustainability, alongside identifying potential partners (Levy & Heiser, 2018).
Additionally, respondents noted considerations for culturally responsive assessment
throughout assessment processes including the learning outcomes phase (McArthur,
2017; Henning & Lundquist, 2018a; Levy & Heiser, 2018; Laird & BrckaLorenz, 2017;
An assessment process that
is not mindful of equity
can risk becoming a tool
that promotes inequities,
whether intentional or
not.
https://www.learningoutcomesassessment.org/wp-content/uploads/2019/02/OccasionalPaper29
https://www.learningoutcomesassessment.org/wp-content/uploads/2019/02/OccasionalPaper29
National Institute for Learning Outcomes Assessment | 5
Williams, E., 2018; Tullier, 2018); questions to ask during data collection to choose
appropriate sources of evidence and include student voices in the assessment process
(Wright, 2017; Williams & Perrone, 2018; Meyerhoff, 2019; Clark & Arimoto, 2018);
and data analyses to check biases, dive deeper into the data, and make meaningful
comparisons (Levy & Heiser, 2018; Williams & Perrone, 2018; Williams, R., 2018;
Roberts, R., 2019). However, NILOA has not been the only vehicle driving the equity
conversation forward.
Indeed, we have seen the field respond to the call for equitable and culturally responsive
assessment through various means. For one, we have seen an increased focus on equity
at many higher education assessment conferences either through conference themes on
equity, diversity, and inclusion such as the 2019 Association for Assessment of Learning
in Higher Education (AALHE) conference and the 2020 Higher Education Assessment
Conference sponsored by New England College, and/or specific presentation tracks
discussing equity in assessment (e.g., 2019 Assessment Institute, 2019 Assessment in
Higher Education Conference, and the pre-conference track at the 2020 Association
for Institutional Research (AIR) Forum to name a few). Coupled with increases in
publications on the topic, including special issues of academic journals devoted to equity
in assessment (e.g., New Directions for Institutional Research Spring 2018 issue, AALHE’s
Fall 2019 and Spring 2020 issues of Intersection). Additionally, the Council for the
Advancement of Standards in Higher Education (CAS) and Campus Labs launched their
Socially Just Assessment podcast, while the Student Affairs Assessment Leaders (SAAL)
and CAS launched the Assessment for Social Justice Project (ASJP) (Henning, 2018)
bringing together multiple organizations across higher education to explore social justice
and assessment. We have even been fortunate to learn that various consortia of institutions
across the country have used the concepts presented in Equity and Assessment to structure
their assessment plans, including a group of Tribal Colleges and Universities utilizing
elements of culturally responsive assessment to advance work within their unique mission
and contexts. Relatedly, our equity work helped launch the Historically Black Colleges
and Universities (HBCU) Collaboration for Excellence in Educational Quality Assurance
(HBCU-CEEQA). Since the Fall of 2017, HBCU-CEEQA has grown to include over
70 members from over 35 colleges and universities across the United States, focused on
assessing student learning and being accountable to stakeholders, while also remaining
true to the special mission of HBCUs and the students they serve (Orr, 2018). And in
partnership with CAS and Campus Labs, NILOA released a call for equity related case
studies to better explore culturally responsive assessment in practice, cases which will be
released throughout the year.
Through these important voices in the equity in assessment conversation we have learned
much and have heard the need for further direction regarding what exactly equity in
assessment is and is not, along with what it looks like for different stakeholders across all
levels of higher education. It is for these reasons that as opposed to a paper focused on
deepening and developing culturally responsive frameworks in assessment, we instead
provide an overview of the various conversations on equity and assessment that emerged
over the course of the past three years, with implications for future directions on where
the larger conversation on the relationship between equity and assessment might lead in
the decade ahead. We present what we gleaned from the insightful questions, comments,
and perspectives shared through conference presentations, webinars, authored guest
responses, and other literature to address: 1) what equitable assessment is and is not; 2)
National Institute for Learning Outcomes Assessment | 5
https://www.aalhe.org/
https://www.aalhe.org/
https://www.nec.edu/higher-education-assessment-conference/
https://www.nec.edu/higher-education-assessment-conference/
https://assessmentinstitute.iupui.edu/
https://www.airweb.org/forum/2020
https://www.airweb.org/forum/2020
https://onlinelibrary.wiley.com/toc/1536075x/2018/2018/177
https://www.aalhe.org/intersection
http://studentaffairsassessment.org/entries/blog/saal-furthering-critical-and-socially-just-assessment
http://studentaffairsassessment.org/entries/blog/saal-furthering-critical-and-socially-just-assessment
https://www.msm.edu/oeoa/ceeqa/index.php
National Institute for Learning Outcomes Assessment | 6
the challenges and barriers to equitable assessment work; 3) where the decade ahead may
lead; and 4) next steps for NILOA and the field of assessment as it relates to issues of
equity.
What Equity-Minded Assessment Is and Is Not
What has become evident as more and more assessment practitioners and thought leaders
engage with the equity conversation is the desire for a unifying definition of what is
meant by equitable assessment. For example, NILOA’s conceptualization of equity in
assessment, along with others (Singer-Freeman, Hobbs, & Robinson, 2019) revolves
around culturally responsive assessment. Furthermore, CAS and Campus Labs approach
the discussion from a socially just assessment perspective (Henning, 2018; Henning
& Lundquist, 2018b). Thought leaders have also conceptualized equitable assessment
through a critical perspective (Heiser, Prince, & Levy, 2017; Hanson 2019). Other efforts
include decolonized assessment (Eizadirad, 2019), bias free assessment (Gibbs & Stobart,
2009), assessment of learning outcomes relevant to indigenous peoples and their cultures
(Small & Willson, 2018), and assessment that ultimately aims to do no harm. But what
exactly does all of this mean? Questions have been posed aiming to uncover differences
and similarities among approaches, and practitioners have requested direct translations
of what exactly these terms mean for everyday practice. For the sake of cataloging the
conversation thus far, brief explorations of culturally responsive, socially just, and critical
assessment are presented below.
Culturally Responsive Assessment
NILOA began with cultural responsiveness for various epistemological and practical
reasons. First, cultural responsiveness is a concept that has been associated with evaluation
and assessment since the 1970’s (Stake, 1975) and gaining prominence in the 1990’s (Hood
& Hopson, 2008). However, the conversation emerging from the evaluation community
examined assessment and cultural responsiveness from the perspective of item validity in
test development. Nevertheless, it provides a space from which to build upon an existing
conversation within assessment on issues of cultural appropriateness. Additionally, the term
“culturally responsive” is practical in nature and connected with teaching and learning.
Stemming from Ladson-Billings’ (1995a; 1995b) seminal conceptualization of culturally
relevant pedagogy, cultural responsiveness calls for practices which respond to the needs
of the contexts in which we teach and learn; including the needs of the students we serve.
If assessment is an integral and connected part of the teaching and learning process,
then conversations on culturally responsive pedagogy are appropriately positioned to help
better explore culturally responsive assessment and is also well aligned with NILOA’s
prior work in transparency of assignments and assignment design (Hutchings, Jankowski,
& Baker, 2018).
However, the focus upon culture left readers curious. Whose culture are we being responsive
to: the department/institution or the students? How does this translate to the tools used
and the policies in place? Both are very important questions, but the answers depend on
context. The reality is that culturally responsive assessment is fully dependent on the context in
which you are assessing. It is a process that requires reflection and planning. What worked
at one institution, program, or classroom may not work the same at another. However,
as posited by Montenegro and Jankowski (2017), it is impossible to do without direct
Culturally responsive
assessment is fully
dependent on the context
in which you are assessing.
It is a process that requires
reflection and planning.
National Institute for Learning Outcomes Assessment | 7
involvement of students, noting that in its core, culturally responsive assessment must:
1. Be mindful of the student population(s) being served and involve
students in the process of assessing learning;
2. Use appropriate student-focused and cultural language in learning
outcomes statements to ensure students understand what is expected
of them;
3. Develop and/or use assessment tools and multiple sources of evidence
that are culturally responsive to current students; and
4. Intentional improvement of student learning through disaggregated
data-driven change that examines structures, demonstrations of
learning, and supports which may privilege some students’ learning
while marginalizing others.
These considerations guide and inform assessment work at each step of the culturally
responsive assessment process to ensure responsiveness to the needs of students and
implementation of meaningful improvements. Some of the elements of culturally
responsive assessment are expanded upon in socially just assessment.
Socially Just Assessment
Socially just assessment stems from the work of SAAL, CAS, Campus Labs, and other
partners in the Assessment for Social Justice Project. Socially just assessment includes the
elements mentioned in the previous section and refocuses them within a framework that
analyzes the interplay between culture, bias, power, and oppression in the assessment
process. Socially just assessment calls for the acknowledgement that assessment takes
place within various departmental and institutional cultures which impact the processes
we follow (Heiser, Henning, & Lundquist, 2018; Henning & Lundquist, 2018b). There
typically are norms, resource constraints, timelines, procedures in place which influence
assessment plans and how those plans are subsequently executed. In addition, personal
biases can influence the types of tools used, the sources of evidence to which more weight
is assigned, and the interpretations drawn from assessment data along with possible
solutions on how to go about improving student learning. Heiser et al. (2018) also note
that the paradigms used to approach assessment—whether conscious or subconscious—
work to affect decisions made and questions asked (e.g., asset-based versus deficit-based
perspectives toward different initiatives, student populations, sources of data).
Socially just assessment uses the concept of deconstructed assessment to not only
understand why our students are achieving, persisting, or stopping-out in the ways
they are, but to also understand the underpinning structures of why these things are
happening in the first place (Henning & Lundquist, 2018b). In order to do this, there
must be an understanding that learning and assessment operate under dynamics of power
and oppression (Henning & Lundquist, 2018a; Heiser et al., 2018). In other words,
assessment is not an apolitical process. We need to first understand how systems of power
and oppression influence how students experience college, engage with the learning
process, and build knowledge before we can understand how to better assess their learning.
This also helps draw appropriate interpretations and conclusions from the data. Power and
oppression can play into the assessment process when selecting whose voices to include
in assessment and the methods we use or processes we follow. Typically, assessment is
National Institute for Learning Outcomes Assessment | 7
National Institute for Learning Outcomes Assessment | 8
planned and carried out by faculty and administrators, and changes are implemented
according to what faculty and administrators assume to be most appropriate. Seldomly
are students involved to verify that the assumptions in play are real, appropriate, or meet
their needs.
The perspectives we include in assessment and decisions of who has a voice at the
table privileges certain ways of knowing while potentially oppressing those who are
not represented. For example, culturally responsive assessment stresses that the student
voice must be included in assessment because students have been typically treated as
the object of the assessment; a mere passive participant in a process that has important
ramifications for their success (Montenegro & Jankowski, 2017). As argued by Zerquera,
Reyes, Pender, and Abbady (2018), “Many of the assessment approaches employed today
are misaligned with social justice agendas, failing to adequately inform decisions about
how best to support marginalized student populations within higher education” (p. 17),
reinforcing the need to actively involve students throughout the process of assessment.
With this in mind, socially just assessment raises awareness on how assessment can be a
process inherent of structures of power and oppression (Henning & Lundquist, 2018b).
Socially just assessment reminds practitioners to be mindful of how the ways in which data
are analyzed can also privilege or oppress. Far too often, if a specific student population
has a small sample size in assessment data, they are removed from analyses (American
Indian College Fund, 2019). This inherently marginalizes specific populations because
they are silenced from analyses, even though much can be learned from the experiences
of the students behind the small ‘n’ (Montenegro & Jankowski, 2017; Heiser et al, 2018).
Furthermore, assessment needs to be more aware of the types of comparisons made
between learners (Levy & Heiser, 2018). Far too often, we compare the outcomes of
students of color to those of white students. White students are then normed as the
population to which others should strive. Or we examine theories of white male student
persistence, pipeline approaches to education and learning, and assume them to be
the theory that should guide our practice to help “the other students” be more like the
“successful students.” These comparisons, especially if not worded or contextualized
appropriately, can send the message that non-white students should strive to be like their
white peers without examining the unique experience of non-white students. The point
raised in social justice focused assessment is that “cultural and social differences influence
whether and how students perform academically and socially at their institution” (Dorimé-
Williams, 2018, p. 42) as well as that “flawed assessment and implementation processes
disadvantage students” while “inaccurate interpretation and reporting of results can lead
to policies with a discriminatory impact” (Dorimé-Williams, 2018, p. 51).
The end goal of socially just assessment is to advance social justice. In other words,
assessment should strive to serve as a mechanism that helps close opportunity, persistence,
and attainment gaps between different student populations. Socially just assessment
should challenge structures of privilege within institutions and society writ large to better
serve and support learners. The goal of assessment is to make data informed decisions
on how to improve teaching and learning, so the goals of social justice and assessment
are very similar. It takes a conscious, intentional approach to make it happen, alongside
potentially hard conversations. Indeed, socially just assessment echoes the same values
expressed in culturally responsive assessment but bounds them within an exploration of
Assessment should strive
to serve as a mechanism
that helps close
opportunity, persistence,
and attainment gaps
between different student
populations.
National Institute for Learning Outcomes Assessment | 9
how power and oppression impact assessment.
Critical Assessment
At its most fundamental level, equitable assessment requires approaching assessment
through a critical lens. Both culturally responsive assessment and socially just assessment
operate from a critical perspective. They strive to challenge the status quo; raise questions
of privilege, power, and oppression; and work to remedy injustices whether purposeful or
accidental. Heiser et al. (2017), as well as Hanson (2019) brought forth various elements
of critical inquiry which can be applied to the assessment process. In short, critical
assessment calls for:
1. Disregarding the objectivity myth and accepting that assessment is
inherently subjective and guided by the biases and experiences of those
conducting assessment;
2. Varying the types of evidence used to assess learning outcomes to not
privilege specific ways of knowing or preferred ways to demonstrate
knowledge;
3. Including the voices of students, especially those who belong to
minoritized populations or those whose voices can often be left unheard,
throughout the assessment process; and
4. Using assessment to advance the pursuit of equity across previously
identified institutional parameters that demonstrate disparate outcomes
across student populations.
What these four tenets outline is the important role that context plays in critical
assessment. The context of the institution/program, the person(s) conducting the
assessment, and the learners need to be understood and reflected upon in order to
properly create and execute an assessment plan that will yield appropriate and equitable
results. However, simply being attentive to issues of equity during the assessment process
is not enough. Results must then be used to improve equity imperatives for the student
populations experiencing inequitable outcomes, in part because the learning outcomes
that institutions list are learning outcomes to which all students strive. Thus, assessment
efforts must be consequential to issues of equity. Culturally responsive assessment and
socially just assessment centralize these tenets of critical assessment. There certainly are
nuances between them, but they each strive to meet the same end goal: increase equity
in assessment.
Bringing it All Together: Equity-minded Assessment
Nomenclature aside, each of the above perspectives on equity and assessment are rooted
in the same core notion of being mindful of equity and actively working to address
inequities. At its core, equitable assessment calls for those who lead and participate in
assessment activities to pay attention and be conscious of how assessment can either
feed into cycles that perpetuate inequities or can serve to bring more equity into higher
education. From this point on, this paper will use the term equity-minded assessment,
similar to how Bensimon (2006) speaks of equity-mindedness, as it encompasses a shared
perspective on equity across the various conversation spaces unfolding. Here, we draw out
principles and elements of culturally responsive assessment, socially just assessment, and
National Institute for Learning Outcomes Assessment | 9
National Institute for Learning Outcomes Assessment | 10
critical assessment that have emerged over the past three years of conversation.
1. Meaningful Student Involvement
Being mindful of equity brings a necessary and embedded focus to questioning the
assessment process to ensure we are not succumbing to biases or established norms
while simultaneously excluding important voices and perspectives. One of the easiest
means by which to check assumptions is to actively involve students in the process of
assessment. In the face of changing enrollments in higher education, continued and
widening opportunity, persistence, and attainment gaps for different student populations
(Condron, Tope, Stiedl, & Freeman, 2013), the “typical” or “traditional” way is simply not
working for specific groups of students. Equity-minded assessment is about challenging
what we think and exploring what others think, need, and are affected by which we may
not understand or experience. Just as authentic assessment requires the use of multiple
sources of evidence (Kuh et al., 2015), equity-minded assessment invites multiple voices
to the table to determine learning outcome statements (Tkatchov, 2019), inform the
appropriateness of assessment (Gipps & Stobart, 2009), and make sense of the results.
A good starting place is with statements of learning outcomes—both in ensuring they are
measurable and that faculty, staff, and students understand and can make sense of those
outcomes. If learning outcome statements serve as the point from which educational
experiences are designed, and the learning outcome statements themselves are not inclusive
or include biases, then the educational design will as well (Rodrigues & Raby, 2019).
Further, students can be involved in determining what could be changed to further their
learning. Institutions are positioned to make more impactful changes by engaging with
students about what would best support their learning as opposed to trial and error, or
implementing “what worked for me when I was a student.” Experiences and perspectives
come with biases which can be embedded into the assessment process. If we do not reflect
on these biases, and take action to challenge them, then we risk acting upon assumptions
which may not be appropriate for students—even with the best intentions.
Listening to the voices of those historically silenced is an essential element of equity-
minded assessment. Fully complementing this is ensuring that everything, from learning
outcomes to data collected and reports are 1) written in a way that can be understood
by students and other stakeholders; and 2) are disseminated and communicated through
channels which can be easily found by students and stakeholders. In other words, equitable
assessment is transparent.
A forthcoming equity case study of Capella University, notes that equity-minded
assessment requires transparency in the assessment process as well as in educational
design—students should know what is being assessed, how it is being assessed, and how
well they achieved the assessment’s expectations. If students are to be active participants
in assessment, then we need to ensure students are also informed of assessment results,
improvements or changes made due to assessment data, and what this means for students.
Capella believes, equitable assessment practice means that all learners have an equal
and unbiased opportunity to demonstrate their knowledge and achievements through
assessment processes that use differentiated methods, transparency, accountability, and
fairness from design to measurement to improvements and dissemination of results.
(NILOA’s Transparency Framework can offer additional insight on how to make
Listening to the voices of
those historically silenced
is an essential element of
equity-minded assessment.
National Institute for Learning Outcomes Assessment | 11
assessment a more transparent process by making information accessible and engaging.)
2. Data Disaggregation, Exploration, and Action
A second element of equity-minded assessment involves exploring assessment data to
uncover potential learning gaps between student populations to make data informed
changes in order to close those gaps. In other words, equity-minded assessment requires
meaningful data disaggregation and subsequent action. In reviewing data, students can be
an active part of the conversation such that their needs and lived realities are present, heard,
and acted upon. Consider the noise added to data on a particular learning outcome if
collected results are not a demonstration of students’ learning on that particular outcome
but are instead based on social capital related to navigating assessment tasks? Should
curricular changes be made based on data about structural inequities as opposed to
learning? Another forthcoming equity case study from the University of North Carolina
at Charlotte notes ways to identify false evidence of achievement gaps in data analysis. The
exploration revealed that differences in student grades were reflective of students’ ability to
navigate college instead of demonstration of competence or achievement of an outcome.
This led assessment professionals to turn their attention to making assignments more
equitable for assessing learning instead of the “hidden curriculum”; a similar sentiment
echoed by R. Roberts (2019) who urged assessment practitioners not to make decisions
from assessment data which only reflects students’ ability to navigate assessments (e.g.,
ask questions and have good test-taking skills) instead of their actual learning.
As learned from the National Association of System Heads (NASH) project on taking
high-impact practices to scale, meaningful data disaggregation is a good first step towards
examining equity issues but is not by itself a practice of equity-minded assessment.
Simply examining disaggregated data without examining if the assessment process is
equitable will lead to continued inequities. Meaningful disaggregation involves deeper
analyses by specific student characteristics, alongside the intersection between and among
them (Roberts, J., 2019). To enable a place where practitioners can dive deeper into the
data, we first need to gather assessment data at a level and in a manner which can be
meaningfully disaggregated. It is incredibly difficult to disaggregate data at an institution-
level if data do not exist. To address issues of disaggregation, institutions might work in a
data sharing consortium in order to explore disaggregation options, or instead undertake
focus groups with students by verbally exploring differences in experience.
The first step is evaluating the depth of assessment data on hand and what can realistically
be done with it. Before collecting more data, it is always wise to fully explore the data
already collected and determine if it should still be collected or if something else is needed.
This can be done in partnership with institutional research or institutional effectiveness
staff as well as assessment professionals (should you be at an institution with so many
positions!). Further, we should consider siloed data which may provide additional insight
if connected to data on student learning whether in the Student Information System,
Learning Management System, course related systems, or other data sets managed by
student affairs and related units. Then, we can begin to determine what analyses to run.
An assessment plan that can yield the data needed would include shared definitions,
variables, and student characteristic data, which takes time and conversations to
determine. We cannot expect to collect all of the data in one round of assessment, but
National Institute for Learning Outcomes Assessment | 11
National Institute for Learning Outcomes Assessment | 12
over time we can begin to collect data that allows us to explore the intersection of various
student characteristics. For example, a deeper dive into our data can show that low-
SES male Latinx students are disproportionately less likely to attain a specific learning
outcome compared to both their low-SES female Latinx peers and their high-SES Latinx
counterparts. Or we might find that low-SES full-time commuter students are lagging
behind both their full-time residential peers and their high-SES full-time commuter peers
in attainment of a particular learning outcome in certain programs based on curricular
paths. Here, both SES and the residential/commuter characteristics are of importance,
coupled with student voice as to why this might be the case.
A cautionary note: In equity-minded assessment, data related conversations should be
interrogated to ensure that data are not weaponized to facilitate self-fulfilling prophecies
where results are used as “proof ” that students are unprepared or disinterested; where
someone can take assessment data and say “see, I told you they can’t learn!” To support
equity-minded assessment discussions informed by data, professional development may
be needed prior to data discussions to interrogate biases and assumptions. An important
aspect of meaningful disaggregation is thus knowing which questions to ask and what
to do with the different findings, as well as who to have participate in the process. This
takes practice, patience, and thrives from collaborating with and learning from others
who have different experiences, skills, and perspectives than our own. For faculty and
assessment practitioners to become comfortable with such conversations, administrators
need to provide spaces with facilitated discussions on structural barriers, inequities, and
practices which can affect student outcomes. Gansemer-Topf, Wilson, and Kirk (2019)
offer various questions assessment practitioners can ask of the data to critically interrogate
data collection and analysis processes.
Second cautionary note: While small samples can inform assessment through trend
data collected over time, equity-minded assessment is responsive to student needs by
examining and helping individual students, when they need it (Maki, 2017a). We cannot
disregard data simply because it is from a small sample. Generalizability is important for
publications but not as much for equity-minded institutional improvement or decision-
making. If the capstone for a major/program has three students in it who will graduate
and all three students struggled with attaining the program learning outcomes, should
something be done, or is the “sample size” (in this case the population of graduates – not
a sample, but an often confused point in these conversations) too small to incite action,
when none of the graduating students met the learning of interest? If ten students out of
500 are not meeting an outcome, it may not be statistically significant, but it is significant
to those students’ and their families and has implications for retention and persistence.
Thus, it is worth exploring. Further, looking at the same student in multiple points
through their learning journey in relation to learning outcomes provides information
not simply on one data point at the end of a program, but many throughout, in order to
examine learning progression over time. Such an approach shifts the unit of analysis from
a student to instances of demonstrated learning, thus one ‘n’ may have many associated
data points.
3. Context-Specific Approaches and Responses
Third, equity-minded assessment requires that we address issues of equity within our
specific context. It is impractical and unlikely that assessment professionals working to
Simply examining
disaggregated data
without examining if
the assessment process
is equitable will lead to
continued inequities.
National Institute for Learning Outcomes Assessment | 13
advance equity and assessment within their institutions will be able to fix the entirety of
the educational system in the United States. In addition, there are many areas and issues
to explore within a specific institutional context on equity and assessment, and it is unwise
to tackle them all at once. Determining an issue of equitable assessment that can bring
faculty, staff, and students together in a space of productive discomfort will lead to more
equitable assessment in the future than charging ahead with full disruption of assessment
processes in the name of equity. Some possible ways to begin include exploratory
analyses—done to see if there are inequities within assessment data and inquire about
how they can be fixed—or purposeful analyses to see if a recent change intended to
close a learning outcomes gap among specific populations achieved its intended purpose.
Starting with changing student demographics as a point of exploration of learning success
or of questions that faculty have about students and their learning opens space to examine
learning while bringing an intentional equity lens to the discussion. Additionally, equity-
minded assessment efforts can align to ongoing department, program, or institutional
initiatives to help meet overarching student success and learning goals, thus helping
to inform a larger issue that the institution needs to address or is already addressing.
Whatever the case, assessment efforts must be mindful of inequities which matter in a
specific context and assessment professionals need to sensitively navigate institutional
initiative space, working to make the case and connect the dots between equity and
assessment for faculty, staff, and administrators (Jankowski & Slotnick, 2015).
4. Embedded in All Things Assessment
Finally, equity needs to be embedded within and throughout the entirety of any assessment
effort. In her Assessment Institute keynote remarks, Tia B. McNair (2019) said that those
doing equity work need to live equity work. In other words, doing equity work is not
something we can step in and out of. It is a mentality and approach that remains central
so that we do not lose sight of it, that others are able to follow by example, and we are
always being critical, reflective, and questioning processes, biases, assumptions, within
ourselves, others, and the processes followed. This equity-mindedness needs to actively
permeate the entire assessment process, and the practice of assessment professionals. To
do equity-minded assessment we need to:
1. check biases and ask reflective questions throughout the assessment
process to address assumptions and positions of privilege;
2. use multiple sources of evidence appropriate for the students being
assessed and assessment effort;
3. include student perspectives and take action based on perspectives;
4. increase transparency in assessment results and actions;
5. ensure collected data can be meaningfully disaggregated and
interrogated; and
6. make evidence-based changes that address issues of equity that are
context-specific.
Equity-minded assessment refers to ways we ensure assessment processes and practices are
appropriate for all students and that we ultimately do no harm in the process. While it
can be challenging to consider the vast differences and needs of our student populations
in our practices, our task as educational providers is to strive to help every student succeed.
National Institute for Learning Outcomes Assessment | 13
National Institute for Learning Outcomes Assessment | 14
What “ensuring that our assessments are appropriate for all students” means is that
equitable assessment should work to ensure that learning outcomes, and how we assess
those outcomes, are done in ways which do not privilege certain students over others;
that data-informed changes are not benefiting one student group over others; and that
assessment efforts are not conducted with only one dominant perspective or voice leading
the process. Once we are aware of inequity in learning or assessment, we should strive
to address it instead of ignoring it; or worse, blaming students. Equitable assessment
means that we interrogate changes for possible disparate impacts on different student
populations and their learning, that we examine the changes with data collected on various
characteristics of interest to examine if learning improved, and provide students with
multiple opportunities to advance in their learning before leaving our institutions (Maki,
2017a). This means that the learning gains we desire are not only for future students, but
the students who are in the active process of learning (Maki, 2017b).
The Barriers and Challenges to Equity in Assessment
Assessment with an embedded focus on equity is attainable, but there are barriers to
advancing the work. Some of those barriers deal with discomfort with engaging in
conversations about privilege, power, oppression, and marginalization. In her book White
Fragility: Why It’s so Hard for White People to Talk About Racism, DiAngelo (2018) explores
the various cognitive barriers some can encounter when equity and related concepts are
discussed. With the many lessons of this book in mind, R. Roberts (2019) notes that
assessment practitioners who avoid equity simply because it is uncomfortable or an
inconvenience makes assessment practitioners accomplices to practices which perpetuate
inequities; especially through sustaining “barriers to student success, retention, graduation,
and, most of all, learning” (p. 3). In her response to Equity and Assessment, R. Roberts
(2019) provides a rich personal experience that exemplifies this behavior:
I recently attended a training focused on equity and data-informed
improvements to instruction…The group excitedly followed along during the
first day of our training when we reviewed information about the efficacy of
the proposed improvements that we could all bring to our colleges. However,
when the presentation shifted to a review of national data about inequity, the
room exploded with anxiety. Suddenly, several white people had seemingly
random objections, others had comments they believed were crucial about
all the aspects of inequity outside of their control, or comments about how
“other faculty” on their campuses would never tolerate reflecting on campus-
wide or course-specific data on inequity. The presenter repeatedly had to
intervene to redirect the conversation… She reminded us of the challenging
but extremely important truth that there are some things as instructors and even
administrators that are absolutely within our control when it comes to improving
equity (emphasis added) (pp. 1-2).
The last sentence is invaluable. Equity work is not someone else’s responsibility. We each
play a role in equity-minded assessment. There are elements we control within our spheres
of work and influence which can alleviate the mechanisms through which inequities exist
and persist. However, responsibility is diverted with comments such as “this is not a
problem at my institution” or “I can see why this would matter for Minority-Serving
Institutions but not us.” Choosing not to see an issue does not mean it is not still there,
Equitable assessment
should work to ensure
that learning outcomes,
and how we assess those
outcomes, are done in
ways which do not privi-
lege certain students over
others.
National Institute for Learning Outcomes Assessment | 15
nor can only one type of institution approach assessment in an equity-minded perspective.
A very real barrier to this work, which most can relate to, is initiative fatigue (Kuh &
Hutchings, 2015). At many institutions, people are wearing multiple hats often being the
lifeline of various projects, students, and other stakeholders in addition to their daily job
responsibilities. For those who work in assessment offices, most are often understaffed,
without much authority to convene faculty or directly impact policy and practices across
the institution. Assessment professionals have to contend with accreditation requirements
and various external calls for accountability while addressing issues of ongoing need
internal to the institution. Being asked to be the “assessment person” supporting one more
institutional initiative can be a significant load to bear. Adding further responsibility to
be the sole driver of equitable assessment can further stretch thin assessment staff, thus,
finding partners in the work before implementation can be key to further success.
Institutional culture plays an important role too; specifically the culture around assessment.
For example, if the assessment culture is about compliance/reporting or positivist-based
scientific views of measurement, then equity might not be an important part of the
conversation aside from item validity or reporting required disaggregated data (Jankowski,
2017). However, if the culture of assessment is focused around improving learning and
teaching, then equity can be a fruitful approach. The point being that depending on
the culture of assessment, the conversation around equity-minded assessment can be
difficult; often requiring someone to lead and facilitate the discussion in language that
resonates with the institutional context. An example of such facilitated approaches are
Assignment Charrettes—intensive assignment design workshops that are led by faculty—
or the Transparency in Learning and Teaching in Higher Education (TILT Higher Ed)
project that helps faculty add transparency in their pedagogy to improve student learning
(Winkelmes, 2016). This way, our assessments and the use of assessment data can both
lead to more equitable learning outcomes.
Another challenge is determining how much to involve students in the assessment process,
coupled with the comfort-level of faculty, staff, and administrators regarding student
involvement in assessment. Assessment practitioners, in partnership with others within
the institution, will need to determine how much and in what ways to involve and recruit
students to be part of different stages of the assessment process. While students may
initially lack assessment literacy (Smith, Worsfold, Davies, Fisher, & McPhail, 2013), or
an understanding of assessment, student involvement in assessment as active participants
with agency over their learning has proven beneficial to student learning and the overall
student experience (Jankowski, Baker, Brown-Tess, & Montenegro, forthcoming; Singer-
Freeman & Bastone, 2019). While recruitment of students may be difficult—because
just like faculty and staff students wear multiple hats and are rather busy— students may
be more likely to participate if they understand the impact(s) and/or benefits of their
involvement. No one wants to waste their time or, worse, share their thoughts and see no
action taken in response; thus adding to feelings of being unheard or unseen.
As the roles of students in assessment are explored for a particular institution, measurement
related concerns towards active involvement of students in the design, administration,
and analysis of assessment related information may arise. Most of these concerns stem
from conflicts of interest and issues of objectivity and bias from involving the students
who are being assessed in the process of measuring their own learning. However, student
National Institute for Learning Outcomes Assessment | 15
https://tilthighered.com/
National Institute for Learning Outcomes Assessment | 16
involvement in assessment at an institution-level involves a variety of students in the
process (Maki, 2017b; Damiano, 2018), and involving students as part of the assessment
process within a course can position assessment as a formative opportunity for learning,
not just summative demonstrations (Hattie, 2009). Involving students in the process
of measuring learning supports learning outcomes of quantitative reasoning along with
written and oral communication as well as involves students in undergraduate research
(Truncale et al., 2018; Welsh, 2013). However, equity will not be attained through
placement of a token student on institutional assessment committees or connections
with student governance without a wider involvement of students from throughout the
institution.
Looking to the Decade Ahead
We are not tasked with changing the world with one assessment effort or verifying that
learning outcome statements are culturally appropriate with every individual student. We
are not expected to survey every single student about intended data-informed changes,
nor does equity-minded assessment call for every single student to participate in data
analysis. Instead, equity-minded assessment is ultimately about being responsive, aware,
and intentional in order to not perpetuate inequalities.
We are aware of the various challenges to this work. In part, there are data issues including a
lack of common definitions by which institutions gather and report student characteristics,
coupled with a lack of data by which to disaggregate—in ways that intersect with multiple
student identities— but also intersect with data on learning. In part, there is a design issue
for embedding equity in assessment as well as the means by which to be transparent to
students about the assessment and learning process. While some faculty and staff actively
engage in universal design for learning and structure curricular and co-curricular learning
experiences such that access and success are not student characteristic dependent, it is not
widespread enough to occur for every student, every time. And finally, in part we are not
measuring in ways that provide the data needed to address issues of equity. It is to address
these challenges in the coming decade that we turn.
Professional Development: The Key to Unlocking the Potential of Equity-Minded Assessment
Professional development can break down barriers to equity-minded assessment. It
can bridge perceived gaps between assessment and context (Levy & Heiser, 2018), and
between knowledge and knower. We cannot assess what students know without also
attempting to understand how culture, context, and the influence of both impact learning
and how we assess that learning (Fisler, 2017; Montenegro & Jankowski, 2017). Levy and
Heiser (2018) suggest that “Institutions may want to create a meta-assessment rubric or
checklist to help ensure assessment practice is following proper process as intended by the
institution in accordance with institutional goals and values” (p. 3). The goal being that by
maintaining a clear vision on good assessment practice, biases may be limited. However,
Tharp (2019) argues the need for individual professional development on issues of equity,
making the case that for issues of equity to be examined within an institution, individuals
must spend time working on understanding their own assumptions and biases first. Thus,
we need to both examine processes and practices and ensure there are checkpoints on
implementation, as well as provide support for individuals to explore issues of equity in
order to meaningfully implement processes and practices in equitable ways.
We need to both
examine processes and
practices and ensure
there are checkpoints on
implementation, as well
as provide support for
individuals to explore
issues of equity in order to
meaningfully implement
processes and practices in
equitable ways.
National Institute for Learning Outcomes Assessment | 17
The truth is that many of us are not comfortable with or well-versed in conversations about
equity. However, the best way to acquire a skill is not by avoiding it, but by immersing
ourselves in it. Having conversations about race/racism or power/oppression—even if
they are just reflective and introspective—can be uncomfortable. None of us want to find
that our practices may be marginalizing students or contributing to inequitable outcomes.
But that fear—or worse yet, complacency and comfort in current practices because they
work for “most” students—prohibits progress and perpetuates inequities and inequality.
Addressing equity is everyone’s job at the institution, and difficult conversations cannot
be passed to someone else. Equity work requires assessment professionals to be courageous
and continuously seek ways to develop skills in equitable assessment, but everyone in the
institution has a role to play.
Professional development is an important partner in equity work because it allows faculty,
staff, and administrators the space to improve their own learning and understanding
in order to improve the learning of students; all the while helping programs and
institutions better meet student success goals. Equity focused professional development
should be communicated as important and supported with targeted incentives to engage
all stakeholders, not just the willing. As R. Roberts (2019) explains, “To develop this
self-awareness…educators should participate in ongoing training in understanding and
supporting equity and how it relates to inquiries about culturally-responsive outcomes,
classrooms, and professional development” (p. 3). In truth, doing equity work is a
continuous process—much like assessment—because our student populations change
with enrollment trends, and gaps in learning change continuously. This means that
equity work, and especially our own continuous development in this area, is a life-long
process that requires “sustained engagement, humility, and education” (DiAngelo, 2018,
p. 9) regardless of race/ethnicity and background. Equity-minded assessment requires a
certain comfort with being uncomfortable; with having tough conversations, engaging in
reflective practices, and implementing a critical mindset throughout.
Relatedly, we could all benefit from professional development on how to involve student
voice to ensure biases are in check and equity is embedded throughout assessment
practice. Biased assessments fail students. They can be unfair to learners who are not fluent
in specific cultural norms and exclude the experiences of linguistically, culturally, and
socioeconomically diverse learners (Williams & Perrone, 2018). Professional development
helps practitioners engage in practices that maintain a conscious understanding of how
practices and decisions are influenced and, in turn, influence the assessment effort.
In the coming decade, research is needed on effective equity focused professional
development supports for various levels within an institution. What might be the
role of centers for teaching and learning in working with faculty in partnership with
assessment professionals on issues of equity and assessment in the classroom? Note:
see Levesque-Bristol et al., 2019 and Kinzie, Landy, Sorcinelli, & Hutchings, 2019 for
some ideas on how centers for teaching and learning could be involved. Where will
assessment professionals find support on issues of equity-minded measurement and
data collection? What is the role of institutional researchers in equitable assessment and
what professional development supports will they need? What professional development
might accreditation related positions need to address issues of equity? And where will
administrators find models for supporting equitable assessment through professional
development? It is our hope that the decade ahead provides answers and resources to the
National Institute for Learning Outcomes Assessment | 17
National Institute for Learning Outcomes Assessment | 18
wide-scale professional development needs to fully support equity-minded assessment
across an entire institution’s assessment process.
Focusing on Equity and Assessment
Since the launch of the equity and assessment conversation, many threads of related
discussion have emerged. In the coming decade, assessment professionals will
continue to unpack issues of equity and measurement; data definition, collection,
and disaggregation; decolonizing assessment and learning outcomes; and indigenous
approaches to assessment. Student affairs will lead in socially just assessment, and groups
such as HBCU-CEEQA will provide insight into the equity-minded practices that have
been unfolding for years within the confines of their member institutions but have not
been acknowledged or implemented more broadly at predominantly white institutions.
And while the use of multiple sources of evidence can be helpful to culturally responsive
assessment, it should not be taken as a simple solution, and instead an opportunity to
explore equity in design and measurement and how to offset different sources of bias, if
possible. It is our intention that over the course of this decade, we will work to elevate
the institutions who have been doing equitable assessment and have models and answers
to scholarly questions, but whom have been silenced or not asked to join the assessment
conversation thus far.
We expect a challenging of measures of institution-level assessment around areas such
as climate, which have been historically presented in ways that ensure “white” students
are comfortable and experiencing “enough” diversity, as opposed to understanding
diverse student experiences or what an equitable climate entails (Phillips & Jones,
2019). And if our practice is guided by theory and our theories are inequitable, we
have a responsibility in the coming decade to develop theories that address as well as
interrogate the norms around student behavior, engagement, and what a “good” student
does to demonstrate their learning in ways that address diverse student populations and
their experiences. Discussions on how to embark on equitable comparison groups along
with related supports and possible changes to see success are all rife for unpacking in
the decade ahead—if as a discipline of assessment we focus on equity and assessment,
at all levels of assessment from classroom to program to college to general education
and institution. We will continue to see examples of ways in which students can be
involved in assessment, ranging from curating their own collections of evidence related
to learning outcomes, participating in transparent assessment design, or simply helping
to rewrite learning outcome statements in student-focused language. It is not enough
to tell our students about the intentional design through transparency approaches if
the design itself remains flawed and inequitable. Assessment is an ongoing process of
improvement helping to continuously refine teaching and learning, as well as assessment
processes and practices, and a focus on equity can help us attain this goal.
Final Thoughts
The equity in assessment conversation is far from over and it will become an increasingly
important practice for higher education as our student populations continue to diversify.
NILOA will continue to embed equity-minded assessment as a central thread in our
efforts. This is especially true as we continue our partnerships with others working in
the equity space. We pledge to continue the dialogue on equity-minded assessment and
We have a responsibility
in the coming decade
to develop theories
that address as well as
interrogate the norms
around student behavior,
engagement, and what a
“good” student does to
demonstrate their learning
in ways that address diverse
student populations and
their experiences.
National Institute for Learning Outcomes Assessment | 19
to continue supporting and learning from the field so that we may advance this work
together. We must all reflect on how our privilege(s) and positionality within society, the
institution, and the classroom intersect with that of students and assessment processes
and practices. We look forward to continued reflection by assessment professionals on
the ways that current assessment efforts either centralize issues of equity or serve to
perpetuate them (Felder, 2017), showcasing examples of implementation, and pushing
the scholarly conversation forward towards wider understanding and action.
National Institute for Learning Outcomes Assessment | 20
References
American Indian College Fund (2019). Creating visibility and healthy learning environments for Native Americans in
higher education: Declaration of native purpose in higher education: An Indigenous higher education equity initiative.
Denver, CO: Author. Retrieved from https://collegefund.org/wp-content/uploads/2020/01/Creating-Visibility-
and-Healthy-Learning-Environments-for-Natives-in-Higher-Education_web
Bensimon, E. M. (2006). Learning equity-mindedness: Equality in educational outcomes. The Academic Workplace,
1(17), 2-21.
Clark, I., & Arimoto, M. (2018, April). An inclusive learning environment (Equity Response). Urbana, IL: University of
Illinois and Indiana University, National Institute for Learning Outcomes Assessment (NILOA). Retrieved from
https://www.learningoutcomesassessment.org/wp-content/uploads/2019/08/EquityResponse-ClarkArimoto
Condron, D. J., Tope, D., Steidl, C. R., & Freeman, K. J. (2013). Racial segregation and the Black/White achievement
gap, 1992 to 2009. The Sociological Quarterly, 54, 130-157.
Damiano, A. (2018, April). Bringing student voices to the table: Collaborating with our most important stakeholders.
Urbana, IL: University of Illinois and Indiana University, National Institute for Learning Outcomes Assessment
(NILOA).
DiAngelo, R. (2018). White fragility: Why it’s so hard for White people to talk about racism. Boston, MA: Beacon Press.
Dorimé-Williams, M. D. (2018). Developing socially just practices and policies in assessment. New Directions for
Institutional Research, (177), 41-56. DOI:10.1002/ir.20255.
Eizadirad, A. (2019). Decolonizing educational assessment models. In A. Eizadriad (Ed.), Decolonizing educational
assessment: Ontario elementary students at the EQAO. Champ , Switzerland: Palgrave Macmillan.
Felder, P. P. (2017, May). On the importance of culturally responsive assessment (Equity Response). Urbana, IL: University
of Illinois and Indiana University, National Institute for Learning Outcomes Assessment (NILOA). Retrieved from
https://www.learningoutcomesassessment.org/wp-content/uploads/2019/08/EquityResponse-Felder
Fisler, J. (2017, December). Questions & thoughts to continue the conversation on culturally responsive assessment
(Equity Response). Urbana, IL: University of Illinois and Indiana University, National Institute for Learning
Outcomes Assessment (NILOA). Retrieved from https://www.learningoutcomesassessment.org/wp-content/
uploads/2019/08/EquityResponse-Fisler
Gansemer-Topf, A. M., Wilson, J., & Kirk, M. (2019). Numbers may not lie, but they can hide: Critically examining
our “number” narratives. Intersection, 2019(2), 5-10.
Gipps C., & Stobart, G. (2009). Fairness in assessment. In C. Wyatt-Smith & J. J. Cumming (Eds) Educational
assessment in the 21st century. Dordrecht, Netherlands: Springer.
Hanson, J. M. (2019). Feminist evaluation: A theoretical framework for culturally responsive higher education
assessment. Intersection, 2019(2), 28-32. Retrieved from https://www.aalhe.org/assets/docs/Intersection/AAHLE_
fall_2019_Intersection_final
Hattie, J. (2009). Visible learning: A synthesis of over 800 meta-analyses relating to achievement. New York, NY: Routledge.
Heiser, C. A., Henning, G., & Lundquist, A. (2018). Beyond culturally responsive assessment: Socially just assessment
(Webinar Presentation, May 30, 2018). Campus Intelligence. Retrieved from https://www.campusintelligence.
com/types/upcoming-webinars/beyond-culturally-responsive-assessment-social-just-assessment/
Heiser, C. A., Prince. K., & Levy, J. D. (2017). Examining critical theory as a framework to advance equity through
student affairs assessment. The Journal of Student Affairs Inquiry, 3(1), 1-17. Retrieved from https://jsai.
scholasticahq.com/article/1621-examining-critical-theory-as-a-framework-to-advance-equity-through-student-
affairs-assessment
Henning, G. W. (2018). The assessment for social justice project. Council for the Assessment of Standards in Higher
Education (CAS) Blog. Retrieved from https://www.cas.edu/blog_home.asp?Display=82
https://collegefund.org/wp-content/uploads/2020/01/Creating-Visibility-and-Healthy-Learning-Environments-for-Natives-in-Higher-Education_web
https://collegefund.org/wp-content/uploads/2020/01/Creating-Visibility-and-Healthy-Learning-Environments-for-Natives-in-Higher-Education_web
https://www.learningoutcomesassessment.org/wp-content/uploads/2019/08/EquityResponse-ClarkArimoto
https://www.learningoutcomesassessment.org/wp-content/uploads/2019/08/EquityResponse-Felder
https://www.learningoutcomesassessment.org/wp-content/uploads/2019/08/EquityResponse-Fisler
https://www.learningoutcomesassessment.org/wp-content/uploads/2019/08/EquityResponse-Fisler
https://www.aalhe.org/assets/docs/Intersection/AAHLE_fall_2019_Intersection_final
https://www.aalhe.org/assets/docs/Intersection/AAHLE_fall_2019_Intersection_final
https://www.campusintelligence.com/types/upcoming-webinars/beyond-culturally-responsive-assessment-social-just-assessment/
https://www.campusintelligence.com/types/upcoming-webinars/beyond-culturally-responsive-assessment-social-just-assessment/
https://jsai.scholasticahq.com/article/1621-examining-critical-theory-as-a-framework-to-advance-equity-through-student-affairs-assessment
https://jsai.scholasticahq.com/article/1621-examining-critical-theory-as-a-framework-to-advance-equity-through-student-affairs-assessment
https://jsai.scholasticahq.com/article/1621-examining-critical-theory-as-a-framework-to-advance-equity-through-student-affairs-assessment
https://www.cas.edu/blog_home.asp?Display=82
National Institute for Learning Outcomes Assessment | 21
Henning, G. W., & Lundquist, A. E. (2018a). Moving towards socially just assessment (Equity Response). Urbana, IL:
University of Illinois and Indiana University, National Institute for Learning Outcomes Assessment (NILOA).
Retrieved from https://www.learningoutcomesassessment.org/wp-content/uploads/2019/08/EquityResponse-
HenningLundquist
Henning, G. W., & Lundquist, A. E. (2018b). Deconstructed assessment: Using assessment to foster social justice
(Webinar Presentation, July 17, 2018). Campus Intelligence. Retrieved from https://www.campusintelligence.
com/types/upcoming-webinars/deconstructed-assessment-using-assessment-to-foster-social-justice/
Hood, S., & Hopson, R. K. (2008). Evaluation roots reconsidered: Asa Hillard, a fallen hero in the “Nobody Knows
My Name” project, and African educational excellence. Review of Education Research, 78(3), 410-426.
Hutchings, P., Jankowski, N. A., & Baker, G. R. (2018). Fertile ground: The movement to build more effective
assignments. Change: The Magazine of Higher Learning, 50(6), 13-19. Retrieved from https://www.tandfonline.
com/doi/abs/10.1080/00091383.2018.1540816
Jankowski, N. A. (2017). Moving toward a philosophy of assessment. Assessment Update, 29(3), 10-11.
Jankowski, N. A., Baker, G. R., Brown-Tess, K., & Montenegro, E. (in press). Student-focused learning and
assessment: Involving students in the learning process in higher education. New York, NY: Peter Lang Publishing.
Jankowski, N. A., & Slotnick, R. (2015). The five essential roles of assessment practitioners. Journal of Assessment
and Institutional Effectiveness, 5(1), 78-100.
Kinzie, J., Landy, K., Sorcinelli, M. D., & Hutchings, P. (2019). Better together: How faculty development and
assessment can join forces to improve student learning. Change: The Magazine of Higher Learning, 51(5),46-54.
Retrieved from https://naspa.tandfonline.com/doi/full/10.1080/00091383.2019.1652076#.Xi87-c5KiUk
Kuh, G. D., & Hutchings, P. (2015, March). Addressing assessment fatigue by keeping the focus on learning. Urbana,
IL: University of Illinois and Indiana University, National Institute for Learning Outcomes Assessment
(NILOA). Retrieved from https://www.learningoutcomesassessment.org/wp-content/uploads/2019/08/
Viewpoint-KuhHutchings
Kuh, G. D., Ikenberry, S. O., Jankowski, N. A., Cain, T. R., Ewell, P. T., Hutchings, P., & Kinzie, J. (2015). Using
evidence of student learning to improve higher education. San Francisco, CA: Jossey-Bass.
Ladson-Billings, G. (1995a). Toward a theory of culturally relevant pedagogy. American Educational Research Journal,
32(3), 465-491.
Ladson-Billings, G. (1995b). But that’s just good teaching! The case for culturally relevant pedagogy. Theory Into
Practice, 34(3), 159-165.
Laird, T. F. N., & BrckaLorenz, A. (2017, May). First and next steps: Moving towards culturally responsive assessment
(Equity Response). Urbana, IL: University of Illinois and Indiana University, National Institute for Learning
Outcomes Assessment (NILOA). Retrieved from https://www.learningoutcomesassessment.org/wp-content/
uploads/2019/08/EquityResponse-LairdBrckaLorenz
Levesque-Bristol, C., Flierl, M., Zywicki, C., Parker, L. C., Connor, C, Guberman, D., Nelson, D., Maybee, C.,
Bonem, E., FitzSimmons, J., & Lott, E. (2019, February). Creating student-centered learning environments and
changing teaching culture: Purdue University’s IMPACT program. Urbana, IL: University of Illinois and Indiana
University, National Institute for Learning Outcomes Assessment (NILOA). Retrieved from https://www.
learningoutcomesassessment.org/wp-content/uploads/2019/05/OccasionalPaper38
Levy, J., & Heiser, C. (2018, March). Inclusive assessment practice (Equity Response). Urbana, IL: University of
Illinois and Indiana University, National Institute for Learning Outcomes Assessment (NILOA). Retrieved
from https://www.learningoutcomesassessment.org/wp-content/uploads/2019/08/EquityResponse_
LevyHeiser
Maki, P. (2017a). Real-time student assessment: Meeting the imperative for improved time to degree, closing the
opportunity gap, and assuring student competencies for 21st-century needs. Sterling, VA: Stylus.
https://www.learningoutcomesassessment.org/wp-content/uploads/2019/08/EquityResponse-HenningLundquist
https://www.learningoutcomesassessment.org/wp-content/uploads/2019/08/EquityResponse-HenningLundquist
https://www.campusintelligence.com/types/upcoming-webinars/deconstructed-assessment-using-assessment-to-foster-social-justice/
https://www.campusintelligence.com/types/upcoming-webinars/deconstructed-assessment-using-assessment-to-foster-social-justice/
https://www.tandfonline.com/doi/abs/10.1080/00091383.2018.1540816
https://www.tandfonline.com/doi/abs/10.1080/00091383.2018.1540816
https://naspa.tandfonline.com/doi/full/10.1080/00091383.2019.1652076#.Xi87-c5KiUk
https://www.learningoutcomesassessment.org/wp-content/uploads/2019/08/Viewpoint-KuhHutchings
https://www.learningoutcomesassessment.org/wp-content/uploads/2019/08/Viewpoint-KuhHutchings
https://www.learningoutcomesassessment.org/wp-content/uploads/2019/08/EquityResponse-LairdBrckaLorenz
https://www.learningoutcomesassessment.org/wp-content/uploads/2019/08/EquityResponse-LairdBrckaLorenz
https://www.learningoutcomesassessment.org/wp-content/uploads/2019/05/OccasionalPaper38
https://www.learningoutcomesassessment.org/wp-content/uploads/2019/05/OccasionalPaper38
https://www.learningoutcomesassessment.org/wp-content/uploads/2019/08/EquityResponse_LevyHeiser
https://www.learningoutcomesassessment.org/wp-content/uploads/2019/08/EquityResponse_LevyHeiser
National Institute for Learning Outcomes Assessment | 22
Maki, P. (2017b, February). Real-time student assessment: Prioritizing enrolled students’ equitable progress toward
achieving a high-quality degree. Urbana, IL: University of Illinois and Indiana University, National Institute for
Learning Outcomes Assessment (NILOA). Retrieved from https://www.learningoutcomesassessment.org/wp-
content/uploads/2019/08/Viewpoint-Maki
McArthur, J. (2017, December). Opportunities for social justice within & through assessment (Equity Response).
Urbana, IL: University of Illinois and Indiana University, National Institute for Learning Outcomes
Assessment (NILOA). Retrieved from https://www.learningoutcomesassessment.org/wp-content/
uploads/2019/08/EquityResponse-McArthur
McNair, T. B. (2019, October). Why equity and assessment must be inextricably linked for student success. Keynote for
the Assessment Institute annual meeting, Indianapolis, IN.
Meyerhoff, L. (2019, March). Redefining putting students first (Equity Response). Urbana, IL: University of Illinois
and Indiana University, National Institute for Learning Outcomes Assessment (NILOA). Retrieved from
https://www.learningoutcomesassessment.org/wp-content/uploads/2019/08/EquityResponse-Meyerhoff
Montenegro, E., & Jankowski, N. A. (2017, January). Equity and assessment: Moving towards culturally
responsive assessment (Occasional Paper No. 29). Urbana, IL: University of Illinois and Indiana University,
National Institute for Learning Outcomes Assessment (NILOA). Retrieved from https://www.
learningoutcomesassessment.org/wp-content/uploads/2019/02/OccasionalPaper29
Orr, V. F. (2018, November). Assessment, accountability, and student learning outcomes at Historically Black Colleges
and Universities. Urbana, IL: University of Illinois and Indiana University, National Institute for Learning
Outcomes Assessment (NILOA). Retrieved from https://www.learningoutcomesassessment.org/wp-content/
uploads/2019/02/CEEQA_Report
Phillips, G. A., & Jones, S. M. (2019). Assessing diversity, equity, and inclusion in higher education. Research Paper
presented at the Association for the Study of Higher Education (ASHE) Conference, Portland, OR.
Roberts, J. (2019). An argument for employing an equity mindset in higher education assessment. Intersection,
2019(2), 11-15. Retrieved from https://www.aalhe.org/assets/docs/Intersection/AAHLE_fall_2019_
Intersection_final
Roberts, R. (2019, September). Making culturally-responsive sense of assessment data: Inquiry about equity (Equity
Response). Urbana, IL: University of Illinois and Indiana University, National Institute for Learning
Outcomes Assessment (NILOA). Retrieved from https://www.learningoutcomesassessment.org/wp-content/
uploads/2019/09/EquityResponse-Roberts
Rodrigues, A. A., & Raby, P. (2019). Lessons learned from indigenizing a media program at an Ontario
Community College. College Quarterly, 22(1). Retrieved from http://collegequarterly.ca/2019-vol22-num01-
winter/indigenizing-a-media-program.html
Rudnick, D. L. (2019, November). Culturally responsive assessment is just good assessment (Equity Response). Urbana,
IL: University of Illinois and Indiana University, National Institute for Learning Outcomes Assessment
(NILOA). Retrieved from https://www.learningoutcomesassessment.org/wp-content/uploads/2019/11/
EquityResponse_Rudnick
Singer-Freeman, K., & Bastone, L. (2019). Increasing equity in general education using self-relevant writing.
Intersection, 2019(2), 24-27. Retrieved from https://www.aalhe.org/assets/docs/Intersection/AAHLE_
fall_2019_Intersection_final
Singer-Freeman, K., Hobbs, H., & Robinson, C. (2019). Theoretical matrix of culturally relevant assessment.
Assessment Update, 31(4), 1-2, 15-16.
Small, S. B., & Wilson, E. (2018). Institutionalizing indigenous learning outcomes at Confederation College. In
F. Deller, J. Pichette, & E. K. Watkins (Eds.), Driving academic quality: Lessons from Ontario’s skills assessment
projects (pp. 65-80). Toronto, ON Canada: Higher Education Quality Council of Ontario.
https://www.learningoutcomesassessment.org/wp-content/uploads/2019/08/Viewpoint-Maki
https://www.learningoutcomesassessment.org/wp-content/uploads/2019/08/Viewpoint-Maki
https://www.learningoutcomesassessment.org/wp-content/uploads/2019/08/EquityResponse-McArthur
https://www.learningoutcomesassessment.org/wp-content/uploads/2019/08/EquityResponse-McArthur
https://www.learningoutcomesassessment.org/wp-content/uploads/2019/08/EquityResponse-Meyerhoff
https://www.learningoutcomesassessment.org/wp-content/uploads/2019/02/OccasionalPaper29
https://www.learningoutcomesassessment.org/wp-content/uploads/2019/02/OccasionalPaper29
https://www.learningoutcomesassessment.org/wp-content/uploads/2019/02/CEEQA_Report
https://www.learningoutcomesassessment.org/wp-content/uploads/2019/02/CEEQA_Report
https://www.aalhe.org/assets/docs/Intersection/AAHLE_fall_2019_Intersection_final
https://www.aalhe.org/assets/docs/Intersection/AAHLE_fall_2019_Intersection_final
https://www.learningoutcomesassessment.org/wp-content/uploads/2019/09/EquityResponse-Roberts
https://www.learningoutcomesassessment.org/wp-content/uploads/2019/09/EquityResponse-Roberts
http://collegequarterly.ca/2019-vol22-num01-winter/indigenizing-a-media-program.html
http://collegequarterly.ca/2019-vol22-num01-winter/indigenizing-a-media-program.html
https://www.learningoutcomesassessment.org/wp-content/uploads/2019/11/EquityResponse_Rudnick
https://www.learningoutcomesassessment.org/wp-content/uploads/2019/11/EquityResponse_Rudnick
https://www.aalhe.org/assets/docs/Intersection/AAHLE_fall_2019_Intersection_final
https://www.aalhe.org/assets/docs/Intersection/AAHLE_fall_2019_Intersection_final
National Institute for Learning Outcomes Assessment | 23
Smith, C. D., Worsfold, K., Davies, L., Fisher, R., & McPhail, R. (2013). Assessment literacy and student learning:
the case for explicitly developing students ‘assessment literacy’. Assessment & Evaluation in Higher Education,
38(1), 44-60.
Stake, R. E. (1975). Program evaluation, particularly responsive evaluation. Urbana, IL: Center for Instructional
Research and Curriculum Evaluation at the University of Illinois Urbana-Champaign.
Tharp, D. S. (2019). Cultural consciousness in data-driven efforts: Why it matters and how to do it. Intersection,
2019(2), 20-23. Retrieved from https://www.aalhe.org/assets/docs/Intersection/AAHLE_fall_2019_
Intersection_final
Tkatchov, M. (2019). Equity in “authentic” assessments: A closer look at defining and assessing learning outcomes
in competency- based education. Intersection, 2019(2), 16-19. Retrieved from https://www.aalhe.org/assets/
docs/Intersection/AAHLE_fall_2019_Intersection_final
Truncale, N. P., Chalk, E. D., Pellegrino, C., & Kemmerling, J. (2018, March). Implementing a student
assessment scholars program: Students engaging in continuous improvement. Urbana, IL: University of
Illinois and Indiana University, National Institute for Learning Outcomes Assessment (NILOA).
Retrieved from https://www.learningoutcomesassessment.org/wp-content/uploads/2019/04/AiP_
TruncaleChalkPellegrinoKemmerlingMarch2018
Tullier, S. (2018, October). Epistemological considerations for student affairs (Equity Response). Urbana, IL: University
of Illinois and Indiana University, National Institute for Learning Outcomes Assessment (NILOA). Retrieved
from https://www.learningoutcomesassessment.org/wp-content/uploads/2019/08/EquityResponse-Tullier
Welsh, J. (2013). Student involvement in assessment: A 3-way win. Urbana, IL: University of Illinois and Indiana
University, National Institute for Learning Outcomes Assessment (NILOA). Retrieved from https://www.
learningoutcomesassessment.org/wp-content/uploads/2019/08/Viewpoint-Welsh
Williams, E. (2018, February). First thing’s first: Privilege, power, and pedagogy the antecedent of assessment (Equity
Response). Urbana, IL: University of Illinois and Indiana University, National Institute for Learning
Outcomes Assessment (NILOA). Retrieved from https://www.learningoutcomesassessment.org/wp-content/
uploads/2019/08/EquityResponse-EWilliams
Williams, R. (2018, December). Culturally responsive assessment: What to tackle first (Equity Response). Urbana, IL:
University of Illinois and Indiana University, National Institute for Learning Outcomes Assessment (NILOA).
Retrieved from https://www.learningoutcomesassessment.org/wp-content/uploads/2019/08/EquityResponse-
RWilliams
Williams, S., & Perrone, F. (2018, April). Linking assessment practices to indigenous ways of knowing (Equity
Response). Urbana, IL: University of Illinois and Indiana University, National Institute for Learning
Outcomes Assessment (NILOA). Retrieved from https://www.learningoutcomesassessment.org/wp-content/
uploads/2019/08/EquityResponse-WilliamsPerrone
Winkelmes, M. (2016, March). Helping faculty use assessment data to provide more equitable learning experiences.
Urbana, IL: University of Illinois and Indiana University, National Institute for Learning Outcomes
Assessment (NILOA). Retrieved from https://www.learningoutcomesassessment.org/wp-content/
uploads/2019/08/Viewpoint-Winkelmes
Wright, M. (2017, May). Steps in the right direction: Additional strategies for fostering culturally responsive assessment
(Equity Response). Urbana, IL: University of Illinois and Indiana University, National Institute for Learning
Outcomes Assessment (NILOA). Retrieved from https://www.learningoutcomesassessment.org/wp-content/
uploads/2019/08/Equity-Response_Wright
Zamani-Gallaher, E. (2017, May ). What is equity? (Video Equity Response). Urbana, IL: University of Illinois and
Indiana University, National Institute for Learning Outcomes Assessment.
Zerquera, D., Reyes, K. A., Pender, J. T., & Abbady, R. (2018). Understanding practitioner‐driven assessment and
evaluation efforts for social justice. New Directions for Institutional Research, (177), 15-40.
https://www.aalhe.org/assets/docs/Intersection/AAHLE_fall_2019_Intersection_final
https://www.aalhe.org/assets/docs/Intersection/AAHLE_fall_2019_Intersection_final
https://www.aalhe.org/assets/docs/Intersection/AAHLE_fall_2019_Intersection_final
https://www.aalhe.org/assets/docs/Intersection/AAHLE_fall_2019_Intersection_final
https://www.learningoutcomesassessment.org/wp-content/uploads/2019/04/AiP_TruncaleChalkPellegrinoKemmerlingMarch2018
https://www.learningoutcomesassessment.org/wp-content/uploads/2019/04/AiP_TruncaleChalkPellegrinoKemmerlingMarch2018
https://www.learningoutcomesassessment.org/wp-content/uploads/2019/08/EquityResponse-Tullier
https://www.learningoutcomesassessment.org/wp-content/uploads/2019/08/Viewpoint-Welsh
https://www.learningoutcomesassessment.org/wp-content/uploads/2019/08/Viewpoint-Welsh
https://www.learningoutcomesassessment.org/wp-content/uploads/2019/08/EquityResponse-EWilliams
https://www.learningoutcomesassessment.org/wp-content/uploads/2019/08/EquityResponse-EWilliams
https://www.learningoutcomesassessment.org/wp-content/uploads/2019/08/EquityResponse-RWilliams
https://www.learningoutcomesassessment.org/wp-content/uploads/2019/08/EquityResponse-RWilliams
https://www.learningoutcomesassessment.org/wp-content/uploads/2019/08/EquityResponse-WilliamsPerrone
https://www.learningoutcomesassessment.org/wp-content/uploads/2019/08/EquityResponse-WilliamsPerrone
https://www.learningoutcomesassessment.org/wp-content/uploads/2019/08/Viewpoint-Winkelmes
https://www.learningoutcomesassessment.org/wp-content/uploads/2019/08/Viewpoint-Winkelmes
https://www.learningoutcomesassessment.org/wp-content/uploads/2019/08/Equity-Response_Wright
https://www.learningoutcomesassessment.org/wp-content/uploads/2019/08/Equity-Response_Wright
National Institute for Learning Outcomes Assessment | 24
About the Authors
Erick Montenegro is a Doctoral Candidate in Education Policy, Organization and Leadership at the University of Illinois
Urbana-Champaign. Erick also serves as the Communications Coordinator and Research Analyst for NILOA where
he is responsible for NILOA’s integrated communications effort including developing media, maintaining the website,
promoting activities that benefit NILOA and its partners, and providing access to resources for NILOA’s various audiences
and stakeholder groups. As a research analyst, Erick conducts timely assessment research and leads NILOA’s equity initiative.
Erick received a dual BS in marketing and business administration with a concentration in international business, and an
EdM in education policy, organization and leadership with a concentration in higher education both from the University of
Illinois. His research interests include issues of equity in assessment, culturally responsive assessment, outcomes assessment
practices at Minority-Serving Institutions, and issues affecting Latinx students in higher education.
Dr. Natasha Jankowski is NILOA’s Executive Director and a Research Associate Professor with the Department of
Education Policy, Organization and Leadership at the University of Illinois Urbana-Champaign. She is co-author, along
with her NILOA colleagues, of the books Using Evidence of Student Learning to Improve Higher Education, and, Degrees That
Matter: Moving Higher Education to a Learning Systems Paradigm. Her main research interests include all things assessment,
organizational evidence use, and evidence-based storytelling. She holds a PhD in higher education from the University of
Illinois, an MA in higher education administration from Kent State University, and a BA in philosophy.
Follow us on social media:
@NILOA_web
@LearningOutcomesAssessment
Sign up to receive our monthly NILOA
Newsletter and stay up to date with our
research and publications.
About NILOA
• The National Institute for Learning Outcomes Assessment (NILOA) was
established in December 2008.
• NILOA is co-located at the University of Illinois and Indiana
University.
• The NILOA website contains free assessment resources and can be found at
http://www.learningoutcomesassessment.org.
• The NILOA research team has scanned institutional websites, surveyed chief
academic officers, and commissioned a series of occasional papers.
• NILOA’s Founding Director, George Kuh, founded the National Survey for
Student Engagement (NSSE).
• The other co-principal investigator for NILOA, Stanley Ikenberry, was pres-
ident of the University of Illinois from 1979 to 1995 and of the American
Council of Education from 1996 to 2001.
NILOA Staff
Natasha Jankowski, Executive Director
Gianina Baker, Assistant Director
Katie Schultz, Project Manager
Erick Montenegro, Communications Coordinator and Research Analyst
Verna F. Orr, Post-Doctoral Researcher
NILOA Senior Scholars
Peter Ewell, Senior Scholar
Pat Hutchings, Senior Scholar
Jillian Kinzie, Senior Scholar
George Kuh, Founding Director and Senior Scholar
Paul Lingenfelter, Senior Scholar
David Marshall, Senior Scholar
Nan Travers, Senior Scholar
NILOA Sponsors
Lumina Foundation for Education
University of Illinois, College of Education
National Institute for Learning Outcomes Assessment | 25
https://twitter.com/niloa_web?lang=en
https://www.facebook.com/learningoutcomesassessment/
https://www.learningoutcomesassessment.org/joinemail/
https://www.learningoutcomesassessment.org/joinemail/
https://www.campusintelligence.com/types/upcoming-webinars/beyond-culturally-responsive-assessment-social-just-assessment/
https://jsai.scholasticahq.com/article/1621-examining-critical-theory-as-a-framework-to-advance-equity-through-student-affairs-assessment
https://www.learningoutcomesassessment.org/joinemail/
https://www.learningoutcomesassessment.org/wp-content/uploads/2019/05/OccasionalPaper38
For more information, please contact:
National Institute for Learning Outcomes Assessment (NILOA)
University of Illinois Urbana-Champaign
51 Gerty Drive
Suite 196, CRC, MC-672
Champaign, IL 61820
learningoutcomesassessment.org
niloa@education.illinois.edu
Phone: 217.244.2155
mailto:niloa@education.illinois.edu
NationalInstitute for Learning Outcomes Assessment | 1
k n o w l e d g e a c c o u n t a b i l i t y c o n n e c t i o n s e l f – r e f l e c t i o n e d u c a t e a c t i o n u n d e r s t a n d c o m m u n i c a t e l i s t e n l e a r n a c c e s s q u a l i t y i n n o v a t i o n s u c c e s s
i n g e n u i t y i n t e l l e c t c u r i o s i t y c h a l l e n g e c r e a t e a c h i e v e m e n t c o n n e c t i o n s e l f – r e f l e c t i o n e d u c a t e a c t i o n u n d e r s t a n d c o m m u n i c a t e l i s t e n l e a r n a c c e s s
q u a l i t y i n n o v a t i o n s u c c e s s i n g e n u i t y i n t e l l e c t c u r i o s i t y c h a l l e n g e k n o w l e d g e a c c o u n t a b i l i t y c o n n e c t i o n u n d e r s t a n d c o m m u n i c a t e l i s t e n l e a r n
a c c e s s q u a l i t y i n n o v a t i o n s u c c e s s i n g e n u i t y s e l f – r e f l e c t i o n e d u c a t e a c t i o n u n d e r s t a n d i n t e l l e c t k n o w l e d g e a c c o u n t a b i l i t y c o n n e c t i o n s e l f – r e f l e c t i o n
e d u c a t e a c t i o n u n d e r s t a n d c o m m u n i c a t e c u r i o s i t y c h a l l e n g e c r e a t e a c h i e v e m e n t c o n n e c t i o n s e l f – r e f l e c t i o n c u r i o s i t y c h a l l e n g e c r e a t e a c h i e v e m e n t
c o n n e c t i o n s e l f – r e f l e c t i o n k n o w l e d g e a c c o u n t a b i l i t y c o n n e c t i o n s e l f – r e f l e c t i o n e d u c a t e a c t i o n u n d e r s t a n d c o m m u n i c a t e l i s t e n l e a r n a c c e s s q u a l i t y
i n n o v a t i o n s u c c e s s i n g e n u i t y i n t e l l e c t c u r i o s i t y c h a l l e n g e e d u c a t e i n n o v a t i o n s u c c e s s i n g e n u i t y i n t e l l e c t c u r i o s i t y c h a l l e n g e c r e a t e a c h i e v e m e n t
k n o w l e d g e a c c o u n t a b i l i t y c o n n e c t i o n s e l f – r e f l e c t i o n e d u c a t e a c t i o n u n d e r s t a n d c o m m u n i c a t e c u r i o s i t y c h a l l e n g e c r e a t e a c h i e v e m e n t c o n n e c t i o n
s e l f – r e f l e c t i o n u n d e r s t a n d c o m m u n i c a t e l i s t e n l e a r n a c c e s s q u a l i t y a c t i o n e d u c a t e a c t i o n u n d e r s t a n d c o m m u n i c a t e l i s t e n l e a r n a c t i o n u n d e r s t a n d
c o m m u n i c a t e l i s t e n l e a r n a c c e s s q u a l i t y i n n o v a t i o n s u c c e s s i n g e n u i t y i n t e l l e c t c u r i o s i t y c h a l l e n g e k n o w l e d g e a c c o u n t a b i l i t y c o n n e c t i o n a c c e s s
q u a l i t y s e l f – r e f l e c t i o n c u r i o s i t y c h a l l e n g e c r e a t e a c h i e v e m e n t l e a r n a c c e s s q u a l i t y i n n o v a t i o n s u c c e s s i n g e n u i t y s e l f – r e f l e c t i o n e d u c a t e a c t i o n
u n d e r s t a n d i n t e l l e c t k n o w l e d g e a c c o u n t a b i l i t y c o n n e c t i o n s e l f – r e f l e c t i o n e d u c a t e a c t i o n u n d e r s t a n d k n o w l e d g e a c c o u n t a b i l i t y c o n n e c t i o n s e l f –
r e f l e c t i o n e d u c a t e a c t i o n u n d e r s t a n d c o m m u n i c a t e l i s t e n l e a r n a c c e s s q u a l i t y i n n o v a t i o n s u c c e s s i n g e n u i t y i n t e l l e c t c u r i o s i t y c h a l l e n g e c o n n e c t i o n
k n o w l e d g e a c c o u n t a b i l i t y c o n n e c t i o n s e l f – r e f l e c t i o n e d u c a t e a c t i o n u n d e r s t a n d c o m m u n i c a t e l i s t e n l e a r n a c c e s s q u a l i t y i n n o v a t i o n s u c c e s s i n g e n u i t y
c h a l l e n g e c r e a t e a c h i e v e m e n t c o n n e c t i o n s e l f – r e f l e c t i o n e d u c a t e a c t i o n u n d e r s t a n d c o n n e c t i o n s e l f – r e f l e c t i o n u n d e r s t a n d c o m m u n i c a t e l i s t e n l e a r n
a c c e s s q u a l i t y a c t i o n c r e a t e a c h i e v e m e n t c o n n e c t i o n s e l f – r e f l e c t i o n e d u c a t e a c t i o n u n d e r s t a n d c o m m u n i c a t e l i s t e n l e a r n a c c e s s q u a l i t y i n n o v a t i o n
s u c c e s s e d u c a t e a c t i o n c o m m u n i c a t e l i s t e n l e a r n a c c e s s q u a l i t y a c t i o n e d u c a t e a c t i o n u n d e r s t a n d c o m m u n i c a t e e d u c a t e i n n o v a t i o n s u c c e s s s e l f –
r e f l e c t i o n k n o w l e d g e a c c o u n t a b i l i t y c o m m u n i c a t e l i s t e n l e a r n a c h i e v e m e n t c o n n e c t i o n s e l f – r e f l e c t i o n e d u c a t e a c t i o n u n d e r s t a n d c o m m u n i c a t e l i s t e n
l e a r n a c c e s s q u a l i t y i n n o v a t i o n s u c c e s s i n g e n u i t y i n t e l l e c t a c c e s s q u a l i t y i n n o v a t i o n s u c c e s s s e l f – r e f l e c t i o n c u r i o s i t y c h a l l e n g e c r e a t e a c h i e v e m e n t
c o n n e c t i o n s e l f – r e f l e c t i o n u n d e r s t a n d e d u c a t e a c t i o n u n d e r s t a n d c o m m u n i c a t e l i s t e n l e a r n a c t i o n u n d e r s t a n d c o m m u n i c a t e l i s t e n l e a r n a c c e s s
q u a l i t y i n n o v a t i o n s u c c e s s i n g e n u i t y c u r i o s i t y c h a l l e n g e c r e a t e a c h i e v e m e n t c o n n e c t i o n s e l f – r e f l e c t i o n u n d e r s t a n d c o m m u n i c a t e l i s t e n l e a r n
Equity and Assessment:
Moving Towards
Culturally Responsive Assessment
Erick Montenegro and Natasha A. Jankowski
National Institute for Learning Outcomes Assessment
January 2017
O c c a s i o n a l P a p e r # 2 9
www.learningoutcomesassessment.org
National Institute for Learning Outcomes Assessment | 2
NILOA Mission
The National Institute for Learning
Outcomes Assessment’s (NILOA) primary
objective is to discover and disseminate
the ways that academic programs and
institutions can productively use
assessment data internally to inform and
strengthen undergraduate education, and
externally to communicate with policy
makers, families, and other stakeholders.
Abstract….3
Equity and Assessment:
Moving Towards Culturally Responsive Assessment…4
Limiting Learning Demonstration…6
Culturally Responsive Assessment…8
Student Learning Outcomes Statements…11
Assessment Approaches…12
Use of Assessment Results…13
Final Thoughts…14
References…17
NILOA National Advisory Panel…21
About NILOA…22
Table of Contents
Please cite as: Montenegro, E., & Jankowski, N. A. (2017, January). Equity and assessment: Moving towards culturally
responsive assessment. (Occasional Paper No. 29). Urbana, IL: University of Illinois and Indiana University, National
Institute for Learning Outcomes Assessment (NILOA).
Erick Montenegro is a doctoral candidate in the Education Policy, Organization and Leadership program at the
University of Illinois. He is responsible for NILOA’s integrated communications effort including developing media,
maintaining the website, promoting activities that benefit NILOA and its partners, and providing access to resources for
NILOA’s various audiences and stakeholder groups. Erick received a dual B.S. in Marketing and Business
Administration with a concentration in International Business, and an Ed.M. in Education Policy, Organization and
Leadership with a concentration in Higher Education both from the University of Illinois. His research interests
include issues of equity in assessment, culturally responsive assessment, outcomes assessment practices at Minority-
Serving Institutions, and issues affecting Latinx students in higher education.
Dr. Natasha Jankowski is Director and Research Assistant Professor with the Department of Education Policy,
Organization and Leadership at the University of Illinois Urbana-Champaign, has presented at numerous national
conferences and institutional events, and written various reports for NILOA. Her main research interests include
assessment and evaluation, organizational evidence use, and evidence-based storytelling. She holds a PhD in Higher
Education from the University of Illinois, an M.A. in Higher Education Administration from Kent State University,
and a B.A. in philosophy from Illinois State University. She previously worked for GEAR UP Learning Centers at
Western Michigan University and worked with the Office of Community Research and Leadership studying
community colleges and public policy.
About the Authors
National Institute for Learning Outcomes Assessment | 3
Abstract
As colleges educate a more diverse and global student population, there is increased need to ensure every student
succeeds regardless of their differences. This paper explores the relationship between equity and assessment,
addressing the question: how consequential can assessment be to learning when assessment approaches may not be
inclusive of diverse learners? The paper argues that for assessment to meet the goal of improving student learning
and authentically document what students know and can do, a culturally responsive approach to assessment is
needed. In describing what culturally responsive assessment entails, this paper offers a rationale as to why change
is necessary, proposes a way to conceptualize the place of students and culture in assessment, and introduces three
ways to help make assessment culturally responsive.
National Institute for Learning Outcomes Assessment | 4
Equity and A ssessment:
Moving Towards Culturally Responsive A ssessment
Erick Montenegro and Natasha A. Jankowski
Introduction
College enrollment has become increasingly diverse in terms of students’
race, ethnicity, gender identity, socioeconomic status, sexual orientation,
age, ability, etc. This trend is only expected to continue as the United
States moves into a majority-minority nation by the year 2050, and college
enrollments continue to increase. Conducting assessment in a manner that
takes into consideration the various needs of different student populations is
a responsibility of higher education. For one, underrepresented students are
more likely to be low-income and first-generation (Del Rios & Leegwater,
2008; Li & Carroll, 2007; Benitez, 1998), and there are vast differences
between the needs of lesbian, gay, bisexual, transgender, queer, intersex, asexual
(LGBTQIA) (Check & Ballard, 2014; Mallory, 2009), undocumented (Kim
& Diaz, 2013; Perez, 2010), nontraditional (Macqueen, 2012), and special-
needs students attending higher education institutions (Froese-Germain &
McGahey, 2012). Further, students are increasingly mobile, with transfer
students coming from mostly traditionally underrepresented backgrounds,
attending multiple institutions (Backes & Velez, 2015; Shapiro et al, 2012)
and facing their own challenges in higher education (Tobolowski & Cox,
2012).
Various areas of higher education are aware of the need to accommodate
different student populations because “individual differences are clearly
important to student success” (Strange & Banning, 2015, p. 61). For
example, approaches to teaching, student development, student services,
and campus programs have been analyzed and altered to improve outcomes
for specific student groups (Ladson-Billings, 1995a; Ladson-Billings, 1995b;
Schuh, Jones, Harper, & Associates, 2011; Kezar, 2011; Lara & Wood,
2015; Bowen, Chingos, & McPherson, 2009, Aronson & Laughter, 2016).
Within the field of campus advising, the issue of microaggressions through
lack of cultural awareness has been raised (Chu, 2016) and the work of the
Association of American Colleges and Universities (AAC&U) in their Equity
Imperative outlines the need to understand who students are, disaggregate
data to look for inequities, and explore policy changes for unintended
impacts on student groups. Conversations in K-12 have addressed the notion
of equity from the standpoint of equity traps (McKenzie & Scheurich, 2004)
within schools and the need to prepare school leaders to not only expose but
address them through courageous conversations about inequities (Singleton,
2012). In a literature review of culturally responsive school leadership,
Khalifa, Gooden, and Davis (2016) argue that culturally responsive leaders
need to continuously support minoritized students through examination of
assumptions about race and culture. Further, they argue that as demographics
continue to shift, so should practice that responds to student needs, finding
that it is “deleterious for students to have their cultural identities rejected in
school and unacknowledged as integral to student learning” (p. 1285).
Conducting assessment in
a manner that takes into
consideration the various
needs of different student
populations is a responsibility
of higher education.
National Institute for Learning Outcomes Assessment | 5
Culturally relevant and culturally responsive pedagogies sought to outline
ways in which teachers could address unique learning needs of diverse student
populations. Gloria Ladson-Billings (1995a, 1995b) recommends using
culturally relevant pedagogy as a way to allow students in populations outside
of the majority to maintain their cultural integrity all-the-while succeeding
academically. Culturally relevant pedagogy aims to “produce students who
can achieve academically…demonstrate cultural competence, and develop
students who can both understand and critique the existing social order”
(Ladson-Billings, 1995a, p. 474). In culturally responsive pedagogy, teachers
use aspects of students’ cultures in an asset-based approach as opposed to
deficit-based to make the course material relevant to them, and increase
their skill acquisition, engagement, and learning outcomes (Ladson-Billings,
1995a). Yet, Geneva Gay (2010) has argued that solely modifying teaching
practices cannot solve the challenges faced by ‘minoritized’ students.
In terms of assessing student learning, the field has been largely quiet when
it comes to issues of equity. Assessment, if not done with equity in mind,
privileges and validates certain types of learning and evidence of learning
over others, can hinder the validation of multiple means of demonstration,
and can reinforce within students the false notion that they do not belong in
higher education. For equity gaps to be addressed, an entire institution needs
to explore the combination of solutions and supports needed for students
to be successful (Jones, 2015; Methvin & Markham, 2015), of which
assessment is one. However, little of the conversation thus far has focused on
the connection points between demonstration of student learning and issues
of equity. Instead, assessment has remained largely unchanged in regards to
inclusivity, and little urgency has been given to ensuring that students are
provided with just and equitable means to demonstrate their learning. There
is a difference between assessing all students in the same way in relation to a
specific outcome of interest and making sure assessments are appropriate and
inclusive of all students. Being attentive to how students may understand
questions, tasks, and assignments differently, as well as feedback regarding
their learning, is not only beneficial to students but to internal improvement
efforts as well. Intentionally choosing appropriate assessment tools or
approaches that offer the greatest chance for various types of students to
demonstrate their learning so that assessment results may benefit students
from all backgrounds advances our collective interest in student success.
Without examining issues of equity the students who may stand the most to
gain from assessment efforts may have the least benefit since their learning
is not accurately assessed and feedback may not be relevant to impact
learning. If assessments are to be holistic in their goal of improving student
learning, then incorporating a culturally responsive approach to assessment
is a priority. As C. Carney Strange and James Banning (2015) state, student
cultures “can play an important role, for good or otherwise, in introducing
students to and maintaining their engagement in the learning process” (p.
53). It also creates opportunities for students to experience deep learning
(Entwistle, 2001) by honoring students’ prior knowledge and experience.
However, before we present the concept of culturally responsive assessment,
it is useful to unpack an assumption that hinders consideration of diverse
learner needs within assessment—that while learners may take multiple
paths to and through learning, they must demonstrate their knowledge and
skills in the same way.
Assessment, if not done with
equity in mind, privileges
and validates certain types
of learning and evidence
of learning over others,
can hinder the validation
of multiple means of
demonstration, and can
reinforce within students the
false notion that they do not
belong in higher education.
National Institute for Learning Outcomes Assessment | 6
Limiting Learning Demonstration
There is an assumption at play within the field of assessment that while
there are multiple ways for students to learn, students need to demonstrate
learning in specific ways for it to count. For instance, in a specific course
different approaches may be used to engage students in the material, but
demonstration of a students’ knowledge, skills, and abilities are done
uniformly in the same assignment or approach—so while there may be
multiple approaches and methods used across a program or institution for
assessing student learning, at each instance of demonstration a single approach
is employed. Regardless of the literature on the multiple ways students acquire
knowledge, assessment asks students, at each instance of demonstration,
to show they have the knowledge and skills of interest through the same
means. William Sedlacek (1994) discusses the need for the development of
multicultural assessment standards within the Association for Assessment in
Counseling (AAC). While the focus is upon assessment within the context
of counseling support and services, the interest of addressing the needs of
those with “cultural experiences different from…White middle-class men
of European descent, those with less power to control their lives, and those
who experience discrimination in the United States” (p. 550), remains the
same for assessment of or for learning in higher education. Sedlacek (1994)
identifies five fallacies related to culture and assessment, stressing that most
measures were not designed with nontraditional or underserved populations
in mind, that few assessment specialists are trained in developing measures
for use with nontraditional populations, and that larger issues exist that
need to be explored and addressed when promoting diversity, equity, and
inclusivity through assessment. Of note is the fallacy referred to as the three
musketeers, which is the idea that in order to make a measure equally valid
for everyone, everyone completes the same measure—all for one and one
for all—as a means to ensure fairness instead of using different measures
for different groups. Yet, Sedlacek (1994) argues, “if different groups have
different experiences and different ways of presenting their attributes and
abilities…it is unlikely that we could develop a single measure or test item
that would be equally valid for all” (p. 550); further arguing that there is no
need to employ the same measure when what is desired is equity of results,
not process.
There are institutions providing students with support and opportunity to
choose from a variety of approaches or even design how they will be assessed
in cooperation with faculty members, presenting students with agency and
choice in the assessment process (Singer-Freeman & Bastone, 2016) and
most institutions use a combination of assessment methods to gauge learning
(Kuh et al., 2014). In a study at the University of East London, students were
allowed to choose how they were assessed, significantly improving attainment
among learners without an academic background (Grove, 2016). Instead of
completing exams based on coursework, students were given the option to do
a presentation, poster, or debate. Using the alternative assessment techniques
“helped mitigate the fact that many first-year students had not been in
formal education for some time” allowing them space to demonstrate their
learning, not their exam-taking abilities (Grove, 2016). Further, a similar
approach was used at the University of Dublin where students were able to
make a poster instead of taking an exam. In both instances, students had to
There is an assumption at play
within the field of assessment
that while there are multiple
ways for students to learn,
students need to demonstrate
learning in specific ways for it
to count.
National Institute for Learning Outcomes Assessment | 7
The manner in which students
demonstrate learning is
irrelevent when student
demonstration is held to the
same learning outcomes and
evaluative criteria.
demonstrate their learning on the same learning outcomes and evaluative
criteria, but the manner in which they did so was irrelevant. Rubrics were
used such that the evaluation of the work was the same, thus quality ensured,
but the demonstration could be different. In Canada, a study was undertaken
within a large, third-year psychology class regarding differentiated evaluation
to examine student engagement, quality of learning experience, and address
challenges associated with increased student diversity (Gosselin & Gagné,
2014). Differentiated evaluation allowed students to choose how they would
be evaluated though all students were still required to take mid-term and
final exams. Students had the option of adding a term project through
preparing a mini-class or participating in a community service learning
program. The study found positive impact on student achievement and on
the learning experience, with students performing below class average seeing
grade improvement when completing a term project. Further, students who
completed the project performed better on the final exam in comparison to
those that did not, and the option helped to alleviate stress of sitting for an
exam. Qualitative responses from students that selected the project option
indicated that they saw the alternative as an opportunity to demonstrate
their learning through a format over which they felt more control. Gosselin
and Gagné (2014) argued that there are “methods of assessment that can
foster inclusiveness and academic success whilst upholding high standards
for the quality of student learning” yet interestingly “most innovations in
this context have focused on teaching rather than on student learning” (p. 6).
The differentiated evaluation approach complemented the existing structure
and allowed the relationship between faculty and student to shift to one
of collaboration instead of power, regarding decisions about how students
demonstrate their learning.
The need to fold in culture and student experience into assessment is stressed
in the everyday expertise framework—a perspective of learning that takes
into account how students demonstrate knowledge and skills in their daily
life with the other people around them (Toomey Zimmerman & Bell,
2012). The framework allows for learning to have multiple dimensions
including individual, social, and cultural, requiring a broad consideration
of how people learn within and across learning environments, noting that
learners do not act with equal competency in all settings, even if the content
is the same. Toomey Zimmerman and Bell (2012) argue that the difference
in performance indicates that learners competent in informal and everyday
settings may falter in more formalized learning settings, requiring alternative
means to demonstrate their knowledge outside of the traditional classroom.
Beyond the many benefits from engaging students in co-curricular
experiences (Meents‐DeCaigny & Sanders, 2015; Schuh, Jones, Harper, &
Associates, 2011; Schuh, 2009; Pascarella & Terenzini, 2005; Del Rios &
Leegwater, 2008), co-curricular learning provides a means to address the
issues raised by the everyday expertise framework by widening our lens of
where learning happens to include experiences beyond the classroom. In
addition to conceptions shifting where learning happens, there has been a
rise in competency-based education (CBE) which releases the time structure
in which learning occurs in terms of credit hours. CBE programs stress that
authentic artifacts, or demonstrations of student learning, need to come
from a variety of sources to engage learners with curricula and assessment
National Institute for Learning Outcomes Assessment | 8
that reflect not just multiple ways to learn but multiple ways to demonstrate
mastery of a competency (Jobs for the Future, 2016). However, there are
calls for significant research to determine how best to design assessments
for underprepared learners that also elevate and validate their skills through
alternative measures (Person, Goble, & Bruch, 2014). While learning may
happen anywhere and learners may need different lengths of time in their
learning process, there is still the issue of who gets to validate that learning
has occurred, or that demonstrations of learning are of the ‘right type.’
While there is movement to more inclusive means of assessment and active
engagement with students as partners in learning, it is clear that the challenges
of various minority groups on campus differ from those of the majority
(Ellis & Chen, 2013; Rankin, Weber, Blumenfeld, & Frazer, 2010; Miller,
Bradbury, & Pedley, 1998), yet higher education still privileges certain types
of learners, certain ways of demonstrating knowledge, and certain learning
spaces by not consistently offering transparency, differentiated assessments,
or empowering students in their own learning. Students need to develop
and apply their knowledge and skills across multiple contexts in different
courses through a range of methods (Newman, Carpenter, Grawe, & Jaret-
McKinstry, 2014, p. 14) with integrative liberal learning requiring students
to engage in “ongoing demonstration to themselves and to others, of the
gains made through curricula, programs, and the educational experience
as a whole” (Ferren & Paris, 2015, p. 5). Yet, the signals education sends
to students about what is validated or counts as demonstration of learning
can be detrimental and reinforce for marginalized students that they do not
belong because their learning ‘doesn’t count.’ What is needed is collaboration,
where students, faculty and staff “draw together their life experiences and
aspirations with classroom, co-curricular, and community opportunities”
(Ferren & Paris, 2015, p. 20).
Culturally Responsive Assessment
Defining “culture” and explaining what is meant by culturally responsive
assessment is complicated. The issue is that culture, whether speaking
about it in terms of an organization, a campus, or an individual, has been
historically difficult to define. Higher education has a tendency to group
student differences and issues around race under the term ‘diversity,’ which
is often discussed in relation to benefits to White students as opposed to
African Americans, Latinx, Asian Americans, and Native Americans who
continue to be underrepresented in higher education (Dowd & Bensimon,
2015). While diversity efforts on college campuses have brought attention
to the vast differences among students—including gender, religion, sexual
orientation, etc.—the term diversity fails to address issues surrounding race/
ethnicity and does not account for the different histories, needs, interests, and
issues affecting distinct groups of students on campus. With this in mind,
one can see why it would be beneficial to use culture instead of diversity as
the imperative to refocus assessment into a more inclusive endeavor.
This paper draws from and expands on past definitions of culture to develop
an understanding that culture should be thought of as: (1) the explicit
elements that makes people identifiable to a specific group(s) including
behaviors, practices, customs, roles, attitudes, appearance, expressions of
While learning may happen
anywhere and learners
may need different lenghts
of time in their learning
process, there is still the issue
of who gets to validate that
learning has occurred, or that
demonstrations of learning are
of the ‘right type.’
National Institute for Learning Outcomes Assessment | 9
identity, language, housing region, heritage, race/ethnicity, rituals, religion;
(2) the implicit elements that combine a group of people which include
their beliefs, values, ethics, gender identity, sexual orientation, common
experiences (e.g. military veterans and foster children), social identity;
and (3) cognitive elements or the ways that the lived experiences of a
group of people affect their acquisition of knowledge, behavior, cognition,
communication, expression of knowledge, perceptions of self and others,
work ethic, collaboration, and so on. The culturally relevant component
involves assuring that the assessment process—beginning with student
learning outcome statements and ending with improvements in student
learning—is mindful of student differences and employs assessment methods
appropriate for different student groups. Underlying the culturally relevant
component is the focus on students—the importance of keeping students at
the center, which requires their involvement at every step in the assessment
process and builds upon their lived experience.
In addition, it is important to understand the concept of intersectionality and
its effect on culture. Traditionally, intersectionality is thought of in racial/
ethnic identity intersecting with class, gender, and sexual orientation to shape
how people of color experience oppression (Solorzano & Yosso, 2002; Huber,
2010; Cho, 1997). However, for purposes of this paper, intersectionality is
the way that aspects of a person’s identity cannot be fully separated from one
another, play a central role in peoples’ experiences and making meaning of
those experiences. This is related to Susan Jones and Marylu McEwen’s (2000)
multiple dimensions of identity which treats a student’s identity as dynamic
and changing depending on the relative contextual salience of other elements
of one’s identity (e.g. race/ethnicity, sexual orientation, gender, religion);
with no single aspect of one’s identity understood singularly, but only in
relation to the other dimensions. For example, a White male that identifies
as a member of the lesbian, gay, bisexual, transgender, queer, intersex, and
asexual (LGBTQIA) community and practices Judaism is shaped by the
intersectionality of these four elements. A Latina that is a single-mother from
a low-socioeconomic background is shaped by the intersectionality of these
elements. An undocumented English as a second language, first-generation
student will experience college, acquire knowledge, and demonstrate
knowledge differently than an international English as a second language
first-generation student. The culture—the explicit, implicit, and cognitive
elements—of the people in these examples shape their college experiences,
and while one aspect of their culture may manifest itself more than another
in specific contexts, they all affect the outcomes being assessed.
Thinking of culture in the way that it is defined here can serve as a reference
point for what to consider when engaging in assessment and developing/
choosing/implementing assessment tools and methods. Culture is by no
means simple, and it is by no means easily definable. It is dependent on
the context in which culture is discussed. Culture permeates the individual,
group, entire institutions, countries, and continents; and at the same time
the individuals that comprise cultural groups are multicultural through
intersectionality. Perhaps Lang (1997) stated it best when he said “attempts
at defining culture in a definite way are futile” (p. 389). However, developing
an inclusive understanding of culture, and making it explicit that culture is
much more than race/ethnicity and affects students’ lives on multiple levels,
The culturally relevant
component involves assuring
that the assessment process—
beginning with student
learning outcome statements
and ending with improvements
in student learning—is
mindful of student differences
and employs assessment
methods appropriate for
different student groups.
National Institute for Learning Outcomes Assessment | 10
including learning and how they demonstrate learning, will help ensure
culturally responsive assessment and increase the effectiveness and impact of
learning outcomes assessment efforts.
In addition to the term of culture, it is important to note the use of responsive
to indicate “an action-based, urgent need to create contexts and curriculum
that responds to the social, political, cultural, and educational needs of
students; it is affirmative and seeks to identify and institutionalize practices
that affirm indigenous and authentic cultural practices of students” (Khalifa,
Gooden, & Davis, 2016, p. 1278). Students who experience validation from
faculty and integrate academically and socially are more likely to persist and
be successful (Karp, Hughes, & O’Gara, 2011). Assessment approaches
and processes can help reinforce a sense of belonging or add to students’
belief that they do not belong because their learning or experiences are not
deemed as valid or important. Susan Headden and Sarah McKay (2015)
stress this point, arguing that student motivation is connected to student’s
beliefs that they are able to do the work and have a sense of control over
the work. For first-generation college goers and African American students
“stereotypes about academic performance can turn into self-fulfilling
prophecies…even feedback on papers can reinforce or foster learning…that
students are cared about and respected as learners” (Headden & McKay,
2015, p. 15). An environment focused on students’ unique learning interests
and needs enables students to incorporate prior and everyday experiences in
meaning construction (Land, Hannafin, & Oliver, 2012). Involvement with
culture is also important as Cathleen Spinelli (2008) argues that there are a
disproportionate number of students with cultural and linguistic differences
that are misidentified as learning disabled. As a result, students are classified
incorrectly, not academically challenged, and do not receive appropriate
services. Spinelli (2008) further argues that when looking specifically at the
case of English language learners, informal assessment provided a solution
to the need of assessment of learning, but in a manner adaptable to language
and cultural diversity, individual learning styles, and personal challenge
while also informing instruction.
Culturally responsive assessment is thus thought of as assessment that is
mindful of the student populations the institution serves, using language
that is appropriate for all students when developing learning outcomes,
acknowledging students’ differences in the planning phases of an assessment
effort, developing and/or using assessment tools that are appropriate for
different students, and being intentional in using assessment results to improve
learning for all students. Culturally responsive assessment involves being
student-focused, which does not simply mean being mindful of students.
Instead, being student-focused calls for student involvement throughout the
entire assessment process including the development of learning outcome
statements, assessment tool selection/development process, data collection
and interpretation, and use of results. An essential aspect of maintaining
focus on students is truly understanding the student population at the
institution and/or level at which the assessment is being conducted. Once we
understand who our students are we can begin to tailor assessment processes
and materials to have the greatest impact for their learning. Institutions with
high enrollment of traditionally underrepresented students have already
begun tailoring their learning outcomes assessment approaches based on
Assessment approaches and
processes can help reinforce
a sense of belonging or add
to students’ belief that they
do not belong because their
learning or experiences are not
deemed as valid or important.
National Institute for Learning Outcomes Assessment | 11
the student populations that they serve (Montenegro & Jankowski, 2015;
Nunley, Bers, & Manning, 2011; Baker, Jankowski, Provezis, & Kinzie,
2012). Minority-Serving Institutions (MSIs) have been found to increase
self-esteem, engagement, critical thinking skills, leadership skills and
opportunities, and help the identity formation processes for traditionally
underrepresented students; which helps increase students’ persistence
through college (Conrad & Gasman, 2015; Del Rios & Leegwater, 2008;
Conrad et al, 2013). The work at these institutions can serve as guideposts
for the development of culturally responsive assessment practices.
Student Learning Outcomes Statements
Learning outcomes assessment as a process begins with developing learning
outcome statements that clearly state what students should know and be
able to demonstrate upon completion of a course, academic program,
college, making use of student services, etc. To develop student learning
outcomes statements using a cultural lens necessarily involves students in
the development process. Poorly constructed learning outcomes make it
difficult for students to demonstrate their learning for a myriad of reasons
(e.g. not understanding what is expected of them, not understanding how
the course/program is expected to contribute to their learning). In addition,
it is students that will directly benefit from the feedback they receive as a
result of assessment. Clarity of outcomes and curricular structure matters
in general education (Gaston, 2015), assignment design (Winkelmes et al,
2016), co-construction of knowledge for deep learning (Juvova et al, 2015;
von Glasersfeld, 2005), and new course design models like competency-based
education (Jobs for the Future, 2016). Further, in the National Research
Council report, How People Learn, (2000) principles for designing learner-
centered environments emphasized the importance of individual social and
cultural contexts in learning. Such perspectives require different approaches
to curricular design, teaching, and assessment, and squarely place learner
preconceptions and experiences as an integral part of the learning process.
Assessment is a field of alignment, and this also originates from learning
outcomes statements. Hutchings (2016) defines alignment as “the linking of
intended student learning outcomes with the processes and practices needed
to foster those outcomes” (p. 5). Similarly to how academic programs, student
services, and other institutional programs aim to align with and promote
the mission of the college or university, learning outcomes statements of
departments, programs, and courses should align with those of the institution.
Outcome statements need to be culturally responsive because they align with
assignments, evaluative criteria, and institutional and departmental goals. If
outcome statements are not culturally responsive, then there are implications
for various levels of the institution; not just for students. Learning outcome
statements which are written to inform educational policy and practice,
and are clear about expected proficiencies make it possible for programs,
departments, institutions, and students to meet their goals (National
Institute for Learning Outcomes Assessment, 2016). The language and
operative verbs in learning outcome statements serve as a guide for students
to understand departmental/program expectations, as well as understand
how their educational experiences prepare them for their careers and lives
after college (National Institute for Learning Outcomes Assessment, 2016).
However, if learning outcomes statements are not written with attention to
To develop student learning
outcomes statements using
a cultural lens necessarily
involves students in the
development process.
National Institute for Learning Outcomes Assessment | 12
cultural relevancy, then it becomes difficult to accurately infer the learning
gains of different groups of students.
Cliff Adelman (2015) speaks about the importance of being intentional and
mindful of language when writing learning outcomes statements as this can
lead to creating assignments that allow for genuine judgement of student
achievement. However, taking this a step further and being mindful of how
the language of learning outcomes statements might be appropriate for/
inclusive of certain student groups but not others can lead to more holistic
assessments. Flawed assessment designs may unintentionally skew scores
for certain student populations and ensuring this does not happen begins
with the writing of culturally responsive learning outcomes statements that
consider students, their different ways of learning, and the diverse ways
they demonstrate learning. One way to make statements more culturally
responsive is to explicitly define terms and use scenarios or examples that
are relatable to various student groups. A sample tool that incorporates these
elements of being intentional and explicit in writing learning outcomes and
clearly defining learning is the Degree Qualifications Profile (DQP) and
Tuning process. The DQP was developed by Lumina Foundation (2014),
and by coupling Tuning processes to it serves as a way to clearly outline
what students know and should be able to do after attaining a degree (Ewell,
2013).
Traditionally, learning outcomes statements are written by and for faculty
and administrators. As a result, faculty and administrators define the
intended learning outcomes and what it looks like to demonstrate those
outcomes. If, instead, we write learning outcome statements for and with
students, then we increase the chances of students understanding what is
expected of them. In addition, instead of students’ knowledge conforming
to how we traditionally measure it, students would now have agency in how
to demonstrate learning. This would result in learning outcomes, as well as
the assessment process, becoming a more inclusive endeavor.
Assessment Approaches
There is a need for assessments that allow students to demonstrate their
learning in various ways while also being transparent about the learning
that is taking place, help students reflect on their learning experiences, and
allow students to actively participate in the learning and assessment process.
Course-level assessments such as culturally responsive rubrics, portfolios, and
capstone projects can lead to more valid, appropriate, holistic, and formative
assessment where results are more indicative of what all students can do
or lead to more targeted improvements in teaching and learning. Rubrics,
which help instructors gauge student learning, skills development, and
acquisition of learning outcomes, provide criteria by which to assess whether
or not the learning outcome was demonstrated. Rubrics, when they undergo
a culturally conscious development process and are shared with students,
can be a way to accurately assess learning for all students while allowing
variation in how the learning is demonstrated. While rubrics are at times
created by individual faculty members to fit the context of specific courses or
programs, the Association of American Colleges and Universities (AAC&U)
Valid Assessment of Learning in Undergraduate Education (VALUE)
Rubrics serve as examples of rubrics for institutions to employ. In addition,
There is a need for assessments
that allow students to
demonstrate their learning
in various ways while also
being transparent about the
learning that is taking place,
help students reflect on their
learning experiences, and allow
students to actively participate
in the learning and assessment
process.
National Institute for Learning Outcomes Assessment | 13
capstones—which can be entire courses or student projects—can also be
an avenue through which the learning of students can be better gauged by
allowing students to design their own projects in partnership with faculty.
They provide “tangible, visible, self-explanatory evidence of exactly what
students have and haven’t learned” (Suskie, 2004, p. 95). As comprehensive,
culminating experiences, capstones allow students to demonstrate a wide-
range of skills and knowledge that oftentimes draws from previous work,
experiences, and learning that occurred throughout their coursework.
Finally, portfolios offer a similar freedom for students to demonstrate
their learning and provide a more holistic representation of what students
know and can do. The use of portfolios provides students the option to
select demonstrations and add commentary and reflection, furthering
their agency in the process and selection of assessment evidence. Portfolios
represent student work over time and demonstrate various forms of learning
(Kuh et al, 2015; Banta, Griffin, Flateby, & Kahn, 2009) which may not
be easily captured by other forms of assessment. Portfolios are “authentic
assessment that draws on the work students do in regular course activities
and assignments” and “reconnect assessment to the ongoing work of
teaching and learning and to the work of faculty, raising the prospects for
productive use” (Kuh et al, 2015, p. 36). Portfolios provide the opportunity
to get students invested into the course beyond grade attainment, and help
to deepen students’ educational experiences through allowing them to
make connections between conceptual issues, theoretical knowledge, and
real world experiences (Singer-Freeman & Bastone, 2016). Additionally,
portfolios can be made available online. Eportfolios can be easily accessed
by potential employers, as well as other institutions, which provides students
in the job market or looking to transfer a means to easily demonstrate their
knowledge and skills. Kuh et al (2015) mention a few of the advantages
that portfolios have for assessment, including advancing student success,
catalyzing change, and making learning more visible for students. These
impacts can be furthered by applying a cultural lens when assessing student
portfolios. By being mindful of how culture affects students’ meaning-
making processes, cognition, and demonstrations of learning, we can better
understand and appreciate the learning gains that students make. In fact,
at the program-level, assessment approaches such as rubrics and portfolios
are used more often than surveys and other approaches (Ewell, Paulson, &
Kinzie, 2011).
Use of Assessment Results
Implementing formative assessment methods means very little if assessment
data are not used to inform learning at various levels of the institution or if it
has no meaning to students to improve their own learning. The first step in
creating change is analyzing the data by student populations. Disaggregating
the data is instrumental in informing changes to higher education. While
the data may tell a positive story about overall learning, disaggregation may
yield the observation that first-generation students are struggling in a course,
female students are making use of resources aimed at supporting their
education at disproportionate rates, or Latinx students are not reaching the
same institutional learning outcomes as other racial/ethnic groups. In either
hypothetical case, disaggregating the data allows researchers, administrators,
and practitioners to see themes that they otherwise would have missed and
By being mindful of how culture
affects students’ meaning-
making processes, cognition, and
demonstrations of learning, we can
better understand and appreciate
the learning gains that students
make.
National Institute for Learning Outcomes Assessment | 14
could inform changes that would positively impact students’ education.
In addition, disaggregation of assessment data should not only be used to
uncover surface-level findings such as Latinx students excel at “ABC” while
first-generation students struggle with “XYZ.” As Aydin Bal & Audrey Trainor
(2016) state, “researchers must also include an examination of processes
(e.g. the racialization of disability… and the institutional acts of exclusion
based on ability differences) and institutions…that reproduce, regardless of
intentionality, disparities” (p. 330-331). This means that disaggregating data
should explore why the condition exists in the first place, and then be used
to inform/develop possible solutions.
In using assessment results, it is also useful to be mindful of our own
assumptions. Similar to how a researcher’s bias cannot be fully removed from
his/her/zer’s study and can either harm or enhance his/her/zer’s research, so
can the biases of faculty and staff affect assessment efforts and use of results.
It is unrealistic and counterproductive for assessment professionals to think
they are approaching their work from an impartial stance or to assume that
the students being assessed also operate from an impartial stance. Failing to
recognize how culture and our own experiences affect the assessment process
can limit the impact of assessment. In discussing the need for faculty to be
attentive to the changes in the institution’s student population, Goldrick-
Rab and Cook (2011) warn against comparing all students against the
researcher’s subconscious idea of what students do/should do. Failing to be
aware of our own biases or subconscious ideas and failing to disaggregate
assessment data in a culturally responsive manner may cause the assessment
endeavor to implement outdated norms as a means of comparison, which
can misclassify certain students as underachievers, confusing, or outliers; and
can also lead to the mistake of failing to connect the data to the actual lived
experiences and realities of the students the institution serves (Goldrick-Rab
& Cook, 2011). This can also lead to unintentionally reinforcing negative
assumptions about certain student groups. Treating different racial/ethnic
groups under an aggregate umbrella, as has been the recent case with the
term “underrepresented minorities,” minimizes the voice of various groups
and ignores their salient differences (Dowd & Bensimon, 2015) which
impact their needs, experiences, learning, and demonstration of that
learning. Finally, it would be worthwhile to connect assessment results to
other campus assessment strategies. While certain data collection efforts
on campus may seem unrelated, occurrences on campus seldom happen in
isolation. Connecting different assessment efforts and resulting data sets can
better inform issues related to student attrition, success, campus climate,
pedagogy, and others (Hurtado & Halualani, 2014).
Final Thoughts
Students’ college experiences are inseparable from other daily experiences such
as those encountered at work, microaggressions endured on campus, family
life, and employment. More often than not, students’ college experiences are
affected by students’ own culture and cultural differences with faculty, staff,
and peers. It has long been known that students of different backgrounds
experience college differently and respond differently to similar situations,
stimuli, experiences, requests, questions, etc. So, if we also know that students
from different cultures who have similar education backgrounds respond
Students’ college experiences
are inseparable from other
daily experiences such as
those encountered at work,
microaggressions endured
on campus, family life, and
employment.
National Institute for Learning Outcomes Assessment | 15
and perform significantly different, why would we design assessments,
execute them, and then make changes based on assessment results without
considering the cultural relevance of the assessment effort and analyze how
the assessment might affect all students/benefit certain population(s) and
hinder others? Why would we not include students in the assessment process
to improve our approaches?
The focus of assessment as a means to improve student learning is an agreed
upon purpose of the work. As Kuh, Ikenberry, Jankowski, Cain, Ewell,
Hutchings, and Kinzie (2015) state, “gathering information about collegiate
outcomes has a practical goal: using it to improve both student learning
and institutional performance” (p. 51); and “harnessing evidence of student
learning, making it consequential in the improvement of student success
and strengthened institutional performance is what matters” (p. 4). Yet,
how consequential can assessment truly be when assessment approaches are
minimally inclusive of our current student populations? Using assessment
tools and approaches that work for the majority of students but are less
mindful of students identifying with groups outside of the majority
population places a significant portion of students at a disadvantage, leads
to a decrease in the quality of education, creates a disconnect between
students and the institution, and contributes to achievement gaps (Slee,
2010; Sullivan, 2010; Qualls, 1998). Assessment that overlooks issues of
diversity and equity contributes to inequalities in outcomes (Bal & Trainor,
2016). The same can be said for assessment approaches that do not take into
account students’ culture.
Students have different ways to demonstrate their knowledge and we need
to use assessment metrics that appropriately elicit demonstrations of what
students know. One example of the diverse ways students can demonstrate
learning comes from Nick Sousanis’ (2015) published dissertation exploring
how people construct knowledge. Instead of writing a typical manuscript,
Sousanis demonstrated his knowledge in a graphic novel format. At times,
the illustrations said more than the words on the page, and both pictures
and words united to tell a powerful academic story. This way of presenting
scholarly work, while unconventional in academia, is still a powerful
demonstration of learning. Sousanis’ chosen method of demonstrating his
knowledge on a specific topic is not wrong, it is just different. We undo
boundaries through the awareness that “it is our [own] vision, and not what
we are viewing, that is limited” (Sousanis, 2015, p. 42). How assessment is
often operationalized or experienced by students has not moved to a position
where it continuously regards students’ diverse methods of demonstrating
knowledge as appropriate. Instead, different can often be marked as wrong.
If assessment is about demonstrating learning, then we need to allow students
the space to show their knowledge. Students are highly varied in customs,
identity, and understanding, and it is all shaped by culture which affects
learning; and thus, should affect how we measure learning. If assessment
is done for improvement and with the goal of using the results to benefit
student learning, then having outcome assessments that appropriately tell
the stories of what students know and can do is of imperative importance.
Our assessments approaches—how we assess and the process of assessment
itself—should align with the students we have, empowering them with
narratives to share and document their learning journey.
Assessment that overlooks
issues of diversity and equity
contributes to inequalities in
outcomes. The same can be
said for assessment approaches
that do not take into account
students’ culture.
National Institute for Learning Outcomes Assessment | 16
What is needed is not to help
learners conform to the ways
of higher education, thus
reinforcing inequities and
expectations based on ideologies
the students may ascribe to,
but to empower students for
success through intentional
efforts to address inequality
within our structures, create
clear transparent pathways,
and ensure that credits and
credentials are awarded by
demonstration of learning, in
whatever form that may take.
In summary, assessing students in the same way without paying attention
to their differences works if students are all privy to the same educational
opportunities, are all at the same academic standing, have similar experiences
on campus, work through knowledge in similar fashion, understand questions
in similar ways, and benefit from the same programs, pedagogical styles,
support services, and interactions. However, we know this is not the case.
“While absolute growth in the college-going population helped shape today’s
college milieu, compositional changes also impacted the college experience,
turning it into a set of highly diverse experiences that led to very different
outcomes” (Goldrick-Rab & Cook, 2011, p. 257). Sara Goldrick-Rab &
Marjorie Cook (2011) continue to say that “as the student body grew more
diverse, so did the kinds of colleges and universities serving them; at the same
time, opportunities both expanded in number and became more distinct
and disparate, reflecting and preserving key aspects of the inequality of
opportunity and outcomes” (p. 255). Continuing to assess students as if there
are no differences will only work to preserve key aspects of inequality and
widen the achievement gap. It is no secret that there is a disparity between the
academic attainment of students based on race/ethnicity (Bowen, Chingos,
& McPherson, 2009; Condron, Tope, Steidl, & Freeman, 2013; Santiago,
Galdeano, & Taylor, 2015; Carnevale & Strohl, 2013) and social class (Kezar,
2011; Bowen, Chingos, & McPherson, 2009). We need to ask ourselves, is it
that we want students to demonstrate their knowledge and skills or attainment
of learning outcomes in a particular way, or that they demonstrate their
learning? What is needed is not to help learners conform to the ways of higher
education, thus reinforcing inequities and expectations based on ideologies
the students may not ascribe to, but to empower students for success through
intentional efforts to address inequality within our structures, create clear
transparent pathways, and ensure that credits and credentials are awarded by
demonstration of learning, in whatever form that may take.
National Institute for Learning Outcomes Assessment | 17
References
Adelman, C. (2015). To imagine a verb: The language and syntax of learning outcomes statements. Urbana, IL: University of
Illinois and Indiana University, National Institute for Learning Outcomes Assessment (NILOA).
Aronson, B., & Laughter, J. (2016). The theory and practice of culturally relevant education: A synthesis of research across
content areas. Review of Education Research, 86(1), 163-206.
Backes, B., & Velez, E. D. (2015). Who transfers and where do they go? Community college students in Florida. Washington,
DC: National Center for Analysis of Longitudinal Data in Education Research (CALDER).
Baker, G. R., Jankowski, N., Provezis, S. & Kinzie, J. (2012). Using assessment results: Promising practices of institutions
that do it well. Urbana, IL: University of Illinois and Indiana University, National Institute for Learning Outcomes
Assessment (NILOA).
Bal, A., & Trainor, A. A. (2016). Culturally responsive experimental intervention studies: The development of a rubric for
paradigm expansion. Review of Education Research, 86(2), 319-359.
Banta, T. W., Griffin, M., Flateby, T. L., & Kahn, S. (2009, December). Three promising alternatives for assessing college
students’ knowledge and skills. Urbana, IL: University of Illinois and Indiana University, National Institute for Learning
Outcomes Assessment.
Benitez, M. (1998). Hispanic-serving institutions: Challenges and opportunities. New Directions for Higher Education,
102(2), 57-68.
Bowen, W. G., Chingos, M. M. & McPherson, M. S. (2009). Crossing the finish line: Completing college at America’s public
universities. Princeton, NJ: Princeton University Press.
Bransford, J. D., Brown, A. L., & Cocking, R. R. (Eds.). (2000). How people learn: Brain, mind, experience, and school.
Washington, DC: National Academy Press.
Carnevale, A. P., & Strohl, J. (2013). Separate and unequal: How higher education reinforces the intergenerational reproduction
of White racial privilege. Washington, DC; Georgetown University, Georgetown Public Policy Institute, Center on
Education and the Workforce.
Check, E., & Ballard, K. (2014). Navigating emotional, intellectual, and physical violence directed toward LGBTQQ
students and educators. Art Education, 67(3), 6-11.
Cho, S. K. (1997). Converging stereotypes in racialized sexual harassment: Where the model minority meets Suzie Wong.
In A. K. Wing, (Ed). Critical Race Feminism: A Reader (2nd. Ed). New York, NY: New York University Press.
Chu, J. Y. (2016, October 21). The need to be culturally aware when advising students (essay). Inside Higher Ed. Retrieved
from https://www.insidehighered.com/views/2016/10/21/need-be-culturally-aware-when-advising-students-essay.
Condron, D. J., Tope, D., Steidl, C. R., & Freeman, K. J. (2013). Racial segregation and the Black/White achievement
gap, 1992 to 2009. The Sociological Quarterly, 54, 130-157.
Conrad, C., Gasman, M. Lundberg, T., Nguyen, T., Commodore, F., & A. C. Samayoa (2013). Using educational data
to increase learning, retention, and degree attainment at Minority Serving Institutions. Philadelphia, PA: Penn Center for
Minority Serving Institutions. Retrieved from http://www.gse.upenn.edu/pdf/cmsi/using_educational_data .
Conrad, C., & Gasman, M. (2015). Educating a diverse nation: Lessons from Minority Serving Institutions. Cambridge,
MA: Harvard University Press.
Del Rios, M., & Leegwater, L. (2008). Increasing student success at minority serving institutions: Findings from the BEAMS
project. Washington, DC: Institute for Higher Education Policy (IHEP).
Dowd, A. G., & Bensimon, E. M. (2015). Engaging the “race question”: Accountability and equity in U.S. higher education.
New York, NY: Teachers College at Columbia University.
Ellis, L. M., & Chen, E. C. (2013). Negotiating identity development among undocumented immigrant college students:
A grounded theory study. Journal of Counseling Psychology, 60(2), 251-264.
Entwistle, N. (2001). Styles of learning and approaches to studying in higher education. Kybernetes, 30(5/6), 593-603.
Ewell, P. T. (2013). The Lumina Degree Qualifications Profile: Implications for assessment. Urbana, IL: University of Illinois
and Indiana University, National Institute for Learning Outcomes Assessment.
Ewell, P. T., Paulson, K., & Kinzie, J. (2011). Down and in: Assessment practices at the program level. Urbana, IL: University
of Illinois and Indiana University, National Institute for Learning Outcomes Assessment (NILOA).
National Institute for Learning Outcomes Assessment | 18
Ferren, A. S., & Paris, D. C. (2015). Faculty leadership for integrative liberal learning. Washington, DC: Association of
American Colleges and Universities.
Froese-Germain, B., & McGahey, B. (2012). Optimizing conditions of teacher’s professional practice to support students with
special educational needs. Ottawa, ON, Canada: Canadian Teacher’s Federation. Retrieved from http://files.eric.ed.gov/
fulltext/ED532565
Gaston, P. L. (2015). General education transformed: How we can, why we must. Washington, DC: Association of American
Colleges and Universities.
Gay, G. (2011). Culturally responsive teaching: Theory, research, and practice (2nd ed.). New York, NY: Teacher College
Press.
Goldrick-Rab, S., & Cook, M. A. E. (2011). College students in changing contexts. In P. G. Altbach, P. J. Gumport, &
R. O. Berdahl, (Eds) American higher education in the twenty-first century: Social, political, and economic challenges (3rd
Ed.) pp. 254-278. Baltimore, MD: The John Hopkins University Press.
Gosselin, J., & Gagné, A. (2014). Differentiated evaluation: An inclusive evaluation strategy aimed at promoting student
engagement and student learning in undergraduate classrooms. Toronto, Canada: Higher Education Quality Council of
Ontario.
Grove, J. (2016, September, 29). Mature students ‘do better with non-written assessment’. Times Higher Education.
Retrieved from https://www.timeshighereducation.com/news/mature-students-do-better-non-written-assessment.
Headden, S., & McKay, S. (2015, July). Motivation matters: How new research can help teachers boost student engagement.
Stanford, CA: Carnegie Foundation for the Advancement of Teaching.
Huber, L. P. (2010). Using Latina/o critical race theory (LatCrit) and racist nativism to explore intersectionality in the
educational experiences of undocumented Chicana college students. Educational Foundations, 24, 77-96.
Hurtado, S., & Halualani, R. (2014). Diversity assessment, accountability, and action: Going beyond the numbers.
Diversity & Democracy, 17(4), 8-11.
Hutchings, P. (2016). Aligning educational outcomes and practices. Urbana, IL: University of Illinois and Indiana University,
National Institute for Learning Outcomes Assessment (NILOA).
Jobs for the Future. (2016, February). Expanding competency-based education for all learners. Washington, DC: Author.
Jones, S. (2015). The game changers: Strategies to boost college completion and close attainment gaps. Change: The
Magazine of Higher Learning, 47(2), 24-29.
Jones, S. R., & McEwen, M. K. (2000). A conceptual model of multiple dimensions of identity. Journal of College Student
Development, 41, 405-414.
Juvova, A., Chudy, S., Neumeister, P., Plischke, J., & Kvinotova, J. (2015). Reflection of constructivist theories in current
educational practice. Universal Journal of Educational Research, 3(5), 345-349.
Karp, M. M., Hughes, K. L., & O’Gara, L. (2010). An exploration of Tinto’s integration framework for community
college students. Journal of College Student Retention: Research, Theory & Practice, 12(1), 69-86.
Kezar, A. (2011). Recognizing and serving low-income students in higher education: An examination of institutional policies,
practices, and culture. New York, NY: Routledge.
Khalifa, M. A., Gooden, M. A., & Davis, J. E. (2016). Culturally responsive school leadership: A synthesis of the literature.
Review of Educational Research, 86(4), 1272-1311.
Kim, E., & Diaz, J. (2013). Undocumented students and higher education. ASHE Higher Education Report, 38(6), 77-90.
Kuh, G. D., Ikenberry, S. O., Jankowski, N. A., Cain, T. R., Ewell, P. T., Hutchings, P., & Kinzie, J. (2015). Using evidence
of student learning to improve higher education. San Francisco, CA: Jossey-Bass.
Kuh, G. D., Jankowski, N., Ikenberry, S. O., & Kinzie, J. (2014). Knowing what students know and can do: The current
state of student learning outcomes assessment in US colleges and universities. Urbana, IL: University of Illinois and Indiana
University, National Institute for Learning Outcomes Assessment (NILOA).
Ladson-Billings, G. (1995a). Toward a theory of culturally relevant pedagogy. American Educational Research Journal,
32(3), 465-491.
Ladson-Billings, G. (1995b). But that’s just good teaching! The case for culturally relevant pedagogy. Theory Into Practice,
34(3), 159-165.
National Institute for Learning Outcomes Assessment | 19
Land, S. M., Hannafin, M. J., & Oliver, K. (2012). Student-centered learning environments: Foundations, assumptions and
design. In D. Jonassen, & S. Land (Eds.). Theoretical foundations of learning environments (2nd ed.) pp. 3-26. New York,
NY: Routledge.
Lang, A. (1997). Thinking rich as well as simple: Boesch’s cultural psychology in semiotic perspective. Culture & Psychology,
3, 383–394.
Lara, R., & Wood, J. L. (2015). Impact of non-cognitive behaviors on student engagement for Latino male students attending
a designated two-year Hispanic-serving institution. In J. P. Mendez, F. A. Bonner II, J. Mendez-Negrete, & R. T. Palmer
(Eds) Hispanic-serving institutions in American higher education: Their origin, and present and future challenges. Sterling, VA:
Stylus Publishing, LLC.
Li, X., & Carroll, C. D. (2007). Characteristics of minority-serving institutions and minority undergraduates enrolled in these
institutions. Washington, DC: National Center for Education Statistics, Institute of Education Sciences, U.S. Department
of Education.
Lumina Foundation (2014). The Degree Qualifications Profile. Indianapolis, IN: Author.
Macqueen, E. (2012). Studying the experiences of non-traditional students in teacher education. Sydney, Australia: Joint AARE
APERA International Conference. Retrieved from http://files.eric.ed.gov/fulltext/ED544492 .
Mallory, A. (2009). Access and equity for all students: Meeting the needs of LGBT students. California Postsecondary Education
Commission. Retrieved from http://files.eric.ed.gov/fulltext/ED507508 .
McKenzie, K. B., & Scheurich, J. J. (2004). Equity traps: A useful construct for preparing principals to lead schools that are
successful with racially diverse students. Educational Administration Quarterly, 40, 601-632.
Meents-DeCaigny, E., & Sanders, L. (2015). Framing research and assessment of co-curricular learning. New Directions for
Institutional Research, 164, 15-25.
Methvin, P., & Markham, P. N. (2015). Turning the page: Addressing the challenge of remediation. Change: The Magazine
of Higher Learning, 47(4), 50-56.
Miller, R., Bradbury, J., & Pedley, J. (1998). Academic performance of first and second language students: Disadvantages and
underpreparedness. South African Journal of Science, 95, 30-34.
Montenegro, E., & Jankowski, N. A. (2015). Focused on what matters: Assessment of student learning outcomes at minority-
serving institutions. Urbana, IL: University of Illinois and Indiana University, National Institute for Learning Outcomes
Assessment (NILOA).
National Institute for Learning Outcomes Assessment. (2016, May). Higher education quality: Why documenting learning
matters. Urbana, IL: University of Illinois and Indiana University, National Institute for Learning Outcomes Assessment.
Newman, L. E., Carpenter, S., Grawe, N., & Jaret-McKinstry, S. (2014). Creating a culture conducive to integrative learning.
Peer Review, 16(4), 14-15.
Nunley, C., Bers, T, & Manning, T. (2011, July). Learning outcomes assessment in community colleges. Urbana, IL: University
of Illinois and Indiana University, National Institute for Learning Outcomes Assessment (NILOA).
Pascarella, E. T., & Terenzini, P. T. (2005). How college affects students. San Francisco, CA: Jossey-Bass.
Perez, W. (2010). Higher education access for undocumented students: Recommendations for counseling professionals.
Journal of College Admission, 206, 32-35.
Person, A. E., Goble, L., & Bruch, J. (2014, April). Developing competency-based program models in three community colleges.
Oakland, CA: Mathematica Policy Research.
Qualls, A. L. (1998). Culturally responsive assessment: Development strategies and validity issues. The Journal of Negro
Education, 67(3), 296-301.
Rankin, S., Weber, G., Blumenfeld, W., & Frazer, S. (2010). 2010 state of higher education for lesbian, gay, bisexual, and
transgender people. Charlotte, NC; Campus Pride.
Santiago, D. A., Galdeano, E. C., & Taylor, M. (2015). The condition of Latinos in education: 2015 factbook. Washington,
DC; Excelencia in Education.
Schuh, J. H. (2009). Assessment methods for student affairs. San Francisco, CA: Jossey-Bass.
National Institute for Learning Outcomes Assessment | 20
Schuh, J. H., Jones, S. R., Harper, S. R., & Associates. (2011). Student services: A handbook for the profession (5th Ed). San
Francisco, CA: Jossey-Bass.
Sedlacek, W. E. (1994). Issues in advancing diversity through assessment. Journal of Counseling & Development, 72, 549-553.
Shapiro, D., Dundar, A., Chen, J., Ziskin, M., Park, E., Torres, V., & Chiang, Y. (2012). Completing college: A national view
of student attainment rates. Herndon, VA: National Student Clearinghouse Research Center.
Singer-Freeman, K. & Bastone, L. (2016, July). Pedagogical choices make large classes feel small. (Occasional Paper No. 27).
Urbana, IL: University of Illinois and Indiana University, National Institute for Learning Outcomes Assessment (NILOA).
Singleton, G. E. (2012). More courageous conversations about race. Thousand Oaks, CA: Corwin.
Slee, J. (2010). A systemic approach to culturally responsive assessment practices and evaluation. Higher Education Quarterly,
64(3), 246-260.
Solorzano, D. G., & Yosso, T. J. (2002). Critical race methodology: Counter-storytelling as an analytical framework for
education research. Qualitative Inquiry, 8(1), 23-44.
Sousanis, N. (2015). Unflattening. Cambridge, MA: Harvard University Press.
Spinelli, C. G. (2008). Addressing the issue of cultural and linguistic diversity and assessment: Informal evaluation measures
for English language learners. Reading & Writing Quarterly, 24(1), 101-118.
Strange, C. C., & Banning, J. H. (2015). Designing for learning: Creating campus environments for student success (2nd Ed.).
San Francisco, CA: Jossey-Bass.
Sullivan, A. L. (2010). Preventing disproportionality: A framework for culturally responsive assessment. Communique, 39(3),
24-26.
Suskie, L. (2004). Assessing student learning: A common sense guide. Boston, MA: Anker Publishing Company.
Tobolowski, B. F., & Cox, B. E. (2012). Rationalizing neglect: An institutional response to transfer students. Journal of
Higher Education, 83(3), 389-410.
Toomey Zimmerman, H., & Bell, P. (2012). Everyday expertise: Learning within and across formal and informal settings.
In D, Jonassen & S. Land (Eds.). Theoretical foundations of learning environments (2nd ed.) pp. 224-241. New York, NY:
Routledge.
von Glasersfeld, E. (2005). Instroduction: Aspects of constructivism. In C. T. Fosnot (Ed.), Constructivism: Theory, Perspectives,
and practice (2nd ed.), pp. 3-7. New York, NY: Teachers College Press.
Winkelmes, M., Bernacki, M., Butler, J., Zochowski, M., Golanics, J., & Harriss Weavil, K. (2016). A teaching intervention
that increases underserved college students’ success. Peer Review, 18(1-2), 31-36.
National Institute for Learning Outcomes Assessment | 21
NILOA National Advisory Panel
James Anderson
Interim Dean
Edward William and Jane Marr Gutsgell
Professor
University of Illinois Urbana-Champaign
Wallace Boston
CEO
American Public University System
Molly Corbett Broad
President
American Council on Education
Judith Eaton
President
Council for Higher Education Accreditation
Richard Ekman
President
Council of Independent Colleges
Keston Fulcher
Director of the Center for Assessment and
Research Studies
Associate Professor, Graduate Psychology
James Madison University
Paul Gaston, III
Trustees Professor
Kent State University
Mildred Garcia
President
California State University,
Fullerton
Susan Johnston
Executive Vice President
Association of Governing Boards
Norman Jones
Professor
Utah State University
Peggy Maki
Higher Education Consultant
George Mehaffy
Vice President for
Academic Leadership and Change
American Association of State Colleges and
Universities
Lynn Pasquerella
President
Association of American Colleges &
Universities
George Pernsteiner
President
State Higher Education Executive Officers
Association
Mary Ellen Petrisko
President
WASC Senior College and University
Commission
Kent Phillippe
Associate Vice President, Research and
Student Success
American Association of Community Colleges
Robert Sheets
Research Professor
George Washington Institute of Public Policy
Ralph Wolff
Founder and President
The Quality ASsurance Commons for Higher
and Postsecondary Education
Ex-Officio Members
Peter Ewell
President Emeritus
National Center for Higher Education
Management Systems
Stanley Ikenberry
President Emeritus and Regent Professor
University of Illinois
Natasha Jankowski
Director, NILOA
Research Assistant Professor
George Kuh
Founding Director, National Institute for
Learning Outcomes Assessment
Adjunct Research Professor, University of
Illinois Urbana-Champaign
Chancellor’s Professor of Higher Education
Emeritus, Indiana University
Paul Lingenfelter
President Emeritus
State Higher Education Executive Officers
NILOA Mission
NILOA’s primary objective is to
discover and disseminate ways that
academic programs and institutions
can productively use assessment data
internally to inform and strengthen
undergraduate education, and exter-
nally to communicate with policy
makers, families and other stake-
holders.
NILOA Occasional Paper Series
NILOA Occasional Papers
are commissioned to examine
contemporary issues that will inform
the academic community of the
current state-of-the art of assessing
learning outcomes in American higher
education. The authors are asked to
write for a general audience in order
to provide comprehensive, accurate
information about how institutions and
other organizations can become more
proficient at assessing and reporting
student learning outcomes for the
purposes of improving student learning
and responsibly fulfilling expectations
for transparency and accountability
to policy makers and other external
audiences.
Comments and questions about this
paper should be sent to:
niloa@education.illinois.edu
National Institute for Learning Outcomes Assessment | 22
About NILOA
• The National Institute for Learning Outcomes Assessment (NILOA) was estab-
lished in December 2008.
• NILOA is co-located at the University of Illinois and Indiana
University.
• The NILOA website contains free assessment resources and can be found at http://
www.learningoutcomesassessment.org/.
• The NILOA research team has scanned institutional websites, surveyed chief
academic officers, and commissioned a series of occasional papers.
• NILOA’s Founding Director, George Kuh, founded the National Survey for
Student Engagement (NSSE).
• The other co-principal investigator for NILOA, Stanley Ikenberry, was president
of the University of Illinois from 1979 to 1995 and of the American Council of
Education from 1996 to 2001.
NILOA Staff
NATIONAL INSTITUTE FOR LEARNING OUTCOMES ASSESSMENT
Stanley Ikenberry, Co-Principal Investigator
George Kuh, Founding Director, Senior Scholar, and Co-Principal Investigator
Natasha Jankowski, Director
Gianina Baker, Assistant Director
Filip Przybysz, Communications Coordinator
Katie Schultz, Project Manager
Peter Ewell, Senior Scholar
Pat Hutchings, Senior Scholar
Jillian Kinzie, Senior Scholar
Paul Lingenfelter, Senior Scholar
David Marshall, Senior Scholar
Erick Montenegro, Research Analyst
Verna F. Orr, Research Analyst
Anthony B. Sullers, Jr., Research Analyst
Emily Teitelbaum, Research Analyst
Terry Vaughan III, Research Analyst
Karie Brown-Tess, Research Analyst
NILOA Sponsors
Lumina Foundation for Education
University of Illinois, College of Education
Produced by Creative Services | Public Affairs at the University of Illinois for NILOA. 10.032
National Institute for Learning Outcomes Assessment | 23
National Institute for Learning Outcomes Assessment
For more information, please contact:
National Institute for Learning Outcomes Assessment (NILOA)
University of Illinois at Urbana-Champaign
360 Education Building
Champaign, IL 61820
learningoutcomesassessment.org
niloa@education.illinois.edu
Phone: 217.244.2155
k n o w l e d g e a c c o u n t a b i l i t y c o n n e c t i o n s e l f – r e f l e c t i o n e d u c a t e a c t i o n u n d e r s t a n d c o m m u n i c a t e l i s t e n l e a r n a c c e s s q u a l i t y i n n o v a t i o n s u c c e s s i n g e n u i t y
i n t e l l e c t c u r i o s i t y c h a l l e n g e c r e a t e a c h i e v e m e n t c o n n e c t i o n s e l f – r e f l e c t i o n e d u c a t e a c t i o n u n d e r s t a n d c o m m u n i c a t e l i s t e n l e a r n a c c e s s q u a l i t y i n n o v a t i o n
s u c c e s s i n g e n u i t y i n t e l l e c t c u r i o s i t y c h a l l e n g e k n o w l e d g e a c c o u n t a b i l i t y c o n n e c t i o n u n d e r s t a n d c o m m u n i c a t e l i s t e n l e a r n a c c e s s q u a l i t y i n n o v a t i o n s u c c e s s
i n g e n u i t y s e l f – r e f l e c t i o n e d u c a t e a c t i o n u n d e r s t a n d i n t e l l e c t k n o w l e d g e a c c o u n t a b i l i t y c o n n e c t i o n s e l f – r e f l e c t i o n e d u c a t e a c t i o n u n d e r s t a n d c o m m u n i c a t e
c u r i o s i t y c h a l l e n g e c r e a t e a c h i e v e m e n t c o n n e c t i o n s e l f – r e f l e c t i o n c u r i o s i t y c h a l l e n g e c r e a t e a c h i e v e m e n t c o n n e c t i o n s e l f – r e f l e c t i o n k n o w l e d g e a c c o u n t a b i l i t y
c o n n e c t i o n s e l f – r e f l e c t i o n e d u c a t e a c t i o n u n d e r s t a n d c o m m u n i c a t e l i s t e n l e a r n a c c e s s q u a l i t y i n n o v a t i o n s u c c e s s i n g e n u i t y i n t e l l e c t c u r i o s i t y c h a l l e n g e e d u c a t e
i n n o v a t i o n s u c c e s s i n g e n u i t y i n t e l l e c t c u r i o s i t y c h a l l e n g e c r e a t e a c h i e v e m e n t k n o w l e d g e a c c o u n t a b i l i t y c o n n e c t i o n s e l f – r e f l e c t i o n e d u c a t e a c t i o n u n d e r s t a n d
c o m m u n i c a t e c u r i o s i t y c h a l l e n g e c r e a t e a c h i e v e m e n t c o n n e c t i o n s e l f – r e f l e c t i o n u n d e r s t a n d c o m m u n i c a t e l i s t e n l e a r n a c c e s s q u a l i t y a c t i o n e d u c a t e a c t i o n
u n d e r s t a n d c o m m u n i c a t e l i s t e n l e a r n a c t i o n u n d e r s t a n d c o m m u n i c a t e l i s t e n l e a r n a c c e s s q u a l i t y i n n o v a t i o n s u c c e s s i n g e n u i t y i n t e l l e c t c u r i o s i t y c h a l l e n g e
k n o w l e d g e a c c o u n t a b i l i t y c o n n e c t i o n a c c e s s q u a l i t y s e l f – r e f l e c t i o n c u r i o s i t y c h a l l e n g e c r e a t e a c h i e v e m e n t l e a r n a c c e s s q u a l i t y i n n o v a t i o n s u c c e s s i n g e n u i t y
s e l f – r e f l e c t i o n e d u c a t e a c t i o n u n d e r s t a n d i n t e l l e c t k n o w l e d g e a c c o u n t a b i l i t y c o n n e c t i o n s e l f – r e f l e c t i o n e d u c a t e a c t i o n u n d e r s t a n d k n o w l e d g e a c c o u n t a b i l i t y
c o n n e c t i o n s e l f – r e f l e c t i o n e d u c a t e a c t i o n u n d e r s t a n d c o m m u n i c a t e l i s t e n l e a r n a c c e s s q u a l i t y i n n o v a t i o n s u c c e s s i n g e n u i t y i n t e l l e c t c u r i o s i t y c h a l l e n g e
c o n n e c t i o n k n o w l e d g e a c c o u n t a b i l i t y c o n n e c t i o n s e l f – r e f l e c t i o n e d u c a t e a c t i o n u n d e r s t a n d c o m m u n i c a t e l i s t e n l e a r n a c c e s s q u a l i t y i n n o v a t i o n s u c c e s s
i n g e n u i t y c h a l l e n g e c r e a t e a c h i e v e m e n t c o n n e c t i o n s e l f – r e f l e c t i o n e d u c a t e a c t i o n u n d e r s t a n d c o n n e c t i o n s e l f – r e f l e c t i o n u n d e r s t a n d c o m m u n i c a t e l i s t e n l e a r n
a c c e s s q u a l i t y a c t i o n c r e a t e a c h i e v e m e n t c o n n e c t i o n s e l f – r e f l e c t i o n e d u c a t e a c t i o n u n d e r s t a n d c o m m u n i c a t e l i s t e n l e a r n a c c e s s q u a l i t y i n n o v a t i o n s u c c e s s
e d u c a t e a c t i o n c o m m u n i c a t e l i s t e n l e a r n a c c e s s q u a l i t y a c t i o n e d u c a t e a c t i o n u n d e r s t a n d c o m m u n i c a t e e d u c a t e i n n o v a t i o n s u c c e s s s e l f – r e f l e c t i o n k n o w l e d g e
a c c o u n t a b i l i t y c o m m u n i c a t e l i s t e n l e a r n a c h i e v e m e n t c o n n e c t i o n s e l f – r e f l e c t i o n e d u c a t e a c t i o n u n d e r s t a n d c o m m u n i c a t e l i s t e n l e a r n a c c e s s q u a l i t y i n n o v a t i o n
s u c c e s s i n g e n u i t y i n t e l l e c t a c c e s s q u a l i t y i n n o v a t i o n s u c c e s s s e l f – r e f l e c t i o n c u r i o s i t y c h a l l e n g e c r e a t e a c h i e v e m e n t c o n n e c t i o n s e l f – r e f l e c t i o n u n d e r s t a n d
e d u c a t e a c t i o n u n d e r s t a n d c o m m u n i c a t e l i s t e n l e a r n a c t i o n u n d e r s t a n d c o m m u n i c a t e l i s t e n l e a r n a c c e s s q u a l i t y i n n o v a t i o n s u c c e s s i n g e n u i t y c u r i o s i t y
c h a l l e n g e c r e a t e a c h i e v e m e n t c o n n e c t i o n s e l f – r e f l e c t i o n u n d e r s t a n d c o m m u n i c a t e l i s t e n l e a r n a c c e s s q u a l i t y a c t i o n c r e a t e a c h i e v e m e n t c o n n e c t i o n s e l f –
r e f l e c t i o n e d u c a t e a c t i o n u n d e r s t a n d c o m m u n i c a t e l i s t e n l e a r n a c c e s s q u a l i t y i n n o v a t i o n s u c c e s s e d u c a t e a c t i o n c o m m u n i c a t e l i s t e n l e a r n a c c e s s q u a l i t y
a c t i o n e d u c a t e a c t i o n u n d e r s t a n d c r e a t e a c h i e v e m e n t c o n n e c t i o n s e l f – r e f l e c t i o n u n d e r s t a n d c o m m u n i c a t e l i s t e n l e a r n a c c e s s q u a l i t y a c t i o n c r e a t e a c h i e v e m e n t
c o n n e c t i o n s e l f – r e f l e c t i o n e d u c a t e a c t i o n u n d e r s t a n d c o m m u n i c a t e l i s t e n c o m m u n i c a t e e d u c a t e i n n o v a t i o n s u c c e s s s e l f – r e f l e c t i o n k n o w l e d g e a c c o u n t a b i l i t y
c o n n e c t i o n s e l f – r e f l e c t i o n e d u c a t e a c t i o n u n d e r s t a n d c o m m u n i c a t e l i s t e n l e a r n a c c e s s q u a l i t y i n n o v a t i o n i n g e n u i t y i n t e l l e c t c o n n e c t i o n s e l f – r e f l e c t i o n
u n d e r s t a n d c o m m u n i c a t e l i s t e n l e a r n a c c e s s q u a l i t y a c t i o n c r e a t e a c h i e v e m e n t c o n n e c t i o n s e l f – r e f l e c t i o n e d u c a t e a c t i o n u n d e r s t a n d c o m m u n i c a t e l i s t e n l e a r n