W#15 Health Promotion

Assignment: Faculty and Students as Vulnerable Populations: Can We Predict Violence with a Screening Tool?

Read the following articles (Article #1 and Article #2, See attachments), and watch the video noted below: 

Video:  https://hls.ted.com/talks/2681.m3u8?preroll=newshortintro_053119&qr

Note: If you are not able to watch the video, please see the attached document with the “Video transcript”.

Articles attached:

• Article # 1:  An Analysis of US School Shootings (1840–2015) (Paradice, 2017) 
• Article # 2:  Development of the SaFETy Score: A Clinical Screening Tool for Predicting Future Firearm Violence Risk (Goldstick et al., 2017).

Answer three of the following questions in your initial post. (Please include at least three scholarly sources within your initial post.) 

1. Has your community been directly affected by gun violence/mass shootings? Discuss the affects you have noted. 
2. Discuss what you believe to be the greatest risk factor(s) in your community that could be associated with a potential for violence.
3. What are your thoughts about the Predictive Firearm Violence Risk Scale?
4. Will you use this scale in your practice? Why or why not?
5. What type of changes has your community made to help improve safety?
6. What are your thoughts about the Columbine shooter’s mother’s TED Talk?

Note: I live in South Florida, Miami-Dade County.

INSTRUCTIONS

– Please include at least 3 scholarly sources within your initial post.

– APA style.

(This is not a turnitin assignment)

Note: My background for you to have as a reference: I am currently enrolled in the Psych Mental Health Nurse Practitioner Program, I am a Registered Nurse. I work in a Psychiatric Hospital. 

health promotionadvanced nursing

My Son was a Columbine Shooter. This is My Story Video Transcript

Sue Klebold: The last time I heard my son’s voice was when he walked out the front door on his way to school. He called out one word in the darkness, “Bye.” It was April 20th, 1999. Later that morning, at Columbine high school, my son Dylan and his friend Eric killed 12 students and a teacher and wounded more than 20 others before taking their own lives. 13 innocent people were killed leaving their loved ones in a state of grief and trauma. Others sustained injuries, some resulting in disfigurement and permanent disability. But the enormity of the tragedy can be measured only by the number of deaths and injuries that took place. There’s no way to quantify the psychological damage of those who were in the school or who took part in rescue or cleanup efforts. There’s no way to assess the magnitude of a tragedy like Columbine, especially when it can be a blueprint for other shooters who go on to commit atrocities of their own.

Columbine was a tidal wave and when the crash ended, it would take years for the community and for society to comprehend its impact. It has taken me years to try to accept my son’s legacy. The cruel behavior that defined the end of his life showed me that he was a completely different person from the one I knew. Afterwards, people asked, “How could you not know? What kind of a mother were you?”I still ask myself those same questions. Before the shootings, I thought of myself as a good mom. Helping my children become caring, healthy, responsible adults was the most important role in my life. But the tragedy convinced me that I failed as a parent and it’s partially this sense of failure that brings me here today.

Aside from his father, I was the one person who knew and loved Dylan the most. If anyone could have known what was happening, it should have been me. Right? But I didn’t know. Today, I’m here to share the experience of what it’s like to be the mother of someone who kills and hurts. For years after the tragedy, I comb through memories trying to figure out exactly where I failed as a parent, but there are no simple answers. I can’t give you any solutions. All I can do is share what I have learned.

When I talk to people who didn’t know me before the shootings, I have three challenges to meet. First, when I walk into a room like this, I never know if someone there has experienced loss because of what my son did. I feel a need to acknowledge the suffering caused by a member of my family who wasn’t here to do it for himself. So first, with all of my heart, I’m sorry if my son has caused you pain.

The second challenge I have is that I must ask for understanding and even compassion when I talk about my son’s death as a suicide. Two years before he died, he wrote on a piece of paper in a notebook that he was cutting himself. He said that he was in agony and wanted to get a gun so he could end his life. I didn’t know about any of this until months after his death. When I talk about his death as a suicide, I’m not trying to downplay the viciousness he showed at the end of his life. I’m trying to understand how his suicidal thinking led to murder. After a lot of reading and talking with experts, I’ve come to believe that his involvement in the shootings was rooted not in his desire to kill, but in his desire to die.

The third challenge I have when I talk about my son’s murder-suicide is that I’m talking about mental health, excuse me. Excuse me. Is that I’m talking about mental health or brain health as I prefer to call it because it’s more concrete and in the same breath I’m talking about violence. The last thing I want to do was to contribute to the misunderstanding that already exists around mental illness. Only a very small percent of those who have a mental illness are violent toward other people. But of those who die by suicide, it’s estimated that about 75% to maybe more than 90% have a diagnosable mental health condition of some kind.

As you all know very well, our mental healthcare system is not equipped to help everyone and not everyone with destructive thoughts fits the criteria for a specific diagnosis. Many who have ongoing feelings of fear or anger or hopelessness are never assessed or treated. Too often, they get our attention only if they reach a behavioral crisis. If estimates are correct, that about 1% to 2% of all suicides involves the murder of another person. When suicide rates rise as they are rising for populations, than murder-suicide rates will rise as well.

I wanted to understand what was going on in Dylan’s mind prior to his death. So I look for answers from other survivors of suicide loss. I did research and volunteered to help with fundraising events and whenever I could, I talked with those who had survived their own suicidal crisis or attempt. One of the most helpful conversations I had was with a co-worker who overheard me talking to someone else in my office cubicle. She heard me say that Dylan could not have loved me if he could do something as horrible as he did.

Later when she found me alone, she apologized for overhearing that conversation, but told me that I was wrong. She said that when she was a young single mother with three small children, she became severely depressed and was hospitalized to keep her safe. At the time, she was certain that her children would be better off if she died, so she had made a plan to end her life. She assured me that a mother’s love was the strongest bond on earth and that she loved her children more than anything in the world. But because of her illness, she was sure that they would be better off without her.

What she said, and what I’ve learned from others is that we do not make the so-called decision or choice to die by suicide in the same way that we choose what car to drive or where to go on a Saturday night. When someone is in an extremely suicidal state, they are in a stage four medical health emergency. Their thinking is impaired and they’ve lost access to tools of self-governance. Even though they can make a plan and act with logic, their sense of truth is distorted by a filter of pain through which they interpret their reality. Some people can be very good at hiding this state and they often have good reasons for doing that.

Many of us have suicidal thoughts at some point, but persistent ongoing thoughts of suicide and devising a means to die are symptoms of pathology. And like many illnesses, the condition has to be recognized and treated before life is lost. But my son’s death was not purely a suicide, it involved mass murder. I wanted to know how his suicidal thinking became homicidal. But research is sparse and there are no simple answers. Yes, he probably had ongoing depression. He had a personality that perfectionistic and self-reliant and that made him less likely to seek help from others. He had experienced triggering events at the school that left him feeling debased and humiliated and mad. He had a complicated friendship with a boy who shared his feelings of rage and alienation and who was seriously disturbed, controlling, and homicidal.

On top of this period in his life of extreme vulnerability and fragility, Dylan found access to guns even though we’d never owned any in our home. It was appallingly easy for a 17-year-old boy to buy guns, both legally and illegally without my permission or knowledge. And somehow, 17 years and many school shootings later, it’s still appallingly easy.

What Dylan did that day broke my heart and as trauma so often does, it took a toll on my body and on my mind. Two years after the shootings, I got breast cancer. Two years after that, I began to have mental health problems. On top of the constant perpetual grief, I was terrified that I would run into a family member of someone Dylan had killed or be accosted by the press or by an angry citizen. I was afraid to turn on the news, afraid to hear myself being called a terrible parent or a disgusting person. I started having panic attacks.

The first bout started four years after the shootings when I was getting ready for the depositions and would have to meet the victims’ families face to face. The second round started six years after the shootings when I was preparing to speak publicly about murder-suicide for the first time at a conference. Both episodes lasted several weeks. The attacks happened everywhere in the hardware store, in my office, or even while reading a book in bed. My mind would suddenly lock into this spinning cycle of terror and no matter how hard I tried to calm myself down or reason my way out of it, I couldn’t do it. It felt as if my brain was trying to kill me and then being afraid of being afraid consumed all of my thoughts. That’s when I learned firsthand what it feels like to have a malfunctioning mind. And that’s when I truly became a brain health advocate.

With therapy and medication and self-care, life eventually returned to whatever could be thought of as normal under the circumstances. When I look back on all that had happened, I could see that my son’s spiral into dysfunction probably occurred over a period of about two years. Plenty of time to get him help. If only someone had known that he needed help and known what to do.

Every time someone asks me, “How could you not have known?” It feels like a punch in the gut. It carries accusation and taps into my feelings of guilt that no matter how much therapy I’ve had, I will never fully eradicate. But here’s something I’ve learned. If love were enough to stop someone who was suicidal from hurting themselves, suicides would hardly ever happen. But love is not enough and suicide is prevalent. It’s the second leading cause of death for people aged 10 to 34 and 15% of American youth report having made a suicide plan in the last year.

I’ve learned that no matter how much we want to believe we can, we cannot know or control everything our loved ones think and feel. And the stubborn belief that we are somehow different, that someone we love would never think of hurting themselves or someone else can cause us to miss what’s hidden in plain sight. And if worst-case scenarios do come to pass, we’ll have to learn to forgive ourselves for not knowing or for not asking the right questions or not finding the right treatment. We should always assume that someone we love may be suffering regardless of what they say or how they act. We should listen with our whole being without judgment and without offering solutions.

I know that I will live with this tragedy, with these multiple tragedies, for the rest of my life. I know that in the minds of many, what I lost can’t compare to what the other families lost. I know my struggle doesn’t make theirs any easier. I know there are even some who think I don’t have the right to any pain, but only to a life of permanent penance. In the end, what I know comes down to this, the tragic fact is that even the most vigilant and responsible of us may not be able to help, but for love’s sake, we must never stop trying to know the unknowable. Thank you.

AN ANALYSIS OF US SCHOOL SHOOTING DATA
(1 8 4 0 – 2 0 1 5)

D a v i d Pa r a d i c e
Auburn University

This paper describes the construction and descriptive analysis o f a data
set o f United States school shooting events. Three hundred forty-three
shooting events are included, spanning 175 years o f United States ed-
ucational history. All levels o f US educational institution are includ-
ed. Events are included when a firearm is discharged, regardless o f
whether an injury occurs. The analysis defines a mass shooting as an
event in which four or more persons, excluding the shooter, are injured
or killed. It defines a mass murder as an event in which four or more
persons, excluding the shooter, are killed. The data reveals that US
high schools are where most shooting events occur. Relatively speak-
ing, there have been few mass murder events in US campuses, but
they have occurred with much greater frequency in the last 50 years.
In most cases, shootings are premeditated. No prescription related to
firearms at educational institutions is made.

Keywords: school shooting, mass shooting, mass murder, shooter,
firearms on campus, campus carry

Introduction

On January 9, 2011, police were called to
a fraternity house at Florida State University
at 1:16 AM to respond to an accidental shoot-
ing report. A student was showing his rifle to
friends when the rifle accidentally discharged.
The bullet went through the chest o f one
student, killing her, and hit a second student
in the wrist. On November 19, 2014, on the
same campus, a mentally ill former student
went on a shooting rampage at the library,
injuring three students, including one who
is permanently paralyzed. Police responded
within minutes, killing the shooter.

Advocates against allowing firearms on
college campuses point to the 2011 event,
and others like it, as evidence enough that
allowing firearms on a college campus is
a bad policy. They argue that alcohol and
youthful recklessness mix together to form
a deadly combination that leads to tragedy.
Advocates in favor o f allowing firearms on

college campuses point to the 2014 event, and
others like it, as examples o f situations that
could have been ended faster if citizens were
allowed to carry firearms on campus.

The National Conference o f State Legis-
latures maintains information on concealed
carry weapon laws and college campuses
(h ttp ://w w w .n csl.o rg /rese arc h /e d u catio n /
guns-on-campus-overview.aspx), but policy
in this area is particularly difficult to construct
because emotions run high on this issue. On
the one hand, proponents o f “campus carry”
laws point to the Second Amendment as
confirming a right to bear arms. On the oth-
er hand, faculty, administration, and police
opposition to allowing firearms to be carried
on some campuses has been significant. On
the one hand, accidental shootings only occur
when guns are present. On the other hand, a
person who is properly trained in firearm use
can only use a firearm for protection from a
shooter when allowed to carry the firearm.

135

136/ Education Vol. 138 No. 2

A wide range o f scholarly articles have
been published on topics related to school
shooting events. The legal aspects o f firearm
possession on campus, school zones and
the Second Amendment (Arnold, 2015) and
historical perspectives that review American
university policies in the 1700’s and 1800’s
(Cramer, 2014) have been examined. As-
mussen and Creswell (1995) have studied
how a campus reacts to a shooting event, and
Weiler and Armenta (2014) surveyed school
principals to assess their feelings about arm-
ing school personnel. Neuman et al. (2015)
present a way to profile school shooters using
automatic text-based analysis o f their writing.
Towers et al. (2015) found statistically signif-
icant results that indicate high profile school
shootings lead to more o f the same.

Some scholarly articles focus on the most
violent events (see Kalish and Kimmel, 2010),
which certainly warrant study, but they do not
describe the most common type o f shooting
event that occurs at an educational institution.
Some scholarly articles read more like opin-
ion pieces, containing unsupported statements
such as this:

Most mass shooters are young men or
occasionally women – usually teens –
who are emotionally unstable and want
to exact revenge on society fo r some
harm that they have suffered (real or
imaginary), commit suicide in a blaze
o f gun fire, and get national media at-
tention fo r their last act. (Nedzel, 2014)

In fact, the data do not support most o f
the implicit assertions in this excerpt. Most
shooters are men, but only in high schools are
most o f the shooters teens. Few have been di-
agnosed as emotionally unstable; most shoot-
ers are just angry about something. Twice as
many mass shooters are arrested at or near
the scene o f the shooting as complete suicide.
When asked why they did what they did, they
rarely mention a desire for media attention.

The quote above exposes another prob-
lem. What is a “mass shooting”? The terms
“mass shootings”, “mass killings”, and “mass
murders” are used almost interchangeably
and the definitions vary.

In their study o f school shooter offender
and offense characteristics, Gerard et al.
(2016) defined a school shooting event as
an attack by someone against two or more
victims with at least one firearm on school
grounds. They went on to write:

This broad definition will be adopted in
order to include as many cases as pos-
sible in the sample; because o f the rare
nature o f school shooting incidents,
fe w cases are available.

This is a European-based study, but these
researchers included US data (79% of their
events occurred in the US). They were able to
find only 28 cases of shooting events between
1988 and 2009 that met their criteria. By com-
parison, the data set in this study contains 48
events that meet their criteria. All o f them are
US events. This discrepancy highlights the
lack o f a systematic review of these events.

What has been missing is a comprehen-
sive, data-driven analysis o f what actually
occurs in shooting events at educational in-
stitutions. The National School Safety Cen-
ter (http://www.schoolsafety.us/media-re-
sources/school-associated-violent-deaths)
published a report summarizing school
violence during the academic years from
1992-1993 to 2009-2010, but it is only a list
o f the events. The CDC has data on school
associated student homicides from roughly
the same period (1992-2006) and also pro-
duces other relevant reports on source o f
weapons and warning signs (http://www.
cdc.gov/violenceprevention/youthviolence/
schoolviolence/savd.html).

This study was undertaken to provide
in one place a single accounting o f school
shooting events at educational institutions.

An Analysis Of Us School Shooting Data (1840 – 2015) /13 7

No prior analysis has focused on describing
the shooting events at educational institutions
that have occurred over a long period o f time
in a consistent manner.

Method

Constmcting a dataset that completely
documents every shooting at an educational
institution is an impossible task. There is
simply no way to identify every such event
that has occurred. When one searches for data
related to “school shootings” or some similar
term, one inevitably comes across one o f the
many variations o f a list o f school shooting
events that begins with a 1764 Indian raid
on a school house. Such data has been used,
even when not explicitly referenced (see, for
example, Duplechain and Morris, 2014). This
study began with one o f those variations, the
list o f shooting events that exists on Wikipe-
dia’s “List o f school shootings in the United
States” entry. The earliest event described in
this list occurred in 1764, but the rest o f the
events in the list occurred in or after 1840.

Entries in the Wikipedia list are not de-
scribed in a consistent manner. Some entries
contain detail about the event while other
entries only indicate that the event occurred.
However, most o f the entries in the list con-
tain a reference to an online data source. In
most cases, that data source is a newspaper ar-
ticle. In order to construct the data used in this
study, the author read all o f the data sources
connected to the Wikipedia entries. The au-
thor also read any other information (e.g.,
court documents) that could be located on
older events. Recent events that are especially
violent often result in hundreds o f articles. No
attempt was made to read all o f the articles
written about those events.

An event is included in the data set as long
as a firearm was discharged in an educational
institution or on its grounds, regardless of the
number o f people wounded or killed, with a
few exceptions. Eleven o f the events included

in the Wikipedia list were discarded because
they described shootings by police (e.g., in re-
sponse to a crime) or other authorities (e.g., the
Kent State University shootings by National
Guardsmen). Four events on the Wikipedia
list were discarded because they occurred at
school board meetings. Twenty-three shoot-
ing events were excluded because something
in the information about them indicated they
were not reasonably related to anything that
would be considered normal educational
campus-related activity or they did not ac-
tually occur on a campus. For example, the
1764 raid by Lanape Indians on a Pennsylva-
nia schoolhouse was removed from the data
set because it is not reasonably relevant to
understanding shooting events at educational
institutions. Similarly, an 1858 shooting “in
the woods near the city” where a school cele-
bration was occurring was removed (not on a
campus), a 1910 shooting o f two schoolboys
who were sledding in a park was removed
(not on a campus), a 1935 suicide by an
administrator was removed (not related to
educational activity and occurred outside
school hours), the 1959 arrest o f twenty-sev-
en men and boys in gang-related activity was
removed (no shots were fired), and the 2014
shooting death in the early morning hours o f
a 16-year-old boy in a middle school parking
lot was removed (occurred in the middle o f
the night), among others.

Some events known to the author were
researched and added to the data set. Other
events were added as they were found, but
there was no concentrated effort to identify all
missing events. In the end, 343 events from
November 12, 1840 through December 31,
2015 are included in the data set. Thirty o f
these events occurred before January 1, 1900.
Table 1 contains descriptions o f the data items
that were sought for each event.

138 / Education Vol. 138 No. 2

Table 1. Data items

Item D escription

Date Date o f the event

Month Month part o f date (for analysis)

Day o f Week

Location

Day part o f date (for analysis)

City, State where event occurred

State State part o f location (for analysis)

Institution
ES = Elementary School, MS = Middle School, HS = High School
SH = School house (commonly used in 1800’s news articles)
U = University (includes all post-high school institutions)

Teachers Killed Number o f teachers killed in the event

Teachers Injured

Students Killed

Number o f teachers injured in the event

Number o f students killed in the event

Students Injured

Others Killed

Number o f students injured in the event

Number o f others killed in the event

Others Injured

Number o f Shooters

Number o f others injured in the event

Number o f shooters

Shooter Gender Male, Female, Unknown

Shooter Age

Shooter Outcome

Age(s) o f shooter(s)

A range o f values such as Arrested, Convicted, Suicide, Identified, Escaped, Killed, etc.

Related to Location
No = the shooter docs not have some prior relationship to the educational institution
Yes = the shooter has some prior relationship to the educational institution

Deaths Total number o f deaths from the event (excluding shooter)

Injuries

Dead + Injured

Total number o f injuries from the event (excluding shooter)

Total number o f deaths and injuries from the event (excluding shooter)

Type

Accident
Premeditated = the shooter came to the institution with an intent to shoot
Spontaneous = the shooter was armed, but did not come to the institution with an intent to
shoot
Unknown

Weapon Handgun, rifle, shotgun, or unknown (combinations arc possible)

Delay
Hours, Next Day, Days Later, or Unknown. I f Type = Premeditated, this is how much time
elapsed from the event that triggered the intent to shoot until the actual shooting event.

Category I The primary reason for the shooting. Values include Accident, Anger, Dispute, Domestic,
Fight, Gang, Self-Defense, etc.

Category II
A secondary reason for the shooting. For example, when Anger is the designation in Cate-
gory I the values include Harassment, Dismissal, Discipline, Revenge, etc.

Comment A brief description o f the event

Sources Links to sources

An Analysis Of Us School Shooting Data (1840 – 2015) /13 9

Many o f the data items in the set are
self-explanatory, but a few warrant additional
information. For example, Shooter Outcome
includes “Identified” as a possible value. In
many o f the early newspaper accounts (i.e.,
mid- to late 1800’s) a shooter would be identi-
fied (or would certainly be known to witness-
es o f the event) but no additional information
about the event or shooter could be found.
Here is a representative example from The
Penny Press (Cincinnati, January 21, 1860):

One School Boy Shoots Another Dead.
– A son o f Col. Elijah Sebree, o f Todd
County, Ky., was killed in the school-
house, at Trenton, a fe w days since.
The boys o f the school had been prac-
ticing upon the credulity and fears o f
one o f their number, by inducing him
to believe young Sebree had been mak-
ing threats against him, and intended
to kill him, whereupon the lad armed
him self and walked deliberately up to
Sebree, in the school-house, and shot
him dead.

In this case, the witnesses to the event
would have been able to identify the shooter.
However, no other mention o f the event could
be found that contained more details. Whether
the shooter was arrested or otherwise held ac-
countable for his action is unknown.

This particular event demonstrates the
value in studying the descriptions o f the event
when it happened. The paragraph above is
the entire newspaper article about the event.
Similar descriptions can be found in other
archives. However, one can also find this
unreferenced description on the web https://
skeptic78240.w ordpress.com /2015/01/28/
govemors-acting-brilliantly-2/:

Todd County. Kentucky. A son o f Col.
Elijah Sebree was shot dead by another
student. Young Sebree was threatening
the other boy and intended to kill him.

Notice that in this version the young Se-
bree was threatening the shooter, a vastly
different scenario than the one described in
greater detail in the newspaper at the time o f
the event.

The Type data item also warrants further
description. The value Premeditated indicates
the shooter brought the weapon to the school
with a purposeful intent to shoot someone or
something. The value Spontaneous indicates
the shooter was carrying a weapon, but did
not have a purpose for using it prior to some
provocation immediately prior to the shooting
event. Premeditated events are characterized
by some prior event that leads up to the
shooting. The choice to carry a weapon used
at Spontaneous events is also typically moti-
vated by some prior event or experience, but
the shooter did not come to campus that day
with an intent to shoot a specific person. In the
spontaneous event, the shooter is responding
to the occurrence o f an event in that moment.

Two examples can illustrate the values.
The following excerpt comes from a Pre-
meditated event from 1960. In this event,
14-year-old Donna Dvorak shot 15-year-old
Bobby Whitford who she believes has threat-
ened a classmate. These statements come
from the article (https://news.google.com/
new spapers?id=jthH A A A A IBA J& sjid=Io-
A M A A A A IB A J& pg=3467,122520&dq=d-
vorak&hl-en):

Deputy Sheriff Bob Miller said that
Donna Dvorak, 14, a petite blonde,
stood up at her desk in the back o f the
room and fired at the Whitford boy,
who sat in the fro n t row about 25 fe e t
away.

The chief deputy said that Kate McCoy,
also 15, told him she had been threat-
ened by Whitford. Miller said the girl s
mother quoted Whitford as vowing “I f
Kate w o n ’t go out with me she w on’t go
out with anyone. ”

140 / Education Vol. 138 No. 2

Miller quoted Donna as saying she
shot the youth because he had threat-
ened Kate.

The following excerpt comes from a
1968 shooting event that was classified as
Spontaneous:

A bullet struck down Linda Lipscomb,
16, in a corridor at Miami Jackson
[High School]. Blanche Ward, also 16,
was held fo r a juvenile hearing.

Detective Tony Fontana said Blanche
told officers Linda threatened her with
a razor during an argument over the
fountain pen and her gun went o ff
during a struggle.

In addition, the following convention was
maintained with respect to criminal activity.
If a shooting occurred in the commission o f
a crime, an attempt was made to determine
whether the perpetrator o f the crime intended
to shoot someone in committing the crime.
One could argue that bringing a weapon
to a crime is a premeditated action and any
resultant shooting should also be considered
premeditated. However, this analysis is an
attempt to distinguish between purposeful
shootings and other shootings. Thus, if a
weapon was discharged while the shooter
was fleeing (or attempting to flee) a crime
scene, for example, the event was placed in
the Spontaneous category. On the other hand,
i f a shooting occurs in the commission o f the
crime (e.g., an armed robbery), then the event
is classified as a Premeditated shooting.

The data items Category I and Category II
are used to document factors underlying the
shooting event. With respect to these factors,
the value “mental illness” was assigned only
when documentation existed that a shooter
had been, or was currently being, treated for
mental illness; or when a shooter was found
not guilty due to an insanity defense; or when
a shooter was found incompetent to stand

trial. The value “mental illness” was assigned
only in Category I. If there was some sug-
gestion o f mental illness, but not enough to
meet the threshold just described, a secondary
factor (Category II) o f “mental issues” was
assigned. The value “mental issues” was as-
signed only in Category II.

The final data set contains information
on 343 shooting events. The majority o f the
shooting events occur in high schools (168
events), followed by university shooting
events (77), middle school shooting events
(25), and elementary school shooting events
(18). Fifty-five shooting events occurred at
a “school house,” a commonly used term in
the late 1800’s and a term used to describe the
location when a more accurate description is
not possible (e.g., a location described origi-
nally as a boarding school or a prep school).

Analysis

In many cases, the data for a specific
event is incomplete. The information may
provide great detail for some o f the data
items but have no information on other data
items. Thus, the percentages given in the
analysis are based on different underlying
population sizes. The population size is giv-
en in parentheses in each case.

When all o f the data is considered
(n=343), there have been 22 shooting events
(6%) in which no one was killed or injured. In
129 o f the shooting events (38%) no one was
killed. In these events where no one is killed,
the shooter is arrested 79 times (61%), es-
capes 14 times (11%), and completes suicide
twelve times (9%), attempts suicide two times
and is killed at the scene two times. In these
non-lethal events the shooter is identified nine
times (7%) but no further determination can
be made about the shooter from the available
information (all o f these events occurred in
1920 or earlier). In ten cases the shooter out-
come is unknown; in three cases the shooter
outcome was not determined due to other

An Analysis Of Us School Shooting Data (1 8 4 0 -2 0 1 5 ) /141

circumstances (e.g., no injury or the shooter
being extremely young).

In total (n=343), there have been 420
shooting deaths and 558 shooting injuries at
educational institutions since 1840. Sixty-one
percent (61%) o f the shootings are classified
as premeditated events. Nineteen percent
(19%) are classified as spontaneous and 8%
are accidental. The rest, 12%, could not be
classified.

There are 277 events in which the weapon
can be identified. Although the weapon in
some events is almost certainly a handgun
(e.g., the news account indicates the shooter
“drew” a weapon), the weapon is classified as
“unknown” in the cases where it is not ade-
quately described for classification. In most
events (n=207), only a handgun is used (75%
o f the events where a weapon is identified).
In 11% o f the events only a rifle is used and
in 9% of the events only a shotgun is used.
In the remaining cases, multiple weapons are
used. For example, a handgun is used with
another weapon in twelve shooting events,
bringing (total) handgun use up to 79% of the
events in which a weapon can be identified.
Rifles are used in 16% o f the shooting events
and shotguns are used in 11% o f the shooting
events. All three of these weapons are used in
only one event.

There is no commonly accepted definition
o f the term “mass shooting.” The Congressio-
nal Research Office defines a mass shooting as
an event in which 4 or more people, excluding
the shooter, are killed. This echoes the FBI
definition for “mass murder.” Advocates for
gun control argue that the term “mass shoot-
ing” should include all victims o f a shooting
event, not just the ones that are murdered.

This study excludes the shooter in all cal-
culations related to deaths or injuries. In an
effort to maintain impartiality (with respect to
gun control advocacy), both “mass murder”
and “mass shooting” analyses are made in
this study. A “mass murder” event is defined

as an event in which at least four people died.
A “mass shooting” event is defined in this
study as a shooting event in which at least
four people are injured (where “injury” can
include death).

There are 20 mass murder events resulting
in 180 deaths and 208 injuries in the data set.
While the number o f mass murder shooting
events accounts for only 6% o f the shooting
events in the data set, they account for 43% o f
the deaths and 37% o f the injuries. From 1840
until 1966, only three mass murder events oc-
cur at an educational institution (resulting in
14 deaths, 4 injuries). A turning point in mass
murder shootings on educational campuses
occurs in 1966, when Charles Whitman,
an engineering student at the University o f
Texas, went on a shooting rampage killing
19 and injuring 28 before he was killed by
responding police officers. Starting with that
event and counting through the end o f 2015,
there are 17 mass murder events that result in
166 deaths and 204 injuries. Put another way,
85% o f the mass murder shooting events have
occurred since 1966, resulting in 92% o f the
mass murder-related deaths and 98% of the
mass murder-related injuries.

There is a general perception that shoot-
ers involved in mass murder events complete
suicide. The data tells us that in 11 shooting
events the shooter completes suicide (55% o f
the events) and in one other shooting event
the shooter attempted to complete suicide. In
4 o f these cases (20%), the shooter is taken
into custody by law enforcement. A common
misconception, however, is that the shooters
in these mass murder events want to die in
some type o f shoot-out with law enforcement.
In fact, in only 2 o f the mass murder events
(10%) was the shooter killed by responders.
There were 2 cases where the outcome o f
the shooter could not be determined. Both o f
these occurred in the 1890’s.

The data provides some insights into the
factors underlying these mass murder events.

142 / Education Vol. 138 No. 2

In seven o f them (35%), the primary factor
(Category I) is “anger.” In five o f the events
(25%) the factor is “mental illness”, in two
events (10%) the factor is “attention”, in two
(10%) it is “fight”, in three of them (15%)
there is some indication o f “domestic” issues,
and one event (5%) was categorized as racial-
ly motivated.

Mass murder events are rare in elemen-
tary school environments. There have been
two o f them, resulting in 31 deaths and 34
injuries. The Sandy Hook shooting in 2012
accounts for 26 o f the deaths. A 1989 event
in California accounts for 32 o f the injuries.
There has been one mass murder event in a
middle school (5 deaths, 10 injuries) and one
in a school house (5 deaths, 5 injuries). Five
mass murders have occurred in high schools
(34 deaths, 54 injuries), with the Columbine
shooting accounting for 13 o f the deaths. Uni-
versities have witnessed nine mass murders
(95 deaths, 103 injuries). The shooting at Vir-
ginia Tech in 2007 resulted in 32 deaths and
23 injuries and the shooting at the University
o f Texas in 1966 resulted in 19 deaths and 28
injuries. These two events account for most
o f the carnage on university campuses that
occurred in a single event.

There are 54 mass shooting events (an in-
crease o f 170% over the mass murder total) re-
sulting in 220 deaths (which is 40 more deaths
added to the mass murder-related death count –
a 22% increase) and 378 injuries (an additional
170 injuries added to the mass murder-related
injury count – an 82% increase) in the data set.
From 1840 until 1966, only five mass shooting
events occur at an educational institution (15
deaths, 21 injuries). Considering the data from
the time o f the 1966 University o f Texas shoot-
ing event forward, there are 49 mass shooting
events. They result in 205 deaths and 357 in-
juries. Four mass shootings have occurred in
elementary schools, five in school houses, four
in middle schools, twenty-five in high schools,
and sixteen in universities.

As a point o f contrast, the shooting events
that are not mass shooting events have the
following characteristics. Using the 1966
University o f Texas event as a reference
point for consistency, there are 92 non-mass
shooting events from 1840 until August, 1966
that result in 74 deaths and 37 injuries. From
August, 1966 through the end o f 2015, there
are 197 non-mass shooting events that result
in 126 deaths and 143 injuries.

High schools are where the most non-mass
shooting events occur, with 143 o f them (49%
o f all non-mass shooting events) occurring at
those locations. Fourteen non-mass shooting
events have occurred in elementary schools,
twenty-one in middle schools, fifty in school
houses, and sixty-one in universities.

Seventy-nine percent (79%) of the deaths
related to shooting events at educational insti-
tutions have occurred since the University o f
Texas shooting event in August, 1966. Ninety
percent (90%) of all injuries due to shooting
events at educational institutions have oc-
curred since that watershed event in 1966.

Where identifying information about the
shooter’s relationship to the campus exists
(n=305), in 91% o f the events the shooter
has some relationship to the campus. Where
information about the shooter’s identity can
be determined (n=318), 70% o f the shooters
are students or former students. Twenty-one
percent (21%) o f the shooters are some adult
other than a teacher or parent. Five percent
(5%) are teachers and 2% are parents. In 2%
o f the shooting events, the shooter is a youth
who is otherwise unidentifiable as a student
(e.g., gang-related activity).

Looking at students as shooters (n=223)
further, in six events a student (or former stu-
dent) was the shooter at an elementary school.
The age o f these shooters ranges from six to
sixteen. A student was the shooter in seven-
teen shooting events at middle schools. At
school houses, twenty-nine shooting events
involved a student shooter. One hundred

An Analysis Of Us School Shooting Data (1840 – 2015) /143

thirty-five high school shooting events in-
volve student shooters. Thirty-six university
shooting events involve student shooters.
Notably, students are shooters in only 50% o f
the shooting events at universities where the
shooter could be classified. In 10% of those
events, the shooter was a teacher; in 40 per-
cent o f the events the shooter is another adult.

The youngest shooter in the data set is 6
years old. This is Dedrick Owens, who found
a .32 caliber handgun in his uncle’s home,
brought it to school, and shot classmate Kayla
Rolland. She died that morning in a hospital.
The oldest shooter is 70-year-old James Fer-
guson, who in 1891 fired a shotgun at a group
o f students in the playground of a parochial
school in New York. He caused minor injuries
to several o f the students.

In 328 o f the shooting events where the
gender o f the shooter can be determined
(n=337), the shooter is male (97%). In the
nine shooting events where the shooter is
female, the ages vary from 14 to 46. One fe-
male shooter’s age could not be determined,
but she was an adult. In eight o f the nine
shooting events involving women shooters,
the event was classified as a premeditated
shooting event.

The primary factor for most shootings is
“anger”, “fight”, and “dispute” (combined).
This combination o f values for Category I ac-
counts for 185, or 61% o f the shooting events
where a factor can be identified (n=304).
These types o f events account for 54% of all
the shooting events in the data set. A second-
ary factor (Category II) in these cases sheds
more light on the situation. Twenty-five o f
these 185 shooting events are related to dis-
cipline (14%), nineteen to harassment (10%),
eighteen to dismissal (or failure or a bad
grade, 10%), fourteen to revenge (8%), seven
to romance (4%) and four to some domestic
issue (i.e., domestic abuse or some other do-
mestic issue) (2%). In 39 o f the 343 events
(11%) a cause could not be identified. Eight

percent (8%, n=29) of the events are acci-
dents. Seven percent (7%, n=25) are classified
as mental illness cases. Six percent (n=12) o f
the shooting events are classified as criminal
activity. Five percent (n=17) are classified as
gang activity. The rest result from miscella-
neous factors such as alcohol consumption,
seeking attention, self-defense, and so forth.

When considering the “anger”, “fight”,
and “dispute” factors further, one finds that
96 (or 52%) o f the 185 events characterized
this way occurred in high schools. O f those 96
shooting events, 17 are related to disciplinary
measures (17%), 14 to harassment (15%), 7
to revenge (7%), 6 to romance (6%), and 4 to
dismissal (or failure or a bad grade, 4%).

Mental illness is listed as a primary factor
(Category I) in 25 o f the 304 shooting events
where a factor was identified, accounting for
8% o f these events (or 7% o f all shooting
events in the data set). Domestic issues are
the primary factor in 15 shooting events (5%
o f identifiable factor events; 4% of all events
in the data set.

What happens to the shooter can be de-
termined in 324 shooting events. In over half
(56% or 193 events) o f the shooting events, the
shooter is taken into custody by law enforce-
ment. In 33 shooting events (10%) the shooter
escapes and typically is not ever known. In 30
events (9%) the shooter is identified, but one
cannot determine from the information avail-
able whether the shooter is arrested. Most o f
these cases occur before 1920, but a few o f
them are more recent. In 45 shooting events
(14%), the shooter completes a suicide.

In elementary school shooting events
where the shooter completes suicide, the
shooter is more likely to be an adult. In two
middle school shooting events the shooter
completed a suicide attempt and in both cases
the shooter was a student. Shooter suicide
is common in high school shooting events
(n= 15) and university shooting events (n=21).
The high school shooters who complete

144 / Education Vol. 138 No. 2

suicide are more likely to be students (n=13)
than in other educational institutions. By
comparison, there were 10 student shooters
in universities who completed suicide (out of
21 shooters who completed suicide). In four
more events (1%), the shooter attempts sui-
cide. In eight events (2%) the shooter is killed
at the scene o f the shooting. The rest of the
time the shooter outcome data item is “NA”,
typically due to the event being accidental.

Accidental shootings have happened in
almost every decade for 150 years. The first
o f the 29 events placed in the Accident Type
category occurred in 1867. In this event, a
13-year-old boy brought a pistol to a New
York City school to shoot a dog that the boy
claimed had bitten him. While playing with
the gun, he accidentally shot and injured a
classmate. The last accident in the data set
occurred in 2012, when a third-grade girl
was shot. In that event, a fellow student had
brought a handgun to school in his backpack.
The gun discharged when the boy dropped
the backpack. The young girl was treated for
6 weeks in a hospital after surgery to remove
a bullet from her spine. Accidents account
for 17 deaths and 16 injuries. Fourteen o f the
accidental shootings occurred in high schools;
only one occurred at a university. In 19 o f the
29 accidents, a handgun was involved.

Conclusion

This study describes shooting events at ed-
ucational institutions in an objective manner.
The data was compiled from news accounts
o f the events, along with the use o f court doc-
uments and other primary source materials
when found. The study does not seek to ad-
vocate for or against a specific policy related
to the possession o f firearms at educational
institutions. Instead, the author is hopeful that
the study will inform decision makers and
policy makers in a way that resources can be
allocated wisely.

References
Arnold G. (2015) Arming the Good Guys: School Zones

and the Second Amendment, BYU Education and
Law Journal, pp. 481-506.

Asmussen K J and Creswell J W (1995) Campus Re-
sponse to a Student Gunman, The Journal o f Higher
Education, Vol. 66, No. 5, pp. 575-591.

Cramer CE (2014) Guns on Campus: A History, Academ-
ic Questions, 27, pp. 411 -425.

Duplcchain R and Morris R (2014) School Violence: Re-
ported School Shootings and Making Schools Safer,
Education, 135(2), pp. 145-150.

Weiler SC and Armenta AD (2014) The Fourth R – Re-
volvers: Principal Perceptions Related to Armed
School Personnel and Related Legal Issues, The
Clearing House, 87, pp. 115-118.

Neuman Y, A ssaf D, Cohen Y, Knoll JL (2015) Profiling
School Shooters: Automatic Text-Based Analy-
sis, Frontiers in Psychology, 6(86): doi: 10.3389/
fpsyt.2015.00086

Towers S, Gomez-Licvano A, Khan M, Mubayi A, Castil-
lo-Chavez C (2015) Contagion in Mass Killings and
School Shootings. PLoS ONE 10(7): c0117259.doi:
10.1371/joumal.ponc.0117259

Kalish R and Kimmel M (2010) Suicide by Mass Murder:
Masculinity, Aggrieved Entitlement, and Rampage
School Shootings, Health Sociology Review, 19(4),
pp. 451464.

Ncdzel NE (2014) Concealed Carry: The Only Way to
Discourage Mass School Shootings, Academic Ques-
tions, 27, pp. 4 2 9435.

Gerard FJ, Whitfield KC, Porter LE, Browne KD (2016)
Offender and Offence Characteristics o f School
Shooting Incidents, Journal o f Investigative Psychol-
ogy and Offender Profiling, 13, pp. 22-38

URLs
National Conference o f State legislatures (http://www.

ncsl.org/research/education/guns-on-cam pus-over-
view.aspx),

The National School Safety Center (http://
w w w . s c h o o l s a f c t y . u s / m e d i a – r e s o u r c e s /
school-associated-violent-deaths)

CDC (http://www.cdc.gov/violenceprevcntion/youthvio-
lence/schoolviolcncc/savd.html).

URL (https://ncws.google.com/newspapers? id=
vB o M A A A A IB A J& sjid = v lw D A A A A IB A J& p –
g=3895,165958&dq=lipscomb&hl=en)

Copyright of Education is the property of Project Innovation, Inc. and its content may not be
copied or emailed to multiple sites or posted to a listserv without the copyright holder’s
express written permission. However, users may print, download, or email articles for
individual use.

Development of the SaFETy Score: A Clinical Screening Tool for
Predicting Future Firearm Violence Risk
Jason E. Goldstick, PhD; Patrick M. Carter, MD; Maureen A. Walton, MPH, PhD; Linda L. Dahlberg, PhD;
Steven A. Sumner, MD, MSc; Marc A. Zimmerman, PhD; and Rebecca M. Cunningham, MD

Background: Interpersonal firearm violence among youth is a
substantial public health problem, and emergency department
(ED) physicians require a clinical screening tool to identify high-
risk youth.

Objective: To derive a clinically feasible risk index for firearm
violence.

Design: 24-month prospective cohort study.

Setting: Urban, level 1 ED.

Participants: Substance-using youths, aged 14 to 24 years,
seeking ED care for an assault-related injury and a proportion-
ately sampled group of non–assault-injured youth enrolled from
September 2009 through December 2011.

Measurements: Firearm violence (victimization/perpetration)
and validated questionnaire items.

Results: A total of 599 youths were enrolled, and presence/
absence of future firearm violence during follow-up could be
ascertained in 483 (52.2% were positive). The sample was ran-
domly split into training (75%) and post–score-construction vali-
dation (25%) sets. Using elastic-net penalized logistic regression,
118 baseline predictors were jointly analyzed; the most predic-
tive variables fell predominantly into 4 domains: violence victim-

ization, community exposure, peer influences, and fighting. By
selection of 1 item from each domain, the 10-point SaFETy (Se-
rious fighting, Friend weapon carrying, community Environment,
and firearm Threats) score was derived. SaFETy was associated
with firearm violence in the validation set (odds ratio [OR], 1.47
[95% CI, 1.23 to 1.79]); this association remained (OR, 1.44 [CI,
1.20 to 1.76]) after adjustment for reason for ED visit. In 5 risk
strata observed in the training data, firearm violence rates in the
validation set were 18.2% (2 of 11), 40.0% (18 of 45), 55.8% (24
of 43), 81.3% (13 of 16), and 100.0% (6 of 6), respectively.

Limitations: The study was conducted in a single ED and
involved substance-using youths. SaFETy was not externally
validated.

Conclusion: The SaFETy score is a 4-item score based on clini-
cally feasible questionnaire items and is associated with firearm
violence. Although broader validation is required, SaFETy shows
potential to guide resource allocation for prevention of firearm
violence.

Primary Funding Source: National Institute on Drug Abuse
R01024646.

Ann Intern Med. 2017;166:707-714. doi:10.7326/M16-1927 Annals.org
For author affiliations, see end of text.
This article was published at Annals.org on 11 April 2017.

Firearm violence has been identified by health andlegal professionals as a critical public health prob-
lem (1). Homicide is the third leading cause of death in
the United States among youth aged 15 to 24 years,
with more than 86% of these deaths due to firearms (2).
Furthermore, firearm violence results in substantial
monetary cost; for example, medical and work-loss
costs of nonfatal firearm injuries treated in U.S. emer-
gency departments were estimated to exceed $2.9 bil-
lion in 2010 (3). Mitigating this public health issue
requires novel hospital and community-based interven-
tions that are focused on at-risk youth, especially those
in urban communities. Urban emergency departments
(EDs) have been identified as a critical access point for
identifying and intervening with such youth (4). Firearm
violence encompasses interpersonal, self-directed, and
unintentional firearm-related incidents, but in this study
we focus on interpersonal firearm violence, which we
refer to simply as “firearm violence” throughout.

Although previous ED-based research (5) has iden-
tified risk factors associated with firearm violence in-
volvement among high-risk youth, the field of hospital
and ED-based youth violence prevention programs
lacks a short, clinically relevant screening tool that can
be applied as part of routine clinical care in urban set-
tings. Such a tool could play a key role in determining

where to focus prevention or intervention efforts. Youth
identified during an ED visit, particularly violently in-
jured youth, are at elevated risk for future firearm vio-
lence (5) and thus would benefit most from early inter-
vention, including case management and therapeutic
services. Previous screening tools for youth violence
(6 – 8) primarily focused on primary care settings, lack a
specific focus on firearm violence, or are too lengthy for
practical use in a busy ED setting. Furthermore, re-
search on the construction of violence screening tools
(6, 8) has been limited by small sample sizes and has
not considered out-of-sample predictive power in de-
vising the screen. Developing an ED/hospital-based
clinical screening tool that is focused on assessing risk
for future firearm violence will enable ED and hospital
health systems to better focus prevention resources on
patients at the highest risk.

In the current study, we seek to develop a clinical
screening tool for future risk for firearm violence by ex-

See also:

Web-Only
Supplement

Annals of Internal Medicine ORIGINAL RESEARCH

© 2017 American College of Physicians 707

amining data collected as part of a 2-year prospective
study of youth aged 14 to 24 years seeking ED care.
First, we used machine learning methods to determine
which variables measured at the baseline of a 2-year
longitudinal study were most predictive of future fire-
arm violence. Second, on the basis of breadth and clin-
ical feasibility, we selected 4 items from among the
most predictive variables. Third, we developed cut-
points and assigned point values to each level based
on their relative effects, resulting in the SaFETy (Serious
fighting, Friend weapon carrying, community Environ-
ment, and firearm Threats) score. Finally, we examined
the relationship between the SaFETy score and rates of
future firearm violence within training and internal vali-
dation data sets.

METHODS
Study Design and Setting

Data were collected during the Flint Youth Injury
study (9 –11), a 2-year prospective cohort study of
assault-injured youth (age 14 to 24 years) with any drug
use in the past 6 months and a comparison group of
non–assault-injured, drug-using youth seeking ED care
at a level 1 trauma center in Flint, Michigan. The parent
study focused on service needs and utilization among
substance (predominantly marijuana) users. Although
this potentially limits generalizability, we note that most
youth who seek care for assault injuries in this setting
are substance users (9). Patients were recruited from
December 2009 through September 2011, 24 hours
per day on Thursday through Monday and from 5 a.m.
to 2 a.m. on Tuesday and Wednesday. Youth who
sought care for sexual assault, child abuse, suicidal ide-
ation or attempt, or any conditions that preclude con-
sent (such as altered mental status) were excluded. In-
stitutional review boards at the University of Michigan
and Hurley Medical Center approved the study. A Na-
tional Institutes of Health (NIH) Certificate of Confiden-
tiality (COC) was obtained.

Potential participants were ascertained through
electronic patient logs and approached by research as-
sistants in waiting or treatment areas. All assault-injured
youth, including those who were initially unstable but
stabilized with 72 hours of presentation, were ap-
proached and screened for study eligibility. In se-
quence, the next available age group (14 to 17, 18 to
20, and 21 to 24 years) and sex-matched, non–assault-
injured ED entrant was screened for the comparison
group. Those providing consent (or assent with paren-
tal consent for those younger than age 18 years) pri-
vately self-administered the screening survey using a
tablet device and received a $1.00 gift for participation.
Individuals who self-reported drug use in the past 6
months (98% used marijuana) were considered eligible
and consented to the subsequent 2-year longitudinal
study. Appendix Figure 1 (available at Annals.org)
shows a flow chart of the original study. Remunera-
tion was $20 for completion of a subsequent self-
administered baseline survey. Follow-up assessments
were conducted at 6, 12, 18, and 24 months, and par-

ticipants were compensated $35, $40, $40, and $50 for
each sequential follow-up. Baseline characteristics (9)
and 2-year outcomes (5, 10) are reported elsewhere.

Measures
The following measures were assessed:
The outcome variable was a binary indicator of fire-

arm violence (victimization, perpetration, firearm injury
requiring medical care, or firearm death) during the 24-
month follow-up period, ascertained through a com-
posite of self-report, medical chart review, and vital re-
cords databases (see Carter and colleagues [5] for
greater detail). Both peer and partner firearm violence
was included. Although the dynamics of peer and part-
ner violence differ, we justify combining them by noting
the large overlap between victims (12, 13) and perpe-
trators (14 –16) of peer and partner violence.

Candidate predictor variables were taken from
baseline self-report surveys; in addition to age, sex, and
reason for ED visit (assault-injured/non–assault-injured),
we included 115 survey items. Other variables that
were measured but judged unlikely to be assessed ac-
curately and truthfully (for example, serious violence
perpetration) without an NIH Certificate of Confidenti-
ality were not considered. See the Supplement (avail-
able at Annals.org) for question wording and response
options for all items described below.

1. Violence items (13 items) from the National Lon-
gitudinal Study on Adolescent Health (17) captured the
frequency of received threats/violence, perpetrated
threats, fighting, and carrying a weapon while intoxi-
cated in the past 6 months.

2. Partner aggression (13 items) was assessed with
Conflict Tactics Scale items (18), which measured the
frequency of partner violence victimization in the past
6 months.

3. Nonpartner aggression (13 items) was assessed
with questions modified from the Conflict Tactics Scale
(18), measuring the frequency of nonpartner violence
victimization in the past 6 months.

4. Community violence exposure (5 items) included
assessment of the frequency of exposure to violence
and neighborhood crime in the past 6 months (19).

5. Mental health (12 items) was measured with the
Brief Symptom Inventory checklist (20), which assessed
severity of depression and anxiety in the past week.

6. Drug and alcohol efficacy (16 items) assessed
confidence in avoiding drug (8 items) or alcohol (8
items) use in various situations (21, 22).

7. Alcohol use (10 items) was assessed with the Al-
cohol Use Disorders Identification Test (AUDIT), which
measures the frequency of alcohol consumption and
alcohol-related consequences in the past 6 months
(23, 24).

8. Peer influences (11 items) included items from
the Flint Adolescent Study (25) regarding the number
of friends providing positive (4 items) and negative (7
items) influences; positive items were reverse coded.

9. Parental behavior (10 items) included items from
the Flint Adolescent Study (25) assessing parental sup-

ORIGINAL RESEARCH Development of the

SaFETy Score

708

Annals of Internal Medicine • Vol. 166 No. 10 • 16 May 2017 Annals.org

port (6 items) and level of parental drug/alcohol use (4
items); parental support was reverse coded.

10. Retaliatory attitudes (7 items) included items as-
sessing willingness to engage in violent retaliation;
higher scores indicate greater willingness (26, 27).

11. Fight self-efficacy (5 items) assessed perceived
ability to avoid conflicts (28).

Statistical Analysis
We first randomly split the data into a training set

(75%) and a validation set (25%), ensuring that the
prevalence of firearm violence was equivalent in each.
In determining variable importance, developing cut-
points, and assigning point values, we used only the
training data. Multiple imputation by chained equations
(29), implemented in the R statistical software package
mice (29), was used to impute missing outcomes (there
were no missing values in the predictors measured at
baseline) for inclusion in the training data. Variable im-
portance analyses and determination of risk score con-
tributions were based on estimates pooled across 5

0

multiply imputed data sets.

To determine variable importance, we used elastic-
net penalized logistic regression, a common machine
learning approach to binary classification, which shrinks
the regression coefficient estimates to improve out-of-
sample prediction. The elastic net (30) penalizes both
the absolute value of the coefficients, which performs
automatic variable selection by shrinking irrelevant co-
efficients to zero (31), and the squared size of the coef-
ficient, which limits the effect of collinearity (32). The
elastic net is particularly applicable when the number
of predictors is large relative to the sample size and
when interest lies in variable selection wherein the ef-
fect of collinearity is reduced (30). To evaluate out-of-
sample prediction accuracy, we used leave-one-out
cross-validation (LOOCV). In LOOCV, the model is fit to
the entire sample, minus 1 point, and is used to predict
the excluded case; this is repeated so each point is left
out once. The level of coefficient shrinkage that mini-
mizes the LOOCV error rate was chosen. Model fitting
and cross-validation were performed by using the R
statistical software package glmnet (33).

After determining the optimally penalized model,
we ranked predictor importance by using the size of
the standardized regression coefficient estimates. Us-
ing the variable importance rankings and the variable
domains, we selected 4 variables that were predictive
and covered distinct content. We justify the use of 4
items by noting that exploration with 3 item scores (not
shown) indicated that the 4-item score yielded a more
thorough risk gradient (in the training data), and this
was not notably improved by adding a fifth item; be-
cause clinical feasibility is paramount, item scores of 6
or more were not explored. Cut-points for each item
were chosen by cycling through all possible categori-
zations of each variable, fitting a logistic regression
model of future firearm violence (in the training set)
using the categorized variables, and choosing the cat-
egorization that minimized the finite-sample– corrected
Akaike information criteria (34). To avoid overfitting the

training data, a maximum of 3 categories were consid-
ered for each variable and overly small categories (<

20

people) were not considered. Finally, we determined
score contributions by 1) entering the categorized pre-
dictors into a single logistic regression model and 2)
scaling by the minimum regression coefficient and
rounding to an integer (as in reference 35). Properties
of the risk score in the validation set (sensitivity, speci-
ficity, and odds ratio [OR] with firearm violence) were
examined and stratified by assault-injured/non–assault-
injured group.

Role of the Funding Source
Our funding sources had no role in the design,

conduct, or analysis of our study or the decision to sub-
mit the manuscript for publication.

RESULTS
Sample Characteristics

In total, 599 youth (349 assault-injured and
250 non–assault-injured) participated in the study.
Follow-up rates were 85.3%, 83.7%, 84.2%, and 85.3%
at 6, 12, 18, and 24 months. Among participants, 483
(80.6%) could be definitively classified as having been
involved with firearm violence (n = 252 [52.2%]) or not
(n = 231 [47.8%]) during the follow-up period. Of the
participants, 57.3% were male and 62.5% were African
American; the average age at baseline was 19.9 years
(SD, 2.4 years); greater detail is published elsewhere
(36). One fourth of those with (n = 63) and those with-
out (n = 58) firearm violence were randomly removed
for post–score-construction validation; there were no
significant demographic differences between the train-
ing and validation data.

Variable Importance Analysis
Appendix Figure 2 (available at Annals.org) shows

the relative size of the top 20 largest standardized co-
efficients. Table 1 lists the selected items and their uni-
variate associations with firearm violence. Selected
items largely fell into 4 domains that were observed
post hoc: 1) violence victimization (peer and partner),
2) community violence exposure, 3) peer/family influ-
ences, and 4) fighting.

Risk Score Construction
We narrowed the 20 variables to 4 items from dif-

ferent domains to construct the SaFETy score. From vi-
olence victimization, we chose the highest-ranking
item: being threatened with a firearm. For practical use
this could be combined with the number 2–ranked and
number 9 –ranked items into “threatened or shot you”
because of similar content. “Friend weapon carrying”
was chosen from the peer influence domain. Among
the community violence items, we chose the lower-
ranking item—frequency of hearing gunshots— because
of the greater likelihood of a truthful response in a clin-
ical setting (compared with the frequency of seeing
someone shot). Similarly, we chose “frequency of being
in a serious fight” over “frequency of putting someone
in the hospital” from the fighting domain.

Development of the SaFETy Score ORIGINAL RESEARCH

Annals.org Annals of Internal Medicine • Vol. 166 No. 10 • 16 May 2017

709

Table 2 shows the derived cut-points and weights
for each category, and how the selected items corre-
spond to the SaFETy mnemonic. “Received firearm
threats” was divided into “never,” “once,” and “2+
times”; “hearing gunshots” was divided into “less than
many times” and “many times”; “fight frequency” was
stratified into “never,” “1–5 times,” and “6+ times”; and
“friend weapon carrying” was collapsed into whether or
not “many, most, or all” friends carry weapons. The
largest weights corresponded to received firearm vio-
lence threats and high-frequency serious fighting.

Risk Score Performance
In the validation set, a 1-point increase in SaFETy

score corresponded to higher risk for firearm violence
(OR, 1.47 [95% CI, 1.23 to 1.79]). The area under the
receiver-operating characteristic curve was 0.73, indi-
cating reasonable out-of-sample discriminatory power.
The Figure shows the distribution of SaFETy scores
among those with and without firearm violence in the
validation set. Table 3 shows the sensitivity and speci-
ficity of SaFETy score in the validation set for each cut-
point between 0 and 10 (Appendix Table 1, available at
Annals.org, shows training sample results). Informal ex-
amination of the training data indicated 5 risk strata:

SaFETy scores of 0, 1 to 2, 3 to 5, 6 to 8, and 9 to 10;
the same risk gradient is apparent in the validation set
(Appendix Figure 3, available at Annals.org), with each
level corresponding to future firearm violence rates of
18.2% (2 of 11), 40.0% (18 of 45), 55.8% (24 of 43),
81.3% (13 of 16), and 100.0% (6 of 6), respectively.

Sensitivity Analysis
Because membership in the assault-injured group

is itself associated with future firearm violence (Appen-
dix Tables 2 and 3, available at Annals.org), we present
several stratified analyses. First, we conducted an om-
nibus test comparing the model with all 20 of the top
variables that do versus do not allow the effects to vary
by assault-injured/non–assault-injured group, which
yielded a nonsignificant result (P = 0.09). Appendix Ta-
bles 4 and 5 (available at Annals.org) show univariate
associations between each variable and firearm vio-
lence, stratified by assault-injured/non–assault-injured
group. Second, we estimated the joint effects of SaFETy
score and assault-injured group membership (Appen-
dix Table 3) and found that 1) with SaFETy score in-
cluded in the model, assault-injured group member-
ship was not significant and 2) the effect of SaFETy
score was similar after the inclusion of the assault-

Table 1. Highest-Ranked Prognostic Factors for Future Firearm Violence

Prognostic Factor Response
Type*

Importance
Rank

Timeframe Odds Ratio
(95% CI)†

Standardized
Odds Ratio‡

Received threats
Someone pulled a gun on you Frequency (0–6) 1 6 mo 2.44 (1.89–3.20) 2.90
Someone used a gun on you Frequency (0–6) 2 6 mo 3.31 (1.98–5.52) 2.22
Someone pulled a knife on you Frequency (0–6) 6 6 mo 1.89 (1.40–2.54) 1.81
Someone shot you Frequency (0–6) 9 6 mo 2.92 (1.69–5.06) 1.73
Someone threw something at you Frequency (0–6) 12 6 mo 1.58 (1.21–2.07) 1.58
Someone cut/stabbed you Frequency (0–6) 14 6 mo 2.22 (1.47–3.34) 1.73

Community
I have seen someone shot Frequency (0–3) 3 6 mo 1.87 (1.44–2.42) 1.78
I have heard guns shot Frequency (0–3) 5 6 mo 1.55 (1.27–1.90) 1.61
Seen gangs in neighborhood Frequency (0–3) 18 6 mo 1.31 (1.11–1.55) 1.

40

My house was broken into Frequency (0–3) 20 6 mo 1.70 (1.23–2.34) 1.4

5

Friends
My friends carry weapons Number (1–5) 10 Current 1.56 (1.26–1.92) 1.62
My friends smoke marijuana Number (1–5) 15 Current 1.27 (1.07–1.50) 1.34
Friend legal trouble (drug-related) Number (1–5) 17 Current 1.63 (1.25–2.12) 1.53

Partner violence
Partner used a knife on you Frequency (0–6) 13 6 mo 3.46 (1.46–8.24) 2.37

Fighting
Been in a serious fight Frequency (0–6) 4 6 mo 1.46 (1.25–1.71) 1.76
Put someone in the hospital Frequency (0–6) 7 6 mo 1.71 (1.35–2.16) 1.85
Drank before fighting Frequency (0–6) 8 6 mo 1.78 (1.33–2.38) 1.72

Other
Understand another’s point of view Agree (1–5) 11 6 mo 1.38 (1.16–1.64) 1.49
Today’s ED visit for violent injury Yes/no 16 Current 1.89 (1.23–2.89) NA
Unable to stop drinking Frequency (0–4) 19 6 mo 1.56 (1.10–2.20) 1.39

ED = emergency department; NA = not available.
* Frequency (0 – 6) measures frequency on a 7-point scale from 0 (never) to 6 (20+ times). Frequency (0 –3) measures frequency on a 7-point scale
from 0 (never) to 3 (many times). Frequency (0 – 4) measures frequency on a 5-point scale from 0 (never) to 4 (daily). Number (1–5) measures
frequency on a 5-point scale from 1 (none) to 5 (all). Agree (1–5) measures agreement on a 5-point scale from 1 (very true) to 5 (not true). Yes/no
denotes a binary (1/0) indicator.
† CIs with lower bounds of 1.00 are entirely above 1.00.
‡ Standardized odds ratios were those obtained by using the standardized predictors.

ORIGINAL RESEARCH Development of the SaFETy Score

710 Annals of Internal Medicine • Vol. 166 No. 10 • 16 May 2017 Annals.org

injured group. Appendix Tables 6 and 7 (available at
Annals.org) show frequency distributions of SaFETy
scores and sensitivity/specificity estimates, respectively,
stratified by assault-injured group (corresponding train-
ing sample results are shown in Appendix Tables 8 and
9, available at Annals.org).

DISCUSSION
In the current study, we used machine learning

methods and data from a prospective cohort of ED pa-
tients to identify factors predictive of future firearm vio-
lence that could be incorporated into a brief clinical
screening tool for ED use. This data set is unique in that
high-risk urban youth were followed successfully over 2
years and that the interviews capture not only firearm
violence that resulted in injury but incidents that re-
sulted in near-injury (for example, being threatened
with a gun). SaFETy clearly defines a gradient for future
firearm violence risk in this population; this steady in-
crease (rather than a sharp increase at 1 point) inhibits
determination of a threshold with strong combined
sensitivity/specificity but creates a strong basis for allo-
cation of prevention resources.

Coupling risk stratification with effective prevention
tools is an important potential use of the SaFETy score.
Very-high-risk individuals (e.g., those with a SaFETy
score !6) may represent sensible candidates for entry
into resource-intensive programs (for example, 1-year
wraparound programs), whereas individuals in the mid-
dle range (a SaFETy score !1 but “5) may benefit from
graduated levels of targeted interventions designed to
interrupt a negative trajectory. Programs focusing on
primary prevention may be appropriate for lower-
scoring (a SaFETy score of 0) individuals. Although

SaFETy has predictive power among both assault-
injured and non–assault-injured youth, it is most appli-
cable among non–assault-injured youth because there
is no other current means of stratifying risk in that
group. Furthermore, given the excess risk for future vic-
timization among those presenting with assault injury,
particularly firearm injuries (37), prevention resources
should also be considered for this group, even among
those with low SaFETy scores. Previous violence pre-
vention programs have been shown to be cost-effective
(38 – 42), specifically with regard to the costs of treating
repeated violent injuries (38, 39) and preventing incar-
ceration due to violence-related offenses (38, 40).
Given that the average ED visit for a firearm assault
costs $1200 and average inpatient costs approach
$24 000 (43), even a moderately effective prevention
program directed at individuals in higher-risk strata
would be cost-effective.

The items selected for the screening tool confer
strong face validity to the data-derived prediction tool.
Two items— history of receiving gun threats and hearing
gun violence in one’s community— confirm the impor-
tance of previous violence exposure in the risk for fu-
ture firearm violence. In addition, peer influences,
whose importance is most pronounced during the pe-
riod of adolescence and emerging adulthood (44), is
an important prognostic factor, in the form of friend
weapon carrying. The emergence of serious fighting as
a strong predictor agrees with prior violence screening
tools (7) and underscores the role of impulse control
and aggression in firearm violence. These results high-
light the broader importance of incorporating commu-
nity and peer factors into prevention programs, in ad-
dition to addressing psychological distress stemming

Table 2. Rules for Calculation of the SaFETy Score

Mnemonic Category Question/Scale Levels SaFETy
Contribution

S Serious Fighting In the past 6 mo, including today, how often did you get into a
serious physical fight?

0 (never) 0
1 (once) 1
2 (twice) 1
3 (3–5 times) 1
4+ (6 or more times) 4

F Friend Weapon Carrying How many of your friends have carried a knife, razor, or gun?
1 (none) 0
2 (some) 0
3+ (many, most, or all) 1

E Community Environment In the past 6 mo, how often have you heard guns being shot?
0 (never) 0
1 (once or twice) 0
2 (a few times) 0
3 (many times) 1

T Firearm Threats How often, in the past 6 mo, including today, has someone pulled
a gun on you?

0 (never) 0
1 (once) 3
2+ (twice or more) 4

SaFETy = Serious fighting, Friend weapon carrying, community Environment, and firearm Threats.

Development of the SaFETy Score ORIGINAL RESEARCH

Annals.org Annals of Internal Medicine • Vol. 166 No. 10 • 16 May 2017 711

from victimization history, consistent with trauma-
informed care practices (45).

We also note that male sex, which is consistently
identified as a risk factor for gun violence (5, 46, 47),
was not predictive of future gun violence. This suggests
that sex differences in firearm violence risk may be sub-
sumed by other risk exposure (such as affiliation with
high-risk peers). Similarly, depression and anxiety were
not identified as important prognostic factors, suggest-
ing that their association with firearm violence may also
be largely subsumed by proximal exposures to high-
risk community violence. We note that self-reported
ability to understand another’s point of view was highly
associated with future gun violence but was not se-
lected for the final screen because of concerns about
its accurate assessment and wording that may be mis-
understood in a brief clinical screen.

The results here are encouraging because they rely
only on items considered feasible to ascertain in a clin-
ical setting. The feasibility of screening on the basis of
such measures as “Have you shot someone in the last 6
months?” is limited when the respondents may not per-
ceive themselves as having the same degree of confi-
dentiality as in a clinical study with an NIH Certificate of
Confidentiality. In addition, that question is not likely to
be asked or answered in a way that is perceived as
nonjudgmental or nonincriminating by staff or patients.
Asking youth about peer behavior that they are more
likely to report—and is not incriminating— has been
done for other risk behavior tools, such as the National
Institute on Alcohol Abuse and Alcoholism alcohol
screening tool for adolescents (48). None of the mea-
sures selected specifically ask respondents to 1) incrim-

inate themselves, 2) incriminate any specific person, or
3) embarrass themselves. Although a superior predic-
tive tool may be derived by including a broader class of
measures, this gain may be offset by reduced willing-
ness to answer or response accuracy.

Our study had several limitations. First, our sample
is limited to a single urban ED; validation in other high-
risk populations is required. Second, the analytic sam-
ple is limited to drug-using youth. Although we cannot
ascertain the efficacy of this screen in non– drug-using
youth, we note that substance use has been linked to
both gun carrying (36) and violence (49) and that a
large majority of individuals screened into the study be-
cause of marijuana use. To mitigate this limitation, cli-
nicians could first inquire about marijuana use in the
past 6 months, a standard history and physical exami-
nation question. Alternatively, given the lack of valid
screening instruments for firearm violence and the
knowledge that this is the leading cause of death for
youth in urban communities, it would be clinically rea-
sonable to suggest that high-scoring urban youth, even
those who have not used marijuana in the past 6
months, warrant preventive services. However, future
research is needed to validate this tool among non–
substance-using youth.

Third, self-reported data were a large component
of identifying those with versus those without firearm
violence during the follow-up period, which is a con-
cern for underreporting of firearm violence. This limita-
tion is partly mitigated by the use of full validated
scales, such as the Conflict Tactics Scale (18), which
were privately administered on a tablet. Fourth, our
mental health assessments focused only on depression
and anxiety. Because such symptoms as suspicious-
ness, delusions, and extreme anger have been linked
to violence and gun carrying (50, 51), future work is
needed to assess their power to predict future gun vi-
olence. Finally, our missing-data imputation relies on
the untestable missing-at-random assumption. Noting
the high follow-up rate and that no covariates that
made up the SaFETy score differed significantly in
terms of missing versus nonmissing cases, the role of
nonrandom attrition was likely minimal.

In conclusion, we used machine learning methods
to determine the most important predictors of future

Figure. Distribution of SaFETy scores among youth with
and without firearm violence during the follow-up period
in the validation data.

0
0
5

10

D
is

tr
ib

u
ti

o
n
o

f
Sa

FE
Ty

S
co

re
s,

%

SaFETy Score

No firearm violence (n = 63)
Firearm violence (n = 58)

15

20

25

1 2 3 4 6 7 9 105

SaFETy = Serious fighting, Friend weapon carrying, community Envi-
ronment, and firearm Threats.

Table 3. Sensitivity and Specificity for SaFETy Score
Thresholds Between 1 and 10 in the Validation Set

Threshold Sensitivity, n/N (%)
(95% CI)

Specificity, n/N (%)
(95% CI)

1 61/63 (96.8 [88.0–99.4]) 9/58 (15.5 [7.8–27.9])
2 53/63 (84.1 [72.3–91.7]) 27/58 (46.6 [33.5–60.0])
3 43/63 (68.3 [55.2–79.1]) 36/58 (62.1 [48.3–74.2])
4 36/63 (57.1 [44.1–69.3]) 40/58 (69.0 [55.3–80.1])
5 32/63 (50.8 [38.0–63.5]) 48/58 (82.8 [70.1–91.0])
6 19/63 (30.2 [19.6–43.2]) 55/58 (94.8 [84.7–98.7])
7 7/63 (11.1 [5.0–22.2]) 57/58 (98.3 [89.5–100.0])
8 6/63 (9.5 [3.9–20.2]) 58/58 (100.0 [92.3–100.0])
9 6/63 (9.5 [3.9–20.2]) 58/58 (100.0 [92.3–100.0])
10 5/63 (6.3 [2.1–16.3]) 58/58 (100.0 [92.3–100.0])

SaFETy = Serious fighting, Friend weapon carrying, community Envi-
ronment, and firearm Threats.
ORIGINAL RESEARCH Development of the SaFETy Score

712 Annals of Internal Medicine • Vol. 166 No. 10 • 16 May 2017 Annals.org

firearm violence in a high-risk ED sample. This is the
first scale to provide risk stratification for firearm vio-
lence and the first developed in and specifically for an
ED setting (rather than primary care). Previous risk
scores were developed to predict related but distinct
behaviors, such as nonspecific violent injury (6, 8) and
firearm carrying (7). The common thread between
SaFETy and previous scales is the importance of fight-
ing (6 – 8) and received threats (7) as prognostic factors.
The SaFETy instrument, which can be administered in 1
to 2 minutes, defines a gradient of future firearm vio-
lence risk that can be adapted to a variety of settings.
Emergency departments have been previously used as
opportunities for identifying high-risk individuals for
other types of violence (52, 53), but the current lack of
an easily administered screening tool for firearm vio-
lence has limited our ability to harness the same oppor-
tunity for firearm violence. Our results suggest that
SaFETy fills this gap.

From University of Michigan School of Medicine and Univer-
sity of Michigan School of Public Health Ann Arbor, Michigan;
Centers for Disease Control and Prevention, Atlanta, Georgia;
and Hurley Medical Center, Flint, Michigan.

Disclaimer: The findings and conclusions in this manuscript
are those of the authors and do not necessarily represent the
official position of the Centers for Disease Control and
Prevention.

Acknowledgment: The authors thank the staff and patients of
Hurley Medical Center for their support of this project and
Wendi Mohl, BS, and Sonia Kamat, MS, for their assistance
with manuscript preparation.

Grant Support: National Institute on Drug Abuse grant R01
024646 (principal investigator, Rebecca M. Cunningham), 1
June 2009 to 30 April 2014; National Institutes of Health/Na-
tional Institute on Drug Abuse (NIDA) grants K23DA039341
(principal investigator: Patrick M. Carter) and 16IPA605200
(principal investigator: Jason Goldstick), 1 July 2016 to 30
June 2017.

Disclosures: Dr. Walton reports grants from NIDA during
the conduct of the study. Authors not named here have dis-
closed no conflicts of interest. Disclosures can also be viewed
at www.acponline.org/authors/icmje/ConflictOfInterestForms
.do?msNum=M16-1927

Reproducible Research Statement: Study protocol and data
set: Not available. Statistical code: Available from Dr. Gold-
stick (e-mail, jasoneg@umich.edu).

Requests for Single Reprints: Jason Goldstick, PhD, University
of Michigan Injury Center, University of Michigan School of
Medicine, 2800 Plymouth Road, NCRC G10-080, Ann Arbor,
MI 48109; e-mail, jasoneg@umich.edu.

Current author addresses and author contributions are avail-
able at Annals.org.

References
1. Weinberger SE, Hoyt DB, Lawrence HC 3rd, Levin S, Henley DE,
Alden ER, et al. Firearm-related injury and death in the United States:
a call to action from 8 health professional organizations and the
American Bar Association. Ann Intern Med. 2015;162:513-6. [PMID:
25706470] doi:10.7326/M15-0337
2. Centers for Disease Control and Prevention. Injury Prevention &
Control: Data & Statistiscs (WISQARS). Leading causes of death.
2015. Accessed at www.cdc.gov/injury/wisqars/leading_causes
_death.html on 6 June 2016.
3. Centers for Disease Control and Prevention. Data & Statistiscs
(WISQARS): Cost of Injury Reports. 2014. Accessed at https://wisqars
.cdc.gov:8443/costT on 23 June 2016.
4. Cunningham R, Knox L, Fein J, Harrison S, Frisch K, Walton M,
et al. Before and after the trauma bay: the prevention of violent injury
among youth. Ann Emerg Med. 2009;53:490-500. [PMID:
19162376] doi:10.1016/j.annemergmed.2008.11.014
5. Carter PM, Walton MA, Roehler DR, Goldstick J, Zimmerman MA,
Blow FC, et al. Firearm violence among high-risk emergency depart-
ment youth after an assault injury. Pediatrics. 2015;135:805-15.
[PMID: 25847808] doi:10.1542/peds.2014-3572
6. Sege R, Stringham P, Short S, Griffith J. Ten years after: examina-
tion of adolescent screening questions that predict future violence-
related injury. J Adolesc Health. 1999;24:395-402. [PMID:
10401967]
7. Hayes DN, Sege R. FiGHTS: a preliminary screening tool for ado-
lescent firearms-carrying. Ann Emerg Med. 2003;42:798-807. [PMID:
14634605]
8. Sigel E, Hart J, Hoffenberg A, Dodge M. Development and psy-
chometric properties of a violence screening tool for primary care.
J Adolesc Health. 2011;48:358-65. [PMID: 21402264] doi:10.1016
/j.jadohealth.2010.07.024
9. Cunningham R, Ranney M, Newton M, Woodhull W, Zimmerman
M, Walton M. Cunningham RM, Ranney M, Newton M, Woodhull W,
Zimmerman M, Walton MA. Characteristics of youth seeking emer-
gency care for assault injuries. Pediatrics. 2014;133:e96-105. [PMID:
24323994] doi:10.1542/peds.2013-1864;133(1):96-105.
10. Cunningham RM, Carter PM, Ranney M, Zimmerman MA, Blow
FC, Booth BM, et al. Violent reinjury and mortality among youth
seeking emergency department care for assault-related injury: a
2-year prospective cohort study. JAMA Pediatr. 2015;169:63-70.
[PMID: 25365147] doi:10.1001/jamapediatrics.2014.1900
11. Bohnert KM, Walton MA, Ranney M, Bonar EE, Blow FC, Zimmer-
man MA, et al. Understanding the service needs of assault-injured,
drug-using youth presenting for care in an urban Emergency Depart-
ment. Addict Behav. 2015;41:97-105. [PMID: 25452051] doi:10.1016/j
.addbeh.2014.09.019
12. Finkelhor D, Turner H, Hamby S, Ormrod R. Polyvictimization:
children’s exposure to multiple types of violence, crime, and abuse.
Office of Juvenile Justice and Delinquency Prevention and Centers
for Disease Control and Prevention. Juvenile Justice Bulletin. Na-
tional Survey of Children’s Exposure to Violence. 2011. Accessed at
www.ncjrs.gov/pdffiles1/ojjdp/235504 on 13 March 2017.
13. Hamby S, Grych J. The Web of Violence: Exploring Connections
among Different Forms of Interpersonal Violence and Abuse. New
York: Springer; 2013.
14. Klevens J, Simon TR, Chen J. Are the perpetrators of violence
one and the same? Exploring the co-occurrence of perpetration of
physical aggression in the United States. J Interpers Violence. 2012;
27:1987-2002. [PMID: 22328658] doi:10.1177/0886260511431441
15. Bossarte RM, Simon TR, Swahn MH. Clustering of adolescent
dating violence, peer violence, and suicidal behavior. J Interpers
Violence. 2008;23:815-33. [PMID: 18252941] doi:10.1177
/0886260507313950
16. Foshee VA, McNaughton Reyes L, Tharp AT, Chang LY, Ennett
ST, Simon TR, et al. Shared longitudinal predictors of physical peer
and dating violence. J Adolesc Health. 2015;56:106-12. [PMID:
25287983] doi:10.1016/j.jadohealth.2014.08.003
17. Bearman P, Jones J, Udry J. Add Health. 1997. Accessed at www
.cpc.unc.edu/projects/addhealth on 22 December 2016.

Development of the SaFETy Score ORIGINAL RESEARCH

Annals.org Annals of Internal Medicine • Vol. 166 No. 10 • 16 May 2017 713

18. Straus MA, Hamby SL, Boney-McCoy S, Sugarman DB. The re-
vised Conflict Tactics Scale (CTS2): development and preliminary
psychometric data. J Fam Issues. 2016;17:283-316.
19. Richters JE, Saltzman W. Survey of Exposure to Community Vio-
lence: Self-Report Version. Rockville, MD: JE Richters; 1990.
20. Piersma HL, Boes JL, Reaume WM. The Brief Symptom Inventory
as an Outcome Measure for Adolescent Psychiatric Inpatients. As-
sessment. 1994;1:151-8. [PMID: 9465145]
21. Belgrave FZ, Reed MC, Plybon LE, Corneille M. The impact of a
culturally enhanced drug prevention program on drug and alcohol
refusal efficacy among urban African American girls. J Drug Educ.
2004;34:267-79. [PMID: 15648887]
22. Conners NA, Plybon LE, Pegg P. Specific Events Drug and Alco-
hol Refusal Self-Efficacy Scale. Richmond, VA: Virginia Common-
wealth Univ; 2003.
23. Babor TF, Higgins-Biddle JC, Saunders JB, Monteiro MG. The
Alcohol Use Disorders Identification Test: Guidelines for Use in Pri-
mary Care. Geneva: World Health Organization; 2001.
24. Chung T, Colby SM, Barnett NP, Rohsenow DJ, Spirito A, Monti
PM. Screening adolescents for problem drinking: performance of
brief screens against DSM-IV alcohol diagnoses. J Stud Alcohol.
2000;61:579-87. [PMID: 10928728]
25. Zimmerman M. Flint Adolescent Study (FAS): a longitudinal study
of school dropout and substance use, 1994-1997. Ann Arbor, MI:
University of Michigan; 2014.
26. Hill HM, Noblin V. Children’s Perceptions of Environmental Vio-
lence. Washington, DC: Howard Univ; 1991.
27. Copeland-Linder N, Jones VC, Haynie DL, Simons-Morton BG,
Wright JL, Cheng TL. Factors associated with retaliatory attitudes
among African American adolescents who have been assaulted.
J Pediatr Psychol. 2007;32:760-70. [PMID: 17403911]
28. Bosworth K, Espelage D. Teen Conflict Survey. Bloomington, IN:
Center for Adolescent Studies, Indiana Univ; 1995.
29. van Burren S, Groothuis-Oudshoorn K. mice: Multivariate impu-
tation by chained equations in R. J of Stat Softw. 2011;45:1-67. doi:
10.18637/jss.v045.i03
30. Zou H, Hastie T. Regularization and variable selection via the
elastic net. Journal of the Royal Statistical Society: Series B. 2005;67:
301-320.
31. Tibshirani R. Regression shrinkage and selection via the lasso.
J Royal Stat Soc: Series B. 1996;58:267-288.
32. Hoerl AE, Kennard RW. Ridge regression: applications to nonor-
thogonal problems. Technometrics. 1970;12:55-67.
33. Friedman J, Hastie T, Tibshirani R. Regularization Paths for Gen-
eralized Linear Models via Coordinate Descent. J Stat Softw. 2010;
33:1-22. [PMID: 20808728]
34. Burnham KP, Anderson DR. Multimodel inference: understand-
ing AIC and BIC in Model Selection. Sociol Methods Res. 2004;33:
261-304.
35. Sullivan LM, Massaro JM, D’Agostino RB Sr. Presentation of mul-
tivariate data for clinical use: The Framingham Study risk score func-
tions. Stat Med. 2004;23:1631-60. [PMID: 15122742]
36. Carter PM, Walton MA, Newton MF, Clery M, Whiteside LK, Zim-
merman MA, et al. Firearm possession among adolescents present-
ing to an urban emergency department for assault. Pediatrics. 2013;
132:213-21. [PMID: 23837181] doi:10.1542/peds.2013-0163
37. Rowhani-Rahbar A, Zatzick D, Wang J, Mills BM, Simonetti JA,
Fan MD, et al. Firearm-related hospitalization and risk for subse-
quent violent injury, death, or crime perpetration: a cohort study.
Ann Intern Med. 2015;162:492-500. [PMID: 25706337] doi:10.7326
/M14-2362
38. Cooper C, Eslinger DM, Stolley PD. Hospital-based violence in-
tervention programs work. J Trauma. 2006;61:534-7. [PMID:
16966983]

39. Chong VE, Smith R, Garcia A, Lee WS, Ashley L, Marks A, et al.
Hospital-centered violence intervention programs: a cost-
effectiveness analysis. Am J Surg. 2015;209:597-603. [PMID:
25728889] doi:10.1016/j.amjsurg.2014.11.003
40. Shibru D, Zahnd E, Becker M, Bekaert N, Calhoun D, Victorino
GP. Benefits of a hospital-based peer intervention program for vio-
lently injured youth. J Am Coll Surg. 2007;205:684-9. [PMID:
17964444]
41. Sharp AL, Prosser LA, Walton M, Blow FC, Chermack ST, Zim-
merman MA, et al. Cost analysis of youth violence prevention. Pedi-
atrics. 2014;133:448-53. [PMID: 24515518] doi:10.1542/peds.2013-
1615
42. Purtle J, Rich JA, Fein JA, James T, Corbin TJ. Hospital-Based
Violence Prevention: Progress and Opportunities. Ann Intern Med.
2015;163:715-7. [PMID: 26301734] doi:10.7326/M15-0586
43. Howell EM, Abraham P. The Hospital Costs of Firearm Assaults.
Washington, DC: The Urban Inst; 2013.
44. Steinberg L, Monahan KC. Age differences in resistance to peer
influence. Dev Psychol. 2007;43:1531-43. [PMID: 18020830]
45. Ursano RJ, Benedek DM, Engel CC. Trauma-informed care for
primary care: the lessons of war. Ann Intern Med. 2012;157:905-
6. [PMID: 22964876] doi:10.7326/0003-4819-157-11-201212040
-00542
46. Walton MA, Cunningham RM, Goldstein AL, Chermack ST, Zim-
merman MA, Bingham CR, et al. Rates and correlates of violent be-
haviors among adolescents treated in an urban emergency depart-
ment. J Adolesc Health. 2009;45:77-83. [PMID: 19541253] doi:10
.1016/j.jadohealth.2008.12.005
47. Pickett W, Craig W, Harel Y, Cunningham J, Simpson K, Molcho
M, et al; HBSC Violence and Injuries Writing Group. Cross-national
study of fighting and weapon carrying as determinants of adolescent
injury. Pediatrics. 2005;116:e855-63. [PMID: 16322143] Pickett W,
Craig W, Harel Y, et al. Cross-national study of fighting and weapon
carrying as determinants of adolescent injury. Pediatrics. 2005;
116(6):855-853.
48. National Institute on Alcohol Abuse and Alcoholism. Alcohol
Screening and Brief Intervention for Youth: A Practitioner’s Guide.
Accessed at www.niaaa.nih.gov/Publications/EducationTraining
Materials/Pages/YouthGuide.aspx on 22 December 2016.
49. McGinty EE, Choksy S, Wintemute GJ. The Relationship Between
Controlled Substances and Violence. Epidemiol Rev. 2016;38:5-31.
[PMID: 26905893] doi:10.1093/epirev/mxv008
50. Swanson JW, Sampson NA, Petukhova MV, Zaslavsky AM, Ap-
pelbaum PS, Swartz MS, et al. Guns, Impulsive Angry Behavior, and
Mental Disorders: Results from the National Comorbidity Survey
Replication (NCS-R). Behav Sci Law. 2015;33:199-212. [PMID:
25850688] doi:10.1002/bsl.2172
51. Swanson JW, McGinty EE, Fazel S, Mays VM. Mental illness and
reduction of gun violence and suicide: bringing epidemiologic re-
search to policy. Ann Epidemiol. 2015;25:366-76. [PMID: 24861430]
doi:10.1016/j.annepidem.2014.03.004
52. Walton MA, Chermack ST, Shope JT, Bingham CR, Zimmerman
MA, Blow FC, et al. Effects of a brief intervention for reducing vio-
lence and alcohol misuse among adolescents: a randomized con-
trolled trial. JAMA. 2010;304:527-35. [PMID: 20682932] doi:10
.1001/jama.2010.1066
53. Cunningham RM, Chermack ST, Zimmerman MA, Shope JT,
Bingham CR, Blow FC, et al. Brief motivational interviewing interven-
tion for peer violence and alcohol use in teens: one-year follow-up.
Pediatrics. 2012;129:1083-90. [PMID: 22614776] doi:10.1542/peds
.2011-3419

ORIGINAL RESEARCH Development of the SaFETy Score

714 Annals of Internal Medicine • Vol. 166 No. 10 • 16 May 2017 Annals.org

Current Author Addresses: Drs. Goldstick, Carter, and Cun-
ningham: University of Michigan Injury Center, University of
Michigan School of Medicine, 2800 Plymouth Road, NCRC
G10-080, Ann Arbor, MI 48109.
Dr. Walton: Department of Psychiatry, University of Michigan
Addiction Research Center, University of Michigan School of
Medicine, 4250 Plymouth Road, Ann Arbor, MI 48109.
Dr. Zimmerman: Michigan Youth Violence Prevention Center,
University of Michigan School of Public Health, 1415 Washing-
ton Heights, Ann Arbor, MI 48109.
Drs. Dahlberg and Sumner: Division of Violence Prevention,
National Center for Injury Prevention and Control, Centers for
Disease Control and Prevention, 4770 Buford Highway, NE,
MS-F64, Atlanta, GA 30341.

Author Contributions: Conception and design: P.M. Carter, R.
Cunningham, L.L. Dahlberg, J. Goldstick, M. Walton, M.A.
Zimmerman.
Analysis and interpretation of the data: P.M. Carter, R. Cun-
ningham, J. Goldstick, S.A. Sumner.
Drafting of the article: P.M. Carter, L.L. Dahlberg, J. Goldstick,
M.A. Zimmerman.
Critical revision for important intellectual content: J. Goldstick,
S.A. Sumner, M.A. Zimmerman.
Final approval of the article: P.M. Carter, R. Cunningham, L.L.
Dahlberg, J. Goldstick, S.A. Sumner, M. Walton, M.A.
Zimmerman.
Provision of study materials or patients: M. Walton.
Statistical expertise: J. Goldstick, S.A. Sumner.
Obtaining of funding: M. Walton, M.A. Zimmerman.
Administrative, technical, or logistic support: P.M. Carter, R.
Cunningham, S.A. Sumner.
Collection and assembly of data: M. Walton.

Annals.org Annals of Internal Medicine • Vol. 166 No. 10 • 16 May 2017

Appendix Figure 1. Flint Youth Injury study flow chart.

Ineligible for screening (n = 474)
Sexual assault or child abuse: 128
Mental health issue precluding
consent (e.g., schizophrenia): 88
Parental consent not granted: 65
Screened previously: 58
Injured >72 h ago: 41
Currently incarcerated: 30
Other: 64

Missed (n = 76)
Study staff screening another
participant: 45
Discharged before being located
by study staff: 15
Youth could not be located: 5
Other: 11

Declined (n = 131)
Did not feel well enough: 62
Unwilling to participate: 29
Family did not allow access: 16
Other: 24

Declined (n = 33)
Did not complete baseline after
agreeing to do so: 22
Decided not to participate: 6
Left hospital/discharged before
completing baseline: 3
Other: 2

Assault-injured
youth seeking ED

care (n = 1718)

The non–assault-injured group was
sampled so that the next available age- and

sex-matched non–assault-injured
youth was approached

Eligible for screen
(n = 925)

Approached
(n = 849)

Approached
(n = 846)

Declined (n = 116)
Did not feel well enough: 51
Unwilling to participate: 31
Family did not allow access: 17
Discharged/didn’t want to
stay: 10
Other: 7

Declined (n = 27)
Did not complete baseline after
agreeing to do so: 15
Decided not to participate: 7
Left hospital/discharged before
completing baseline: 5

Screened
(n = 718)

Screened
(n = 730)

Eligible
(n = 388)

Baseline complete
(n = 349)

Baseline complete
(n = 250)

Note: n = 70 arrived for
firearm injury

Eligible
(n = 278)

Assault-injured youth
arriving during

recruitment shifts
(n = 1399)

Ineligible for screening (n = 599)
Parental consent not granted:

60

Mental health issue precluding
consent (e.g., schizophrenia): 81
Screened previously: 66
Other: 392

Excluded (n = 6) Excluded (n = 1)

ED = emergency department.

Annals of Internal Medicine • Vol. 166 No. 10 • 16 May 2017 Annals.org

Appendix Figure 2. Variable importance determined by the predictive model, expressed as standardized regression
coefficients (divided by the minimum so that the smallest score is 1).

Standardized Regression Coefficient (Scaled by the Minimum)
0 5 15 20 25 30 35 40 45 5010

My house was broken into

Unable to stop drinking

Seen gangs in neighborhood

Friends in legal trouble (drugs)

Violent injury

My friends smoke marijuana

Someone cut/stabbed you

Partner used knife on you

Someone threw something at you

Understand other’s POV

My friends carry weapons

Someone shot you

Drank before a serious fight

Put someone in the hospital

Someone pulled a knife on you

Heard shots in neighborhood

Got into serious fight

Seen someone shot

Someone used a gun on you

Someone pulled a gun on you

POV = point of view.

Appendix Table 1. Sensitivity and Specificity for SaFETy Score Thresholds Between 1 and 10 in the Training Set

Threshold Sensitivity (95% CI) Specificity (95% CI)

1 186/189 = 98.4% (95.1%–99.6%) 29/173 = 16.8% (11.7%–23.4%)
2 160/189 = 84.7% (78.5%–89.3%) 77/173 = 44.5% (37.0%–52.2%)
3 132/189 = 69.8% (62.7%–76.2%) 125/173 = 72.3% (64.9%–78.6%)
4 122/189 = 64.6% (57.2%–71.3%) 134/173 = 77.5% (70.4%–83.3%)
5 109/189 = 57.7% (50.3%–64.7%) 140/173 = 80.9% (74.1%–86.3%)
6 74/189 = 39.2% (32.2%–46.5%) 162/173 = 93.6% (88.6%–96.6%)
7 35/189 = 18.5% (13.4%–25.0%) 167/173 = 96.5% (92.3%–98.6%)
8 19/189 = 10.1% (6.3%–15.5%) 171/173 = 98.8% (95.4%–99.8%)
9 15/189 = 7.9% (4.7%–13.0%) 173/173 = 100.0% (97.3%–100.0%)
10 10/189 = 5.3% (2.7%–9.8%) 173/173 = 100.0% (97.3%–100.0%)

SaFETy = Serious fighting, Friend weapon carrying, community Environment, and firearm Threats.
Annals.org Annals of Internal Medicine • Vol. 166 No. 10 • 16 May 2017

Appendix Table 2. Joint Distribution of Violent Injury and
Future Firearm Violence Status

Status AI Non-AI Total

Future firearm violence 167 85 252
No future firearm violence 116 115 231
Unknown 66 50 116
Total 349 250 599

AI = assault-injured.

Appendix Table 3. Relationship Between Future Firearm
Violence and SaFETy Score and Assault Injury
Presentation

Model Validation (95% CI) Training (95% CI)

1: Assault injury 2.14 (1.03–4.45) 1.89 (1.24–2.89)

2: SaFETY score 1.47 (1.23–1.79) 1.56 (1.41–1.75)

3
Assault injury 1.49 (0.67–3.32) 1.23 (0.75–2.00)
SaFETY score 1.44 (1.20–1.76) 1.54 (1.39–1.73)

SaFETy = Serious fighting, Friend weapon carrying, community Envi-
ronment, and firearm Threats.

Appendix Figure 3. Future firearm violence rates in the
validation data set (dashed line) in 5 risk strata identified
by using the training data set (solid line).

Training sample (n = 362)
Validation sample (n = 121)

0 1–2 3–5 9–106–8

100

80

60
40
20
0

SaFETy score

P
ar

ti
ci

p
an

ts
W

it
h
F

u
tu

re
F

ir
ea

rm
V

io
le

n
ce

,
%

SaFETy = Serious fighting, Friend weapon carrying, community Envi-
ronment, and firearm Threats.
Annals of Internal Medicine • Vol. 166 No. 10 • 16 May 2017 Annals.org

Appendix Table 4. Description of Highest-Ranked Factors for Future Firearm Violence Among Those Presenting
for Violent Injury

Factor Response
Type†

Importance
Rank

Timeframe Odds Ratio
(95% CI)*

Standardized
Odds Ratio

Received threats
Someone pulled a gun on you Freq (0–6) 1 6 mo 1.91 (1.41–2.61) 2.17
Someone used a gun on you Freq (0–6) 2 6 mo 2.44 (1.40–4.26) 1.81
Someone pulled a knife on you Freq (0–6) 6 6 mo 1.37 (0.99–1.67) 1.34
Someone shot you Freq (0–6) 9 6 mo 2.37 (1.27–4.43) 1.56
Someone threw something at you Freq (0–6) 12 6 mo 1.41 (1.03–1.92) 1.41
Someone cut/stabbed you Freq (0–6) 14 6 mo 1.80 (1.15–2.83) 1.50

Community
I have seen someone shot Freq (0–3) 3 6 mo 1.92 (1.34–2.73) 1.82
I have heard guns shot Freq (0–3) 5 6 mo 1.28 (0.99–1.67) 1.31
Seen gangs in neighborhood Fred (0-3) 18 6 mo 1.19 (0.95–1.49) 1.24
My house was broken into Freq (0–3) 20 6 mo 1.31 (0.86–1.99) 1.23

Friends
My friends carry weapons Number (1–5) 10 Current 1.44 (1.09–1.91) 1.50
My friends smoke marijuana Number (1–5) 15 Current 1.24 (0.99–1.54) 1.30
Friend legal trouble (drug-related) Number (1–5) 17 Current 1.43 (1.00–2.05) 1.37

Partner violence
Partner used a knife on you Freq (0–6) 13 6 mo 3.20 (1.07–9.61) 2.24

Fighting
Been in a serious fight Freq (0–6) 4 6 mo 1.21 (0.99–1.47) 1.32
Put someone in the hospital Freq (0–6) 7 6 mo 1.33 (1.00–1.77) 1.39
Drank before fighting Freq (0–6) 8 6 mo 1.37 (1.00–1.88) 1.35

Other
Understand another’s point of view Agree (1–5) 11 6 mo 1.37 (1.10–1.72) 1.48
Today’s ED visit for violent injury Yes/No 16 Current NA NA
Unable to stop drinking Freq (0–4) 19 6 mo 1.53 (0.94–2.50) 1.20

ED = emergency department; Freq = frequency; NA = not available.
* CIs with lower bounds of 1.00 are entirely above 1.00.
† Freq (0 – 6) measures frequency on a 7-point scale from 0 (never) to 6 (!20 times); Freq (0 –3) measures frequency on a 7-point scale from 0 (never)
to 3 (many times); Freq (0 – 4) measures frequency on a 5-point scale from 0 (never) to 4 (daily); Number (1–5) measures frequency on a 5-point scale
from 1 (none) to 5 (All); Agree (1-5) measures agreement on a 5-point scale from 1 (very true) to 5 (not true); Yes/No denotes a binary (1/0) indicator.

Annals.org Annals of Internal Medicine • Vol. 166 No. 10 • 16 May 2017

Appendix Table 5. Description of Highest-Ranked Factors for Future Firearm Violence Among Those Not Presenting for
Violent Injury

Factor Response
Type†
Importance
Rank
Timeframe Odds Ratio
(95% CI)*
Standardized
Odds Ratio

Received threats
Someone pulled a gun on you Freq (0–6) 1 6 mo 3.82 (2.14–6.85) 4.96
Someone used a gun on you Freq (0–6) 2 6 mo 7.03 (1.69–29.16) 3.66
Someone pulled a knife on you Freq (0–6) 6 6 mo 4.25 (2.09–8.63) 3.87
Someone shot you Freq (0–6) 9 6 mo 3.54 (1.06–11.79) 1.91
Someone threw something at you Freq (0–6) 12 6 mo 1.85 (1.13–3.04) 1.85
Someone cut/stabbed you Freq (0–6) 14 6 mo 3.44 (1.29–9.17) 2.33

Community
I have seen someone shot Freq (0–3) 3 6 mo 1.75 (1.20–2.55) 1.67
I have heard guns shot Freq (0–3) 5 6 mo 2.01 (1.41–2.85) 2.12
Seen gangs in neighborhood Fred (0-3) 18 6 mo 1.53 (1.17–1.99) 1.70
My house was broken into Freq (0–3) 20 6 mo 2.40 (1.43–4.01) 1.84

Friends
My friends carry weapons Number (1–5) 10 Current 1.83 (1.30–2.56) 1.94
My friends smoke marijuana Number (1–5) 15 Current 1.38 (1.05–1.81) 1.49
Friend legal trouble (drug-related) Number (1–5) 17 Current 2.00 (1.32–3.03) 1.84

Partner Violence
Partner used a knife on you Freq (0–6) 13 6 mo 3.73 (0.94–14.84) 2.49

Fighting
Been in a serious fight Freq (0–6) 4 6 mo 1.80 (1.36–2.37) 2.41
Put someone in the hospital Freq (0–6) 7 6 mo 2.62 (1.65–4.16) 3.02
Drank before fighting Freq (0–6) 8 6 mo 3.37 (1.71–6.67) 3.13

Other
Understand another’s point of view Agree (1–5) 11 6 mo 1.29 (0.97–1.73) 1.38
Today’s ED visit for violent injury Yes/No 16 Current NA NA
Unable to stop drinking Freq (0–4) 19 6 mo 1.68 (1.02–2.75) 1.46

ED = emergency department; Freq = frequency; NA = not available; OR = odds ratio.
* CIs with lower bounds of 1.00 are entirely above 1.00.
† Freq (0 – 6) measures frequency on a 7-point scale from 0 (never) to 6 (!20 times); Freq (0 –3) measures frequency on a 7-point scale from 0 (never)
to 3 (many times); Freq (0 – 4) measures frequency on a 5-point scale from 0 (never) to 4 (daily); Number (1–5) measures frequency on a 5-point scale
from 1 (none) to 5 (All); Agree (1-5) measures agreement on a 5-point scale from 1 (very true) to 5 (not true); Yes/No denotes a binary (1/0) indicator.

Appendix Table 6. Sensitivity and Specificity in the Validation Set, Stratified by AI or Non-AI Group (95% CIs)

Threshold Sensitivity (AI) Specificity (AI) Sensitivity (Non-AI) Specificity (Non-AI)

1 39/41 = 95.1% (82.2–99.2%) 0/27 = 0.0% (0.0%–15.5%) 22/22 = 100.0% (81.5%–100.0%) 9/31 = 29.0% (14.9%–48.2%)
2 36/41 = 87.8% (73.0–95.4) 7/27 = 25.9% (11.9%–46.6%) 17/22 = 77.3% (54.2%–91.3%) 20/31 = 64.5% (45.4%–80.2%)
3 31/41 = 75.6% (59.4–87.1%) 13/27 = 48.1% (29.2%–67.6%) 12/22 = 54.5% (32.7%–74.9%) 23/31 = 74.2% (55.1%–87.5%)
4 26/41 = 63.4% (46.9–77.4) 16/27 = 59.3% (39.0%–77.0%) 10/22 = 45.5% (25.1%–67.3%) 24/31 = 77.4% (58.5%–89.7%)
5 23/41 = 56.1% (39.9%–71.2%) 21/27 = 77.8% (57.3%–90.6%) 9/22 = 40.9% (21.5%–63.3%) 27/31 = 87.1% (69.2%–95.8%)
6 12/41 = 29.3% (16.6%–45.7%) 26/27 = 96.3% (79.1%–99.8%) 7/22 = 31.8% (14.7%–54.9%) 29/31 = 93.5% (77.2%–98.9%)
7 5/41 = 12.2% (4.6%–27.0%) 27/27 = 100.0% (84.5%–100.0%) 2/22 = 9.1% (1.6%–30.6%) 30/31 = 96.8% (81.5%–99.8%)
8 5/41 = 12.2% (4.6%–27.0%) 27/27 = 100.0% (84.5%–100.0%) 1/22 = 4.5% (0.2%–24.9%) 31/31 = 100.0% (86.3%–100.0%)
9 5/41 = 12.2% (4.6%–27.0%) 27/27 = 100.0% (84.5%–100.0%) 1/22 = 4.5% (0.2%–24.9%) 31/31 = 100.0% (86.3%–100.0%)
10 3/41 = 7.3% (1.9%–21.0%) 27/27 = 100.0% (84.5%–100.0%) 1/22 = 4.5% (0.2%–24.9%) 31/31 = 100.0% (86.3%–100.0%)

AI = assault-injured.
Annals of Internal Medicine • Vol. 166 No. 10 • 16 May 2017 Annals.org

Appendix Table 8. Sensitivity and Specificity in the Training Set, Stratified by AI or Non-AI Group (95% CIs)

Threshold Sensitivity (AI) Specificity (AI) Sensitivity (Non-AI) Specificity (Non-AI)

1 126/126 = 100.0% (96.3%–100.0%) 5/89 = 5.6% (2.1%–13.2%) 60/63 = 95.2% (85.8%–98.8%) 24/84 = 28.6% (19.5%–39.6%)
2 113/126 = 89.7% (82.7%–94.2%) 24/89 = 27.0% (18.4%–37.6%) 47/63 = 74.6% (61.8%–84.4%) 53/84 = 63.1% (51.8%–73.2%)
3 93/126 = 73.8% (65.1%–81.0%) 54/89 = 60.7% (49.7%–70.7%) 39/63 = 61.9% (48.8%–73.6%) 71/84 = 84.5% (74.6%–91.2%)
4 86/126 = 68.3% (59.3%–76.1%) 60/89 = 67.4% (56.6%–76.8%) 36/63 = 57.1% (44.1%–69.3%) 74/84 = 88.1% (78.8%–93.8%)
5 75/126 = 59.5% (50.4%–68.1%) 61/89 = 68.5% (57.7%–77.7%) 34/63 = 54.0% (41.0%–66.4%) 79/84 = 94.0% (86.0%–97.8%)
6 46/126 = 36.5% (28.3%–45.6%) 80/89 = 89.9% (81.2%–95.0%) 28/63 = 44.4% (32.1%–57.4%) 82/84 = 97.6% (90.9%–99.6%)
7 22/126 = 17.5% (11.5%–25.5%) 84/89 = 94.4% (86.8%–97.9%) 13/63 = 20.6% (11.9%–33.0%) 83/84 = 98.8% (92.6%–99.9%)
8 10/126 = 7.9% (4.1%–14.5%) 87/89 = 97.8% (91.4%–99.6%) 9/63 = 14.3% (7.1%–25.9%) 84/84 = 100.0% (94.6%–100.0%)
9 8/126 = 6.3% (3.0%–12.5%) 89/89 = 100.0% (94.8%–100.0%) 7/63 = 11.1% (5.0%–22.2%) 84/84 = 100.0% (94.6%–100.0%)
10 5/126 = 4.0% (1.5%–9.5%) 89/89 = 100.0% (94.8%–100.0%) 5/63 = 7.9% (3.0%–18.3%) 84/84 = 100.0% (94.6%–100.0%)

AI = assault-injured.

Appendix Table 9. Frequency Tables of the SaFETy Score
in the Training Data Set, Stratified by Group Membership

Score Non-AI Group AI Group

0 27 (18.4%) 5 (2.3%)
1 42 (28.6%) 32 (14.9%)
2 26 (17.7%) 50 (23.3%)
3 6 (4.1%) 13 (6.0%)
4 7 (4.8%) 12 (5.6%)
5 9 (6.1%) 48 (22.3%)
6 16 (10.9%) 28 (13.0%)
7 5 (3.4%) 15 (7.0%)
8 2 (1.4%) 4 (1.9%)
9 2 (1.4%) 3 (1.4%)
10 5 (3.4%) 5 (2.3%)
Total 147 (100.0%) 215 (100.0%)

AI = assault-injured; SaFETy = Serious fighting, Friend weapon carry-
ing, community Environment, and firearm Threats.

Appendix Table 7. Frequency Tables of the SaFETy Score
in the Validation Data Set, Stratified by Group
Membership

Score Non-AI Group AI Group

0 9 (17.0%) 2 (2.9%)
1 16 (30.2%) 10 (14.7%)
2 8 (15.1%) 11 (16.2%)
3 3 (5.7%) 8 (11.8%)
4 4 (7.5%) 8 (11.8%)
5 4 (7.5%) 16 (23.5%)
6 6 (11.3%) 8 (11.8%)
7 2 (3.8%) 0 (0.0%)
8 0 (0.0%) 0 (0.0%)
9 0 (0.0%) 2 (2.9%)
10 1 (1.9%) 3 (4.4%)
Total 53 (100.0%) 68 (100.0%)

AI = assault-injured; SaFETy = Serious fighting, Friend weapon carry-
ing, community Environment, and firearm Threats.
Annals.org Annals of Internal Medicine • Vol. 166 No. 10 • 16 May 2017

Copyright © American College of Physicians 2017.