Proposal and Purpose #2
Discuss the importance of assessing study limitations when reviewing study results. What considerations should be included when thinking about generalizing study findings to your population? Provide an example of a study and discuss the generalizability of the results. Could they be generalized and to which population?
In the last column we discussed the use of pooling to get a better
estimate of the standard deviation of the measurement method, es-
sentially the standard deviation of the raw data. But as the last column
implied, most of the time individual measurements are averaged and
decisions must take into account another standard deviation, the stan-
dard deviation of the mean, sometimes called the “standard error” of the
mean. It’s helpful to explore this statistic in more detail: fi rst, to under-
stand why statisticians often recommend a “sledgehammer” approach
to data collection methods; and, second, to see that there might be a
better alternative to this crude tactic. We’ll also see how to answer the
question, “How big should my sample size be?”
For the next few columns, we need to discuss in more detail the ways
statisticians do their theoretical work and the ways we use their results.
I often say that theoretical statisticians live on another planet (they don’t,
of course, but let’s say Saturn), while those of us who apply their results
live on Earth. Why do I say that? Because a lot of theoretical statistics
makes the unrealistic assumption that there is an infi nite amount of data
available to us (statisticians call it an infi nite population of data). When we
have to pay for each measurement, that’s a laughable assumption. We’re
often delighted if we have a random sample of that data, perhaps as many
as three replicate measurements from which we can calculate a mean.
That last sentence contains a telling phrase: “a random sample of that
data.” Statisticians imagine that the infi nite population of data contains
all possible values we might get when we make measurements. Statisti-
cians view our results as a random draw from that infi nite population of
possible results that have been sitting there waiting for us. If we were
to make another set of measurements on the same sample, we’d get
a different set of results. That doesn’t surprise the statisticians (and it
shouldn’t surprise us if we adopt their view)—it’s just another random
draw of all the results that are just waiting to appear.
On Saturn they talk about a mean, but they call it a “true” mean. They
don’t intend to imply that they have a pipeline to the National Institute
of Standards and Technology and thus know the absolutely correct value
for what the mean represents. When they call it a “true mean,” they’re
just saying that it’s based on the infi nite amount of data in the popula-
tion, that’s all.
Statisticians generally use Greek letters for true values—μ for a true
mean, σ for a true standard deviation, δ for a true diff erence, etc.
The technical name for these descriptors (μ, σ, δ) is parameters. You’ve
probably been casual about your use of this word, employing it to refer to,
Statistics in the Laboratory:
Standard Deviation of the Mean
say, the pH you’re varying in your experiments, or the yield you get from
those experiments, or maybe even constraints (“We have to stay within
out budgetary parameters”). You can’t be sloppy like that when you work
with a statistician: the word parameter has a very strict meaning.
Because parameters are based on an infi nite amount of data, there is no
uncertainty in their values. (We’ll see why in a minute.)
So, you’re saying to yourself, “I’m confused. And why would I even worry
about what to call these things if I don’t have that infi nite amount of data
and can’t calculate them, anyway?”
Good point. Here’s a key thing, though. Even though we’ll never know
the values of these parameters, we can still use a limited sample of data
to guess at their true values. It’s a process called estimation, so the results
are called parameter estimates, also called sample statistics.
We use a Roman letter to represent individual measurements (e.g., x1 =
3.6), and we put a “bar” above the letter when we want to indicate an
arithmetic average (a mean). For example, if x2 = 4.8, and x3 = 4.5, we would
write the mean of x1 through x3 as x
_
= 4.3. Thus,we say that the statistic x
_
is
an estimate of the parameter μ. Because there is uncertainty in the mea-
sured values that have been “drawn from the population at random,”
there is uncertainty in these parameter estimates.
Backing up a bit, how do we measure the uncertainty in measured
values? As we discussed in the last column, the estimate s of the true
standard deviation σ is given by the familiar equation:
where the Greek capital letter sigma (Σ) is the summation operator, and
its index i indicates the measurement number from 1 to n. For x1 through
x3, s = 0.6807.
Now, let’s go to Saturn for a few minutes. On Saturn we can play with
the infi nite population of data. Let’s suppose that for the measurements
we’ve been making, μ = 4.76 (exactly) and σ = 0.30 (exactly). The estimate
of s = 0.6807 seems a bit high in comparison to σ = 0.30, but parameter
estimates can be quite variable when n is small (and to a statistician n = 3
is small), so it isn’t anything to worry about.
We won’t live long enough to look at all of the data in the infi nite popu-
lation, so let’s look at only one million pieces of data and say that’s
by Stanley N. Deming
AL
20 AMERICAN LABORATORY MARCH 2019
2121 AMERICAN LABORATORY MARCH 2019
AL
21
exhibit less variability than the raw data. The relationship between sx-, s,
and n is a “reciprocal square-root” function, the statistician’s “one-over-
the-square-root-of-n” effect:
Clearly, as n increases, the uncertainty in the mean decreases.
This relationship holds on Saturn, as well, and shows why on Saturn
there is no uncertainty in the mean—if n = ∞ then σ x- = 0:
This equation can be rearranged to show in general how the ratio of the
standard deviation of the mean to the standard deviation of the raw data
decreases as 1/√n :
Figure 4 illustrates this 1/√n effect. Clearly, as n increases, σx – decreases.
Doing a few replicates can reduce the uncertainty in the mean by quite
representative enough. The Gaussian distribution in Figure 1 was ob-
tained by drawing at random one million data points (statistical samples
of size n = 1) from the infinite population with μ = 4.76 and σ = 0.30.
The data have been “binned” to generate the “histogram distribution”
shown in Figure 1. The bin size is 0.04 on the horizontal axis. There are
100 bins from 3 to 7. If a sample mean had a value between 4.00 and
4.04, for example, it would be placed in bin number 26. The height of
each contiguous histogram bar represents the number of data points
that end up in that bin. Note that the mean of the one million data points
is 4.760 (to three decimal places), and the standard deviation of the one
million data points is 0.300 (to three decimal places). Figure 1 is what we
would expect to see for the individual measurements. No surprises here.
Figure 2 is a little bit diff erent. For this fi gure, we didn’t pull out only one
data point, but we pulled out two data points at a time and binned their
means. So, Figure 2 is based on two million data points, or one million
means for which n = 2. The “grand mean,” the “average of the averages”
(represented by the symbol x with two bars above it) is equal to 4.760, as
expected, but now we see that the “standard deviation of the means” sx – =
0.212, less than 0.30. Interesting.
For Figure 3 we pulled out four data points at a time and binned their
means. The grand mean is again 4.760, but sx- = 0.150, exactly half of
σ = 0.30 for the raw data. What’s going on here?
When data points are averaged, the negative deviations of some of the
data points cancel the positive deviations of other data points. Thus,
the estimated means tend to be closer to the true mean and therefore
Figure 1 – The distribution of 1,000,000 individual pieces of data (n = 1)
drawn at random from an infi nite population with μ = 4.76 and σ = 0.30.
See text
for discussion.
Figure 2 – Yellow: the distribution of 1,000,000 means, each estimated
from two pieces of data (n = 2) drawn at random from an infi nite popu-
lation with μ = 4.76 and σ = 0.30. Green in background: the underlying
distribution of raw data. See text for discussion.
2222 AMERICAN LABORATORY MARCH 2019
STATISTICS IN THE LABORATORY continued
marginal improvement in σx – decreases. Stated differently, the first few
replicates give a lot of bang for the buck; after that, it gets more and more
expensive to decrease σx -.
Many researchers want to know how big their sample size should be (a
legitimate request). Suppose a researcher asks a statistician this ques-
tion, expecting to get a simple answer: e.g., n = 3. Instead, the statistician
turns around and silently walks off in disgust. Why do statisticians be-
have this way? Because they know there is no simple answer to this
question, and they’re going to have to work with the researcher to try
to get information that the researcher might not have. Experience has
taught them that the best time to get out of a bad deal is at the begin-
ning. They don’t want to go through this excruciating process again.
The researcher might have a pooled estimate of σ for the measurement
process, but the researcher’s mean is probably going to be used to make
a decision. The question then becomes, “How uncertain can the reported
mean be and still make a good decision?” That is, how small does σx- have
to be? It’s my opinion that because of the ways companies compart-
mentalize their functions, the researcher making the measurements is
often not aware of this last piece of information. It then becomes the
statistician’s task to move across the company to discover this piece of
information so the sample size can be determined. If you know σ and σx -,
you can calculate the sample size n yourself. At this point, you don’t need
the statistician.
Here’s an example. The percentage of toluene in 500 chemical samples
of gasoline is to be estimated by making multiple gas chromatographic
measurements for each gasoline sample and using the sample mean as
an estimate of the toluene percentage. Each measurement costs $50.
Previous experience has indicated that individual measurements have
a standard deviation of 0.10% toluene (this is σ, the method standard
deviation). However, the client requires a standard deviation of 0.025%
toluene (this will be σx-). How big should your sample size be?
You can almost calculate n in your head. If the ratio of σ x- to σ is
0.025%/0.10% = 1/4, then √n = 4 and n = 16. You must make 16 replicate
measurements on each of the 500 chemical samples for a total of 8,000
measurements. But this will cost $400,000. Your client is going to balk at
this. They’ll ask, “Isn’t there a cheaper way to get the results we need?”
Of course there’s a cheaper way. To get there, let’s look at an assumption
statisticians usually make when they solve sample size questions like
this. They assume σ is what it is, and it can’t be changed. They then apply
the 1/√n sledgehammer to come up with a sample size, as we did above.
But statisticians are often wrong about their assumption, and σ can be
changed. Suppose we bought a better chromatograph that gave mea-
surements with σ = 0.025% toluene instead of 0.10% toluene. With that
new chromatograph, the calculation of sample size would be n = 1. Only
500 measurements would be needed, and the cost running the samples
would be only $25,000.
Figure 5 illustrates the idea. Suppose you start out making 16 measure-
ments per sample ($800/sample) using the old chromatograph and
you suddenly realize you could save money if you bought a better
a bit. For example, when n = 4, σx – is decreased by a factor of 2. But to
decrease σx – by another factor of 2, the number of experiments must be
quadrupled to 16. Clearly, as the number of replicates is increased, the
Figure 4 – Illustration of the “one-over-the-square-root-of-n” effect. The
ratio σ x – / σ decreases as 1/√n.
Figure 3 – Yellow: the distribution of 1,000,000 means, each estimated
from four pieces of data (n = 4) drawn at random from an infinite popu-
lation with μ = 4.76 and σ = 0.30. Green in background: the underlying
distribution of raw data. See text for discussion.
2323 AMERICAN LABORATORY MARCH 2019
AL
samples 101 through 220 (the yellow rectangle labeled RECOVER). After
that, it’s pure SAVINGS, spending only $50 per sample rather than $800
per sample (the green rectangle). The total cost of the project (red area)
will be $190,000 ($90,000 for the chromatograph and $100,000 for the
measurements). This is a lot better than the $400,000 it was going to
cost. (The total cost would have been only $115,000 if you’d realized the
benefits of a better chromatograph at the beginning of the project.)
Don’t try to do with statistics what you can do cheaper with an improved
measurement method. The 1/√n sledgehammer isn’t always the best
way to solve sample size problems.
In conclusion: a) σx- is important for most decision-making, b) you can
make σx- as small as you want by using a large enough sample size,
c) you can calculate your sample size yourself, and d) sometimes it’s less
expensive to make σx- small just by using a better measurement method
with a smaller σ.
I n t h e n e x t m o d u l e w e ’ l l s e e h o w σ x- c a n b e u s e d t o c a l c u l a t e a
confidence interval for the mean.
Stanley N. Deming, Ph.D., is an analytical chemist masquerading as a stat-
istician at Statistical Designs, El Paso, Texas, U.S.A.; e-mail: standeming@
statisticaldesigns.com; www.statisticaldesigns.com
chromatograph. By the time you’ve finished your 100th sample (1600
measurements up to this point, an integrated COST of $80,000), you’ve
put together the funding (the upper yellow rectangle in the figure,
$90,000) and the new chromatograph you’re purchased has just arrived.
Starting with sample 101 you use the new chromatograph and start sav-
ing 15 measurements × $50 per measurement = $750 per sample, which
you can use to recover the $90,000 cost of the new chromatograph from
Figure 5 – An illustration of financial considerations when deciding
whether or not to use a more precise measurement method. See text
for discussion.
Copyright of American Laboratory is the property of CompareNetworks, Inc. and its content
may not be copied or emailed to multiple sites or posted to a listserv without the copyright
holder’s express written permission. However, users may print, download, or email articles for
individual use.
1046080.44