Sample size

Students recognize the value of using a large, random, representative sample of the group under study.  The recognize that it is critical for the sample size to be large, even if the sample is a small percentage of the population under study.  A random sample of one thousand is probably big enough, and a random sample of one hundred is probably too small (Paulos, Mathematician, 153). 

Types of studies

Students differentiate among basic types of studies:

• Randomized experiments assign subjects into two groups (the control group and the treatment group).
• Within-subject randomized experiements observe individual differences under different conditions.
• Longitudinal studies observe the same group over time.
• Cross-sectional studies compare two comparable groups at a single point in time.

Evaluation of data

Students evaluate data on which statistics are based, and analyze data sets for common errors.

• Selection bias: How is the sample gathered?  Is the process of randomly placing into treatment vs. control actually random?  Has the sample self-selected?
• Publication bias: “Positive finding are more likely to be published than negative findings" (Wheelan, Naked Statistics, 120).  And the really strange ones will get a lot of publicity.
• Recall bias: Studies based on self-reported memories may not be accurate.
• Truth bias: Is the study designed in such a way to elicit true answer from subjects?  Question phrasing, level of anonymity, and presentation of the questions all might affect the willingness of subjects to tell the truth about sensitive issues.
• Survivorship bias: Particularly in a longitudinal study, some or many of the participants may fall out of the sample, which will change the nature of the group and therefore affect the results.
• Healthy user bias: To what do we attribute difference: Is it the action?  Or is it the kind of person who does that
  action?  Someone who is healthy is more likely to be able to exercise, so does that mean that exercise makes you healthy?

Questions

Which sample is more predictive of its population: A) a random sample of 500 people taken from the entire population of the U.S. or B) a random sample of 50 people taken from a population of 3,000?  Answer!