MKSAP Quiz: Validity of a study
During a routine health examination, a patient asks about an article that recommended avoiding statin therapy because of the risk for memory loss. The findings were based on cross-sectional data analysis of a well-validated national health survey. What is the most likely threat to the validity of this study?
During a routine health examination, a patient asks about an article that recommended avoiding statin therapy because of the risk for memory loss. The findings were based on cross-sectional data analysis of a well-validated national health survey, which was conducted by random sampling of patients according to zip code of residence. The analysis showed that patients who self-reported memory loss were more likely to also report having taken statin drugs (odds ratio, 1.8; 95% CI, 1.2 to 2.7; P=0.046).
Which of the following is the most likely threat to the validity of this study?
A. Confounding
B. Selection bias
C. Self-reported data
D. Statistical significance
MKSAP Answer and Critique
The correct answer is A. Confounding. This content is available to MKSAP 18 subscribers as Question 44 in the General Internal Medicine section. More information about MKSAP is available online.
The most likely threat to the validity of this cross-sectional study is confounding. Cross-sectional studies evaluate the relationship between exposures and health outcomes in a population of interest. These studies are characterized by the measurement of factors and outcomes at a single point in time. The validity of cross-sectional studies is particularly susceptible to recall bias and confounding. Recall bias is a systematic error that is introduced into a study by differences in the accuracy of the recollections of study participants; participants who have unpleasant experiences may recall past events differently than those who do not have similar experiences. Because cross-sectional studies are observational and not experimental, there is also no opportunity to randomly distribute factors that might influence the relationship being studied. Although statistical techniques can be used to control for known potential confounders, unknown confounders remain a threat to the validity of the conclusions. As such, cross-sectional studies are best suited to identifying potentially significant associations that can be more rigorously tested in experimental studies. Finally, because there is no way to verify that the purported cause (statin therapy) preceded the effect (memory loss), cross-sectional studies cannot prove cause-and-effect relationships.
Selection bias occurs when the study participants do not accurately reflect the population being studied, usually because the choice to participate is influenced by the clinical question. Selection bias can compromise the validity of observational study designs; however, in this study, the random sampling according to zip code of residence minimizes the possibility of selection bias.
Although self-reported data are less robust than measured data, well-validated survey designs may use self-reported data to determine the presence or absence of conditions, risk factors, or behaviors in a population.
The conventional level of statistical significance is a P value less than or equal to 0.05, and an odds ratio of 1 implies the absence of a significant relationship. In this case, the confidence interval for the odds ratio does not include the value 1, which supports the statistical significance of the findings.
Key Point
- The validity of cross-sectional studies is particularly susceptible to recall bias and confounding.