My bias: I am a skeptic (“one who suspends judgment”) about any new piece of research, until I have critiqued it.
I. A few general questions to pose of research studies:
* What is the audience for the study? Is the style and format of the study appropriate for them?
* Does a larger study/sample size mean a stronger study? Is the sample too convenient?
* Are correlation and causation confused?
* Does “risk factor abuse” occur?
* Has the study been repeated? If there have been multiple studies, how/why did they differ?
* Are the claims of the authors wildly overstated?
* Do they misuse statistics? Or are the statistics incomprehensible?
II. Questions to ask when evaluating a research study:
1. What specific question(s) does the research address? Are these questions stated unambiguously? Is the scope of the question appropriate?
2. Is the review of the relevant literature thorough? Does it develop a sufficient theoretical framework for the problem under investigation?
3. What key concepts and variables did the authors formulate and study? How appropriately did the authors formulate the key concepts and variables? Is there an explicit set of hypotheses?
4. What methodology was used to gather data? Was this the best way to collect the data needed to answer the question posed in the study? Is it described adequately in the study?
5. Was there a pilot study performed, or is this a replication of a previous study?
6. Is the sample size appropriate? Have sample-related threats to internal validity been assessed?
7. Are other threats to internal and external validity considered explicitly in the study summary? Did the researchers attempt to minimize these threats? Were they successful?
8. What methodology was used to analyze the data? Was this the best analysis technique under the circumstances?
9. Critique the overall design of the research: what does it do well, and where does it fail?
10. Critique the implementation of the research design: did the researchers carry out their research design effectively?
11. Critique the presentation of the study and its results. Is it clearly written and logically organized? Are the conclusions suggested by the authors consistent with and substantiated by the results obtained? Are weaknesses in the study revealed and discussed? Are findings inconsistent with previous research discussed?
12. In your opinion, are the results of the research credible?
13. What was missing from the summary that may have been helpful to the process of critiquing the research?
Fennick, John H. (1997). Studies Show. Amherst , NY : Prometheus Books.
Giere, Ronald N. (1997). Understanding Scientific Reasoning , 4 th ed. New York : Holt, Rinehart, and Winston.