Apply methodological screening criteria to select highest quality studies possible: (p. 56-63)
To find the highest quality articles, reviewers should ask the following questions first (p. 56):
- Is this study's research design internally & externally valid?
- Are the data sources used in the study reliable & valid?
- Are the analytical methods appropriate given the characteristics & quality of the study's data?
- Are the results meaningful in practical & statistical terms?
Factors to keep in mind (p. 57-98):
- Determine research study design - the way the subjects of a study are organized and measured
- Experimental - uses a control group vs subjects undergoing the new program or intervention - adds new knowledge - more vigorous. Examples include:
- Randomized controlled trial - subjects assigned to groups randomly
- Parallel controls or nonrandomized controlled trials - subjects not assigned to groups randomly
- Self-control or longitudinal - pretests before and posttests after intervention
- Historical Control - use of normative (preexisting) data compared to a group
- Observational - uses existing information to draw conclusions on past or existing conditions & activities - describes current conditions - less vigorous
- Cohort - provide data about changes in a specific population
- Case Controls - determine similarities & differences between two groups (i.e. one with a medical condition, the other without)
- Cross-section - descriptive or survey data at one fixed point in time
- Address bias within the study - arise from unanticipated and/or unrecognized characteristics
- Selection bias - occurs when selection of experiment subjects leads to a result that is systematically different to the target population.
- Randomization - gold standard in reducing biases at the start of the experiment
- Blinding - gold standard for reducing bias during the experiment
- External validity - experimental results apply to the target population - avoid the following: (p. 82-84)
- Reactive effects of testing - a premeasure can sensitize participants to the aims of an intervention
- Interactive effects of selection - an intervention and the study's participants are a unique mixture
- Reactive effects of innovation - study subjects act uncharacteristically due to the artificiality of the experiment
- Multiple-program interference - difficulty in isolating the effects of the intervention because of the possibility that subjects are in other complementary activities or programs.
- Internal validity means the study is free from nonrandom error or bias - below are the potential threats to study accuracy (p. 81-82)
- Maturation - age of subjects over course of experiment & aging's effect on the results
- Selection - how people are chosen & assigned to groups for the study
- History - events in the world can bias study results
- Instrumentation - measures used to collect data are dependable
- Statistical regression - tendency of very high or low values to move toward the mean or average.
- Attrition - loss of data from subject's inability to finish study (they move, die, etc.)
- Sampling - a portion or subset of a larger group called a population (p 84-98)
- Random sampling - every member of a target population has an equal chance of being selected - least biased
- Systematic sampling - selecting every 5th, 10th or 100th subject from a randomized list
- Stratified sampling - population is divided into subgroups (strata) and a random sample then selected from each strata
- Cluster sampling - naturally occurring group such as schools, clinics, cities, states, etc.
- Convenience samples - probability of selection is unknown