Conclusions arrived at by individual studies are often limited by constraints such as small sample size or confounding variables. A meta-analysis is a statistical approach that combines data from multiple studies in order to reveal variation or consistency in effects. Through meta-analysis, hypotheses can be addressed with a larger, more comprehensive sample, and the aggregation of information can result in conclusions that are are more broadly applicable than for a single study.
The first step in conducting a meta-analysis is completing a systematic review of the literature. The steps of a systematic review are:
- Formulate a research question. In order for meta-analysis to be effective, the question should be one that has been addressed by multiple individual studies.
- Come up with well-defined inclusion criteria for selecting studies that could be used to answer this question.
- Choose a database for conducting the search. Google Scholar is not the best option for this purpose. Consider instead more carefully curated resources, such as Web of Science.
- Define your search terms. Use Boolean operators (AND, OR, NOT). Make sure to write down every combination of search terms you try, and the number of results that the search returns. Also, specify the time range and type of literature (i.e. peer-reviewed journal articles) that define your search. It is very important to record all the details of your search methods – anyone should be able to replicate the searches later on.
- Start scanning studies, pulling those that meet the criteria as you go. Be sure to add clear details to the study inclusion criteria as questions arise.
After compiling studies that meet your inclusion criteria, the second step is to pull out effect sizes. An effect size is a quantitative measure of the effect that some event has on an outcome. Larger effect sizes indicate stronger associations between two variables. Examining effect sizes from different studies can reveal a pattern of association between two variables across different samples and study conditions.
Effect sizes come in many forms, and the effect size you pull from each paper will depend on whether the data are continuous or categorical, raw or presented as a model, etc. Examples of types of data that may be encountered are:
- effect sizes that the authors calculated within the study, such as Fisher's Z, Hedge's g, and Cohen's d.
- a correlation coefficient, which can in turn be converted to Fisher's Z.
- a table of outcomes in treatment and control groups, which can be used to calculate an odds ratio.
Figuring out how to extract effect sizes from publications is one of the most difficult parts of the process. Comprehensive Meta-Analysis is a user friendly, point-and-click software that allows you to choose from a long list of options for entering effect sizes. CMA is available on the computer in the Nunn lab, and although it is not as elegant and versatile as R, it is great software for getting started.
Once you have a list of effect sizes, it is recommended that you conduct your meta-analysis with the "metafor" package in R. This excellent manual provides information regarding all the options available with the metafor package. In addition, the package has a website containing updates, FAQ's, and examples.
Attached below are some examples of Nunn lab meta-analyses.