| Steps in a Meta-Analysis |
|
There are four basic steps to any good meta-analysis: We will discuss each of these steps below. The first step in a meta-analysis is to find all of the pertinent articles on your topic. Important sources of information for a meta-analysis include:
While MEDLINE, the database of the National Library of Medicine, is a good starting point, it is not the only source of information. MEDLINE indexes approximately 4100 journals, dating from 1966 to the present. It also has an excellent feature called clinical queries. There are also CD-ROM based search engines from BRS Colleague, WinSPIRS, and others which offer different search options, but use essentially the same underlying database. The European version of MEDLINE is called EMBASE, and is a Dutch/English collaboration. Depending on the topic, it may be appropriate to search the more specialized National Library of Medicine databases, such as CancerLit, AIDSLine, and ToxLine. The Cochrane Collaboration Controlled Trials Register, established in 1993, is an important source of studies for a meta-analysis. The Register includes abstracts of thousands of trials. It includes all controlled trials in the MEDLINE and EMBASE, as well as the results of hand searches by Cochrane Collaboration volunteers of thousands of journals not indexed by MEDLINE or EMBASE. MSU students, faculty and staff can best access the Cochrane through the MSU Library. Remember Index Medicus? I'm old enough to recall the intimidating row of thick (and I mean THICK) books that were my only way to find medical research articles as a medical student in the 1980's. They can still be useful when it is important to search for articles published before 1966, when MEDLINE and the other electronic databases were established. Finally, there are other sources of "fugitive literature" that may be important for the author of a meta-analysis (some of which may be found in the Cochrane Controlled Trials Register):
It's important to know that different search strategies can result in different results (Table 4.5, Petitti):
Note: CTR = Controlled Trials Register If you are thinking about doing a meta-analysis of your own, it is important to enlist the aid of an expert Medline searcher such as a medical librarian. The above table also highlights the importance of using the Cochrane Controlled Trials Register. Once the author of a meta-analysis has assembled a large number of studies, it is important to select the right ones! There are a variety of possible inclusion (also called eligibility) criteria:
Once an appropriate group of studies has been identified, the author(s) have to abstract the relevant data from each study. There are many sources of potential error in data abstraction:
A good meta-analysis will take some or all of the following steps to minimize errors:
Bias can also creep into a meta-analysis. For example, the authors may be biased in favor of (or against!) well known researchers. Also, prominent journals may be given greater weight or authority (rightly or wrongly). It is therefore best (although not often done) to have identifiers eliminated from articles. Finally, part of the data abstraction phase is an assessment of study quality. Chalmers has proposed a fairly complex set of criteria which apply well to randomized controlled trials. Simpler criteria may be sufficient. For example, in a diagnostic meta-analysis, simply assuring a high quality gold standard, independent assessment of reference and study tests, and blinding may be adequate. Too often, the quality assessment is done, then ignored! Ideally, the results of the quality assessment should inform the analysis and interpretation of results. There are many issues and controversies in the analysis of meta-analytic data. First, let's define some important terms:
Fixed and random effects models can give very different answers, and you can create examples where either model gives counterintuitive results (see Petitti, page 92). Usually, though, the answers provided by these different modeling assumptions are similar. Differences only arise when studies are not homogenous. In a comparison of 22 meta-analyses, fixed and random effects models gave the same answer in 19 out of 22. In 3 cases, fixed effects models were significant while random effects models were not (Berlin, 1989 in Petitti textbook, pg 94). When there is significant heterogeneity, the between-study variance becomes much larger than the within, and studies of different sample size receive relatively similar weight. When there is homogeneity, sample size dominates, and both models give similar results. Random effects models are therefore more "conservative" and generate a wider confidence interval. Put another way, a random effects model is less likely to show a significant treatment effect than a fixed effects model. In general, if the studies are homogenous, the researchers should use a fixed effects model. If the studies are heterogeneous, the researchers (and you, the reader) should first ask why! While it may be appropriate to do a random effects analysis on all of the studies, it may be better to identify an important subgroup difference (i.e. studies using one dose showed significant effect, while lower dose did not) and then do a fixed effects analysis of each and report all of the results. A term you will encounter in many meta-analyses is "sensitivity analysis". A sensitivity analysis is a way of looking at only certain studies, certain groups of patients, or certain interventions. For example, a meta-analysis of aspirin in prevention of acute MI might first analyze all studies, but then also look separately at only studies of men and studies of women. The article by Hasselblad is an excellent starting point for budding meta-analysts, with lots of examples and formulae. Meta-analysis of diagnostic tests is another area of growing interest - how do you combine sensitivities, specificities, and so on. However, details of calculations for homogeneity, fixed effects models, and random effects models are beyond the scope of this course. Advanced students and those with a special interest in this topic may wish to review the following sections: |