by Michael Turlik, DPM1 
The Foot and Ankle Online Journal 2 (6): 5
Different types of reviews are described and it is suggested that systematic reviews/meta-analysis are the best type of review to use to determine treatment efficacy. How to evaluate a systematic review is explained using as an example an article from the foot and ankle literature. This is another article in the ongoing evidence-based medicine series produced for The Foot and Ankle Online Journal.
Key words: Evidence-based medicine, review article, meta-analysis
Accepted: May, 2009
Published: June, 2009
ISSN 1941-6806
doi: 10.3827/faoj.2009.0206.0005
Is this treatment effective? The best way to answer this question is to review level I evidence for therapeutic decisions. [1] Previous articles in this series dealt with an introduction to evidence-based medicine2 and the critical analysis of randomized controlled trials. [3,4,5] Randomized controlled trials can be considered level I evidence for therapeutic decision-making. Depending on the sample size and the event rate randomized controlled trials may lack precision. Therefore, combining several randomized controlled trials on the same subject into a review article with a pooled estimate of effect if done so that bias can be minimized results in a more precise estimate of treatment effect.
A systematic review is a research article which uses explicit searching, evaluating and reporting criteria to minimize bias. The methods section of the article should clearly explain the criteria used to conduct the study. For therapeutic interventions if the articles used in the systematic review are limited to randomized controlled trials this type of study can be considered level I evidence. [1]
Narrative reviews/textbooks are usually expert-based rather than evidence-based. They are conducted using background questions rather than foreground questions. Trials may be selectively reported to match the author’s background and views. Typically these reviews do not report the method by which they have been compiled as a result, the conclusions of the review may be biased and it does not allow other researchers to replicate the results. Narrative reviews/textbooks lack transparency.
When critically analyzing a systematic review the general structure of the process is similar to a randomized controlled trial. [1] Does the study minimize the likelihood of bias (Internal Validity)? What are the results of the review? Can and should the results be applied to clinical practice (External Validity)?
Internal Validity
The planning and design of a systematic review follows well-defined criteria. (Table 1)
Table 1 Elements of a Systematic Review.
This allows for limiting bias and provides for transparency of the review process. A specialized type of systematic review which quantitatively pools results from randomized controlled trials is a meta-analysis. While all meta-analysis begin with a systematic review not all systematic reviews is a meta-analysis. Sometimes the studies are so different (heterogeneity) that the results may not be able to be pooled quantitatively. The purpose of this article is to provide information to allow podiatric physicians to critically analyze a systematic review. This will be accomplished by critically analyzing a meta-analysis of extra corporeal shock wave therapy (ESWT) for mechanically induced heel pain. [6] The following article in this series will specifically address the meta-analysis section of the ESWT paper.
Clinical Question
The first step in critically analyzing a systematic review is to evaluate the clinical question which the authors develop to guide the review process. The clinical question the authors formulate should be a focused, answerable, foreground question which utilizes the P (patient) I (intervention)
C (comparison) O (outcome) method. (PICO)
The question posed by the authors in our review article is “Our aim was to determine if ESWT is effective in the treatment of patients with plantar heel pain when compared with a control group”. [6] The question appears to be a focused, answerable, foreground question. Using the PICO method the patient, intervention and outcome are clearly defined however, the comparator was not. Was the comparator a placebo or a gold standard?
Inclusion and Exclusion Criteria
Which articles to include? Criteria which are too broad will increase the chances of heterogeneity. Which studies to exclude? Criteria which are too restrictive may result in loss of important studies. There are many different criteria to consider when performing a systematic review. (Table 2)
Table 2 Variables for inclusion and exclusionary criteria.
When reviewing the criteria for inclusion and exclusion the reader needs to consider the following question; would you expect the results to be similar across the range of patients included, the intervention studied, and the different outcomes measured? If the answer is no then the study criteria are probably too broad. The authors should specify with great clarity inclusionary and exclusionary criteria for the systematic review.
The authors of the ESWT study6 specified inclusion criteria for: the type of study, characteristics of the participants, outcome measure and type of comparator used. Exclusionary criteria consisted of heel pain caused by conditions other than mechanically induced. The other variables in table 2 were not addressed by the authors in the methods section. It would be likely that the results should be similar across the range of patients specified, the intervention selected and the outcomes used in the study.
Literature search
The search strategy of a systematic review should be comprehensive, detailed and exhaustive. The literature search should not be confined only to a MEDLINE review. (Table 3)
Table 3 Potential sources of information for systematic reviews.
In the methods section the authors should describe with sufficient detail the different types of databases which were queried and provide the search string used. Limiting the study to published articles and published articles only in English overestimates the treatment effect. An incomplete search results in retrieval bias.
It is common that journals selectively publish trials which demonstrate positive results. It is also more common to have positive trials published which have been sponsored by a pharmaceutical or device manufacturer which has a financial stake in the outcome. [7] Failure to publish trials or portions of trials with negative results leads to reporting bias. [8,9] The failure to publish an entire study because of results obtained is referred to as publication bias. When results and outcomes are selectively published by the authors this is referred to as selective outcome reporting bias. The result of reporting bias is an exaggerated treatment effect in the systematic review/meta-analysis.
The authors of the ESWT study describe in the methods section several different databases which were searched for original as well as, pre-appraised literature. Dissertations and reference lists of articles retrieved were searched as well (unpublished studies). The search was not limited to articles published in the English language. The search string was provided by the authors. A reference was provided which includes more details regarding the search strategy.
Article acquisition
Typically a single author will review the title and abstract of all references obtained from the search. The author will determine if the article meets the inclusionary and exclusionary criteria defined earlier in the study.
A complete copy of each article found to meet the predetermined criteria based upon the title and abstract then will be obtained for data abstraction. This information usually will be presented in the methods section.
The authors of the ESWT study did not clearly explain the article acquisition process.
Data abstraction
This refers to the process by which the data from the relevant articles is transferred for analysis in the systematic review. The process of data abstraction should be clearly defined in the methods section of the paper. Questions to consider when evaluating a systematic review for data abstraction are listed in table 4.
Table 4 Data abstraction questions.
It is important that more than one person participate in the data abstraction process to limit random and systematic errors.
The authors of the ESWT study described in detail in the methods section the data abstraction process. Two reviewers independently abstracted each of the randomized controlled trials obtained from the search. The authors describe clearly the data which was to be abstracted from each article for the study. Resolution of disagreements was explained and the process of contacting the authors for additional information was described. Other aspects of table 4 were not clearly reported by the authors.
Study Quality
An important part of the systematic review is to assess the quality of the studies selected for the review. Peer review and subsequent journal publication does not guarantee the quality of the published trial. The quality of systematic review is only as good as the studies used! Less rigorous studies overestimate the treatment effect.
There is no universal agreement as to the instrument which should be used to assess study quality. There are several important criteria to look at when evaluating the quality of a randomized controlled trial. (Table 5) It is important for the authors to describe and reference in the methods section the instrument used, if and how it was modified, and whether it was validated.
Table 5 Study quality criteria.
The authors of the ESWT study described and referenced in the methods section their study quality instrument. It included all of the elements in table 4.
Summary
The authors of the ESWT study have described in sufficient detail the methods used by which the study sought to limit bias. Therefore, it is reasonable to conclude that the inferences drawn from the study should be valid.
References
1. Levels of Evidence. 3 [Online], Accessed 2/17/2009.
2 .Turlik M: Introduction to evidence-based medicine. The Foot and Ankle Online Journal 2 (2): 4, 2009.
3.Turlik M: Evaluating the internal validity of a randomized controlled trial. The Foot and Ankle Online Journal 2 (5): 5, 2009.
4. Turlik M: How to interpret the results of a randomized controlled trial. The Foot and Ankle Online Journal 2 (4): 4, 2009.
5. Turlik M: How to interpret the external validity of a randomized controlled trial. The Foot and Ankle Online Journal 2 (5): 5, 2009.
6. Thompson C, Crawford F, Murray GD: The effectiveness of extra corporeal shock wave therapy for plantar heel pain: a systematic review and meta-analysis. BMC Musculoskeletal Disorders 6:19, 2005.
7. Rising K, Bacchetti P, Bero L. Reporting bias in drug trials submitted to the Food and Drug Administration: A review of publication and presentation. PloS Med 5: e217, 2008.
8. Guyatt, G, Drummond R, Meade M, Cook D (editors): Users’ Guides to the Medical Literature. McGraw-Hill medical, New York, 2008.
9. Hasenboehler E, Choudhry IK, Newman JT, Smith WR, 1 Ziran BH, Stahel PF: Bias towards publishing positive results in orthopedic and general surgery: a patient safety issue? Patient Safety in Surgery 1:4, 2007.
Address correspondence to: Michael Turlik, DPM
Email: mat@evidencebasedpodiatricmedicine.com
1 Private practice, Macedonia, Ohio.
© The Foot and Ankle Online Journal, 2009
