Evaluation of Clinical Practice Guidelines

linical practice guidelines (CPG) are systematically developed documents which are published to provide specific recommendations to standardize the process of diagnosis and treatment of common clinical disorders for clinicians, patients and healthcare administrators. They are derived from the best available evidence and current best practice. Authors of CPG’s gather appraise and combine evidence similar to systematic reviews however, unlike systematic reviews CPG’s provide actionable recommendations at a clinical level. It is hoped that by publishing guidelines that there is a decrease of ineffective care with a corresponding increase of effective care. Outcomes of CPG’s would be: increased consistency, higher quality care and more predictable healthcare processes. CPG’s are developed by various methods, by diverse stakeholders for different purposes. The process of developing a CPG should be transparent and minimize bias. The recommendations for patient care made should be clear and understandable. Each recommendation should reference the source of the information used to generate the recommendation.

The information cited by the guideline developers for each recommendation should be appraised for its quality and graded for its strength.Primary studies used to make the recommendations in the CPGs may be valid however, the strength will be graded lower if they demonstrate a small imprecise effect or have substantial risks and costs.Using the information provided by the authors of the guideline regarding study quality and grade the podiatric physician will be able to have confidence in the recommendations generated.CPGs are not flawless or a substitute for clinical judgment.CPG's should not be considered as mandatory statements since not all recommendations may be linked to high quality evidence.In addition, not all podiatric physicians may function in the same environment with the same patients used to develop the CPG's.CPG's may be valid but not relevant to the specific clinician or patient.
If CPG's are used by podiatric physicians they should have an understanding of how to critically analyze the guideline for validity, interpret the results and determine its generalizability to their unique situation.Systematic errors in the development of the CPG can distort the recommendations away from the truth.What can go wrong; incorrect search strategies can result in loss of important papers, inadequate appraisal and synthesis of the papers found can result in incorrect recommendations, wording, format or structure which is confusing can lead to misunderstanding.Finally, it is important to the podiatric physician to determine who paid for the study, if there was any conflict of interest involving the authors and what was done about it.The purpose of this paper is to provide instruction for podiatric physicians in evaluating CPG's.Two different CPG's dealing with a common podiatric complaint heel pain 1,2 will be compared and contrasted for validity and relevance.

Evaluating clinical practice guidelines
There are several published instruments in use to evaluate clinical practice guidelines.The Conference on Guideline Standardization 3 (COGS) developed and published an 18 item instrument to evaluate the validity of clinical practice guidelines.The Grading of Recommendations Assessment, Development, and Evaluation working group 4 (GRADE) was begun in 2000 to develop a common, sensible and transparent approach to grading the quality of evidence and strength of recommendations used in clinical practice guidelines.The AGREE instrument 5 was developed 2001 by an international group of researchers and policymakers.The AGREE instrument will be used in this article to critically analyze the clinical practice guidelines referenced earlier.
The AGREE website 5

Scope and purpose
This domain consists of three separate questions (Q) which evaluate the overall aim of the guideline, the specific clinical questions and target patient population.Answers to these questions allow the podiatric physician to determine if the CPG is relevant to his or her practice setting (generalizability).

Q1
The overall objective (s) of the guideline is (are) specifically described.
The American College of Foot and Ankle Surgeons (ACFAS) heel pain guideline 1 does not explicitly state an overall objective.The American Physical Therapy Association (APTA) guideline 2 is one of a series of guidelines produced by APTA.General purposes for the series of guidelines are explicitly stated.

Q2
The clinical question (s) covered by the guideline is (are) specifically described.
The development of foreground questions utilizing the PICO (Patient, Population, Problem, Intervention, Exposure, Comparison and Outcome) technique have been described elsewhere. 6The ACFAS heel pain guideline 1 does not explicitly state clinical question (s) to be covered.The APTA guideline 2 specifically describes two different tasks which it hopes to accomplish.

Q3
The patients to whom the guideline is meant to apply are specifically described.
Neither the ACFAS heel pain guideline 1 nor the APTA guideline 2 explicitly state the patients in whom the guideline is meant to apply.

Stakeholder Involvement
This domain is composed of a series of questions which focus on the extent to which the guideline represents the views of its intended users.The answers to these questions will help the podiatric physician in determining the CPG's relevance to his or her clinical practice.

Q4 The guideline development group includes individuals from all relevant groups.
The guideline development group should be diverse to include various stakeholders; end users, policy makers and consumers. 7It is of interest to the podiatric physician if the guideline development group includes a podiatrist.
The ACFAS heel pain guideline 1 was developed by podiatrists with membership in the ACFAS.No other groups were involved.The APTA guideline 2 was developed principally by physical therapists some with advanced degrees as well as, an orthopedic surgeon specializing in foot and ankle care.

Q5 The patient's views and preferences have been sought.
It is important in evidence-based medicine to include the values and concerns of patients. 8ere is no evidence that the ACFAS heel pain guideline 1 or the APTA guideline 2 included patient's views and preferences in developing the CPG.

Q6 The target users of the guideline are clearly defined.
From a podiatric physician's viewpoint it is important to consider whether the target user of the guideline specifically lists podiatry.
The ACFAS heel pain guideline 1 does not define target users of the guideline.
The authors of the APTA guideline 2 define the target users as orthopedic physical therapy clinicians, students, residents, academic instructors, clinical instructors, fellows and interns.Podiatric physicians are not mentioned in the APTA guideline as authors, reviewers or intended recipients.

Q7 The guideline has been piloted among targeted users.
The ACFAS heel pain guideline 1 has not been piloted among targeted users.The APTA guideline 2 authors provide a detailed and comprehensive explanation of the review process.The guideline was reviewed by multiple varied healthcare practitioners for feedback prior to being finalized.

Rigor of development
This domain relates to the process to gather and synthesize the evidence, the methods to formulate the recommendations and to update them.The answers to these questions will help the podiatric physician in determining the internal validity of the CPG.
Q8 Systematic methods were used to search for the evidence.
An earlier publication has covered this topic in some detail. 9The ACFAS heel pain guideline 1 provided no information regarding the search strategy in the development of the CPG.The authors of the APTA guideline 2 discussed why a systematic search could not be utilized in the development of the CPG.

Q9
The criteria for selecting the evidence are clearly described.
The ACFAS heel pain guideline 1 does not provide any criteria for selecting the evidence used in the development of the CPG.The APTA guideline 2 in the methods section described the criteria which were used to select the evidence used in the CPG.

Q10
The methods used for formulating the recommendations are clearly described.
An important step in the development of the guideline is the method by which the group's recommendations are made.The process used to arrive at consensus should be clearly described in the methods section.Three common methods used in healthcare are 10 the Delphi technique, the nominalgroup technique and the consensus-development conference.It is important for this process to be transparent to the podiatric physician evaluating the CPG for validity.
Neither the ACFAS heel pain guideline 1 nor the APTA guideline 2 describe the method used for formulating the recommendations.

Q11
The health benefits, side effects, risks have been considered in formulating the recommendations.
The grade of the evidence should be affected by the risks and adverse effects reported and therefore affect the strength of the recommendation. 11ither the ACFAS heel pain guideline 1 nor the APTA guideline 2 described clearly how the health effects, side effects and risks have been considered in formulating the CPG.

Q12 There is an explicit link between the recommendations and supporting evidence.
It is important to know if the recommendation is based upon rigorous clinical trials which minimize bias or consensus of expert opinion.The authors should describe the method by which they arrive at the quality and strength of the evidence and relate this to each recommendation.The GRADE (Grading of Recommendations Assessment, Development and Evaluation) method 11 of determining the quality and strength of the evidence used to make the recommendation is well documented in the evidencebased literature.

© The Foot and Ankle Online Journal, 2009
The ACFAS heel pain guideline 1 provides references in the text of the CPG.However, the evidence is not evaluated for quality or graded.The APTA guideline 2 describes explicitly and clearly the link between their recommendations made in the CPG and the supporting evidence.The supporting evidence is evaluated for its quality and graded.The recommendations based upon expert opinion or lesser studies are clearly stated.The method used to determine the quality and strength of the evidence is described in the methods section.

Q13 The guideline has been reviewed by experts prior to its publication.
There is no indication that the ACFAS heel pain guideline 1 was reviewed by external users prior to its publication.The APTA guideline 2 describes explicitly and clearly the review process prior to publication of the CPG.The CPG was reviewed by a diverse group prior to being implemented.It does not appear this included any podiatric physicians.

Q14 A procedure for updating the guideline is provided.
It is recommended that guidelines be reassessed and updated every three years. 12The ACFAS heel pain guideline 1 does not describe the procedure for updating the guideline.The APTA guideline 2 states that the guideline will be reviewed by 2012 or sooner if new evidence becomes available.A website is referenced for updates to the document prior to the anticipated complete review in 2012.The actual procedure is not described.

Clarity and presentation
This section deals with the language and format of the CPG.A guideline may be internally valid and may be generalizable to multiple clinical situations but, may not be written to allow a clear and unambiguous interpretation.

Q15 The recommendations are specific and unambiguous.
The ACFAS heel pain guideline 1 does make recommendations but they are not unambiguous and not specific.In contrast the APTA guideline 2 makes specific and unambiguous recommendations.

Q16 The different options for management of the condition are clearly presented.
The ACFAS heel pain guideline 1 and the APTA guideline 2 both describe different options for management in its CPG.

Q17
The key recommendations are easily identifiable.
The ACFAS heel pain guideline 1 and does not make the key recommendations easily identifiable however; the APTA guideline 2 does.

Q18 The guideline is supported with tools for application.
The entire document is usually large and cumbersome and if a more condensed easily accessed version is not produced for clinicians and patients it is unlikely that it will be utilized as effectively. 13e ACFAS heel pain guideline 1 does not provide any tools for application.The APTA guideline 2 does provide a single page listing the recommendations with the grade and strength of the evidence at the beginning of the publication.This allows the recommendations to be easily used by interested parties.

Application
The questions in this domain pertain to the likely organizational, behavioral and cost implications of applying the CPG.

Q19
The potential organizational barriers in applying the recommendations have been discussed.
Organizational barriers may limit the usefulness and application of the CPG.A recent article provides an in-depth discussion regarding this topic. 14ither the ACFAS heel pain guideline 1 nor the APTA guideline 2 discussed the potential organizational barriers in applying the CPG.

Q20 The possible cost implications of applying the recommendations have been considered.
Given the rapidly changing dynamics the health care policy of the United States it would be short sighted to not consider the cost implications of recommendations if data were available. 15ither the ACFAS heel pain guideline 1 nor the APTA guideline 2 discussed the cost implications of applying the recommendations.

Q21 The guideline presents key review criteria for monitoring and/or audit purposes.
The ACFAS heel pain guideline 1 does not provide any information concerning criteria for monitoring and/or audit purposes.The APTA guideline 2 recommends the use of validated self-reported instruments to monitor response to treatment and gives several examples.

Editorial independence
Q22 The guideline is editorially independent from the funding body.
The ACFAS heel pain guideline 1 does not discuss who has funded the study but explicit in the document is that it is authored by a committee from the ACFAS.It is not clear if the development of the guideline is editorially independent from the ACFAS.The APTA guideline 2 is authored by members of the orthopedic section of the APTA.It is not clear who has funded the guideline and if the authors are editorially independent of the APTA.

Q23 Conflict of interest of guideline members have been recorded.
It has been shown quite clearly that industry sponsored studies are likely to provide pro industry results. 16It is important for guideline developers to provide information to the users regarding how conflict of interest was dealt with when found. 17It is thought that the most common source of bias in CPG's is financial. 18In a survey of physician authors of CPG's 87% had some form of interaction with the pharmaceutical industry. 19ither the ACFAS heel pain guideline 1 nor the APTA guideline 2 discussed a conflict of interest process.

Response scale
Each of the 23 items of the AGREE instrument are individually evaluated on a four-point scale. 5The scale measures the extent to which the item has been fulfilled.The higher the number the greater the AGREE criteria have been met by the authors of the guideline.Comparing the two different guidelines (Table 2) the guideline produced by ACFAS scored lower using the AGREE instrument when compared to the APTA guideline.In a review of CPG's published by specialty societies the authors found that 88% did not report information regarding the search strategy and 82% did not report recommendations specifically linked to the quality and grade of the evidence used. 20This is consistent with the results of the CPG produced by the ACFAS.Neither guideline scored well in the domains of applicability and editorial independence.This is consistent with other reviews of CPGs 21,22 which found that applicability and editorial independence domains were rated lowest using the AGREE instrument.

Conclusion
Older clinical practice guidelines are characterized by narrative reviews and expert opinions without explicit evaluation of the best evidence available. 23Based upon the results of the AGREE instrument the ACFAS clinical practice guideline follows an older expert based format.The APTA clinical practice guideline follows a more contemporary approach to the development of clinical practice guidelines.It is characterized by its adherence to evidence-based principles.APTA guidelines contain clear explicit actionable recommendations which are linked to evidence which has been evaluated for grade and strength.The APTA guideline does not provide comprehensive recommendations for medical treatment and no recommendations for surgical treatment of heel pain thus limiting its relevance to practicing podiatric physicians.

The Foot and Ankle Online Journal, 2009
©

Table 1
AGREE quality domains.

Table 2
Results of comparison ACFAS / APTA guidelines using the AGREE instrument.