Quality of Palliative Care Guidelines in Patients with Heart Failure: A Systematic Review of Quality Appraisal using AGREE II Instrument

Study design

The present study is part of a larger study conducted to implement palliative care CPGs using the ADAPTE approach; The ADAPTE approach provides a systematic approach to adapting guidelines produced in one setting for use in a different cultural and organisational context.^[31] The study was conducted according to the Preferred Reporting Items for Systematic reviews and Meta-analyses. To assess CPG quality, we initiated a comprehensive guideline review using AGREE II.

Inclusion and exclusion criteria

Criteria for including CPGs were: (1) Clear introduction as a clinical guide, (2) containing palliative measures for patients with heart failure, (3) the target population is adults over 18 years old, (4) preferably interprofessional, focusing on care rather than treatment and (5) approval by a national or international professional organisation. Primary studies, systematic reviews, pathways, textbooks, publications, short summaries, conference abstracts, letters, duplicate guidelines, guidelines that focus on only one dimension of palliative care or focus on diagnosis, definition and treatment were excluded from the study.

Literature search

The searches were conducted in April 2021 using the following databases: Excerpta Medica Database, MEDLINE/ PubMed, CINAHL and Guideline internet sites that were searched, including National Institute for Clinical Excellence, National Guideline Clearinghouse, Scottish Intercollegiate Guidelines Network, Guidelines International Network, Canadian Cardiovascular Society, American College of Cardiology and National Health and Medical Research Council.

Search strategy

Terms searched included (Guideline OR Recommendation OR Protocol OR Pathway) AND (‘Palliative care’ OR ‘Terminal Care’ OR ‘Hospice Care’ OR ‘End of life care’) AND (‘Heart failure’ OR ‘Cardiac Failure’ OR ‘Heart Decompensation’ OR ‘Myocardial Failure’) and no year limit. Language restrictions were not applied. In addition, references to each guideline and grey literature were reviewed for other relevant guidelines.

Data extraction/synthesis of results

Search results were uploaded to Endnote software and duplicates were removed. Two researchers (IB and HY) independently screened the titles and abstracts according to the eligibility criteria. Full texts were retrieved when abstracts gave insufficient information ([Figure 1] shows the study selection process). IB extracted the following information 19 CPGs: Purpose, guidelines titles, authors, publication year, country, the organisation that produced the guideline and main key recommendations. The full text of the 19 CPGs was evaluated by the research team (IB, HY, MB and DSH) during a face-to-face meeting. Disagreements between evaluators were resolved through discussion and seven CPGs were finally selected for quality evaluation [Figure 1].

Close

Figure 1:

Flow chart of CPGs search and selection.

Export to PPT

Data quality

To assess CPG quality, we used the Appraisal of Guidelines for Research and Evaluation (AGREE II) instrument. The AGREE II instrument consists of 23 items organised into six domains: (1) Scope and purpose (items 1–3); (2) stakeholder involvement (items 4–6), (3) rigor of development (items 7–14); (4) clarity of presentation (items 15–17); (5) applicability (items 18–21) and (6) editorial independence (items 22–23).^[32] Reviewers assessed each item and assigned a score from 1 (strongly disagree) to 7 (strongly agree).^[33] The overall quality of the guideline (1–7) and the recommendation was also assessed. A standardised score (0–100%) was calculated by summing up item scores within each domain for each reviewer, then standardising it as a percentage of the maximum possible score.^[34]

Five appraisers independently scored the guidelines using the AGREE II system. Four appraisers had a doctorate in nursing and one appraiser was a heart failure specialist physician. Before the evaluation, two training sessions were held (MB, one of the team members who specialised in using the AGREE II instrument, trained the others) to acquaint appraisers with how to use the AGREE II instrument. The instrument usage guide was provided to the appraisers in a printed file and the contact number of the principal investigator was provided for appraisers to ask any questions. To interpret the scores based on similar studies using AGREE II instrument, the CPGs fall into three categories: (1) Strongly recommended (guidelines that scored 50% or higher in all domains), (2) ‘Recommended with modifications’ (guidelines that scored 50% or more in the overall assessment) and (3) ‘not recommended’ (guidelines that do not score of 50% or more in all the domains or in the overall assessment).^[35]

Description of studies

We identified 1501 records from databases, guideline websites and manual searches. After excluding duplicate and irrelevant records, 66 records were considered to be potentially relevant; after selection, a total of seven CPGs satisfied the inclusion criteria [Figure 1]. [Table 1] summarised the features of each of the selected CPGs: Developing organisation, country, compilation method, year of publication, audience and target patient population.

AGREE II appraisal

[Table 2] contains scores in each domain based on the AGREE II appraisal. The quality of the CPG was relatively moderate-to-high overall. We did strongly recommend four of the seven CPGs, as they received a standardised score of 50% or greater on all domains (Guideline numbers 1, 3, 6 and 7). We recommended one guideline with modifications, as it received an overall assessment of 50% or greater (Guideline number 2). We did not recommend two guidelines, as they neither received a standardised score of 50% or greater on all domains nor an overall assessment of 50% or greater (Guideline numbers 4 and 5) [Table 2 and Figure 2].

Close

Figure 2:

Comparison of the quality of the clinical practice guidelines studied in this report.

Export to PPT

Domain 1. Scope and purpose are concerned with the overall aim of the guideline, the specific health questions and the target population.^[32] The mean score of all CPGs was 88.07% (SD = 13.34) and the scores of all CPGs were higher than 50%. The ‘Palliative care in heart failure New York Heart Association (NYHA) Classes III and IV’ and ‘CPGs for Quality Palliative Care’ guidelines achieved the highest score (100%) in this domain. The objective(s) of the guideline and the population (patients, public, etc.) to whom the guideline is meant to apply are specifically described in all guidelines, but in the three guidelines (2, 4 and 5), the clinical questions addressed by the guideline are not explicitly mentioned.

Domain 2. Stakeholder involvement focuses on the extent to which the CPG was developed by the appropriate stakeholders and represents the views of its intended users.^[32] The mean score of the CPGs was 70.9 % (SD = 24.3) and the scores of five CPGs were higher than 50% and two CPGs lower than 50%. The ‘Palliative care in heart failure NYHA Classes III and IV’ guideline achieved the highest score (100%) in this domain and ‘Consensus document and recommendations on palliative care in heart failure of the Heart Failure and Geriatric Cardiology Working Groups of the Spanish Society of Cardiology’ guideline achieved the lowest score (41.1%). The guideline developers include individuals from all relevant professional groups in all guidelines except number 5. The views and preferences of the target population (patients, public, etc.) have been sought for three guidelines (1, 6 and 7). The target users of the guideline are clearly defined in all CPGs except guideline 4.

Domain 3. The rigor of development relates to the process used to gather and synthesise the evidence and the methods to formulate and update the recommendations.^[32] The mean score of the CPGs was 48.71 (SD = 35.9). The scores of four guidelines were higher than 50%, but the scores of three CPGs were lower than 50%. The ‘Care of dying adults in the past days of life’ and ‘End of life care for adults: service delivery’ guidelines achieved the highest score (93.3%) in this domain. For only three of the seven guidelines (3, 6 and 7), systematic methods were used to search for evidence, the criteria for selecting the evidence and the strengths and limitations of the body of evidence are clearly described. In three guidelines (1, 6 and 7), the methods for formulating the recommendations are clearly described. In four guidelines (1, 3, 6 and 7), the health benefits, side effects and risks have been considered in formulating the recommendations in the guideline and there is an explicit link between the recommendations and the supporting evidence. Only two guidelines (6 and 7) were externally reviewed by experts before their publication. A procedure for updating the guideline is provided in only three guidelines (1, 6 and 7 only).

Domain 4. Clarity of presentation deals with the language, structure and format of the guideline.^[32] The mean score of the CPGs was 79.80% (SD=18.06). Scores of all guidelines were higher than 50% except for the ‘Consensus document and recommendations on palliative care in heart failure of the Heart Failure and Geriatric Cardiology Working Groups of the Spanish Society of Cardiology’ guideline, with a score of 48.8%. In all CPGs except guidelines 4 and 5, the recommendations are specific and unambiguous, the different options for management of the condition or health issue are presented and key recommendations are easily identifiable.

Domain 5. Applicability pertains to the likely barriers and facilitators to implementation, strategies to improve uptake and resource implications of applying the guideline.^[32] The mean score of the CPGs was 56.38% (SD = 26.04); the scores of four guidelines were higher than 50%, but the scores of three guidelines were lower than 50%. The ‘CPGs for Quality Palliative Care’ guideline, with a score of 83.3%, had the highest applicability score. Facilitators and barriers to guideline application are neglected in three of the seven guidelines (2, 4 and 5). Only guidelines 2, 3, 6 and 7 provide advice and/or tools on how the recommendations can be put into practice. The potential resource implications of applying the recommendations have been considered in all guidelines except guidelines 4 and 5. All guidelines, except guideline 2, present monitoring and/or auditing criteria.

Domain 6. Editorial independence is concerned with the formulation of recommendations not being unduly biased by competing interests.^[32] The mean score of the CPGs was 57.11% (SD = 29.23). The scores for five guidelines were higher than 50% and lower than 50% for two guidelines. The ‘Living and dying with advanced heart failure: A palliative care approach’ guideline with a score of 6.6% had the lowest score in editorial independence. Four guidelines (2, 3, 4 and 5) do not mention that the views of the funding body did not influence the content of the guideline. Except for guideline 2, all guidelines documented and expressed the potential conflict of interest of the members of the development team.

Overall assessment includes the rating of the overall quality of the guideline and whether the guideline would be recommended for use in practice.^[32] The mean score of the overall quality of the CPGs was 66.6% (SD = 26.44). Five of seven CPGs (1, 2, 3, 6 and 7) scored higher than 50% in the overall evaluation.^[36-40] In the classification of the quality of the guidelines according to the AGREE II score, the use of four CPGs was strongly recommended.^[36-39] The use of one CPG was recommended with modifications^[40] and the use of two CPGs was not recommended.^[41,42]

Limitations

Some of the appraisers had no previous experience with the AGREE II instrument. In our methodology, there was no blinding to either authors or organisations that developed these CPGs, which may be a potential source of bias. However, the research team members were already familiar with most of the identified CPGs, thus true blinding was not possible. Another limitation was the reliance on computerised searches to identify CPGs and references. Although a robust set of search criteria was formulated and tested before guideline identification, there is a possibility that some CPGs were missed. However, a paper search may have resulted in more human error.

Finally, it should be mentioned that quality scores reflect the appraiser’s assessment of the quality of each CPG. We are aware that the robustness of CPGs is more than just the AGREE II score. It has been a common pitfall to put undue emphasis on any aggregate ‘overall score.’ Moreover, it should be emphasised that no guideline is perfect. Therefore, we decided not to report this aggregate overall score and instead present all results in detail in [Table 2], to show separate results for each of AGREE II questions for every identified guideline.

What this study adds

According to the AGREE II evaluation, the quality of CPGs for palliative care of heart failure is moderate-to-high. We found methodological and applicability limitations affecting CPG quality. The future CPG developers should improve adherence to the AGREE II guidelines.

Author contributions

IB and HY conceived the presented idea. They also planned and conducted the literature search and the data collection. IB and MB contributed to the design and the analytical methods. MB and BI contributed to data interpretation. DSH collaborated in writing and editing the article. HY is responsible for the overall content of the article.