FACT Updates its Clinical Outcomes Corrective Action Plan Policy for Transplant Programs
Clinical transplant programs accredited under the FACT-JACIE International Standards for Hematopoietic Cellular Therapy Product Collection, Processing, and Administration have always been expected to follow and evaluate clinical outcomes of transplantation, including both individual and aggregate data. Since the sixth edition in 2015, Standards have also required that clinical outcomes be assessed against national or international outcome data, that program outcomes should meet or exceed national or international benchmarks, and that programs not meeting the expected range of survival must submit to FACT a corrective action plan (CAP).
In the United States, all allogeneic hematopoietic transplants must be reported to the CIBMTR. Since 2016, the current FACT process for review of clinical outcomes requires that Clinical Programs performing allogeneic transplants must meet the expected range of one-year survival as published in the CIBMTR Transplant Center-Specific Survival Report. If the one-year survival for first allogeneic transplants falls below the risk-adjusted expected range, the program must submit a CAP that is reviewed by the FACT Clinical Outcomes Improvement Committee. The Committee advises the Clinical Program, monitors implementation of the elements of the CAP, and follows the program’s internal data assessing the effectiveness of the CAP and improvements in survival.
The CIBMTR report published in December 2021 included first allogeneic transplants for the years 2017-2019. This was the first report in which all included years were within a timeframe in which the Clinical Outcomes Improvement Committee had been requesting and reviewing CAPs. This report presented an opportunity to evaluate the effectiveness of the Committee’s processes. The Committee reviewed data collated from CIBMTR reports, internal data submitted by Clinical Programs, and Committee determinations on the appropriateness and effectiveness of CAPs. It determined that the likelihood that a CAP will successfully improve one-year survival can be predicted by the duration of time in which the program’s risk-adjusted one-year survival was below the expected range, trends in survival rates based on the program’s internal data, and the ability and effort demonstrated by the program.
Based on this analysis, the FACT Board of Directors approved additional steps added to the Clinical Outcomes Corrective Action Plan Policy which include:
- If the risk-adjusted one-year survival remains lower than the expected range and does not improve for three consecutive reporting years, the Clinical Outcomes Improvement Committee will formally evaluate in more depth the Clinical Program’s ability and effort to improve survival.
- The Committee may determine that ability and effort are sufficient, that more time is required to demonstrate the required improvement, and/or that other contingencies exist.
- If the Committee determines that the program has not demonstrated sufficient ability or effort, the program’s accreditation will be placed on probation. Probation is defined as a period of time during which a program has not met specific defined criteria and is at risk of losing FACT accreditation if additional requirements are not met. This status is documented in the program’s FACT application report and internal FACT records but is confidential and will not be published or released.
- Programs on probation for clinical outcomes remain accredited but may be required to complete a focused reinspection. The scope of the reinspection will be based on the program’s specific deficiencies in evaluating and improving one-year survival. It is our experience that the intensified support provided by the Committee during the reinspection process leads to sustained improvement in survival.
- The timing of the focused reinspection will be determined by the Committee, independent of the program’s normal accreditation cycle.
- If the program does not demonstrate satisfactory correction of the deficiencies, the program’s accreditation may be suspended in accordance with FACT’s policy on Suspension or Termination of Accreditation. The Committee will determine the timeframe in which deficiencies must be corrected and one-year survival must improve. Failure to demonstrate correction of the deficiencies within the established timeframe may result in termination of the accreditation.
We note that the potential for accreditation suspension has always been a possible consequence for low one-year survival as published in the current version of the Clinical Outcomes Corrective Action Plan Policy; however, these changes provide a framework based on data in which the Clinical Outcomes Improvement Committee can formally make such a decision. The ultimate goal of FACT and Clinical Programs throughout this process is to improve outcomes for patients. We believe this process will improve patient survival.
For more details about the CIBMTR Transplant Center-Specific Survival Report, see the article published in November 2022 at https://news.factglobal.org/from-the-archives-understanding-the-cibmtr-outcomes-reports/.
From the Archives: Understanding the CIBMTR Outcomes Reports
The publication of the 2022 CIBMTR Transplant Center-Specific Survival Report is expected in December 2022. To help Clinical Program Directors prepare to evaluate their programs’ results, this volume’s From the Archives feature is, “Understanding the CIBMTR Outcomes Report,” published in October 2016.
Many of the concerns outlined in this article are still presented to FACT today, and we added more information and updated references to address recent questions. Our hope is that increased understanding of the CIBMTR report will transform its reputation as an anxiety-inducing evaluation into a reputation for what it is meant to be: one of many resources available to transplantation programs to understand and improve one-year survival.
There are many great Just the FACTs newsletter articles related to clinical outcomes, and it was difficult to choose! To see more helpful articles about clinical outcomes, click here.
Happy reading!
Understanding the CIBMTR Outcomes Reports
Allogeneic transplant programs in the U.S. are required to submit corrective action plans to FACT when they do not meet the expected range of one-year survival in the annual CIBMTR Transplant Center-Specific Survival Report. To adequately assess lower-than-expected one-year survival, it is necessary to fully understand how the report is generated.
CIBMTR reviews its methodology for the Center-Specific Survival Report every other year at the Center Outcomes Forum. This forum includes a variety of stakeholders, including transplant physicians, and is intended to also increase transparency and understanding. The agendas and summaries of each meeting can be found on the CIBMTR website. CIBMTR also explains the report methodology in the document titled, Methodology Employed for Annual Report on Hematopoietic Cell Transplant Center-Specific Survival Rates, which was last updated in December 2021. The document provides technical information on statistics, but offers the following summary:
“A fixed effects censored data logistic regression model is fitted to survival data for first unrelated and related donor hematopoietic cell transplants at U.S. centers. The model is adjusted for recipient age, recipient race, Karnofsky/Lansky score, Sorror HCT-CI, adult BMI group, pediatric BMI group, recipient CMV status, history of mechanical ventilation, history of invasive fungal infection, prior autologous transplant, disease/stage, AML ELN risk group, AML transformed from MDS or MPN, AML therapy related, number of induction cycles for AML in CR1, interval from diagnosis to transplant in ALL and AML in CR2 and CR3+/relapse, ALL cytogenetic risk group, Philadelphia positive-status in ALL, number of induction cycles for ALL in CR1, MDS IPSS-R risk score at HCT, MDS predisposing condition, CLL and other chronic leukemia disease status, NHL subtype, sensitivity to chemotherapy in NHL and HL, MM ISS stage at diagnosis, plasma cell disorder disease status, year of transplant, donor type/graft type/HLA matching, BM or PBSC donor/recipient sex match, unrelated BM or PBSC donor age at transplant, and recipient median household income. The report on transplant center-specific survival rates helps to identify centers that may have under-performed or over-performed compared to the overall network of transplant centers during this specified time period.”
Each year, CIBMTR provides a comprehensive version of the Center-Specific Survival Report directly to programs prior to publishing the results on the Be the Match website. The report is usually provided to programs annually in mid-December. In each of these reports, CIBMTR outlines in detail the risk factors that are considered and ultimately included in the final multivariate model used to produce the report that year. Each report includes sections on methods, statistical analysis, and results that describe how the model is formulated. Not only does this provide context for individual programs’ results, but it also provides information that any program can use regarding factors that have had a statistically significant impact on one-year survival.
CIBMTR also explains the report, and its tools and resources to help Clinical Programs use their data, during a 2016 FACT webinar recording titled, Using CIBMTR Data to Determine and Evaluate Clinical Outcomes, presented by Stephen Spellman, MBS. This recording gives an overview of how CIBMTR data is used to determine outcomes and how Clinical Programs can use additional data to further evaluate outcomes and improve.
The following are common points Clinical Programs question about the CIBMTR report, and how they can be evaluated in the context of creating corrective action plans to address lower-than-expected one-year survival:
- High-risk patients: Some corrective action plans state the root cause of death to be transplants for high-risk patients. As outlined above, the CIBMTR report is risk-adjusted. High-risk patients should be accounted for within the report. The FACT committee expects corrective actions that specifically address the causes of death. Broad refusal to transplant patients with high risk is not the intent of FACT requirements. As transplants are often the last hope for patients, careful attention to trends in causes of death is particularly important for these patients to improve their outcomes. For example, some programs have determined myeloablative therapy was not necessary or beneficial for a group of frail patients; others adjusted their protocol for preparative regimens.
- Socioeconomic factors: Some programs are located in regions in which their patients have a low socioeconomic status. Social factors do have an impact on a patient’s survival, and FACT understands that these factors are difficult to include in a risk-adjusted methodology. In these cases, FACT would expect to see a corrective action plan that targets social issues identified to be a root cause of patient deaths. Programs have implemented methods to successfully address low resources and lack of caregivers, such as pre-discharge dispensing of prescription medications, bus passes, and expanded clinic hours.
- Small programs: It is difficult to identify trends among a small number of transplants, but FACT will look for a good-faith effort of the program to review data and determine if a trend can be found. One small program found that its patients had a high rate of CNS disease, and educated its network of referring physicians.
- Confidence interval: A common worry is that Clinical Programs will have one-year survival lower than the expected range, through no fault of their own, because of the 95% confidence interval. It is important to realize that each program has its own confidence interval. A defined number does not have to drop out of the curve. Therefore, it is possible for each program to meet expected one-year survival. Small programs typically have a wider expected survival range.
- Delay in reporting: Due to the inherent timeframe of “one-year” survival, the CIBMTR report is delayed by two years because the transplant has to have occurred a year prior, and an additional year is needed to analyze the data. Furthermore, the report uses three years’ worth of data. For example, data analysis may show that a Clinical Program had a particularly bad year in 2019 that resulted in lower-than-expected one-year survival in the 2021 report. That year will affect one-year survival in the next two reports for 2022 and 2023. However, it is still necessary to review the causes of death and their root causes for the timeframe of the report. Programs must submit current internal survival data as part of their corrective action plans. Upward trajectories of internal data are taken into account.
- Overall one-year survival: The CIBMTR report only provides overall one-year survival; however, drilling down into specific diseases will help Clinical Programs determine root causes and which corrective actions may help. This is the same for treatment-related mortality or disease relapse. This type of drill-down has helped programs identify root causes.
- Data errors: Some Clinical Programs have noted that errors in the data submitted to the CIBMTR were the true root of lower than expected one-year survival. Indeed, this can affect results of the algorithm. If data errors are a problem, FACT will want to see corrective actions related to accurate data management and evidence that these corrections have had an impact on the risk-adjusted survival.
Transforming Data into Knowledge: How Sarah Cannon Analyzes Risk-Adjusted and Unadjusted Survival Outcome Trends and Projections
By Therese Dodd, BA, MBA, RN, CPHQ, FNAHQ
Quality and Accreditation/Regulatory Compliance Specialist
Transplant & Cellular Therapy Network
Sarah Cannon
1100 Dr. Martin L. King, Jr. Blvd, Suite 800, Nashville, TN 37203
The centers of Sarah Cannon Transplant & Cellular Therapy Network (SCTCTN) are not alone in their need to understand clinical outcomes, the definitive outcome being patient survival. It falls to the centers to transform an abundance of data into knowledge for performance improvement.
The Center for International Blood & Marrow Transplant Research (CIBMTR®) Transplant Center-Specific Survival Report provides survival data to eligible hematopoietic cell therapy (HCT) centers for first allogeneic transplants for a three-year period to at the end of each calendar year. The report displays a “–1” in the performance status column for centers whose actual survival is lower than the 95% confidence limits for predicted survival, a “1” if performing above the confidence limit, and a “0” if performing as predicted. The SCTCTN centers recognized that reading the report is just a first step. One challenge to understanding the survival results is that these reports only display the information from preceding years as a table (see example in Figure 1). Supplying information to centers in this manner may provide them a false sense of confidence if they do not trend the specific outcomes results. In the figure 1 example, the “-1” assigned in the 2015 report may have been an unpleasant surprise if the center wasn’t rigorously analyzing its clinical outcomes internally.
The SCTCTN centers have created box graphs, similar to the Excel template available on the FACT website in the Clinical Outcomes Resource Center, to track the clinical outcomes from the CIBMTR® Transplant Center-Specific Survival Report . As can be seen in Figure 2, there was a decline in actual % 1 year survival for transplants performed during 2010-12, dropping from greater than the % predicted survival in the previous report to lower than % predicted survival and approaching the lower limit of the 95% confidence interval.
With the 2011-13 report, the 95% confidence interval increased, which indicates lower-risk patients were transplanted. Since the report is risk-adjusted, there was an expectation that patient survival would increase; however, the % actual survival was worse than the lower limit of the 95% confidence interval. Fortunately, the center had already started root cause analysis and identified several opportunities for improvement including, but not limited to, increasing data abstraction and entry accuracy, strengthening survivorship programs, and expanding caregiver support services. The center was able to improve its clinical outcomes by the time of the subsequent report.
Figure 2.
In addition to tracking data from the CIBMTR® Transplant Center-Specific Survival Report, the SCTCTN have overlaid a trend line to the box graphs (see black dotted line in Figure 2) of actual % survival using data from the CIBMTR® enhanced data back to centers (eDBtC). Based on experience, and not surprisingly, SCTCTN centers determined that the eDBtC % actual survival for first allogeneic transplants tracked closely to the CIBMTR® report data year over year. Because of this, the centers can determine an estimate that is prognostic for the actual % survival in the upcoming report by finding the % survival at 12 months using the CIBMTR® eDBtC “Survival Probability – HCT” functionality (see Figure 3).
Figure 3
In addition to annual analysis of the CIBMTR® Transplant Center-Specific Survival Report, the SCTCTN centers monitor actual autologous and allogeneic hematopoietic cell transplant (HCT) % 30-day, 100-day, and 1-year survival and treatment-related mortality in smaller timeframe cohorts on a quarterly basis and more concurrently. Tracking these data (as well as many other quality and outcome metrics) supports rigorous quality management programs that improve the experience for SCTCTN patients.
Assessing Outcomes of Non-Traditional Cellular Therapy Products
As cellular therapies expand, accredited cellular therapy programs often become involved in the development, implementation, and performance of protocols and procedures for indications outside of hematological or malignant diseases. Accredited programs must meet the FACT Standards for these nontraditional therapies as they apply, regardless of the patient population served. The lines can become blurred when non-accredited services are also provided in the institution. While accredited Clinical Programs are expected to utilize collection and processing facilities that meet FACT Standards, the converse is not true. An accredited Apheresis Facility or Processing Facility may serve both accredited and non‐accredited clinical programs. A “clinical program” may be simply a medical service in your institution that is not part of your accredited transplant or immune effector cell program, such as neurology or nephrology.
This article will review the responsibilities of Clinical Programs, Collection Facilities, and Processing Facilities for analyzing outcomes of nontraditional cellular therapies. To start, let’s review the requirements for internal clinical outcome analysis. Standards exist in each part of the Standards (clinical, collection, and processing) and all are similar. They require that documentation and review of outcomes and efficacy are outlined in the Quality Management (QM) Plan and/or policies and procedures, that criteria are determined and reviewed at regular time intervals, and that both individual and aggregate data are evaluated for each type of cellular therapy product and recipient.
B/C/D4.7 The Quality Management Plan shall include, or summarize and reference, policies and Standard Operating Procedures for documentation and review of outcome analysis and cellular therapy product efficacy to verify that the procedures in use consistently provide a safe and effective product.
B/C/D4.7.1 Criteria for cellular therapy product safety, product efficacy, and the clinical outcome shall be determined and shall be reviewed at regular time intervals.
B/C/D4.7.2 Both individual cellular therapy product data and aggregate data for each type of cellular therapy product and recipient type shall be evaluated.
Clinical Programs
If the physicians of the FACT-accredited cellular therapy program have no responsibility for the evaluation or management of patients treated with a cellular therapy, the Clinical Program is not responsible for assessing the efficacy of that cellular therapy because it is administered outside of its control.
However, if the accredited program’s physicians are involved in management of these patients, the Clinical Program is responsible for all aspects of the cellular therapy, whether or not the patients are housed on the accredited program’s unit. If the physicians are involved in the management of these patients, the patients and products are considered part of the program and QM program.
If the patients are housed on the accredited Clinical Program’s unit but the programs’ physicians are not involved, this is essentially “renting space” and considered a “nonaccredited service” ‐ a cellular therapy program existing in the same institution but not under the FACT accreditation. This would not be part of the accredited program or its QM responsibilities.
Collection Facilities
If collection of nontraditional cellular therapy products is performed within a FACT-accredited Apheresis Collection Facility, the facility is responsible for treating them as it would any other product. All standards for collection apply, including written procedures; training; collection orders; parameters; end points of collection; labelling; transport and shipping; chain of identity/chain of custody; release criteria; and processes to prevent mix‐ups, contamination, or cross‐contamination.
As part of the accreditation process, FACT expects that all cellular therapy products collected be listed on the apheresis collection facility grid, regardless of whether the products ultimately went to the accredited Clinical Program or to patients on another service. If the Collection Facility cannot evaluate efficacy, it should at a minimum evaluate product quality by assessing some designated parameters; for example, achievement of collection goals, contamination rates (or lack thereof), proper product labeling and storage, and integrity of the product (e.g., bags are intact).
Processing Facilities
Measures of cellular therapy product efficacy may be the responsibility of the clinical teams, but the FACT-accredited Processing Facility is still responsible for providing a safe product. This includes all of the standard laboratory quality attributes: procedures; training; equipment maintenance and calibration; facility cleanliness; chain of identity/chain of custody; labeling; and prevention of mix‐ups, contamination, and cross‐contamination. If performing any processing or cryopreservation of the product, some measure of post‐processing recovery or post‐thaw viability might be used to assess product quality in addition to testing for bacterial or fungal contamination.
FACT would expect all of these cellular therapy products to be listed on the Processing Facility grid and incorporated into the QM plan and evaluated as any cellular therapy product.
FACT Resources to Assist Your Program Throughout the Accreditation Process
FACT aims to provide organizations with valuable resources to aid in achieving and maintaining their accreditation. Our website has useful documents for organizations in all phases of the accreditation process. However, we understand that finding the right information can become overwhelming at times especially for organizations completing their initial application or for personnel new to the accreditation process.
The following overview provides categories, definitions, and the location of important resources. We encourage you to share this information with all personnel involved in your organization’s FACT accreditation process:
FACT Standards
The FACT Standards webpage contains links to the most recent versions of the four sets of Standards published by FACT:
- HCT Standards – FACT-JACIE International Standards for Hematopoietic Cellular Therapy Product Collection, Processing, and Administration
- Common Standards – Common Standards for Cellular Therapies
- Immune Effector Standards – Standards for Immune Effector Cells
- Cord Blood Bank Standards – NetCord-FACT International Standards for Cord Blood Collection, Banking, and Release for Administration
The pages for each set of Standards contain useful documents relevant to the current editions of FACT Standards, including, but not limited to:
- FACT Standards and Accreditation Manuals
- Summary of Changes, Compare, and Crosswalk documents to identify changes between the previous and current versions of FACT Standards, if applicable.
- Forms used during the compliance application submission, including but not limited to Document Submission Requirements, Educational Activities, Critical Electronic Record Systems, and Facility Site Grids.
The above resources can also be found in the Cellular Therapy Library and Cord Blood Bank Library.
Accreditation Process Requirements
The Cellular Therapy Accreditation Process Requirements and Cord Blood Bank Accreditation Process Requirements provide a structured overview of the accreditation process, from completing the Compliance Application to requirements after the on-site inspection.
Self-Assessment Tools
The Self-Assessment Tools can be used to assess readiness for FACT accreditation. They are located in the Cellular Therapy (CT) and Cord Blood Bank (CB) Libraries, and you will find individual self-assessment tools for each current set of FACT Standards:
- Hematopoietic Cellular Therapy Self-Assessment Tool, Seventh Edition
- Common Standards Self-Assessment Tool, Second Edition
- Immune Effector Cells Self-Assessment Tool, First Edition, Version 1.1
- CB Self-Assessment Tool, Seventh Edition
Educational Resources
FACT’s comprehensive, robust training and development program is responsive to the current needs of inspectors and organizations, providing extensive opportunities for all skill levels in all corners of the world. With a focus on real-life examples and practical application, FACT educational sessions provide tools not only for achieving accreditation, but for ensuring quality patient care in accordance with the Standards. The Education and Resources page contains links to upcoming and previously recorded events:
FACT Accreditation Coordinator
FACT’s Accreditation Coordinators provide these and additional FACT resource links in an email sent to organizations at the beginning of the compliance application process. Your accreditation coordinator is available throughout the accreditation process to answer questions via email or phone. The coordinator’s contact information is available in the accreditation portal.
Clinical Outcomes Resource Center
FACT supports blood and marrow transplant (BMT) programs’ efforts to evaluate one-year survival data to improve patient outcomes, advance the field, and maintain payer and public confidence. The Clinical Outcomes Resource Center includes comparative data sources, educational resources, guidelines for corrective action plans (CAP), as well as CAP examples.
Data Management Resource Center
The FACT-CIBMTR Data Audit Committee reviews implementation, adequacy, and effectiveness of corrective action plans with the goal of providing education and assistance to programs throughout the accreditation cycle to achieve quality improvement in data management. The Data Management Resource Center provides CIBMTR and educational resources, guidelines for data management submissions and examples, and audit report examples and templates.
Immune Effector Cells (IEC) Resource Center
FACT is committed to supporting efforts to make quality immune effector cellular therapy accessible to patients much like it did for blood and marrow transplantation and cord blood banking. Several resources are available to assist with understanding immune effector cells and implementing FACT Standards. The Immune Effector Cells (IEC) Resource Center includes resources to assist with understanding immune effector cells and implementing applicable FACT Standards.
Portal Resource Center
The Portal Resource Center provides access to all available resources and training materials for navigating and assisting organizations in completing applications in the portal.
FACT News
Use the links below to access recent news releases, as well as current and previous volumes of our newsletter:
We hope that you find this information useful as you prepare for your next accreditation cycle. If you have additional suggestions regarding how FACT can better inform organizations of the resources available, please email fact@unmc.edu.
Biology of Blood and Marrow Transplantation Publishes Article on FACT Clinical Outcome Review Process
The journal Biology of Blood and Marrow Transplantation (BBMT) has electronically published the article, Integration of Publicly Reported Center Outcomes into Standards and Accreditation: The FACT Model (LeMaistre et al, 2019). Combining historical information with aggregate data collated from corrective action plans (CAPs) submitted to FACT, the article provides an overview of FACT’s process for evaluating one-year survival and preliminary themes found in CAPs.
The full abstract states:
“The rapid evolution of Blood and Marrow Transplant (BMT), coupled with diverse outcomes associated with heterogeneous groups of patients, led to the formation of two important organizations early in the development of the field: the Center for International Blood and Marrow Transplant Research (CIBMTR) and the Foundation for the Accreditation of Cellular Therapy (FACT). These organizations have addressed two of the nine elements identified by the National Quality Strategy (NQS) [1] for achieving better health care, more affordable care, and healthy people and communities: first, a registry that promotes improvement of care and, second, accreditation based on quality standards. More recently, a federally-mandated database in the United States addresses the third element of the NQS: public reporting of treatment results.
This paper describes the current process by which FACT incorporates patient outcomes reported by the CIBMTR into standards for accreditation, the requirements for accredited programs whose performance is below expected outcomes to maintain accreditation, and preliminary findings in assessing corrective action plans intended to improve outcomes.”
The paper is a useful reference for understanding FACT requirements for evaluating one-year survival, generating ideas for outcome analyses within transplant programs, and understanding the value of setting expectations for patient outcomes within the context of voluntary accreditation programs.
LeMaistre, C.F., Wacker, K., Akard, L., et al. (in press). Integration of publicly reported center outcomes into standards and accreditation: The FACT model. Biology of Blood and Marrow Transplantation. doi: 10.1016/j.bbmt.2019.06.035
Trending Clinical Outcomes
By: Kara Wacker, MBA, FACT Strategic Planning Administrator
Clinical outcome analysis might have started gaining renewed attention in recent years, but FACT has required such analysis since its first Standards in 1996. This puts the HPC transplant field at an advantage as more stakeholders and patients are evaluating health care institutions in the context of actual clinical outcomes. Transplant programs that have voluntarily complied with FACT Standards over the years have outcome data that can be useful not just for retroactive investigations but for anticipating lower-than-expected outcomes in the future . . . and addressing them before undesirable outcome trends result.
When programs are required to submit a corrective action plan to address lower-than-expected one-year survival (per the FACT-JACIE Hematopoietic Cellular Therapy Standards), a common question asked of FACT is, “what time frame should my plan address?” The time frame should include current data, but should also go back in time far enough to detect any trends.
For example, autologous-only or non-U.S. programs often use the C.W. Bill Young Cell Transplantation Program’s U.S. Patient Survival Report or the Be the Match/NMDP Disease-Specific HCT Indications and Outcomes Data as comparative data sources. These reports are by year, but a single year is not long enough to adequately evaluate outcomes to detect problems.
Some allogeneic programs in the U.S. assume the time frame is limited to that used in the Transplant Center-Specific Survival Report published annually by the Center for International Blood and Marrow Transplant Research (CIBMTR). These time frames consist of three calendar years, with the last year being two years prior to report publication. The FACT Clinical Outcomes Improvement Committee does expect that these years be included; however, programs are not limited to this time frame. In fact, survival from before the time frame in the CIBMTR report will help programs detect when a declining trend began. Data from years following the CIBMTR report illustrate a complete picture of the current survival rates of the program, and can be used to determine effectiveness of corrective actions. Such data should readily be available from internal outcome analyses required by the Standards. They can also be obtained from CIBMTR’s Data Back to Centers (eDBtC) resource.
A FACT-accredited program shared a trending tool with FACT, and gave us permission to anonymously make it available to other programs on the FACT website. This Microsoft Excel-format Risk and Non-Risk Adjusted Survival Data and Box Graph has fields for entering figures from multiple CIBMTR reports over time (such as number of first-transplant patients, risk-adjusted survival, and endpoints of the expected range), plus non-risk adjusted one-year survival rates from internal data or eDBtC. After inputting this data, a graph on the second sheet is automatically populated.
An example is below. In this example, the program met clinical outcomes in both the 2013 and 2014 cohorts (2009-2011 and 2010-2012, respectively). However, the program’s survival rate declined. By the next report in 2015, the program did not meet expected one-year survival. As a program’s risk-adjusted actual survival declines and gets closer to the lower limit of the 95% confidence interval, it can begin evaluating causes of death of its patients and implementing corrective actions before one-year survival becomes lower than expected.
If survival rates are declining, questions a program may want to ask include:
- Is there a difference in causes of death, or have any specific causes increased?
- Did disease/patient case mix change?
- Are protocols still being followed?
- Are there any changes in psychosocial patterns?
- Do records adequately follow patients?
- Do protocols need updated to current science?
Programs often fall in and out of expected survival ranges, and they should monitor how close their survival rate is to the lower limit. Avoid complacency just because survival was within the expected range; make a genuine effort to robustly complete required internal outcome analyses to detect problems before the CIBMTR report does. After all, we all know the real goal isn’t a score of “0” on the report. It is improving patient survival.
Evaluating Causes of Death to Improve Survival
Evaluating one-year survival, and the impact of efforts to improve survival, are challenging tasks for both FACT and accredited Clinical Programs. Hematopoietic progenitor cell (HPC) transplantation itself is complex before considering the confounding variables and unique patient characteristics required for outcome analysis. Many programs’ corrective action plans hint at a common struggle – where to start?
Risk-adjusted measures will always have weaknesses. Data definitions are necessary but can be rigid. Socioeconomic and geographic characteristics often have an impact but are difficult to measure. The sickest of patients needed alternatives, too. It is easy to feel that a program’s performance against comparative, risk-adjusted data is somewhat out of its control.
The FACT Clinical Outcomes Improvement Committee suggests programs start with cause of death, in a very literal sense. Identification of specific causes of death serves as a starting point for investigation. A quantitative, data-driven approach prevents inaccurate assumptions, especially in emotionally-charged situations. For example, high-risk patients need alternatives and programs may choose to approve them for transplant. By looking at causes of death, programs can focus on improvements within their control, such as risk-stratified approaches to choosing preparative regimens to address deaths caused by transplant-related mortality.
For U.S. allogeneic transplant programs, cause of death reporting is subject to rules set by the Center for International Blood and Marrow Transplant Research (CIBMTR). CIBMTR provides an easily accessible Forms Instruction Manual at http://www.cibmtr.org/manuals/fim. Among many subjects in this online manual, the CIBMTR lists rules for cause of death codes. There are several training and educational resources, also, and can be found on CIBMTR’s Training and References webpage. (For resources on the broader topic of the CIBMTR and FACT joint data audit program, visit FACT’s Data Management Resource Center.)
Clinical Programs are also encouraged to use internal data for outcome analysis. Internal cause of death data is not just helpful, but important. There are often contributory, secondary causes of death in addition to the primary cause. For example, a pediatric program may have a high rate of deaths caused by multi-organ failure, contributed by infection related to a high proportion of patients transplanted for an immunodeficiency. Programs may seek not only improvements in infection control, but also take actions to shorten the time to transplant or use granulocyte transfusion protocols.
Cause of death data can make the path toward outcome analysis and improvement clearer. Such data prevent incorrect assumptions, are available to both large and small programs, and provide a starting point for the complex process of evaluating one-year survival. Thank you for your efforts to improve survival of transplant recipients, and we will continue to share lessons learned via the FACT accreditation process.
Resources Continuously Added to FACT Website
Topics include clinical outcomes, immune effector cellular therapy, patients and families, and more
To support FACT-accredited organizations and the FACT mission to improve cellular therapy, resources are continuously added to the FACT website. This article highlights just a few of the informational webpages that provide links to journal articles, instructions, examples, and more.
FACT supports blood and marrow transplant (BMT) programs’ efforts to evaluate one-year survival data to improve patient outcomes, advance the field, and maintain payer and public confidence. FACT requires programs to submit a corrective action plan when one-year survival does not at least meet expected outcomes as demonstrated in comparative data. Several resources are available to assist programs with identifying trends in cause of death, the root causes, and corrective and preventive actions. This page includes links to comparative data sources (especially helpful to autologous-only and non-U.S. programs), educational resources, and example corrective action plans. The page also lists the guidelines for corrective action plans for quick reference.
The main objective of FACT’s Immune Effector Cell Standards is to promote quality practice in immune effector cellular therapy. FACT is committed to supporting efforts to make quality immune effector cellular therapy accessible to patients much like it did for bone marrow transplant and cord blood banking. Several resources are available to assist with understanding immune effector cells and implementing FACT Standards. Resources include FACT Standards for Immune Effector Cells, educational on-demand webinars, frequently asked questions, and journal articles. Examples of guidelines and procedures used by organizations for managing immune effector cell programs and patients will soon be available.
Education and assistance for cellular therapy patients and their families are key to promoting desirable clinical outcomes. Direct links to a variety of organizations and informational sources are listed for several topics, including donor registries, information and support, cord blood banking and donation, and clinical trials and access to cell therapies.
Although quality management is relatively new to cellular therapy programs and cord blood banks, the principles of quality assurance and quality control have long been incorporated. For the FACT accreditation process, the real turning point for emphasis on quality management activities was with the release of the third editions of the Cellular Therapy Standards and Cord Blood Standards, which included greatly expanded Quality Management sections. Since that time, FACT has developed and introduced quality-based educational materials to assist your organization with developing and enhancing your Quality Management Program.
Establishing Efficiencies Between IND Requirements and FACT Standards
Many FACT-accredited cellular therapy programs participate in clinical trials, and awareness of such activities has become more pronounced with the advent of FACT’s new voluntary accreditation for immune effector cellular (IEC) therapy. Most IEC products are administered under Investigational New Drug, or IND, requirements.
There are some FACT requirements that are not typically required of INDs but could greatly enhance research, such as a quality management program, accessible procedures, and training. Likewise, some typical IND requirements could be used to also satisfy FACT requirements, and documentation created to comply with IND requirements can be used.
Outcomes
FACT requires outcome analysis, which is also performed in accordance with IND requirements. The program does not need to gather and report its own independent data if this is already performed by the investigator and shared with the program. This could be demonstrated through minutes from a quality meeting where outcomes were reviewed, or within signed outcome analysis reports. The key is that outcome information is reported back to the accredited center’s Quality Management (QM) program in some fashion to allow assessment of program performance (such as personnel training and competency or needed procedural improvements).
Product efficacy based on patient outcome may be difficult to document for novel cellular therapy products, and that assessment will differ for each product type. Predefined outcome criteria for investigational cellular therapy products (e.g., CAR T-cells, vaccines) may be found in the clinical research protocol and may include clinical outcomes or only safety endpoints, depending on the trial phase. Data from each investigational product does not need to be aggregated in total; for example, IECs vary widely in cell type, manipulation, protocol, and indication. However, data regarding similar products may shed important light on process improvements within the control and discretion of the program (protocols must be strictly followed per IND requirements).
Reporting of outcome data also advances the field. FACT Standards recommend that Clinical Programs collect each of the data elements contained in the applicable CIBMTR Cellular Therapy forms or EBMT forms. Use of the actual form and submission to CIBMTR or EBMT is not required unless dictated by applicable laws and regulations. FACT strongly recommends the publication of data and strongly encourages the submission of data to the CIBMTR. In the event that the Clinical Program does not submit data to these registries, it should provide reasonable explanations for not doing so. Examples of the forms currently utilized by the CIBMTR may be found on the organization’s website at http://www.cibmtr.org.
Audits
Similar to outcome analysis, audits are typically performed as directed by INDs and the results of those audits can satisfy FACT Standards if they are forwarded to the QM Program for review. If an IND does not require an audit, then an audit as required by the Standards must take place.
Policies and Procedures
Many programs utilize IND protocols to satisfy FACT requirements for SOPs. At a minimum, programs should have a local, high-level SOP that describes the big picture of the activities to perform. Actual protocols could be referenced and made available to staff, or the detailed procedural steps could be outlined in job-specific SOPs. Some manufacturers have very specific SOPs and provide them to programs, and, in these cases, there is no need to copy the SOP over to the program’s format. However, protocols are not a no-fail way of meeting the intent of the standard. Programs should ensure that the protocols include specific details for staff performance without being burdensomely long, that they are accessible to staff, and that training on the protocols has been performed.
Autologous and Non-U.S. Survival: Ideas for Comparative Data Sources
Unlike allogeneic transplants in the U.S., there is no risk-adjusted report of one-year survival for autologous transplants. There are a variety of risk-adjusted survival registries around the globe, but not for every region. Yet Clinical Programs are expected to benchmark their survival to comparative data and take corrective action. One of the most common questions we receive from autologous-only or non-U.S. programs is, “Where do we find data on one-year survival for autologous transplants?”
The FACT-JACIE Hematopoietic Cell Therapy Standards do not prescribe any specific data source to use for autologous benchmarking. The following are ideas to consider:
C.W. Bill Young Cell Transplantation Program
- S. Patient Survival Report: The U.S. Patient Survival Report displays an estimate of the number of patients who are alive after a bone marrow or umbilical cord blood transplant. The estimates are shown:
- By disease and length of time after transplant (including one year)
- By the type of donor who provided the cells for transplant:
- Autologous (the patient’s cells)
- Related allogeneic (a patient’s sibling or another family member’s cells)
- Unrelated allogeneic (a volunteer donor’s cells)
- The cells used for transplant (bone marrow, peripheral blood, or umbilical cord blood)
Be the Match/National Marrow Donor Program
- Disease-Specific HCT Indications and Outcomes Data: The outcomes data presented here are based on more than 50,000 unrelated donor transplants NMDP facilitated, plus autologous and allogeneic transplant data reported to CIBMTR® (Center for International Blood and Marrow Transplant Research). This CIBMTR data and analysis are based on more than 330,000 patients from U.S. and global transplant programs.
Peer-reviewed Medical Literature
- Scholarly articles examining multi-center one-year survival using relevant disease, donor type, and date range.
Understanding the CIBMTR Outcomes Reports
Allogeneic transplant programs in the U.S. are required to submit corrective action plans to FACT when they do not meet the expected range of one-year survival in the CIBMTR Transplant Center-Specific Survival Report. To adequately assess lower-than-expected one-year survival, it is necessary to fully understand how the report is generated.
CIBMTR explains the report methodology in the document titled, Methodology Employed for Annual Report on Hematopoietic Cell Transplant Center-Specific Survival Rates. The document provides technical information on statistics, but offers the following summary: “A censored data logistic regression model is fitted to survival data for first unrelated and related donor hematopoietic cell transplants at U.S. centers. The model is adjusted for recipient age; recipient race/ethnicity; Karnofsky/Lansky score; coexisting disease (Sorror HCT-CI); recipient CMV status; disease/status; interval from diagnosis to transplant (ALL and AML only); NHL subtype; resistant disease (NHL and HL only); CLL and other chronic leukemia stage; prior autologous transplant; year of transplant; donor type/graft type/HLA matching; donor/recipient gender match (bone marrow or PBSC only); and donor age (unrelated donor bone marrow or PBSC only). The report on transplant center-specific survival rates helps to identify centers that may have performed above or below confidence limits compared to the overall network of transplant centers . . .”
CIBMTR also explains the report, and its tools and resources to help Clinical Programs use their data, in the FACT webinar recording titled, Using CIBMTR Data to Determine and Evaluate Clinical Outcomes, presented by Stephen Spellman, MBS. This recording gives an overview of how CIBMTR data is used to determine outcomes and how Clinical Programs can use additional data to further evaluate outcomes and improve.
The following are common points Clinical Programs make about the CIBMTR report, and how they can be evaluated in the context of creating corrective action plans to address lower-than-expected one-year survival:
- High-risk patients: Some corrective action plans stated the root cause of death to be transplants for high-risk patients. As outlined above, the CIBMTR report is risk-adjusted. High-risk patients should be accounted for within the report. The FACT committee expects corrective actions that specifically address the causes of death. Broad refusal to transplant patients with high risk is not the intent of FACT requirements. As transplants are often the last hope for patients, careful attention to trends in causes of death is particularly important for these patients to improve their outcomes. For example, some programs have determined myeloablative therapy was not necessary or beneficial for a group of frail patients; another adjusted its protocol for preparative regimens.
- Small programs: It is difficult to identify trends among a small number of transplants, but FACT will look for a good-faith effort of the program to review data and determine if a trend can be found. One small program found that its patients had a high rate of CNS disease, and is educating its network of referring physicians.
- Confidence interval: A common worry is that Clinical Programs will have one-year survival lower than the expected range, through no fault of their own, because of the 95% confidence interval. It is important to realize that each program has its own confidence interval. A defined number does not have to drop out of the curve. Therefore, it is possible for each program to meet expected one-year survival. Small programs typically have a wider range of expected outcomes.
- Delay in reporting: Due to the inherent timeframe of “one-year” survival, the CIBMTR report is delayed by two years because the transplant has to have occurred a year prior, and an additional year is needed to analyze the data. Furthermore, the report uses three years’ worth of data. For example, data analysis may show that a Clinical Program had a particularly bad year in 2012 that resulted in lower-than-expected one-year survival in the 2015 report. However, it is still necessary to review the causes of death and their root causes for the timeframe of the report. If survival is showing an upward trajectory since that year, include that information in the corrective action plan for consideration.
- Overall one-year survival: The CIBMTR report only provides overall one-year survival; however, drilling down into specific diseases will help Clinical Programs determine root causes and which corrective actions may help. This is the same for treatment-related mortality or disease relapse. This type of drill-down has helped programs identify root causes.
- Data errors: Some Clinical Programs have noted that errors in the data submitted to the CIBMTR were the true root of low one-year survival. Indeed, this can affect results of the algorithm. If data errors are a problem, FACT will want to see corrective actions related to accurate data management and reporting.
Guidelines for Corrective Action Plans to Improve Low One-Year Survival
The FACT Clinical Outcomes Improvement Committee reviews corrective action plans submitted by Clinical Programs not meeting expected one-year survival. Development and review of these corrective action plans is new to Clinical Programs and to FACT. The emphasis of the committee is to help programs identify ways to improve their outcomes for patients, and it appreciates the programs who submitted the first plans and provided more information when requested. After several months of reviewing corrective action plans, the committee has articulated its expectations for these plans using six guidelines:
- The corrective action plan must identify specific causes of death.
- The corrective action plan must provide quantitative data.
- The assessment must identify reasonable causes of the low one-year survival rate.
- The corrective action plan must address the identified causes.
- There must be a measurable outcome improvement.
- The program must provide updates on corrective actions at the time of inspection, at the time of annual reporting, or as otherwise directed by the committee.
The essential point of these guidelines is to help Clinical Programs write corrective action plans that convey the thought process they employed to identify root causes and implement remedial measures. It is tempting to jump to conclusions and take actions assumed to be beneficial, but those actions will not be effective if incorrect conclusions are made. By providing quantitative data about causes of death, it will be clear to both the program and the FACT committee what the underlying issues are and where opportunities for improvement exist.
FACT will continue to provide advice and examples related to improving clinical outcomes. The next Quality Management Series module is dedicated to Outcome Analysis, and will consist of four webinars from November through February that focus on clinical outcomes. Example corrective action plans, used with permission from Clinical Programs, will also be posted on the FACT website this fall.
