Role of Incident Learning Databases in Process Improvement Sasa Mutic Conflicts of interest • • • • • • Shareholder – Radialogica Partner – TreatSafely Licensing – Modus, Varian Consulting – Varian, ViewRay Grants – Varian, ViewRay, Elekta, Philips Speaking – Varian, ViewRay, Philips WU ILS Database Since July 2007 – 9500+ Events Evidence based QM (us as a discipline) • It is difficult for individual clinics to prioritize their QA/QC/QM activities if the broader field and community is still struggling with what to prioritize • Prioritization requires data • Evidence based medicine is everywhere, QA/QC need to embrace the same approach Example: QA\QC Check Effectivness • An analysis of the effectiveness of common QA/QC checks • IRB between Johns Hopkins University & Washington University • Both institutions started incident learning systems (ILS) at the same time • Data: o Incident reports: 2007-2011 o 4,407 reports o 292 (7%) “high potential severity” E.C. Ford, S. Terezakis, A. Souranis, K. Harris, MD, H. Gay, S. Mutic, Quality Control Quantification (QCQ): A tool to measure the value of quality control checks in radiation oncology, Int. J. Radiat. Oncol. Biol. Phys., 84(3), 263-269, (2012). Common QA\QC checks Physics chart review Therapist chart review Physics weekly chart check Physician chart review EPID dosimetry Port films: check by therapist Timeout by the therapist Port films: check by physician In vivo diode measurements Checklist Chart rounds Online CT: check by therapist SSD check Online CT: check by physician Pre-treatment IMRT QA 0 10 20 30 40 50 Sensitivity (%) 60 70 Literature Search • pubmed.org search on: – (Quality Assurance) AND (Radiation Therapy) AND • (IMRT) Results: 463 • (Chart Checks) Results: 7 • (Chart Review) - Results: 34 • An order of magnitude difference May 2013 Data How would investors use this data? Physics chart review Therapist chart review Physics weekly chart check Physician chart review EPID dosimetry Port films: check by therapist Returns Timeout by the therapist Port films: check by physician In vivo diode measurements Checklist Chart rounds Online CT: check by therapist SSD check Online CT: check by physician Pre-treatment IMRT QA 0 10 20 30 40 50 Sensitivity (%) 60 70 Current IMRT QA Paradigm “We are pretty good at making sure that we can treat a phantom correctly at ~7:00 pm” 1. Transfer patient plan to a QA phantom • Dose recalculated (homogeneous) on phantom – any dose calculation errors would not be revealed 2. Perform QA prior to treatment • Subsequent data changes/corruption may result in systematic errors for all subsequent patients 3. The volume of data impossible to monitor and verify manually • Manual checks do reveal data changes/corruptions, but not reliably 4. The process too laborious with questionable benefits • A systematic analysis and redesign demonstrates possibility of a much more robust and automated process Error spectrum • Publicized - One side of the spectrum, usually large dosimetric errors – NY Times Articles • Semi-publicized – RPC data – Approximately 20% of participating institutions fail the credentialing test at 7% or 4mm* – Approximately 30% fail at 5%* • Unpublicized/unnoted – everyday occurrences – “Small” dosimetric errors and geographic misses – Suboptimal treatment plans (contouring and dose distributions) – Care coordination issues – Unnecessary treatment delays *Molineu et al, Credentialing results from IMRT irradiations of an anthropomorphic head and neck, Med Phys, 40, 2013. Safety in Radiation Therapy Safety is the absence of an unacceptable risk of harm. In our context harm is excess morbidity or sub-optimal tumour control. Note: the definition of safety is controversial and depends on the context. We can take this as a working definition for our purposes. Courtesy: Quality in Radiation Therapy Quality of care is defined as the degree to which health services for individuals and populations increase the likelihood of desired health outcomes and are consistent with current professional knowledge. Institute of Medicine. Volume 1. Committee to Design a Strategy for Quality Review and Assurance in Medicine, Institute of Medicine. Lohr, K. ed. Medicare: A Strategy for Quality Assurance. Washington, D.C.: National Academy Press,1990 Courtesy: Quality in Radiation Therapy Benefit Target Overdose Dose Is this distribution realistic: most patients receive acceptable treatments withtreatments? a minority being harmed? acceptable Underdose Courtesy: Quality in Radiation Therapy Benefit Target Overdose Dose Or is this more realistic: there’s a continuous distribution from acceptable treatments to harmful treatments? Underdose Courtesy: What matters • “High-quality” means minimizing process variation and moving the average closer to the optimum value - Med. Phys., 2007. 34(5): p. 1529-1533. • Stable and well defined processes enable – Standardization – Quantification – Benchmarking – Improvements – Quality Control Courtesy: What are we trying to accomplish? Benefit Harm Underdose Harm Target Dose Overdose Courtesy: Reducing Variability Imaging Treatment Selection Contour Approval Contouring Plan Creation Completion MD Approval The Goal Physics Approval Physics Checks Treatment and Ongoing QA Normative decision theory: Work - Value added 1) Timeline Start with efficiency moveWait to – No value efficacy 2) Uncertainty DMAIC Cycle – Continuous Improvement Courtesy: DMAIC Cycle – Continuous Improvement Courtesy: ILS Process Initial Reporters Report Analysis Report Analysis Various Formats • Explicit events • Random events • Corrective measures Courtesy: Lessons Learned – Reporting Culture 21 • A brand new web-based system was named, “Process Improvement Logs” • Staff quickly provided a nickname “E-Snitch” Learning From Our Mistakes: Radiation oncology reporting survey • Multi-institutional,* IRB-approved – Surveymonkey®, Anonymous, Dec-Jan 2011 – Johns Hopkins – Washington University – University of Miami – North Shore-Long Island Jewish Hospital Harris et al Voluntary Error and Near-miss Reports (Dec-July 2010) Attending physicians Resident physicians *Combined data from all four sites. Total number of reports = 916 0 0 Respondent Characteristics Finding Total number of respondents Response rates (%) Overall –Attending physicians –Resident physicians –Physicists 274 81 89 68 96 –Dosimetrists –Radiation therapists 97 79 –Nurses 70 2/3 with 5 or fewer years of experience Knowledge and Attitudes*: • Errors and near-misses happen in our clinic: 90% • Error and near-miss reporting Is my responsibility 97% Is valued by my colleagues 72% I know what to report 83% I’m too busy to report 27% *No differences between the practice groups Perceived Barriers to Reporting Get my Admitting colleagues Embarrassment Affect liability in trouble (%) reputation (%) (%) 26 Attending physician Resident physician Dosimetrist 41 41 49 35 54 42 58 44 7 28 14 29 Physicist 34 39 36 35 Nurse Radiation therapist 40 20 32 24 47 18 25 25 p=0.0089 p=0.0271 p=0.0019 p=0.0467 Perceived Barriers to Reporting • When thinking about reporting, I am concerned about* – Sanctions……………………….62% *No differences between the practice groups Global Problem “…it calls into question the integrity of hospital systems and their ability to pick up errors and the capability to make sustainable changes.” Sir Liam Donaldson, Chief Medical Officer, Department of Towards Safer Radiotherapy. London: The Royal College of Radiologists, Health 2008. Radiotherapy Risk Profile, Geneva: World Health Organization, 2009 . Is there a benefit in every size facility? • Relatively good communications • Streamlined processes • Great collective memory • Perhaps a limited benefit Single Machine Facility Is there a benefit in every size facility? • Non-uniform communications • Complex processes • Pockets of reliable memory • Potentially significant benefits Large Facilities WU – 350 Faculty and Staff Is there a benefit in every size facility? • Still silos • Non-uniform processes • Unawareness • Potentially significant benefits Networks ASTRO’s 6-Point Plan 32 1. Create a database for RT error reporting 2. Develop new practice accreditation program 3. Expand quality and safety education/training 4. Develop tools for patients 5. Further develop interconnectivity compliance 6. Advocate for expanded legislation Why is a national ILS achievable? 33 • Patient Safety and Quality Improvement Act – Signed into law July 29, 2005 – Share information about patient safety events without liability – Allowed for the creation of Patient Safety Organizations (PSOs) What is a PSO? 34 • An entity listed by AHRQ that meets PSIQA requirements • Operationalize PSIQA for healthcare entities www.claritygrp.com The ASTRO/AAPM System 35 PSO: Patient Safety Organization PSWP: Patient Safety Work Product PSES: Patient Safety Evaluation System Analysis and Reports Provider National Safety Alerts and Reports Database Send to PSO Provider’s PSES Database Analytics and Analysis by RO-HAC Clarity PSO PSES Status of the RO•ILS 36 • Currently Collecting Data • Academic and community centers • Last accounting ~ 30 centers • RO-HAC active • Analyzing Events • Actively working on enrolling more sites • Please join! Conclusions • Need to define the goals for safety and quality in radiation therapy • Need to challenge legacy processes based on the available data • This will take years and this talk is just a very small part of that process