By Derek Dietze, MA, FACEhp, CHCP, President, Metrics for Learning, LLC
In its 2019 data report, the Accreditation Council for Continuing Medical Education (ACCME®) reported that 1,724 accredited providers had offered 188,992 educational activities, with approximately 1.3 million hours of instruction and over 37 million “interactions” with healthcare professionals. These providers reported approximately $3 billion in education investments.1 Using these numbers, the overall average cost per healthcare professional “interaction” was approximately $81.
At a macro level, given the size of this investment (which does not include all other healthcare professional CEhp), is this a cost-effective enterprise? Perhaps the more important question for us as CEhp professionals is, “How cost effective are my educational activities?” My opinion is that most of us do not have credible answers to these questions, but that we need those answers to improve and advance our own programs and the CEhp enterprise as a whole. Additionally, cost pressures continue to be applied within our practice settings.
A good starting point to answer these questions is to establish some definitions that will help target our approach. The definition of cost effective is “producing good results without costing a lot of money.”2 For the purposes of the simplistic calculations shown in this article, “good results” will be defined as “increases in mean percent correct answers for pre-/post-tests.” Since “costing a lot of money” is very vague, and the $81 per healthcare professional interaction calculated from ACCME® data is across all types of activity formats and healthcare professions, at this point we will investigate results for just one specific CEhp activity to benchmark costs. As defined by the Outcomes Standardization Project, a “Completer” is a healthcare provider that finished the core educational content/intervention but may or may not have participated in the post-test or evaluation. A “Learner” is a healthcare provider who has progressed beyond the CEhp front matter and pre-test (if available) and has started to consume/participate in the educational experience.3 My definition of “Credit Earner” is Completers who have finished the post-test and evaluation and have been awarded a certificate of credit. “Target credit earners” are Credit Earners who self-report being involved in the care and treatment of patients with the medical condition discussed in the activity content.
While the CEhp activity used in this example may not apply to your specific CEhp practice setting, the same calculations and principles can be applied to your unique situation. For instance, in a hospital setting, what is the cost for one grand rounds activity, how many clinicians receive credit, and what did it cost to get each percentage point increase in knowledge and/or competence for those clinicians?
CEhp Activity Description and Goals
- Online enduring CEhp activity, one credit
- Education partners: A medical education company, a CME/CE provider and one distribution partner (online platform)
- Designed to increase knowledge and competence
- Assessment: Pre-/post-test with eight multiple-choice questions aligned with content and learning objectives
- Target Learner: Primary care clinicians across the entire United States (physicians, physician assistants and nurse practitioners) who are involved in the care and treatment of patients with Type 2 diabetes (T2D)
- Support: An educational grant from a pharmaceutical company that covers all expenses
- Completer goal: 1,600
- Credit Earner goal: 450
- Cost: $175,000 including all aspects of activity development, implementation and assessment (the entire grant)
CEhp Activity Results
- Completers: 1,750
- Credit Earners: 475
- Target Credit Earners: 380 (80% of Credit Earners — those who indicated caring for patients with T2DM)
- Knowledge change in Target Credit Earners: 46% relative increase in mean percentage correct answers from pre-test (57% correct) to post-test (83% correct) for four multiple-choice knowledge questions (paired data)
- Competence change in Target Credit Earners: 41% relative increase in mean percentage correct answers from pre-test (61% correct) to post-test (86% correct) for four multiple-choice case vignette questions (paired data)
Below are calculations regarding the cost effectiveness of the activity, using information in the activity results.
a. Cost per Credit Earner (Completers who finished the post-test and evaluation and were awarded credit) = $368
Calculation: $175,000 total cost ÷ 475 credit earners
b. Cost per Target Credit Earner (Credit Earners who self-report caring for and treating patients with T2DM) = $461
Calculation: $175,000 total cost ÷ 380 Target Credit Earners
c. Cost per Target Credit Earner per Relative Percent Increase in Knowledge = $10
Calculation: $461 per Target Credit Earner ÷ 46 (the 46% relative increase in mean percentage correct answers)
d. Cost per Target Credit Earner per Relative Percent Increase in Competence = $11
Calculation: $461 per Target Credit Earner ÷ 41 (the 41% relative increase in mean percentage correct answers)
Thus, for this specific CEhp activity, it cost $10 for each percentage point relative increase in knowledge from pre-test to post-test for the Target Credit Earners — the clinicians in the best position to impact patient care. The cost was $11 for each percentage point relative increase in competence. By calculating these costs for each type of CEhp activity (i.e., by design, format and Target Credit Earner), baseline costs can be established against which future CEhp activity results can be compared. Note that in order to do these calculations, you will need learners to answer a question about caring for target patients, typically worded as something like, “How many patients with [medical condition] do you care for in a typical week?” An option to respond “zero patients” or “N/A for me” should always be provided.
A natural next step beyond these types of calculations would be to determine the cost per patient for your Target Credit Earners, assuming you have information on target patients cared for per week, month or year for each of them. In addition, similar calculations can be applied to determine costs per Target Credit Earners for changes in performance and patient outcomes.
Should we care about this? Yes! Using these methods, the most cost-effective CEhp activities can be identified and further improved by reducing costs, better targeting learners, enhancing instructional design and engagement and optimizing the linkage between learning objectives, content and outcomes assessments.
1 Accreditation Council for Continuing Medical Education. 2020. ACCME Data Report: Steady Growth in Accredited Continuing Medical Education – 2019. https://www.accme.org/sites/default/files/2020-07/872_2020%2007%2028_2019_Data_Report.pdf
2 Definition of cost-effective, https://www.merriam-webster.com/dictionary/cost-effective
3 Outcomes Standardization Project Glossary, http://outcomesince.org/glossary/