Personalizing Outcomes Measures to Effectively Assess Impact of SSc-ILD Education

By Brittany Puster, Katlyn Cooper, Natasha Mitchner and Ritesh Kothari

Systemic sclerosis (SSc) is a disfiguring, disabling and potentially fatal rare disease that causes scarring of the skin (scleroderma), lungs (SSc-ILD) and other organs. ILD is the leading cause of mortality in SSc, accounting for nearly 35% of all SSc-related deaths.1 

Given the poor prognosis of SSc-ILD, patients with SSc should receive screening and frequent follow-up to detect lung involvement. But, unfortunately, available guidelines do not include recommendations for screening ILD in patients with SSc, resulting in varied clinical practices. Moreover, although an accurate, early diagnosis is critical for prompt, appropriate management, SSc-ILD patients often experience delays in receiving a diagnosis and initiation of therapy.

To extend survivability and improve the quality of life for these patients, education must support improvements in differential diagnosis and appropriate application of pharmacotherapy, considerate of a multidisciplinary team who manages these complex patients. While the goals and educational content across these audiences are universal, we recognized the way in which each learner type would apply this knowledge varies based on area of practice. For example, improving the differential diagnosis process by a primary care physician could differ from that of a pulmonologist or rheumatologist, as each employs different clinical, patient and/or radiologic features to move toward diagnosis.

In recognition of this, our educational outcomes utilized a personalized methodology to support precision-based analytics —choosing measures of success aligned with each learner’s scope of practice. Unlike traditional activities where outcomes questions are considerate of a “typical” learner within the primary target audience, we used AI technologies to index and populate questions based on the unique learner profile. This approach afforded us the opportunity to segment learners based on expectations for improvement relative to their role in managing SSc-ILD patients and protect against unnatural depression (or inflation) of pre- and post-test scores. Even when the content is static and not personalized, the outcomes questions can be by delineating key knowledge points and considering how each different learner type might employ this knowledge in a clinical setting.

Through the development of a multimedia e-newsletter, content was delivered to the learner using text, video, patient testimonials and practical guidance tips. Goals of education were to:

  • Improve patient prognoses through the timely and accurate differential diagnosis of SSc-ILD
  • Improve patient QOL through personalization of management strategies
  • Improve knowledge of new and emerging therapies for SSc-ILD

Summary of Findings*

Clinical Competency: Diagnosis of SSc-ILD

BIC 1.png

Clinical Knowledge and Competency: Oxygen Requirements & Monitoring (Patient QOL)

BIC2.png

BIC 3.png

Clinical Knowledge: Novel and Emerging Therapies

BIC 4.png

Analysis of Findings

Relative to each of the identified goals, learners in both cohorts demonstrated improvement across all measures:

Goal #1: To improve patient prognoses through the timely and accurate differential diagnosis of SSc-ILD

Both groups of learners demonstrated marked increases in competence with components of an ILD diagnosis. Pulmonology learners recognized the need for HRCT over PFTs, and non-pulmonology learners demonstrated increased recognition of crackles, an early sign of ILD.

Goal #2: To improve patient QOL through personalization of management strategies

Clinician knowledge of recommended approaches to assess patient oxygen requirements was used as an indicator of clinical awareness on contributors of patient QOL. Pulmonology learners demonstrated increased awareness of the limitations of finger pulse oximetry based on faculty discussion. Non-pulmonology learners also recognized the utility of the 6-minute walk test.

As a measure of success relative to this goal, HCP knowledge on patient monitoring strategies was tested. Pulmonology learners demonstrated a greater appreciation for more frequent monitoring of lung function in SSc patients while non-pulmonology learners reported increased intention to support earlier referral of patients to an ILD center.

Goal #3: To improve knowledge of new and emerging therapies for SSc-ILD

Both groups of learners demonstrated increased awareness of the SENCSIS clinical trial and its endpoints, an important measure since the therapy investigated in the trial received U.S. Food and Drug Administration (FDA) approval as the first therapy for SSc-ILD shortly after the activity launched.

To further isolate the success of this education and the impact of the personalized outcomes methodology, the outcomes data from this activity was compared to another SSc-ILD activity that had comparable measures in identical audiences, but through use of a traditional outcomes approach (ie, questions that were audience agnostic).  When comparing measures of the effectiveness, we see the following aggregated results, differentiated by learner audience:

Pulmonology Learners

Non-Pulmonology Learners

Cohens d

Effect Size

Percent of

Non-overlap*

Cohens d

Effect Size

Percent of

Non-overlap*

Traditional Outcomes

0.20 (n=72)

15%

0.35 (n=587)

24%

Personalized Outcomes*

1.88 (n=14)

79%

1.12 (n=135)

60%

* data representative of 3 months of learner activity post-launch 

For purposes of interpreting these outcomes, percent of non-overlap indicates how much more knowledgeable learners were following participation in the activity. In the activity that employed the personalized outcomes strategy, pulmonology learners were 79% more knowledgeable following the intervention (pre to post), while non-pulmonology learners were 60% more knowledgeable. These results are striking when considering outcomes from the traditional activity, where the same set of measures were used to assess all learners irrespective of their role in managing these patients. In this model, the percentages measuring impact/improvement were significantly lower.

Another key metric is confidence. When comparing the effectiveness of a personalized outcomes approach in improving learner confidence in making an early diagnosis of ILD, there was a more pronounced positive shift in the activity that employed a personalized outcomes approach:

 

Pulmonology Learners

Non-Pulmonology Learners

Traditional Outcomes

36% (n=33)

47% (n=587)

Personalized Outcomes

43% (n=14)

57% (n=135)

 

This indicates that measures aligned to the knowledge expectation and practical relevance of diverse learners may foster heightened confidence, which may be foundational to enhanced learner performance.

In conclusion, for a personalized outcomes approach to be successful, the following critical success factors apply:

  • Challenging and meaningful questions delineated and aligned to the expected learner skill and role
  • Establishing relevance of content and testing questions to scope of practice
  • A precision-based approach to analyzing data considerate of varied learner characteristics
  • A testing framework that fosters ongoing pedagogical support at both the learner and provider level

*Only first-attempt posttest scores are reported.

Pre- and posttest responses are paired/matched. Non-completer data has been omitted from the analysis to ensure comparison groups are equivalent

 

Reference

1. Tyndall AJ, et al. Causes and risk factors for death in systemic sclerosis: a study from the EULAR Scleroderma Trials and Research (EUSTAR) database. Ann Rheum Dis 2010;69:1809–1815

Recent Stories
An Examination of State and Federal Opioid Analgesic and Continuing Education Policies: 2016–2018

Commission Meets to Offer Input on Task Force Work To-Date

From the Editor: “V” Is for Virtual (and Sometimes Not Victory)