Qualitative Data and IPE: Assessing a Complex Educational Experience  

Corresponding author: Mary Grantner, MA, CHCP; Co-authors: Agne Paner, MD, Kimberly Lynch, BS, Francis Kwakwa, MA

Abstract

Introduction: The utilization of qualitative data analysis for assessing learning outcomes of “2017 Multiple Myeloma Rounds®” presents an approach to assessment that assists providers in overcoming the barriers to Moore’s Outcomes Levels 5 and 6.

Methods: Components of the CE initiative include traditional CE delivery formats with an essential element of rigorous evaluation over time that includes pre-course evaluation, evaluation immediately following the CE session and subsequent focus group discussion two to three months after the live CE session. Outcome measures are designed to assess ongoing adaptation of content and reinforcement of key learning objectives through the utilization of traditional evaluation forms and focus group input.

Results: Our results after one year of programming suggest that this model is an effective means of assessing the richness of a complex educational experience.

Conclusions: Most important, its design is one that could provide an indirect measure of improved patient health.

Introduction

“2017 Multiple Myeloma Rounds” (MMR) is a partnership of a large academic medical center, a medical education company (MEC), and research foundations to improve the diagnosis and treatment of multiple myeloma. The academic medical center is a 664-bed hospital, a research institution and a university specializing in the health science professions. It is a jointly accredited provider of continuing education through the Accreditation Council for Continuing Medical Education, the American Nurses Credentialing Commission and the Accreditation Council for Pharmacy Education. The MEC is an organization providing continuing education in healthcare with a special focus on oncology providers. The research foundations are not-for-profit entities that focus on supporting scientific research, patient advocacy and education.

The 2017 MMR continuing education (CE) activities provided by these entities seek to educate the interprofessional hematology-oncology team about the rapidly evolving landscape of diagnosis and treatment of multiple myeloma. Based on the initial needs assessment conducted in 2015, the medical center launched this CE programming as a series of live activities in the Chicago area. A high level of satisfaction and learner interest (including a high repeat attendance rate) resulted in expansion of the series to four additional areas including New York, Seattle and Tampa with 40 participating institutions, and to expansion of the faculty advisory committee to include 36 multiple myeloma specialists representing 24 different institutions. Expansion led to development of a new outcomes measurement plan utilizing a combination of traditional CE satisfaction measures, post-activity intention-to-change measures and focus group analysis. The purpose of this paper is to describe the approach to the qualitative data analysis and what they allow us to conclude about learner outcomes.

Theoretical Model

The CE program for 2017 MMR was designed to use a combination of delivery formats such as case-based teaching, audience response technology, and question-and-answer sessions. The Shewhart (or Deming) Learning Cycle model, which is predicated on ongoing needs assessment, is well-established in deploying learner feedback from all of these avenues. The model emphasizes outcomes measures as integral pieces of planning and teaching. We address Moore’s Outcomes Levels 1 through 5 throughout the CE program with self-reported information from learners.1  The innovation of the MMR Model is in utilizing focus group input as part of the Shewhart Cycle, input that is designed to both analyze the application of skills to practice and assess ongoing learning needs.

Initial measures were based on intent-to-change evaluations of learners. Self-reported change has long been established as a valid measure of actual practice change.2  As Johnson, et al, and Williams have demonstrated, the Transtheoretical Model postulates that an educational program evaluation can measure learners’ likeliness to change by assessing the learners’ readiness to change.3,4  (See Table 1.)

Our final point of learner assessment is based on results of focus group analysis two to three months post-activity. Traditional qualitative study is a form of empirical research most often used in the social sciences in which the study design is focused on explaining ‘how’ and ‘why’ a particular behavior operates in a particular context.5   In the CE context, qualitative study is much more limited in use. To our knowledge, its most frequent use in CE is in developing learner needs assessments. For example, Kotwal, et al recently utilized semi-structured interviews to better define clinical excellence in hospital medicine and suggested development of CE activities based on enhancing practice rather than on simple competence.6 The premise of the MMR initiative is to expand the use of qualitative study to assess learning outcomes by conducting structured focus group discussions based on reported learner intention-to-change.

Methods

The first focus group discussion took place three months after learners had attended Session 2 of the 2017 XX MMR cycle. The specific emphasis of Session 2 was on approaches to patients with relapse disease. The focus group included several learners who had also attended Session 1 of MMR, which concentrated on diagnosis and management of amyloidosis in multiple myeloma patients. Session 1 had occurred six months prior to the convening of the focus group.

Focus group participants consisted of eight individuals from a spectrum of professions roughly reflective of overall MMR attendance. Any participant who had attended both Sessions of MMR was invited to participate, with an intention to hold the group to a maximum of 10 participants representing at least two different professions. Broadly across MMR programming, attendees consist of a mix of physicians, nurse practitioners, pharmacists and researchers from both academic centers and community practices. Final participants in the focus group consisted of five physicians (three oncologists and two internists), two advanced-practice nurses specializing in oncology and one research fellow in multiple myeloma, with half of the participants from community-based practices. Discussion was led by the Course Director of 2017 MMR, a physician specializing in hematology/oncology. An audio recording was made, and a member of continuing education staff was on hand to take concordant notes. Following the focus group, analysis consisted of the creation of general emic categories based on the recorded discussion. Group discussion points were then sorted into these categories, with categories of discussion then mapped to learning objectives from Session 1 and Session 2. (See Table 2.)

Results

The Course Director introduced the purpose of the focus group as “an attempt to dive a little deeper and find out what you’ve done differently in your practice.” Specific discussion topics related to management of relapse disease had been prepared in advance, based on outcomes derived from traditional written measures, with an intention to expand to Session 1 topics if needed. Participant surveys show an intention to change, based on Likert scale agree/disagree measures corresponding to the Contemplation or Preparation stages of the Transtheoretical Model. (See Figure 1.) In addition, multiple-choice and true/false test questions indicate an improvement in learner knowledge. (See Table 3.)

Initial discussion was concentrated on the prepared topic of therapies for relapsed multiple myeloma. All participants agreed that triple therapies were now implemented in their practice when addressing relapse disease. The discussion leader had intended to focus on specific outcomes of Session 2, but found that discussion quickly turned to application of principles learned in Session 1. While this was unanticipated, the leader allowed discussion to proceed on the track participants had self-identified.

Approximately 50% of group time (15.5 minutes of a 28-minute discussion) consisted of ideas exchanged about amyloid disease, the topic of Session 1 of MM Rounds. Typical comments included, “I learned so much. How to interpret the echo and the part where they talked about what cardiac amyloid should be ruled out for transplant. I just took notes, notes, notes.” Two participants reflected on how they had altered their diagnostic work-up at baseline as a result of Session 1. One physician noted, “I did a couple biopsies and sent them to Mayo. They’re obviously doing the most in this area. I didn’t do that before.”

Discussion about Session 2 of MM Rounds consumed approximately 25% of the group’s time. The primary observations about altered practice involved somewhat subtle changes in behavior, including understanding how difficult it can be to define “relapse” in multiple myeloma. The group as a whole agreed that relapse in this context is an evolving idea.

The remaining time of the group involved observations about the overall value of the MMR sessions. One internist observed that she now tends to make earlier referrals to oncology if she even suspects a case of multiple myeloma. This is because of her greater understanding that treatment has become much more individualized. The second internist pointed out that his practice includes many long-time patients who come seeking information/confirmation about their oncologist’s treatment recommendations, and so understanding of evolving treatment is essential to maintaining these patient relationships. The group agreed that there is no single treatment regimen that will work best, especially for the relapsed patient, and that the value of MMR is in exchanging ideas about treatment approaches. This value is reflected in the comment, “Keeping up with the research is great, obviously, but just hearing what other people are doing with it, that gets you focused.”

Discussion

We are sensitive to the questions of validity and reliability in our study, questions we had ourselves. One of the most common criticisms of qualitative study is its apparent lack of rigor. In addition, bias can enter any qualitative study at virtually any point, from a sampling error to insufficient depth of final analysis.

For these reasons and others, we were careful to model our study on established techniques.5 For example, from the outset, we maintained meticulous recordkeeping of our data to better illustrate how decisions were made. The make-up of our focus group was as representative of our learners as possible, so as to ensure inclusion of divergent views.7 We attempted to triangulate “hard” data with our qualitative findings, to provide as much supportive evidence as possible.8 When we examined our focus group discussion in the context of more traditional CE outcomes measures, we were able to extrapolate from the data important relationships, with implications for application of the activity content to practice. This also gave us important insights into development of future activities in the same topic area.

Knowledge acquisition measures were positive but mixed, demonstrating overall improvement in knowledge immediately following the activity. These results are of limited use in the context of practice change given that the measures were of simple recall and had not been beta-tested following any validated psychometric model.

Of more interest, our immediate post-activity intention-to-change measures were very strong, with 90% of respondents indicating they either “Agree” or “Strongly Agree” with a statement about applying activity content to practice. (See Figure 1.) Expressing an intention to change, even in an unspecified timeframe, has been correlated to actual change in behavior. The measures taken immediately post-activity correspond to the “Pre-Contemplation” or “Contemplation” phase of the Transtheoretical Model. When we allowed learners in the focus group to direct discussion of their own practice, we saw much stronger indications of change than had been reflected in our initial measures.

From the focus group participants’ specific examples of practice change, we concluded that the MMR were effective in ways we had not fully anticipated. We had expected the topics in Session 2 to be prevalent. Instead, participants spoke specifically and enthusiastically about Session 1 topics, a full six months after the session had occurred. This led us to conclude that our initial measure of intention to change did not allow for elaboration on specifics, assuming that participants would be willing or even able to do so during the immediate post-activity evaluation.

Clearly, change in practice had occurred, and what is likely more important for these learners is the Maintenance phase of the TM. This conclusion will assist us in developing content for MMR moving forward.

Our final conclusion about MMR is that its overall design is highly effective. Corresponding to our understanding of adult learning theory, the MMR are designed to be practical and connect to “real-life” experiences of clinicians by employing a case-based format.11,12   Interactivity of learners with content, learners with faculty and learners with one another are essential components. Focus group participants were quite specific in describing the ways in which this approach affected their practice. Again, this reinforced our observation that actual practice change had occurred in ways we had been unaware of.

Further study will be required to validate our observations.

Conclusion

The traditional outcomes measures we obtained for this interprofessional CE activity were adequate only in that they captured some knowledge transfer and a potential for learner change. What was missing – and what has been missing from most CE outcomes measures – is the ability to capture the richness of the learners’ interprofessional educational experience, the ways in which they do or do not change practice and the ultimate effect of change on patient health. We believe the thoughtful utilization of focus groups will help to complete the missing picture. Our intention is to continue using this assessment model in continuing cycles of this activity.

We are compelled to remember the work of Olson and Tooman, who called for new approaches to examining educational outcomes beyond the “mechanistic input-output models,” and suggested utilizing a metaphor known as “knowledge ecosystem”.13

This is a construct from the profession of strategic management that envisions change as the result of the synthesis of existing local knowledge (ie, the practitioner in a community setting) and external knowledge (ie, the academic expert).14  Our re-imagining of the interprofessional CE environment as part of a knowledge ecosystem has allowed us to implement a more complete measure of learner change and will move us closer to capturing our effect on patient health.

Table 1

Evaluation Question

Associated TM Stage of Change

Will adopt in unspecified time.

Pre-Contemplation

Will adopt in a year.

Contemplation

Will adopt in three months.

Preparation

Adopted since attending activity.

Action

Adopted prior to activity.

Maintenance

 

Table 2

Emic Category

Discussion Time
(in minutes)

Corresponding Learning Objective(s)

Cardiac involvement/adverse events

15.5

Distinguish among the biologic and molecular mechanisms involved in the pathology of MM as it relates to therapeutic targets and treatment selection.

 

Identify disease- and therapy-associated adverse events and utilize evidence-based management strategies.

 

Identify limitations and controversies in the standard of care treatment in MM, and how these pertain to individual patient management.

Definition of/triple therapies in relapse

7.0

Distinguish among the biologic and molecular mechanisms involved in the pathology of MM as it relates to therapeutic targets and treatment selection.

 

Appraise conventional, newly approved, and emerging therapies that have the potential to enhance clinical outcomes.

 

Identify limitations and controversies in the standard of care treatment in MM, and how these pertain to individual patient management.

Value of interaction with colleagues

3.0

Distinguish among the biologic and molecular mechanisms involved in the pathology of MM as it relates to therapeutic targets and treatment selection.

 

Appraise conventional, newly approved, and emerging therapies that have the potential to enhance clinical outcomes.

 

Identify disease- and therapy-associated adverse events and utilize evidence-based management strategies.

 

Identify limitations and controversies in the standard of care treatment in MM, and how these pertain to individual patient management.

Individualized therapies

2.5

Appraise conventional, newly approved, and emerging therapies that have the potential to enhance clinical outcomes.

 

Identify disease- and therapy-associated adverse events and utilize evidence-based management strategies.

 

Identify limitations and controversies in the standard of care treatment in MM, and how these pertain to individual patient management.

                       TOTAL

28

 

 

 Table 3

Session/Question #

Percent Correct, Pre-Activity

Percent Correct, Post-Activity

Session I, 1

49.0

60.7

Session I, 2

Not scored

Not scored

Session I, 3

Not scored

Not scored

Session I, 4

44.4

64.2

Session I, 5

57.1

62.5

 

 

 

Session II, 1

75.0

84.9

Session II, 2

Not scored

Not scored

Session II, 3

59.6

55.7

Session II, 4

Not scored

Not scored

Session II, 5

84.0

84.0

 

 

References

  1. Moore D, Green JS, Gallis HA. Achieving desired results and improved outcomes: integrating planning and assessment throughout learning activities. J Contin Educ Health Prof. 2009; 29 (1):1-15.
  2. Curry L, Purkis IE. Validity of self-reports of behavior changes by participants after a CME course. J Med Educ. 1986 Jul;61(7):579-84.
  3. Johnson SS, Cummins C, Paiva A, and Brown JJ. Measuring effectiveness of continuing medical education using the transtheoretical model of behavior change. CE Meas. 2012;6:32-40.
  4. Williams BW, Kessler HA and Williams MV. Relationship among knowledge acquisition, motivation to change, and self-efficacy in CME participants. J Contin Educ Health Prof, 2015;35: S13–S21.
  5. Miller M, Crabtree BF. Clinical research. A multimethod typology and qualitative roadmap. In: Crabtree BF, Miller M, eds. Doing Qualitative Research. 2nd ed. Thousand Oaks, CA: Sage; 1999.
  6. Kotwal S, Peña I, Howell E, Wright S. Defining clinical excellence in hospital medicine: a qualitative study. J Contin Educ Health Prof, 37(1):3-8, Winter 2017.
  7. Morse J, Barrett M, Mayan M, et al. Verification strategies for establishing reliability validity in qualitative research. Int J Qual Res 2002;1:1–19.
  8. Kuper A, Lingard L, Levinson W. Critically appraising qualitative research. BMJ 2008;337:a1035.
  9. Green J, Thorogood N. Qualitative methods for health research. London: Sage, 2004.
  10. Keshmiri F, et al. Effectiveness of an interprofessional education model based on the transtheoretical model of behaviour change to improve interprofessional collaboration. J Interprof Care. Volume 31, 2017 - Issue 3
  11. Knowles MS. Application in continuing education for the health professions: In: Knowles MS. Andragogy in Action: Applying Modern Principles of Adult Learning. Proquest/Csa Journal Division, 1984.
  12. Collins J. Education techniques for lifelong learning: principles of adult learning. RadioGraphics 2004; 24:1483–1489.
  13. Olson CA and Tooman TR. Didactic CME and practice change: don’t throw that baby out quite yet. Adv in Health Sci Educ (2012) 17:441–451.
  14. Nonaka I, Toyama R, and Hirata T. (2008). Managing flow: A process theory of the knowledge-based firm. Basingstoke [England]; New York: Palgrave Macmillan.

Lessons for Practice

Qualitative data analysis is an effective means of assessing the richness of a complex educational experience.

Re-imagining the interprofessional CE environment as part of a knowledge ecosystem will allow us to implement more complete, innovative measures of learner change and will move us closer to capturing our effect on patient health.

Recent Stories
Wills Eye Opens $1.5M Training Lab

Repackage Your CME Programs for MIPS Compliance

Research to Actions Interview With Dr. Jared Austin, MD