Editor’s Note: This is the second article of a two-part series about a gender-affirming continuing education (CE)/continuing medical education (CME) initiative for healthcare professionals. Whereas the previous article focused on the educational planning and design of a seven-part, CE/CME-certified series of live activities on gender-affirmation healthcare in the HIV care setting, this article will highlight the creation and utilization of a new outcomes methodology that launched in tandem with the initiative.
When the Annenberg Center for Health Sciences at Eisenhower (Annenberg Center) began an educational venture into gender-affirmation healthcare in 2015, it was readily apparent that we needed an outcomes tool that would provide meaningful data on the skills, attitudes and behaviors that influence the adoption of culturally competent care. Since we were unable to find an outcomes methodology that met our expectations, we developed one.
A New Outcomes Model Born from Logic
As discussed in Part 1, the Annenberg Center utilized a logic model in the initial planning of all educational initiatives in 2016. Using the backward-planning process of a logic model (see Figure 1), we defined long-, medium- and short-term desired outcomes or intended results for the educational initiative.
Once the desired results were determined, we established educational goals with which to assess achievement of the intended results. These educational goals, in turn, became the learning objectives for the initiative.
Once the objectives were established, each goal was segmented into two clinical skills that providers would have to perform competently in order to achieve that goal.
For example:
- One of our desired outcomes was for healthcare professionals to “incorporate gender-affirmation terminologies and support strategies in the care of transgender and gender-nonconforming (TGNC) patients” (Figure 2).
- A measurable learning objective was then developed for the goal (i.e., “discuss gender-affirmation strategies to improve the healthcare experience of TGNC patients”).
- We then determined two clinical skills that healthcare professionals needed to be able to support this objective/goal and, ultimately, the desired outcome:
- Implement practice solutions that address stigma, discrimination and other barriers.
- Provide inclusive forms and use gender pronouns and name.
It’s not new to use learning objectives to express the intended changes; however, by identifying supporting clinical skills, content became keenly focused on addressing barriers to change and providing solutions on how to overcome those barriers.
Utilizing this backward-planning method ensured that our educational goals remained practical, relevant and constructive to achieving the desired outcomes.
Assessing Importance, Proficiency and Motivation
At the Annenberg Center, we recognized that it wasn’t enough to create practical, relevant and measurable educational goals and supporting clinical skills; we required an outcomes methodology that could efficiently measure learner’s proficiency with each clinical skill.
Furthermore, with clinical skills related to cultural competency — as in the case of our gender-affirmation educational initiative — it’s just as meaningful that healthcare professional learners recognize the importance of these skills and express a willingness to improve them.
Therefore, a new outcomes methodology — The IPM Model — was born. Utilizing self-assessment prior to and immediately after the delivery of education, The IPM Model is intended as a three-factor outcomes tool:
- The first factor is the Importance that learners assign to each clinical skill.
- The second factor assesses learners’ Proficiency in that clinical skill.
- The third factor profiles learners’ Motivation to implement changes to address the clinical skill.
Utilizing a five-point Likert scale, The IPM Model instructs learners — pre-education — to reflect on their practice over the past few months and rate each clinical skill regarding its importance, their proficiency and their motivation to change (Figure 3).
Post-education, a similar, self-assessment evaluation is conducted in which learners are asked to reflect upon their future practice and again rate each skill regarding its importance, their expected proficiency and their motivation to change.
The IPM Model provides insights on how learners’ attitudes, behavior and even culture affect learning and change implementation.
For example, clinical skills rated high on Importance, low on Proficiency, and high on Motivation to Change illustrates a compelling educational opportunity (Figure 4); the deep V-pattern depicts that learners rate the clinical skill as highly important, admit to having a low proficiency in the clinical skill, and yet express a willingness to implement a constructive practice change were their proficiency in the clinical skill to be improved.
Outcomes from Our Gender-affirmation Educational Pilot Program
Figure 5 shows the pre- and post-education overall mean results for our seven-part pilot program on gender-affirmation healthcare; these data clearly illustrate the deep V-pattern for importance, proficiency and motivation.
Pre-education, learners assigned a high level of importance and motivation for improving cultural competency in gender-affirmation healthcare yet reported a low proficiency in caring for TGNC patients in a culturally competent manner. It should be noted that, given our pilot initiative was designed for HIV specialists and infectious disease specialists to integrate gender-affirmation strategies in the HIV care setting, we were not surprised that learners who self-selected to attend these live activities, initially reported that the education was important to them and that they were already motivated to make necessary changes. Likewise, the pre-education data demonstrating low levels of proficiency in gender-affirmation healthcare from these same healthcare professionals aligned with our data on the significant HIV-care disparities experienced by TGNC persons.
Interestingly, the post-education results show that not only was there a significant gain in overall learners’ proficiency, but both importance and motivation to change continued to climb, despite already high pre-education scores.
Figure 6 represents the distribution of proficiency Likert responses across learners versus the overall mean proficiency responses depicted in Figure 5. In the pre-education evaluation, learners were asked “to think about the past few months in their practice and rate how proficient they were using gender-affirmation strategies to improve the healthcare experience.” Pre-educational intervention, only 22 percent of learners assessed their proficiency to be superior or distinguished (4 or 5 on a 1-5 scale, Figure 6).
Post-education, learners were asked “to think about when they return to their practice, to rate their expected proficiency in using gender-affirmation strategies to improve the healthcare experience”; this time, 57 percent of learners assessed their proficiency to be superior or distinguished. This represents a 35 percent favorable change in proficiency between pre- and post-education evaluation.
The distribution ratings in Figure 6 provides insight on proficiency improvement generated across learners and demonstrates achievement of an important educational goal for the activity. Some open-ended comments for specific practice or behavior changes planned among learners included: “ask [patients] for pronouns,” “improve pronoun awareness,” “change the way I ask questions to promote and increase open discussion regarding TGNC issues and concerns” and “ask questions differently to put TGNC patients at ease,”
The favorable proficiency changes described in the gender-affirming educational goal were also observed with supporting clinical skills. Figure 7 illustrates the distribution of proficiency Likert responses for implementing clinical practices such as “providing inclusive intake forms and using appropriate gender pronouns and names.” A significant increase was reported in self-assessed proficiency between the pre- and post- activity responses. After the educational activity, 61 percent of learners assessed themselves as having superior or distinguished proficiency in this clinical practice versus only 26 percent before the activity.
Conclusions
Our gender-affirmation educational initiative changed the attitudes and behaviors of not just our learners, but of the Annenberg Center, as well. In addition to providing the Annenberg Center with an opportunity to strengthen our own cultural competencies as an organization, this initiative challenged our team to revamp our content development and outcomes processes to efficiently create educational content that aligns with practical, sustainable and measurable outcomes.
The development of a new, three-factor outcomes methodology — The IPM Model — illustrates how skills, attitudes and behavior influence the adoption of practice changes and how the incorporation of practical, measurable items within the outcomes-development process can help achieve the desired outcomes of the educational initiative.
Side Note
Due to the success of our pilot program and data gleaned from it via our new outcomes methodology, the Annenberg Center is set to launch a nationwide gender-affirmation healthcare training and certification program in May 2018, with the intention of engaging not only healthcare professionals, but their entire care practices in adopting sustainable, gender-affirming strategies. TGNC patients will serve as content planners and speakers on this initiative alongside faculty, and it will target both primary care practices and specialty HIV care settings in metropolitan areas with a higher prevalence of TGNC persons. The long-term outcomes include providing greater access to HIV care for TGNC persons and retaining more TGNC clients in HIV care by providing culturally sensitive care at every point in the care continuum.
On a related note, the Annenberg Center was chosen to present — The IPM Model — at the Expert Lecture Series for Innovative Outcome Methodologies at the 2017 American Evaluation Association meeting. More than 7,000 members comprised of evaluator scholars, evaluators and evaluation users from academia, governmental agencies, non-profits, and private business attended this four-day meeting; the Annenberg Center’s IPM Model was well received by this international audience of evaluators.