Jim: Where did you start? How has outcomes reporting evolved within each of your groups?
Karyn: I had been a provider for 10 years and have been working with Regeneron for five years. When I started 15 years ago, there were no outcomes (framework, aggregation, standardization). Building the framework has been a process. Now at Regeneron, after five years, we finally have a stable database where we can draw medical education providers’ outcomes and metrics from in a standardized manner. We then aggregate and do meta analyses across all [Regeneron’s] therapeutic areas. When I started, to not reinvent the wheel, I thought about what metrics providers were already collecting. I decided to focus only on the metrics I knew everyone was already collecting; however, I did layer on another format: the Educational Impact Summary (EIS). The EIS adds a narrative to the outcomes report. Providers may have multiple narratives; however, they are asked to provide what is — to them — the most important narrative. This is the narrative we want communicated when we speak to our internal stakeholders. “What do you feel the impact of your program was?” In the aggregation of data, they are distilling variant narratives and item-level analyses to a 1–3 slide summary. This communicates the impact of the program through a data narrative, which is important because if you cannot connect numbers, statistics, and findings to aspects, behaviors and demographics that matter to your stakeholders and audience, they aren’t going to listen. We are here to impact patient care and improve the quality and focus of the education you create, and [Regeneron] can’t do that without communicating your impact properly.
Jim: [follow-up question to Karyn] What do the best 1–3 slide summaries look like and communicate?
Karyn: The impact summaries that add the qualitative (attitudinal) aspects backed with quantitative data “speak to” internal stakeholders the most. The best impact summaries are the ones that are the simplest, “elegant,” creative and pair the data and interpretation. The better data narratives are those that are digestible to a wide audience including all stakeholders.
Kate: …went to Sanofi-Aventis, which is where the outcomes story really began for me. More recently, I’ve worked with supporters who had an outcomes template. I joined Celgene a few years ago and was hired to aggregate outcomes, along with supporting hematology and oncology therapeutic areas. My role involved pulling together quarterly reports of aggregated outcomes data. I used the grant portal and linked it with Tableau, which pulls data directly from grants. This system has multiple filters, so the grant portal should ask the right questions. “The richer the questions, the richer the outcomes.” This allowed us to share quarterly reports and utilize one infographic per therapeutic area. My current position does not have a template, does not limit the providers on the length of the outcomes reports. I am fortunate to speak at monthly internal medical unit meetings for about 25 minutes, where they are interested in insights, not just quantitative data. We not only provide insights from what we supported, but education on what IME can actually do. I totally support Karyn’s executive summary and think everyone should utilize it. I support the ability to tell a story up front to the team: why the education was done; the gap being filled; supported objectives and impact; and how this relates to internal objectives and filled a gap.
Jim: [follow-up question to Kate] Do you want all of the data/slides in a 1–3 slide summary as the providers pull out what they found interesting, or would you rather pick your own?
Kate: I like both. Slides are lengthy, but the three-slide summary can start off a conversation with the medical units and tell the story of why we supported the grant and what the impact was.
Audience: How much weight or importance do you put on the methods, not just data? The data is only as good as the methodology.
Kate: I don’t think the methods are that important to internal stakeholders where I work, although I would like to educate them on the science of adult learning and have them understand that it isn’t just “butts in seats” and the “old school” way of thinking about outcomes. There is a science and strategy behind why a program is designed and chosen and how the outcomes are measured. In the future, I would like internal stakeholders to understand because that would likely lead to more of a budget for IME at my current company.
Karyn: Regeneron is science focused. They ask a lot about methodology. I think the providers give “solid” methodology. They are evolving in the addition of real-world evidence. Providers are now using complex data, such as claims level data, ICD9 codes and patient journeys. I think the methodology is often not communicated enough, but is extremely important, especially when making statements about patient-level data.
Jim: [follow-up question to Karyn] In general, how are the providers doing?
Karyn: Providers have accomplished so much. They are using T-tests, Tableau, modeling and advanced analytics, natural language processing, publishing, presenting, and utilizing data in grants and needs assessment submissions. Supporters “need to catch up” and keep up with the providers. Providers are giving rich data, and I am not sure supporters have the framework and resources to “give it justice.” Companies are now looking for people with a background in analytics, and it has taken me five years to get “up to speed,” which I feel is a long time to adequately and usefully aggregate data. I believe providers are already “NextGen” with outcomes, but supporters need to catch up.
Jim: Kate, you have been with several organizations. Can you generalize what challenges you faced sharing data, with some more advanced than others? How were you able to get outcomes and understanding about what that means from IME? How do you recommend supporters try and get more time?
Kate: At Celgene, I did a “roadshow” to the medical units first, which explained IME and the methodology, science, and outcomes. I explained the impact and value IME activities can exhibit. When we are heard, IME seems to get more of a voice. I am currently working on a similar process with Takeda. We have a long way to go with education on quantitative and qualitative outcomes data and their impact.
Jim: Karyn, are providers able to increase budgets on what they are charging for outcomes? How should we address that? Is it a budget line item: regular versus robust outcomes at a different cost?
Karyn: If budgets go up, supporters need to justify that. For Regeneron, the increase in outcomes-related line items is fine.
Jim: [follow-up to Karyn] So, the cost of doing business is providing those higher-level outcomes?
Kate: The onus is on us. If we want a certain level of outcomes, we must be willing to pay for it. If someone doesn’t have an outcomes team (e.g., hospital, society), we don’t expect higher-level outcomes from those organizations. I understand there are different expectations from smaller institutions.
Jim: [follow up to Kate] If someone has a great program design … fantastic learner, participation, but terrible outcomes, would you not use them again?
Kate: Not necessarily. I think I would have a conversation about expectations.
Jim: In the future, what are we looking at, and what should we be gearing up for?
Karyn: Data isn’t going away. It is everywhere. Expect some softer metrics and text- based/qualitative analysis. We are headed toward a combination of quantitative and qualitative metrics. If we are going to show an impact on behavior, it will be with real-world evidence.
Jim: Do you see more groups submitting programs that include real-world evidence and de-identified data?
Karyn: We are definitely seeing more submissions including that level of data, and the budgets are not outrageous.
Kate: I agree and recommend providing education and partnering with internal stakeholders and other areas within the company and demonstrating how company teams can support grants together to get real world evidence (e.g., with HEOR, RWE, patient advocacy). We will be able to better educate patients and HCPs and get outcomes from that.
Jim: Let’s hear from providers. What challenges have you faced? What questions do you want to ask the panelists?
Audience: You mentioned aggregating grants data through the portal, Tableau. What trends or key metrics are important to supporters to identify that type of longitudinal data analysis?
Kate: This depends on the company. Some want to see how long it took to approve a grant from receipt to completion, and others want to see how many grants were supported and align with learning objectives in your strategy (key one). If what they are supporting is aligned with the learning objective, it communicates internally that they are aligned to the internal medical strategy.
Audience: What medium do you prefer for outcomes (live, virtual, polling webinar, podcast)? Is the expectation changing for the outcomes based on the mediums?
Karyn: If I am aggregating data, I need to have standardized metrics across all programming, which is the definition of a meta-analysis. For example, a podcast is a challenge for aggregating data.
Jim: For outcomes reports, is it more important to get outcomes sooner than later? What do you trade off? When should the analysis take place?
Kate: I like getting interim outcomes with an analysis, but it does not have a specific timeframe. I present the outcomes to the medical unit teams when they are available and do not wait until the end of the program. The final outcomes report sums it up and tells the story.
Karyn: I agree. I ask for outcomes every quarter after launch. The provider enters data into system and then it aggregates. “The sooner they can report out, the better.”
Moderator Jim Palatine, RPh, MBA, is with Pharmacy Times Continuing Education (PTCE) and Physicians’ Education Resource (PER).
Panelist Karyn Ruiz-Cordell, MA, PhD, is with Regeneron Pharmaceuticals.
Panelist Kate Biles, CHCP, was with Takeda at the time of this Q&A and is now with Regeneron Pharmaceuticals.