By Gary Bird, Ph.D., American Academy of Family Physicians, and Pesha Rubinstein, MPH, CHCP, American Medical Informatics Association
This concluding article in the Almanac biostatistics series, which targets the beginning-level learner, concludes as it started: with a restatement of the objectives of the articles.
It is our hope that the series achieved its goals. In this article, we’ll summarize the topics the series covered, and then we’ll share some evaluation data and recommend additional resources.
Reflection on the series content produces the following definitions and key points:
“Sources of Data in CE,” by Lloyd Myers, RPh, and Simone Karp, RPh
- Think backwards from the measure. If you begin with the end in mind, you can work in reverse to make sure that all of your data elements are reasonably within reach.
- Map your activity’s desired outcomes to valid data sources. For example, if your activity focuses on a tool that should help primary care practitioners improve suicide risk identification in adults with major depressive disorder, use National Quality Forum’s measure description, numerator statement and denominator statement to take the pre- and post-activity measures in this area. Go to www.qualityforum. org/QPS/ and click “Data Source” on the left.
“How to Write Sound Educational Outcomes Questions: A Focus on Knowledge and Competence Assessments,” by Erik D. Brady, PhD, CHCP, and Sandra Haas Binford, M.A.Ed.
- Depending on how a CPD professional writes test questions, he or she can measure either a learner’s knowledge or competence. By using realistic scenarios, the educator can accurately measure a learner’s intent and competence.
“Concepts Involved in Sampling Data,” by Melanie D. Bird, PhD, and Erik D. Brady, PhD, CHCP
- Sampling is the process of selecting participants from a particular population to represent that population as a whole.
- Random sampling is used in CEhp programs, not only to limit bias but for two other tangible reasons: It requires the least amount of forethought in the design of the outcomes tool, and it allows the analyst to report the highest participation possible in the outcomes study.
- The spread of scores around the parameter for our population is called the standard deviation (often abbreviated to SD, or denoted by the Greek letter σ). The spread of scores across the sampling distribution is the standard error (sampling error or SE).
- The lower the standard deviation and the larger the sample size, the smaller the sample error becomes.
- The more diverse the sample is, the larger it will need to be to account for the variability and the more confident we can be in the result.
“Impact of Sampling at Multiple Time Points in Measuring Outcomes of Continuing Education in the Health Professions,” by Gary C. Bird, PhD, and Sandra Haas Binford, M.A.Ed
- As many learners experience information loss after an activity, it is important to gather and report educational outcomes data at longer time points.
- A series of linked CE activities offer opportunities for appropriate and accurate measurement. Outcomes data gathered throughout the series inform us of opportunities to tailor educational events to the evolving needs of learners. Learning occurs only if the content is right and the educational delivery is relevant to the target learner population.
“Basic Concepts of Data Sets,” by Melanie D. Bird, PhD, and Derek T. Dietze, MA, FACEHP, CHCP
- Qualitative data are nonnumeric (words, not numbers) and deemed “categorical,” as they are descriptive in nature, falling into distinct categories such as color, texture, appearance or demographics (sex, type of practice, healthcare specialty). Qualitative data can be described but not measured until they are linked to a numerical scale, which makes them become quantitative.
- Quantitative data are numerical data that can be measured.
“Distribution and Variation in Data Sets,” by Gary C. Bird, PhD; Melanie D. Bird, PhD; and Sandra Haas Binford, MAEd
- A meaningful distribution profile becomes truly apparent when data are graphed. In graphs, data take on a distinctive shape based on the frequency values of the group. A normal distribution, or a bell curve, is most common. However, distributions can also be positively or negatively skewed.
- The distribution of the data determines which statistical test will be most appropriate. For data that follow a normal distribution, parametric tests are used. A parametric test assumes a normal distribution across the population and that the measures are from an equal interval scale. Examples of parametric tests include t-tests and analysis of variance (ANOVA).
- For nominal or ordinal data that may not be distributed normally, nonparametric tests are used. Nonparametric tests are those used to analyze data that have a skewed distribution or when the outcome has limits of detection or outliers. Examples of nonparametric tests include chi-square, the Mann-Whitney test and the Fisher’s exact test.
“How to Analyze Your Baseline, Post-Activity Change Data, Parts 1 and 2: Baseline, Post-Activity Multiple-choice Questions,” by Erik D. Brady, PhD, CHCP, and Derek T. Dietze, MA, FACEHP, CHCP
- A P value (the P means probability) is generated from a test of statistical significance (a mathematical formula). Simply put, the P value represents the role that chance plays in your outcomes.
- In general, a P value of 0.05 or less represents the “gold standard” in scientific research, meaning that 95 percent of the time your findings are statistically significant. This means that there is only a 5 percent likelihood that a calculated change from baseline to post-activity would occur by chance alone if the same education were offered to additional learners of similar demographics.
- A small P value (typically ≤ 0.05) indicates strong evidence that the baseline to post change is real and is not due to chance. A large P value (> 0.05) indicates weak evidence that the baseline to post change is real, and it is more likely due to chance.
“Understanding the Impact of Data and Analysis at the Population Level: How Common Statistical Mistakes Impact Data Interpretation,” by Gary C. Bird, PhD and Melanie D. Bird, PhD
- One of the first things to do as you review data obtained from a CE activity is to ask the question, “Is the data reproducible?” meaning that another person can construct the same study and obtain the same results. Reproducibility is important for generalizing data across a population.
- One of the biggest assumptions made in statistics is that by providing a P value associated with a comparative parameter the data is given a “seal of approval.” However, just because your results are non-significant does not mean there is no effect. Although P values can give an indication of differences, over-reliance on them can prove disastrous!
Since evaluation is part of the cycle that CPD professionals engage in at the conclusion of any activity, we’d like to share in brief the measured outcomes of this series.
The first outcome was a formative qualitative study that the Alliance conducted while gathering Alliance Member Value Statements on this guide. There were five members who chose to comment on the series. All of the feedback was favorable, and we are reprinting two of them here:
- “It was important for me to reach out to you as a longstanding member of the Alliance. I found the statistics series to be journal-worthy! The authors codified complex theoretical constructs and translated these in a manner that supports our collective efforts to integrate these principals in practice. I look forward to the next article.”
- “The new statistical series has added a level of professional value and depth of content that I have shared with my CME planners and certification and instructional design specialists. We recently conducted a staff training utilizing excerpts presented in the July statistics series. Based on feedback from staff, we plan to continue monthly trainings on these topics. Well done!”
A second outcome was the successful submission of a poster to the 2016 World Congress on Continuing Professional Development held in San Diego this past March. The poster focused on the methodology of the series and was organized by Sandra Binford, M.A.Ed. The poster title was “Promoting Adaptive Expertise in Educational Research and Outcomes Analysis Among CEhp Professionals Through a 12-Part Series of Archived Case-Based Newsletter Articles.”
With the positive evaluation feedback on the series, it seems the Almanac has hit upon an important gap in CPD competence. A central concept in the CEhp National Learning Competencies is knowing how to use data for a variety of CPD tasks, from identifying educational gaps to measuring activity success. For those who never studied statistics, the series seems to have opened the door to discussion. For those who studied it a long time ago, the series was a refresher course.
For CPD professionals to continue learning about clinical and educational outcomes measurement, there are many free resources. Search “biostatistics” or “inferential statistics” on the following sites:
- iTunes University
Massive Open Online Course (MOOC) sites make this – and other – topics of relevance to CPD professionals free and accessible for users. As the number of educational resources can be overwhelming, you may find it helpful to embark on this journey with a mentor or study partner.
Find a Study Partner
If you learn better with real rather than virtual students, get together with a friend or colleague. Establish a set time to study together, enroll in an online course together, get ahold of an introductory book like “Biostatistics for Dummies” or create a syllabus based on the topics in the Almanac series. Focus on small increments of material to cover, then practice together by reviewing a journal article that uses the statistical principle you are studying. If you can clearly explain the concept under review to your study partner, your grasp of biostatistics has gotten that much stronger.
And if you and your colleagues complete an entire introductory course or book together, we invite you to write about your experience and its impact on your professional competency in an article for the Almanac.