In their most current survey, Harting and Bowser asked survey participants to address an intriguing open-ended question about what they had noticed as poor or unprofessional practices in other needs assessments they had seen. The correlation between two independent reviewers who each assessed the 67 responses and then categorized them into the predefined categories was very well aligned. The authors do note that the observations given are all secondhand from something seen from the past. Additionally, the qualifications of the writers whose work is in question are unknown.
Survey respondents noted a number of concerns that are just overall poor writing skills (grammar, organization, coherence, readability, sources and referencing); perhaps the writers who responded perceive their skills at writing are better than others they have seen. Other concerns seem to be more about a lack of understanding of the content area (irrelevance or poor focus along with outdated information); although this can easily occur within today’s rapidly expanding information occurring within medicine, it is not an excuse. Although the specifics are lacking, the more concerning comments are of course those referring to plagiarism, fabrication and bias.
The authors have noted some key limitations of this survey and its results. Another consideration is, shouldn’t those who read the needs assessments—the commercial supporters—assess the issue of describing poor needs assessments? Although there has been some discussion at panels that have included supporters at meetings of the Alliance for Continuing Education in the Health Professions (Alliance), the authors have not noted any specific published surveys from the supporter audience. Since 2009, the Alliance member section, the Industry Alliance for Continuing Education (IACE), has created an annual Benchmarking Survey to gain insights to trends and standards applied in commercial support of Independent Medical Education and results reported by their Benchmarking Working Group (BWG). To date, questions about the quality of needs assessment within grants have not been extensively assessed by this survey (personal communication with a BWG member). It would be great to see such a comparison survey done within this member section of Alliance. Until such time, comments are welcomed from commercial supporters who sit on grants committees and read many needs assessment and grants.
In their discussion, the authors suggest that educational opportunities for writers and others on how to conduct a high-quality literature review might be useful. The needs assessment is only one part of an overall grant and must align with the other parts, including the overall goals and objectives of the initiative, the educational tactics being recommended and what will be assessed as part of the outcomes plan.
As the authors have noted, the most disturbing results of this survey are suggestions of plagiarism, fabrication and commercial biases. What is not clear about these “worse case” examples is when they occurred—were they noted from many years ago or from a more recent period? If more recent, this is of even more concern based on today’s environment of ensuring independence of commercial interests. I think it is also necessary to stress that the ultimate responsibility of any proposed continuing education initiative (which includes the needs assessment and entire grant) is that of the accredited provider. Even if being contracted out to another entity or person, they need to ensure proper understanding, have processes for editorial reviews and have oversight for this important step in the process. This is well documented within the Accreditation Council for Continuing Medical Education (ACCME) accreditation criteria that address the needs, formats and independence of all certified educational activities.
The ideas raised by Harting and Bowser in the article about the need for a structured template as well as a need for needs assessment spot check of the continuing education needs assessment (and perhaps the entire grant) are interesting to consider. However, just like the practice of medicine, there is both an art and a science to a well-created needs assessment. It takes practice, time and feedback. Moreover, similar to clinical practice, what we are lacking is feedback and coaching regarding our needs assessments.
Author contact: tbruno@francefoundation.com