What about Kirkpatrick’s four levels of evaluation?
As an education professional, you should be familiar with Kirkpatrick”s four levels of evaluation. It is worth noting that there is a shifting of conceptual gears between the third and fourth elements in Kirkpatrick’s framework. The first three elements center on the learner; their reactions, their learning, and changes in their behavior. The fourth element shifts to a concern with organizational payoffs or business results. Kirkpatrick was careful not to imply that a complete cycle of evaluation from Level 1 to Level 4 can always be completed. He suggested that approximations of benefits in non-measurable terms may be substituted for operational measures if the politics and costs of obtaining operational data and transforming them to financial measures are too complicated.
What about Level 4?
Level 4 of Kirkpatrick’s model reflects the evaluation of educational impact on business results:
increasing sales, reducing accidents, reducing turnover, and reducing waste, etc. Data such as this is often collected via operational performance data, financial reports, or perceptual data. The goal of this type of evaluation is to determine the impact of an intervention on the organizational bottom-line. If the program’s aim is “tangible results, rather than to teach concepts, theories, and principles, then it is desirable to evaluate in terms of results” (Kirkpatrick, 1994, p. 70). However, Kirkpatrick offered little as to methodology for this level of evaluation.
Where does ROI fit into the picture?
The most frequent extension of the Kirkpatrick model draws attention to financial returns on the
educational investment. Focusing on the costs and benefits can provide valuable information to
senior managers who make the decisions on whether to increase or decrease funding for education within their organizations. There is no escaping it, educators are increasingly having to account for dollars spent on education and they are having to do it in terms of business results and return on investment. Keeping this in mind, it is important to approach every educational event with the question “How will this impact the goals of my organization?”
A related variation of the Kirkpatrick model splits the fourth level (business results) into two parts, thereby adding a fifth level - return on investment (ROI). “Level 5” evaluations, as they are sometimes referred to, convert Level 4 evaluation indicators into monetary terms for the purpose of calculating the ROI of educational dollars. In other words, ROI is essentially a method for collecting data regarding the impact of education and calculating the monetary return of this impact on the organization.
How is ROI measured?
While the impact of education can be difficult to measure, learning professionals can still be held
accountable for financial results. Tools, techniques, and reliable processes are available to measure the return on investment in education. Criteria for an effective evaluation of training at the ROI level emphasize conservative approaches. An effective ROI process is simple, economical, credible, theoretically sound, flexible, applicable with all types of data, and inclusive of program costs. Further, an effective ROI process must account for other factors that may have influenced output variables, and the ROI process must have a proven track record. Advocates for financial evaluation of educational efforts may differ in their approach but agree that it is possible.
How will the data be used?
Most organizations spend less than one percent of their budget on measurement and evaluation processes. Interjecting accountability throughout a program requires expenditures around four to five percent of the total education budget. However, ROI will generate measurable savings to help offset the increased expenditure. Examples of potential savings include the following:
- Preventing unnecessary programs
- Redesigning existing programs to make them more effective and less expensive
- Eliminating unproductive and ineffective programs and thereby eliminating costs
- Expanding the implementation of successful programs, which may add value to other parts of the organization
Many organizations keep a running total of the monetary benefits derived from implementing an ROI process. When comparing the benefits of ROI to the cost of implementation, the benefits are significant.
Are you ready for ROI?
Accountability for the cost of education does not apply to any particular type of organization. Being accountable for the cost of educational endeavors is a basic concern for organizations regardless of their product, mission, services, or scope. However, there are some signs that may indicate an organization is ready for an ROI process. These signs are briefly discussed below.
- Organizational size – Do you work for a large organization? Large organizations typically have the budget to develop comprehensive evaluation plans. However, ROI should be built into the accountability process in all organizations, regardless of size.
- Visibility of the educational budget – Is your educational budget large? If an educational department has a large budget, it will likely have the attention of the senior management team. Regardless of how it is measured, (i.e. total budget, expenditure per employee, percentage of payroll, or percentage of revenue),a large budget will cause most executives to demand increased accountability for expenditures.
- Focus on measurement – Does your organization measure other processes via methods such as a Balanced Scorecard, Six Sigma, etc.? If so, your organization is likely measurement-focused and ready to delve into the ROI of educational initiatives.
- Recent failed educational effort – Every organization has had a program that was implemented without success. Failures may prompt the implementation of a measurement and evaluation process or to forecast ROI prior to rolling out new initiatives.
While ROI is not universally used for every educational program, shifting priorities and expectations make the topic very important to educational professionals. Learning as much as you can about the ROI process BEFORE you are required to enact methodology is strongly advised.