At a cost of more than $3 billion a year, high-quality training
courses, tutorials, and multimedia simulations that can keep up
with the rapid pace of technological development are inherently
expensive and time-consuming to create. With new content providers,
technology suppliers, and services emerging on a weekly basis, it
becomes increasingly more difficult for training professionals to
efficiently compare the quality, price, and business value of
dozens or even hundreds of online programs, options, or solutions
to ensure that technology-based learning solutions are
cost-effective and appropriate to business needs.
As executives and managers watch learning budgets grow, there is
prevailing frustration from the lack of evidence showing that
learning and e-learning programs can really help performance.
Sponsors need to know how major investments of time, money, and
resources are paying off and aligning with strategic business
goals. A comprehensive measurement and evaluation process
represents the most promising approach to meet rising
accountability challenges. To that end, the use of
return-on-investment is emerging as an essential part of many
global measurement and evaluation systems. Trends show that
organizations with comprehensive measurement and evaluation systems
in place have enhanced their program budgets while those without
comprehensive measurement and evaluation systems have reduced or
eliminated their program budgets.
The e-learning perspective
Despite heightened application of ROI as a measurement strategy,
there remains much confusion about how to measure the ROI of
e-learning methods. On a practical basis, it is helpful to remember
that e-learning is simply another method of instructional delivery
and that the process of calculating ROI is the same, regardless of
the delivery method. However, in far too many cases, e-learning
investments are justified by simply comparing the cost of the
instructor-led approach to the e-learning approach. This can result
in radically different cost amounts, because of the savings
generated with travel, lodging, classroom facilities, and
instructor cost. For such a comparative analysis to be accurate,
the outcomes of the instructor-led learning must be the same as
those generated with e-learning and that is not always the
situation.
The recommended approach
An accurate ROI calculation of an e-learning program requires data
collection at four levels - reaction, learning, application, and
impact. The impact data is isolated from other influences and
converted to monetary value. This monetary value is then compared
to the cost of the e-learning program. This results in a credible
ROI study. To fully compare the value of e-learning to the value of
an instructor-led program requires two comparable studies with the
same calculation process in which a string of benefits and costs is
developed for both scenarios. If the ROI is higher for the
e-learning delivery alternative, then it represents a good approach
from the economic perspective.
An alternative approach
It is not always feasible to complete an instructor-led comparison
due to the extra effort needed to capture four levels of data and
conduct a comprehensive ROI study. Consequently, an ROI based on
cost savings alone may be proper. The key issue here is determining
the level of success achieved with targeted learning outcomes. For
instance, if instructor-led data describes the extent of desired
learning gains, then the same measures can be used to compare
learning gains from the e-learning approach. In other words, if
participants learn as much or more with the e-learning delivery
method as they do with an instructor-led option, then comparing the
two projects on cost savings alone may be appropriate - with maybe
one more adjustment.
At Level Three, when the evaluation follows how participants have
applied what they learned, we have found that participants in an
e-learning program are less likely to follow through with behaviors
than those in an instructor-led program. There is often a
commitment made between the participant and the instructor in an
instructor-led process, which can increase the likelihood of
participants applying what was learned and increase the likelihood
that data will be provided in a follow-up survey. An e-learning
participant is more disconnected from the instructor and can easily
ignore the computer or computer prompts. This factor tends to
reduce the monetary value of e-learning, because the monetary
benefits will not be the same. The complete ROI study is conducted
using benefits and costs for the e-learning, along with the cost
savings generated by the cost differential of presenting the
program two different ways - this should also include a foot note
suggesting that the e-learning results might not be quite as
impressive, because there will generally be less on-the-job
application. Essentially, the payoff is expressed as a difference
in the cost two different learning approaches, but with a
qualifier.
Implications for practice
These complexities have caused some publications to steer ROI
calculations away from cost comparisons alone. For example, a
recent article in Training Magazine invited readers to
submit ROI studies for inclusion in a future article focused on ROI
in e-learning. The request included a notation that studies must
not be based on cost savings alone, but must also show the full ROI
calculation. This was meant to emphasize the importance of a
complete, accurate ROI analysis.
For designers and developers, there are several lessons about how
to conduct a credible ROI study with an e-learning program. First,
showing the cost versus benefit of an e-learning approach is no
different than any other delivery method. Second, it is often
assumed that converting facilitator-led programs to e-learning
platforms are more cost-effective and practical. To ensure true
cost savings and utility, the ROI of the facilitator-led program
should be compared with the ROI of the e-learning version to
determine its value. Essentially, running the two programs parallel
may be the best way to gather this data. The good news is that cost
reductions are so significant that even if the benefits are not
always on the same level with the e-learning, it still may be a
much more positive ROI study with the e-learning approach.
Note: Portions of this article were adapted from
The Business Value of E-learning by Jack Phillips and
Holly Burkett in elearning (December 2007/January
2008).
Jack Phillips is chairman of the ROI Institute. The ROI
Methodology, developed by Phillips, is a comprehensive step-by-step
process used to assess, measure, and evaluate training, performance
improvement, human resources, quality, and technology. The ROI
Institute is a research, benchmarking, and consulting organization
that provides workshops, publications, and consulting services on
the ROI Methodology. It is the only organization that
supports, updates, refines, and distributes information and
provides services on this methodology; roiinstitute.net.
ASTD Field Editor Holly Burkett, SPHR, CPT manages a consulting
practice in Davis, California, where she helps organizations
measure the business value of WLP efforts. A certified ROI
professional, she is frequent presenter, workshop leader, and
author. Most recently she authored the "Action Planning" chapter in
the ASTD Handbook for Measuring and Evaluating Training.
She holds a PhD in human capital development;
burketth@earthlink.net.
2010 ASTD, Alexandria, VA. All rights reserved.