For most people, DOA refers to Dead on Arrival. For learning
professionals, DOA means "Demanded by Older Authorities." ROI for
training is an "old technology" used by older leaders or maybe just
old "thinking" leaders. It is an outdated process.
Outdated, however, doesn't mean useless. Kerosene lamps are
outdated, but they are far from useless. In fact, in certain
situations, they may be preferred over a flashlight when the power
goes out. While few of us may prefer a kerosene lamp in most
situations, we definitely prefer return on expectation (ROE) in
most training situations.
If a customer has certain expectations and those expectations are
met, then he would deem the training event successful. This basic
idea is the framework for the concept of ROE. If a training
designer has a customer with certain expectations regarding the
training he would like delivered, the best way to assure success is
to build the training to meet the customer's expectations.
But what about ROI?
ROI has been the gold standard with regard to measuring the
effectiveness of a training event and its effect on a business's
bottom line. There are still many large, extremely traditional
corporations that to this day expect an ROI assessment on training.
While it's clear that there are challenges with ROI from the
measurement side, there are also challenges from the implementation
and timing side. That is why we refer to ROI as being DOA.
Some factors have led us to this belief. In a 2009 study conducted
by the ROI Institute, it was discovered that:
- Out of 96 company CEOs surveyed, only 4 percent of those
companies measured ROI
- 58 percent of training managers are not required to report on
the effectiveness of their work
- 69 percent are not required to report on productivity
In an August 2010 T+D magazine article title "ROE's Rising
Star," James and Wendy Kirkpatrick listed some of the differences
between ROI and ROE. They wrote that ROI is a complex, rigid,
expensive, and defensive, isolationist approach.
We agree with the Kirkpatricks' assessment, and add a few of our
own observations to the list: ROI is reactive, is hard to prove,
and takes time and the right metrics to be established to return
any value at all with training.
But ROI is demanded by C-level officials
Actually, ROI is not demanded by C-level officials. In a 2006
ASTD-IBM study ("C-Level Perceptions of the Strategic Value of
Learning") of executives in 26 organizations spanning 11
industries, there were two very interesting findings:
- CXOs are less concerned with quantitative metrics that show
learning's value contribution to business outcomes.
- Perception of stakeholders are a key indicator of learning's
value.
Researchers also found that word-of-mouth support of a particular
training event was the most common measure of training
effectiveness that trainers felt was most important for their
customers.
What do these findings mean for ROE? Since ROE measures the success
of atraining event based on the expectations of the customer and
the customer scores the event using a results contract (as we
suggest), then ROE becomes the perception of the customer. It is
well known that perception, for many folks, is the truth.
A closer look at ROE
While we are biased, we firmly believe that ROE is the best
measuring tool available to discover if a training event has been
effective. The Kirkpatricks agree. In the T+D article,
they list some highlights of ROE:
- It's a proactive, business partnership approach that unifies
teams.
- It defines training as a contributor to key business results.
- Its value is defined by business stakeholders in cooperation
with training.
- It focuses on comprehensive evidence and a compelling story of
value.
- It is easy to understand, flexible, and cost-effective.
Even Jack Phillips, the founder of the ROI Institute, says that ROI
should only be used nominally. In a 2003 Chief Learning
Officer article titled "ROI Best Practices," Phillips states
that ROI should be used minimally, evaluating no more than 5 to 10
percent of training programs.
The results contract
The best way to collect quantitative and qualitative data that will
give you an ROE assessment is through the use of what we call a
"Results Contract (RC)."
The RC systematically walks one through the process of identifying
what is important to the customer as well as giving the customer a
way to score his overall satisfaction. There are two different RCs
(RC and RCf Results Contract with Formula) that can be used for
measurement. Let's look first at the RC.
Figure one is an example of the basic contract that provides the
customer with a score based on responses to general questions,
regarding the design and delivery of the course. The demographics
section collects the key customer data. Additionally, it shows
linkage to the company's strategy. For some organizations, there
may not be an obvious linkage between the company's strategy and
the particular training being requested. If this is the case with
for your organization, then leave this section blank.
The middle section of the contract is essential, and contains a
balanced scorecard link, the business goal that the training is
linked to, the course performance objectives, and the overall
results achieved.
It is possible that your organization does not use a balanced
scorecard. If this is the case, this section may be linked to the
company's strategies, goals, or objectives. If the training event
being requested cannot be linked in some way to a business goal,
then you must ask "Why is the training being requested?"
Out of the six sections on the first page of the RC contract, the
performance objectives section is the most important. It is here
where trainers prove their worth. This section lists the
performance objectives identified by the trainer and her customer
regarding the training initiative. Each of these objectives must
meet the criteria for performance objectives; they must have, at a
minimum (and for the sake of space restrictions in the RC itself),
the criteria and the standard listed. Without these, performance
objectives become merely objectives, and objectives are very hard
to measure.
The final section of the first page is customer commitments. It is
here that we hope to alleviate the problem that occurs frequently
with training: under-attended training or training where key levels
of management or supervision are under-represented when the class
is held.
The information in this section typically contains statements
around the challenges faced by the training organization in getting
support or buy-in for the training initiative. Examples we've
placed in this section include:
- Management will ensure that all required personnel attend this
training.
- Management will not keep invited employees from attending this
training.
- Attendees will actively participate in this training.
- Attendees will arrive on time and leave on time.
We both love this section because sometimes it is the only time
that a trainer will be able to discuss these kinds of issues with
management. It's also great because the customer signs this
document, thus taking responsibility to assure the course is
properly attended.
The second page of the RC is fairly short with only two sections,
but is very important. The first section is the project evaluation.
This section gives the trainer a good idea of how the customer
views the entire training process, from design through delivery. It
also includes a section for the participants. This is typically
filled out by the customer or attendees as course feedback.
The service guarantee is where commitment from the customer is
gained through their signature. Their signature virtually assures a
successful ROE measurement because the customer has defined the
performance objectives that will be measured with you and has
agreed to make sure that the appropriate people show up for, and
participate in, the training event.
Quantifying the training value (Tv)
While we truly believe in the power of ROE with the above listed
RC, we also realize that for some individuals, facts and hard
numbers are essential and very comforting. It is for this reason
that we devised a second RC (the RCf - with formula) that
incorporates the ability to quantify the training outcomes using a
weighted scale. With the RCf, a weighted score is established by
the customer, focusing more of the trainer's time or energy on the
more important performance objectives. These may be 5s or 3s. If
you have any 1s it is good to discuss the real value of that
performance objective and how important it is to deliver.
Now that you have additional data collected (allowing us to provide
a quantifiable value to each performance objective) we can now use
the following formula:
The lowest possible overall training value, we believe, that should
occur with the training is no less than 80 percent.
Conclusion
Based on the data from many CEOs and the difficulty of establishing
an ROI, we believe ROE should be the preferred approach for
evaluating training. It is a collaborative, proactive, and
customer-oriented way of ensuring the training delivered and the
training expected is in synch with your customer.
-------------------------------------------------------------------------------------------------------------------------------------------------------------
Scott Knutson, MBA, CPLP, is a senior manager
of training operations for Elan Drug Technologies; scott.knutson@elan.com .
Steven A. Jacobs, MBA, RPh, is president of
Global BioPharm Solutions; steven_jacobs@comcast.net .