As evidenced by this quote, George Bernard Shaw seems to have
grasped the importance of measurement long before the workplace
learning field did.
There are no learning experts out there who want us to think that
the measurement itself is the key to success in workplace learning.
But analyst and author Josh Bersin says it's time to discard the
old models and ideas and take a more systematic - even, gasp,
businesslike - approach to measurement. He outlines his beliefs in
a new book, The Training Measurement Book: Best Practices,
Proven Methodologies, and Practical Approaches (Pfeiffer),
which was released this summer. (See a short review in this issue.)
"When I started work as an analyst, one of the first things I ran
into were people asking how to measure training and how to measure
e-learning," he explains. "I began to attend various conferences,
and I read Donald Kirkpatrick's book. I was astounded as to how
limited the approaches were and how frustrated people were about
the concept of Level 3 (behavior) and Level 4 (results)."
According to Bersin, Kirkpatrick offers a model, not real
direction. "I realized that if you ignore Kirkpatrick and you look
at training as a business process - like any other business process
- there are some very important pragmatic things you can do to
measure it," he says. "Companies are, in fact, doing a lot of
interesting things by measuring a lot of aspects of training,
including measuring impact. And they are measuring it in a lot of
interesting ways."
Bersin realizes that he is stepping on a lot of toes by saying so,
but that he believes that training is mostly a support role. "The
book is a business-driven approach to measuring training. It
accepts the fact that these days training is a support function,"
he explains. "So, a lot of measurements of a successful training
programs are based on how well they support the business behind it,
not the instructional designer or training manager's perception of
what the outcome should be."
And while he's at it, Bersin is also willing to dismiss those who
worship at the altar of return-on-investment. "A lot of the focus
on measurement that takes place in corporations, particularly among
the training groups, is focused on ROI and trying to cost-justify
the training investment," he notes. "What I found is that ROI
actually doesn't get you very far. The companies that seem to
measure ROI often do it because someone is unhappy with the
programs they are building. The projects often (produce) ROIs that,
in the end, are unbelievable."
There is nothing wrong with measuring ROI, he says. "I think it is
a good thing to do. But you can spend a lot less time on ROI and
more on operational measurements of training - what most people
call alignment - and get a lot better value out of your investment
in training measurement.
Bersin simply notes: "Measurement itself is expensive. The book
addresses the limitations of ROI and other approaches of measuring
business impact besides ROI."
Get real
If you really want to measure what your learning efforts are doing,
Bersin suggests looking at the entire process. "You have to
identify the problem, you have to design the program, you have to
engage managers and employees to get them to attend the program,
and then they have to finish it, apply it and have it reinforced.
And somewhere during that process you see some results."
"If you want to create an actionable measurement program, as
opposed to just a number, you want to look at all of those stages,"
he said. "It's like measuring any other business process: you take
all the steps and determine what the criteria are for success.
Most of that information is relatively easy to determine, notes
Bersin. "What the really good companies do is measure different
elements of this process. They take that information and use it in
a way that is a process, not a project - they capture the
information continuously, not only to you see a number but also to
see a trend. They use the measurement data to take very specific
action."
Adds Bersin: "I realized when I was talking to all of these
companies that the reason so many companies struggle with
measurement is because some guy or gal is appointed to be the
measurement person. So they decided to see what they can measure."
These people find areas where they are missing data, he says. "They
need to think about measurement as a part of reengineering the
training process itself."
In companies that don't have a well-aligned CLO, the training group
- usually made up of passionate and hard-working people - works
very hard at measurement. "Then they show it to the business unit
people who don't have much use for it," adds Bersin. "They find
themselves wrapped around the axle on measurement when what they
really needed to do first was alignment. I find that companies that
did a good job with alignment measured less and got more out of
it."
In most companies there are usually two or three training programs
that are so important that if you dropped everything else you would
still be doing the right thing, Bersin suggests. "I found that
companies that had effective measurement programs could identify
those key offerings and measure them very carefully and keep them
aligned. I suggest in the book that organizations try to compute
the ROI before they develop a training program, not measure it
afterwards. It's not that difficult."