At ASTD 2010 in Chicago, three of the biggest names in measurement
and evaluation - Jack Phillips of the ROI Institute, Jim
Kirkpatrick of Kirkpatrick Partners, and Robert Brinkerhoff of
Western Michigan University - joined together to discuss what
learning professionals need to do to be respected, appreciated, and
valued as business partners. The panel impressed upon the group the
importance and urgency of maximizing business value. This is an
excerpt from that panel discussion.
Brinkerhoff: We have an opportunity to shine and
reframe and transform the way our organizations view and value the
learning and development that we do. A couple of years ago, I did
a program evaluation of a very expensive leadership development
with a Fortune 50 company. They put 50 leaders through a very
extensive multiple-stage program where they visited a lot of places
and went to the London School of Economics. It cost about
$25,000-$30,000 per participant. While I had some very good news
for the client, I had some bad news too. The very good news was
that this training had led to some really outstanding and profound
business impact. In fact, in one instance we had documented that
the training had led to one of their employees closing a $500
million sale and opening an entirely new market for the company,
and that was directly related to this program. The bad news was
that only six of the 50 who had participated in the program used
anything they learned to change behavior on the job.
So I asked all of those who I interviewed a common question: "Now
that you are back from participating, what would happen to you if
you never used anything from this leadership program in your
position? The answer was "nothing." What I concluded from that is
that there was an attitude among these participants that learning
and development was a rite of passage as you are progressing to
higher levels. They loved the experience and they said they learned
from it, but none of them had an expectation or accountability to
use what they learned. So I figured out from that had even six more
used the training half as well as the six who did, it would have
accrued another $25 million in business value for their
organization. I concluded that training is largely viewed as a
staff benefit, not necessarily a driver of business performance. A
staff benefit is overhead - a cost, so it is one of the first
things to be cut when times are tough.
Our challenge is, how do we begin to change the culture of
expectations in our organizations where instead of being held
accountable for participating in learning and development
activities, people will be held accountable for applying and using
and changing their job performance?
It is up to us as a profession to help our internal customers see
that they should demand more from training, they should expect that
it will make a difference in the business, and they need to
understand that the only thing that produces business value from
learning and development is application on the job to change
behavior. They should expect that. They should hold their people
accountable for that. How do we change the culture? How do we raise
the expectations of our customers? We do that through measurement
and evaluation. We do that by demonstrating the difference that
training can make when it works. We should show our executives that
training is paying off, and if we do this aggressively, we will
begin to change the culture.
Phillips: We have made progress in our track
record but your track record shows very vividly when we have a
recession. In reality, the time to add resources is in tough
economic times. We should be spending more money during a recession
on training and development because there are fewer people and they
have more things to do. They need more skills to be the best they
can be, but executives don't see it that way. They don't value it.
We need to work on the value.
What is the value? We conducted a very important survey last year
with top executives. I understand exactly what they see as value
for investment in learning and development. The number one measure
that they want to see is connection to business - business impact,
yet that is one of the least important ones we give them. We've got
to deliver that, but to do that we have to change our approach.
What business do you think we are in? If we had to put what we do
into one of four boxes, which one would we choose:
- Entertainment business
- Learning and development business
- Change behavior business
- Business driver
Obviously, we are in all four boxes, but if we have to push one as
the number 1 box, what would it be? In a survey with CEOs, in terms
of outcome measures the number one measure we report to them is our
reaction measure. More than 50 percent said they have reaction
data, but only 22 percent said they really want it. When it comes
to business impact (application, behavior, change), only 11 percent
said they have it now, but 61 percent said they want to have that
data. Only 8 percent said they have business impact data, but 96
percent say they want it. When it comes to ROI, 4 percent said they
have it, 74 percent say they want it.
As you can see, what we are measuring is not what our executives
want, and what they want, we are not measuring so well. But it can
be done. We have to change our processes.
Kirkpatrick: Two weeks ago I was working in
Vancouver for a certification program and one of the senior
training managers invited a senior business partners to sit in and
answer some questions from the training professionals who were
attending the workshop. They asked him what he wanted to see. "We
have smile sheets, models, and pre- and post-test data." He said,
"You need to show me the results and the evidence that these
results have come from training. If you can do that, I will give
you all the funding that you need. You need to connect the dots
about what people learn, what they do, and its impact on the
business."
The problems with the economy have not caused training problems and
cutbacks. It has exposed our model. The training model is not
working. When the economy is good and people are happy with their
results, we get a free pass. But in these circumstances, training
is now on trial and our training model is now on trial. All three
of us believe that belief in the power of the training event is not
working anymore and our senior executives are reaffirming that. We
certainly look at this as a crisis in training, but it is an
opportunity in training that we have not had. The enemy of great is
good. We have been good for a long time, but not good enough. As an
industry, we are no longer good enough until we start to
demonstrate the value in terms that are meaningful to our
stakeholders - not what we believe is meaningful but what is
meaningful in the eyes of the stakeholders.
We need to create value in our programs and demonstrate that value.
We all have jury of groups or individuals who are sitting there
judging us of whether we are guilty of our costs exceeding our
value.
We have a chance to rewrite our legacy and a lot of it has to do
with busting the old training model and reassigning our resources
for those mission-critical programs.
Q. What do you expect to happen if we maintain the status
quo?
Kirkpatrick: We are at the tipping point. The
reason why we are all here today is because there is a sense of
urgency. If we maintain the status quo, training is going to be
outsourced. Our industry will be in jeopardy. Training will not
exist as we know it today.
Phillips: I think we could lose our influence. We
have already lost a lot of it. We had a great opportunity, and we
missed that. Obviously, we will lose resources, support, and
respect if we don't change our processWe are going to continue to
lose some of the influence that we lost during the recession.
Q. Is it really possible to connect learning and
development co-brands to business measures with
some reasonable amount of resources?
Brinkerhoff: I think it is more than possible, it
is absolutely mandatory. What we have to be careful to do is think
about linkage to the business as a nominal linkage. That is, we
have to improve customer service as a business goal so that we have
a program that improves customer service or we need to reduce costs
so we create a program that helps employees reduce costs. That's
nominal linkage. What Jim and Jack are talking about is what I
refer to as deep business linkage and that is who needs to change
behavior in order to drive business results. We need to be asking,
"Who are the key roles that need to change their behavior, what
behaviors do they need to change, what better results can we
expect, and how are those tied to the business goals?" That deep
business linkage is simply a logical analysis. It is very
inexpensive to produce. It just means you need to ask those
questions of your internal customers and document them. Once you've
done that, you've solved 90 percent of your measurement problems
because you have identified behaviors that need to change and they
are the easiest things to track because you have people tracking
those behaviors on a regular basis.
Phillips: I think there is really three points we
have to link to business: business measure, business need, and
business issue. We need to make sure that our programs or solutions
connect to that business measure. We need to do that on the
front-end analysis that describes clearly what you want to
accomplish. The second time is during the program. We need to keep
everyone focused on the business measure throughout the process. We
do that best by having impact objectives - that gives everyone
involved from the participants to the facilitator, to the manager
of the participants the understanding that we are driving them to
the business measure. Then on the follow-up, we track the success
of that business measure. We have to sort out how much we have
actually caused that success. That's the isolation issue - we must
show how much we contribute so we can see the connection we made.
This validates the business provider. That's the third part.
This is not very expensive. What you have to think about is how
much do we spend on it now. Think about your learning and
development budget. How much do you spend on measurement and
evaluation? A lot of you would probably say zero, and that's not
good. You could have a best practice, comprehensive process where
you measure 100 percent of your programs at Level 1, 60 to 80
percent at Level 2, maybe 30 percent at Level 3, 10 percent at
Level 4, and 5 percent at Level 5. You can have that type of
profile and spend about 3 to 5 percent of your total budget. Are
you willing to spend 3 percent to see how your 97 percent is
operating? I think we have to invest more but not that much.
Brinkerhoff: I would like to add to what Jack has
said. There are real costs to not doing it, and Jim talked about
how they are analyzing Level 3. Well, the nominal question in Level
3 is "Who is using it?" but the really critical questions are "When
they are using it, why are they using it and what is helping them
use it?, and "When they are not using it, what is keeping them from
using it?" I remember a dialogue recently where I was trying to
sell a client on doing some more evaluation for the program. I had
already figured out that when only one of these purchasing
executives used their training, it dropped $250,000 to the bottom
line in one quarter. We were going to do the evaluation for $40,000
and I said, "All I have to do is get one more person to use their
training one-fifth as well as the four people who are already doing
it and we pay for this evaluation three times over.
Just like you should expect an ROI from learning and development,
you should expect an ROI from your evaluation. Your evaluation
should pay off in information that will help your client get more
from their learning and development investment.
Kirkpatrick: One thing that I would like to add is
that what helps to save money is identifying what success will look
like to the client. One executive recently said to me, "We'll know
it when we see it." The training department was forced to throw
everything but the kitchen sink at their workforce in hopes that
the right needles would move. The much more cost-effective way is
to know what your target goals are. Negotiate with your
stakeholders - don't just ask them because typically they don't
know - what success will look like. What are the needles they want
to see moved at the end of the initiative? That will help you
tighten your budget by targeting ways to move those needles.
Q. What is the first step that learning professionals need
to take to become a strategic business
partner?
Kirkpatrick: Find one executive that gets it and
is willing to be your advocate, then you can do an ROE or ROI study
with them and then showcase the results. That will create some
momentum for others to follow.
Brinkerhoff: I would agree completely with Jim.
Choose one internal executive who you can forge a relationship with
and then do everything you can to make sure you help them succeed
with an issue that they are facing. You also need to realize that
you are not a supplicate begging them to help you do your training.
You are a partner with them, saying "Let me talk to you about what
I can do to help you solve your problems." Don't ask your partners
help you do training; ask them what you can do to help them achieve
their business results. That is the first step in being a business
partner.
Phillips: The situation is difficult sometimes
because you are trying to partner with someone who really doesn't
want to partner with you. One thing you have to do is make them
understand that you know the business quite well. The more you
know, the more you get their attention. You are ultimately showing
them results for the business partner - which for them are their
KPI (key performance indicators). If you have helped them, you have
become a friend, a colleague, and a contributor.