Many education leaders have implemented terrific learning events
and completed Level 1 and Level 2 evaluations. However, very few
have truly obtained Level 3 and Level 4 results. This Level 2
plateau keeps many educators from "wearing the crown" of business
partner in their organization. Are you stuck at level 2? Do you
feel like you are kissing frogs but never getting the prize? That
is exactly how I felt.
Working hard at kissing frogs
I had evidence that people attended and enjoyed our classes (Level
1) and had obtained knowledge (Level 2); however, I did not have
evidence that I was truly obtaining outcomes (Levels 3 and 4). I
work in the healthcare field and in medicine, we base our decisions
on evidence and outcomes. To survive in this industry, I needed
evidence-based learning outcomes. Without this evidence, I was
working hard but I was not perceived as a true business partner. In
essence, I was working hard at kissing frogs without obtaining the
crown
I knew about the Kirkpatrick 4 Levels training evaluation model and
had effectively been completing Levels 1 and 2. But how was I
really going to get to level 4? I was on a mission.
Knowledge: how to obtain the crown
I reviewed the results of Brent Peterson's 2004 learning
effectiveness study conducted at the University of Phoenix, which
compared the amount of time spent developing training against the
activities that truly contributes to learning effectiveness. The
typical organization invests 85percent of its resources in the
training event, yet those events only contribute 24 percent to the
learning effectiveness of the participants. The activities with the
largest impact to learning effectiveness were follow-up activities
that occurred after the training event.
The typical learning investment pretty much mirrored our
organization. I was providing some pre-event information to prepare
my end-users, and I felt we were providing excellent events but
doing almost nothing to provide follow-up after the event.
This next chart showing the results of the ASTD 2006 State of
the Industry Report on the causes of training failure is also
quite interesting.
According to the evaluation data below, only 20 percent of failure
can be attributed to preparedness and 10 percent to the event
itself. That leaves 70 percent to activities that follow the
"event." This really coincided with the University of Phoenix
study. Again, I felt I was doing a pretty good job of preparing
end-users before they attended an event, and I felt our events were
really quite stellar. So, that means I needed to concentrate on the
"after event" activities to obtain outcomes.
My epiphany came while attending a Kirkpatrick Business Partnership
Certification Program and learning about the Kirkpatrick Business
Partnership Model (www.kirkpatrickpartners.com).
The model emphasizes not only the four levels of evaluation but
also beginning with the end in mind. I needed to understand what my
stakeholders expected so I would know how to target what we were
teaching and how to measure to know if I was meeting expectations.
Only then would my kissing frog efforts gain me a crown.
Performance: frog to crown
With my next big assignment, I was determined to begin with the end
in mind. (I had been asked to provide the learning event for a
system and hardware implementation to ensure safe medication
administration.) I sat down with my stakeholders and asked them
what their expectations were. Their response was, "We need you to
train x number of users on x program by x date." I rephrased my
question. "Why are we doing this?" A nurse in my stakeholder group
spoke up and said, "because we never want an incorrect dosage of
medication to be given to a patient."
"Ah, you want your clinicians to utilize this software to reduce
error rate," I said. I had found the frog and continued our dialog
to negotiate what success would look like. We agreed that 90
percent compliance in utilizing the software would be our early
indicator that our program was successful.
I was off and running. I understood that the nurses not only needed
to attend our class offering and understand what was being taught
but also implement this knowledge on the job. I could measure
implementation by monitoring how many of the nurses actually
utilized the new software. I would use company statistics gathered
over the next year to determine the ultimate impact of increased
patient safety.
We worked with our supervisors to prepare their staff for training
and to ensure their attendance. Next, we provided a great learning
event that received excellent marks and all users passed their
assessments at 100 percent or received remediation until they
could. To help with retention, our instructors spent four hours in
the classroom a day and four hours on the unit providing
after-the-event coaching. This proved to be a very valuable review
and remediation opportunity. By streamlining our classroom content
to include only critical information, we were able to free up
instructor time for this valuable step.
Two weeks after the go-live, we measured the performance of
critical behaviors to ensure that staff was really using their
newfound knowledge. To our dismay, only about 60 percent were in
compliance, which is way below our goal of 90 percent. We talked
with learners to figure out what went wrong. They could perform the
task, but they just weren't doing it. We were still kissing frogs.
At least we knew we had failed, which gave us an opportunity to
adjust. In the past, we would have stopped at Level 2 and would
never have known if our efforts had made an impact.
Kiss that frog one more time!
With further investigation, we realized we had failed to properly
partner with the supervisors. They could not tell who was using the
new system except by visual confirmation, which was very time
consuming. We met with the supervisors and created a report to show
early adopters and stragglers. Once supervisors were armed with the
tracking "rate of adoption" tool, they could hold their staff
accountable.
When we ran the report again, all but two units were above the 90th
percentile. The supervisors were now part of the team and
contributed to the success of the training effort. At that point we
had truly obtained buy-in and trust. Since then, supervisors have
requested our assistance with other projects - we were no longer
kissing frogs because we had obtained the crown.
Wearing the crown
As with any initiative, there were many factors that contributed to
the overall success of the project. Learning can never take sole
ownership of success; it is simply a partner in the success.
Using the Kirkpatrick Business Partnership Model truly helped us
move beyond kissing frogs to the crown of evidence-based learning
outcomes. This experience has raised the bar for learning in our
organization. Our leaders now expect evidence on all four levels
for every major learning initiative.
Are you ready to stop kissing frogs and start partnering for a
crown?
-------------------------------------------------------------------------------------------------------------------------------------------------------------
References
Kirkpatrick, James D & Kayser-Kirkpatrick, Wendy (2010).
"Training on Triall: how workplace learning must reinvent itself to
remain relevant." New York, NY; AMACOM
Kirkpatrick, Donald L. (1959). "Techniques for Evaluating Training
Programs." Journal of the American Society of Training Directors.
American Society for Training and Development (ASTD) (2006). State
of the Industry Report 2006. Alexandria, VA: ASTD publishing.
Peterson, Brent. (2004). Unpublished paper from Apollo Consulting
Group. University of Phoenix.
-------------------------------------------------------------------------------------------------------------------------------------------------------------
Linda Hainlen is manager of Clarian Learning
Solutions; lhainlen@clarian.org .