Here's an overview of six red flags that will impact your ability
to conduct effective Kirkpatrick level 3 assessments.
Consider the following situation:
You recently designed, developed, and delivered a blended learning
workshop. The pre-work was online, and participants applied those
online concepts in a classroom-based workshop. The VP of Sales
catches you in the hall one day and asks you to do one of those
"Level 3 assessment things." This assessment will determine
whether his salespeople have applied the skills from the related
workshop and online materials. The VP of Sales also indicated that
it would be best if you, the course developer, send the email
requesting that salespeople participate in the assessment. The VP
concludes by stating: "and try not to bother the sales managers, I
have them working on a very important project."
If a similar situation has caused you to lose sleep, then you
understand some of the factors that will impede a Level 3
Examine the situation
First, let's examine the facts and the challenges regarding this
- VP of Sales is interested in a Kirkpatrick Level 3 assessment:
Great! At least they recognize the significance of linking training
to application of skills.
- The VP of Sales wants you to drive the assessment process. This
may not be effective, because you probably need a senior manager to
set expectations, answer questions, and enforce negative
consequences for non-participation.
- You are at a slow point in the sales cycle. Therefore,
salespeople may not have the opportunity to demonstrate the skills
- You have to "bother" the sales managers. In fact, you will
request that they take part in an evaluation training session, and
that they evaluate the performance of salespeople.
- You developed the course for a level one and level two
assessment. The outcomes were:
- Level 1 assessment did not indicate any significant course
content or delivery problems
- Level 2 assessment effectively examined learner ability to
propose solutions given realistic sales scenarios.
The good news: Thanks to solid instructional design and excellent
input from your subject matter experts, the existing learning
objectives can be used for a Level 3 assessment, even if you did
not originally plan to conduct one.
Next, let's learn more about the workshop:
- Performance gap: Via direct observation, 70 percent of existing
salespeople do not currently possess minimum skills regarding the
assessment of customer needs, development of a sales solutions, or
the ability to articulate sales solutions.
- Business problem: Some 15 percent of lost sales are due to the
salesperson's inability to effectively articulate an effective
solution to address customer needs. A lost sale usually means the
loss of a customer for at least 24 months.
- Terminal objectives: Each salesperson will have the ability to
- assess customer needs
- propose a feasible solution to address customer needs
- effectively articulate the solution to the customer.
- Delivery method: The delivery method uses a blended approach.
There are three online product basics courses with pre-reading
supplied via intranet links. This is followed by a one-day
- Assessments: The completed assessments include an end-of-course
evaluation (Level 1), as well as pre/post assessment (Level 2).
Because we have a good grasp of the situation, let's review
Kirkpatrick's four levels of assessment.
- Level 1 - Reaction and planned action, which
measures learner reaction and outlines specific plans for
- Level 2 - Learning Measures, which examine skills
knowledge or attitude changes.
- Level 3 - Application and Implementation Measures,
which examine changes in learner behavior on the job and specific
application and implementation.
- Level 4 - Business Impact Measures, which examine
business impact in relation to the program.
To perform an effective Level 3 assessment, it's necessary to
consider how Level 1, Level 2, and Level 3 data are linked
together. In addition, you can effectively set expectations by
getting the necessary stakeholder, observer, and learner support in
the beginning of the project. Most important, always remain focused
on the business problem that you're trying to address.
Link learning to the business problem
First, there should be a clear link between the business problem
and the learning objectives. For example: Subject matter experts
indicate that salespeople lacking solution selling skills are not
able to effectively both fit products to customer needs and close
sales. The specific selling skills required are tied directly to
the course learning objectives.
Second, the course and learning objectives should be designed with
the Level 3 assessment in mind. This means that you have stated
your learning outcomes in observable measurable terms. In our
situation, all three of the terminal objectives can be broken into
specific measurable, observable behaviors. This will help you when
you attempt to collect and analyze Level 3 performance data. That
said, it's necessary to create a plan for capturing and analyzing
performance data. To accomplish this task, I strongly suggest a
combination of data collection methods.
Data collection can be through a combination of direct and indirect
observations. Indirect observation is often in the form of data
that results from the desired behavior or performance. For example,
you have access to sales reports that indicate "closed sales" and
"lost sales." You also have access to sales post mortem reports
that indicate if skills were applied or not applied and how they
affected the sale.
It is also possible to use direct observation, which involves
witnessing the learner's behavior in an actual setting. For
example, the sales managers will perform "ride-alongs" with their
salespeople, and they will be responsible for observing and rating
the salespersons performance during customer sales calls. Sales
managers were prepared for this via a brief training session
involving evaluation practice and proper techniques for providing
The next challenges pertain to opportunities to demonstrate the
newly learned sales skills. If you are actually in a slow sales
period, it may be necessary to move the Level 3 assessment to the
busier sales months. However, if too much time passes between
training and application of skills, you may need to conduct a
Finally, organizations get the behaviors that they reward, not
necessarily the behaviors they want. This means that if salespeople
are not provided with the proper rewards and consequences, they may
not choose to apply the desired sales techniques. As a training
professional you understand that this is outside of your domain.
However, as a performance consultant, you can communicate the
linkages between behaviors, rewards, and consequences to key
The following table provides six red flags that will impact your
ability to conduct a Level 3 assessment.
Undefined problem statement/no linkage to a business problem.
Always make certain that there is an arguable linkage to a business
problem that is constantly reviewed during project milestones.
Unclear performance gap
Make certain that you understand, document, and communicate both
the current learner capabilities and the desired levels of
performance upon completion of the learning materials.
Unclear, vague or non-existent: Program goals, performance
outcomes, learning objectives or a lack of stakeholder "buy in"
Work with key stakeholders to define program goals, and have key
stakeholders and the SME help to define the desired performance
outcomes. In addition, communicate to key stakeholders the
consequences of not having measurable solid learning objectives.
Finally, if the stakeholder lacks buy-in or is not interested, then
you shouldn't be interested either. They probably really want a
Lack of motivation from the: Subject Matter Expert, observer or
Document problems; discuss them with the SME, observer, or learner.
If necessary inform your key stakeholder and request that they
assist you with resolving the issue.
Few opportunities to demonstrate skill, behavior or knowledge
Try to identify this before the training occurs and attempt to time
the training no more than 1 month before the application. Refresher
courses and quick reference materials may be required.
No consequences (+/-) for behavior/non-behavior
To be successful with any behavioral change you must have
meaningful consequences (+/-) linked to the behavior demonstrated.
Without consequences, behavior will not change.
If you face a similar situation, you now understand typical factors
that can impact your ability to conduct effective Level 3
assessments. The key to your success: Do not be afraid to
communicate your needs and requirements to key stakeholders. This
requires you to put on your consultant hat and prepare to discuss
the trade offs and consequences if your Level 3 assessment
requirements are not met.