Level 3 Assessments: Six Red Flags

Here’s an overview of six red flags that will impact your ability to conduct effective Kirkpatrick level 3 assessments.

Consider the following situation:

You recently designed, developed, and delivered a blended learning workshop. The pre-work was online, and participants applied those online concepts in a classroom-based workshop. The VP of Sales catches you in the hall one day and asks you to do one of those “Level 3 assessment things.” This assessment will determine whether his salespeople have applied the skills from the related workshop and online materials. The VP of Sales also indicated that it would be best if you – the course developer – send the email requesting that salespeople participate in the assessment. The VP concludes by stating: “…and try not to bother the sales managers, I have them working on a very important project.”

If a similar situation has caused you to lose sleep, then you understand some of the factors that will impede a Level 3 assessment.

Examine the situation

First, let’s examine the facts and the challenges regarding this situation:

• VP of Sales is interested in a Kirkpatrick Level 3 assessment: Great! At least they recognize the significance of linking training to application of skills.
• The VP of Sales wants you to drive the assessment process. This may not be effective, because you probably need a senior manager to set expectations, answer questions, and enforce negative consequences for non-participation.
• You are at a slow point in the sales cycle. Therefore, salespeople may not have the opportunity to demonstrate the skills gained.
• You have to “bother” the sales managers. In fact, you will request that they take part in an evaluation training session, and that they evaluate the performance of salespeople.
• You developed the course for a level one and level two assessment. The outcomes were 
  • Level 1 assessment did not indicate any significant course content or delivery problems
  • Level 2 assessment effectively examined learner ability to propose solutions given realistic sales scenarios.

The good news: Thanks to solid instructional design and excellent input from your subject matter experts, the existing learning objectives can be used for a Level 3 assessment—even if you did not originally plan to conduct one.

Next, let’s learn more about the workshop.

• Performance gap: Via direct observation, 70 percent of existing salespeople do not currently possess minimum skills regarding the assessment of customer needs, development of a sales solutions, or the ability to articulate sales solutions.
• Business problem: Some 15 percent of lost sales are due to the salesperson’s inability to effectively articulate an effective solution to address customer needs. A lost sale usually means the loss of a customer for at least 24 months.
• Terminal objectives: Each salesperson will have the ability to
  • assess customer needs
  • propose a feasible solution to address customer needs
  • effectively articulate the solution to the customer.
• Delivery method:  The delivery method uses a blended approach. There are three online product basics courses with pre-reading supplied via intranet links. This is followed by a one-day classroom-based workshop.
• Assessments: The completed assessments include an end-of-course evaluation (Level 1), as well as pre/post assessment (Level 2).

Because we have a good grasp of the situation, let’s review Kirkpatrick’s four levels of assessment.

• Level 1 – Reaction and planned action, which measures learner reaction and outlines specific plans for implementation.
• Level 2 – Learning Measures, which examine skills knowledge or attitude changes.
• Level 3 – Application and Implementation Measures, which examine changes in learner behavior on the job and specific application and implementation.
• Level 4 – Business Impact Measures, which examine business impact in relation to the program.

To perform an effective Level 3 assessment, it’s necessary to consider how Level 1, Level 2, and Level 3 data are linked together. In addition, you can effectively set expectations by getting the necessary stakeholder, observer, and learner support in the beginning of the project.  Most important, always remain focused on the business problem that you’re trying to address.

Link learning to the business problem

First, there should be a clear link between the business problem and the learning objectives. For example: Subject matter experts indicate that salespeople lacking solution selling skills are not able to effectively both fit products to customer needs and close sales. The specific selling skills required are tied directly to the course learning objectives.

Second, the course and learning objectives should be designed with the Level 3 assessment in mind. This means that you have stated your learning outcomes in observable measurable terms. In our situation, all three of the terminal objectives can be broken into specific measurable, observable behaviors. This will help you when you attempt to collect and analyze Level 3 performance data. That said, it’s necessary to create a plan for capturing and analyzing performance data. To accomplish this task, I strongly suggest a combination of data collection methods.

Data collection can be through a combination of direct and indirect observations. Indirect observation is often in the form of data that results from the desired behavior or performance. For example, you have access to sales reports that indicate “closed sales” and “lost sales.” You also have access to sales post mortem reports that indicate if skills were applied or not applied and how they affected the sale.

It is also possible to use direct observation, which involves witnessing the learner’s behavior in an actual setting. For example, the sales managers will perform “ride-alongs” with their salespeople, and they will be responsible for observing and rating the salespersons performance during customer sales calls. Sales managers were prepared for this via a brief training session involving evaluation practice and proper techniques for providing performance feedback.

The next challenges pertain to opportunities to demonstrate the newly learned sales skills. If you are actually in a slow sales period, it may be necessary to move the Level 3 assessment to the busier sales months. However, if too much time passes between training and application of skills, you may need to conduct a refresher course.

Finally, organizations get the behaviors that they reward—not necessarily the behaviors they want. This means that if salespeople are not provided with the proper rewards and consequences, they may not choose to apply the desired sales techniques. As a training professional you understand that this is outside of your domain. However, as a performance consultant, you can communicate the linkages between behaviors, rewards, and consequences to key stakeholders.

Red flags

The following table provides six red flags that will impact your ability to conduct a Level 3 assessment.

Results

If you face a similar situation, you now understand typical factors that can impact your ability to conduct effective Level 3 assessments. The key to your success: Do not be afraid to communicate your needs and requirements to key stakeholders.  This requires you to put on your consultant hat and prepare to discuss the trade offs and consequences if your Level 3 assessment requirements are not met.