What do we mean by evaluation success? Chances are the definitions
will change over time, and from situation to situation. There are
some common definitions out there, however. Success with evaluation
does not mean that the training or learning activity being
evaluated was itself necessarily a success. One reason you may be
evaluating is precisely because the training was not a
success! In general there are 5 steps to evaluation success. They
1. Develop a goal statement and baseline.
2. Conduct a comprehensive stakeholder analysis.
3. Develop an impact model.
4. Select the right methods.
5. Implement with grace and present the results.
5. Build for the future; follow up.
Step 1: Develop a Goal Statement and Baseline
The starting point for any evaluation is the 'why'. Why intervene
in the first place? Knowing this will tell us plenty about what
needs to be evaluated.
For each project you tackle, you need to draft a goal
statement and a stakeholder analysis. Why have clarity about the
measures? Well, if you aren't clear about the measures,
specifically what to measure, how the measure is defined, and how
to measure, then how sure can you be that your study is actually
Where it is possible, a baseline measure of performance is taken
and included in the goal statement. A baseline is a key element in
any evaluation that sets out to compare performance before and
after an intervention. It is not always possible to develop a
baseline measure and when this happens, it should be noted in any
report. This does not prevent an assessment, but it will have less
certainty and credibility.
Step 2: Conduct a Comprehensive Stakeholder
One of the first steps toward training success is to identify the
stakeholders and find out their wants and needs. The same is true
of evaluation. Who is interested in the evaluation of the training?
Why? What do they want to know? What is their agenda?
If you have the answers to the above questions, you can undertake
an evaluation study to meet all these requirements, likely to be
lengthy and costly, or undertake a limited and targeted evaluation
study that addresses the needs of just one or two primary
stakeholders. But you still need to manage the major expectations
of the other stakeholders!
In finding out what the various stakeholders require, you will be
better placed later to select the most appropriate evaluation
method. This is critical if you are to ask and answer the right
questions, and to do so effectively and efficiently.
The stakeholder analysis also will tell you how the evaluation
study should be reported and how often and by which methods you
should keep each stakeholder informed of progress. This is often
known as a communications plan.
Step 3: Develop an Impact Model
An impact model is simply a description, a forecast, of what
performance will look like if the intervention is successful. It
does this in more detail and depth than the goal statement in Step
1 because it makes explicit the links between the desired outcome
and what actually has to be done.
The impact model provides guidance about what data may need to be
collected and also how, surveys, interviews, and so forth. This
detail also can help you select the appropriate evaluation
approach, which is the subject of Step 4.
Whether the intervention has already taken place, or (more
ideally), has yet to take place (and so evaluation can be woven in
to it), the impact model is a key part of the success of the
intervention and its evaluation. For interventions yet to be
developed or delivered, it helps identify the key objectives, in
particular the training or learning objectives. For interventions
already in place, it helps establish what should have happened
against what an evaluation can assess for effect and efficiency.
For the best effects the impact model should make a reasonably deep
investigation in to root causes behind the gap. If you aren't clear
about this, it is quite possible to measure some very clearly
defined measures as described in Step 1, but that have only a
passing influence on the indications of the problem you clarified
at the start.
Step 4: Select the Right Evaluation Methods
Different stakeholders may have different requirements. This will
have an impact on the evaluation methods you use. And there's no
reason why you can't use several methods together. For example, one
stakeholder may be the CFO who requires a return on investment.
Other stakeholders will certainly be the trainee and the trainee's
line manager. They are likely to be more interested in whether the
training is helping the trainee to perform better and be better
placed for career progression.
Step 5: Implement With Grace and Present the
Once you select the appropriate methods, you have to implement
them. On many occasions where evaluation studies have been less
than 100 percent effective, the methods employed have either not
been the right ones or they have been poorly implemented,
especially where other people are involved with data collection.
All methods require some data collection, and require some focus on
individual people. Forgetting the human factor can result in
approaches that are seen as annoying if not antagonistic.
There's little point in doing an evaluation if you aren't going to
present your findings, but what is the best way to do this? It
depends on the audience, the stakeholders identified in Step 2.
They will help you decide what to present and in what format.
You can present the evaluation findings in any of the following
- detailed reports
- executive summary
- general audience
- single page reports
And of course you can always deliver a stand-up slide presentation
and provide opportunities for questions to be taken and so build
greater understanding. In such cases, don't go overboard on the
amount of detail; keep it simple.
Step 5: Building for the Future
You may wonder why this isn't step 6. This last step is not
critical to the success of one specific evaluation study, but it is
useful to take if you are looking to conduct future training and
evaluation with this client or part of the organization.
Over time, more data will come to light about the longer term
success of any training you have evaluated. From this you can learn
to improve the evaluation approaches in the future as well as help
improve the impact of the training itself.
The trick is to develop and maintain a relationship with the
stakeholders once the initial evaluation study is completed. So, if
you can, stay in touch with the stakeholders. Use it to gather
information and to share the experiences of your other evaluation
studies to continue to add value to all your clients/stakeholders.
Follow the 5 steps (yes, even the last step!), to ensure
training contributes to better performance, and you build your