There are numerous metrics that serve specific purposes. Some of
the most critical include performance metrics, operations metrics,
financial metrics, and cultural metrics. As the names indicate,
performance metrics deal with the actual performance of learning
programs relative to a set of criteria, operations metrics describe
what is going on in the organization, financial metrics catalogue
the investments made, and cultural metrics tell the story about the
overall organization.
How to Collect This Data
Once you understand these four categories of metrics, you have
several possible methods to collect data, including surveys,
interviews, focus groups, and empirical research. Depending on your
particular situation, each method has its advantages and
disadvantages.
Surveys are easy to scale. Items on the survey should conform to
the metrics of interest, and not just collect data in a vacuum.
Instead, the survey should collect what is needed. Another benefit
of surveys is that they can be automated, freeing up time for
analysis based on the data.
Focus groups are valuable sources of information as they allow you
to dig deeper with follow-up questions. The most important thing to
remember when using focus groups is structure; you should maintain
a list of questions that conform with your metrics and follow it.
The drawback to focus groups is that they are time intensive and
are not as easy to scale.
Empirical research can serve as a goldmine of information. While
this method can be time intensive, the results often are highly
credible. Some examples of empirical research include control-group
studies or statistically linking training to quantifiable
organizational outcomes. You may need to engage an outside
consultant to help with an empirical study, but this can be worth
it for costly, visible, and/or strategic programs where credibility
and precision in the resulting metrics are paramount.
Interpreting the Data
Once you collect the data, you must properly interpret it,
aggregate it, and populate the desired metrics.
Aggregation refers to the level you choose to examine the data. For
example, if you want to look at a level 1 metric, instructor
performance, you can look at the data at the class level: What was
the performance for this particular class? You also can look at
this from the instructor level: What is this instructor's
performance overall across classes? Having a clear idea of what
levels of aggregation may be needed for various metrics before data
collection begins is important.
Aggregation can be thought of as filtering the data. You can only
filter, or aggregate, based on certain criteria that you collect
with the rest of the data. Knowing which metrics need to be
aggregated and presented at what levels ahead of time can save you
headaches down the road.
Frame of Reference
Even with the best metrics and data, they should not be interpreted
in a vacuum. It is essential to establish context for quantitative
metrics.
Benchmarking against an external data set from similar learning
programs can put metrics in perspective. This information can be
crucial to maintaining a human capital edge in a competitive
industry. Internal benchmarks also can provide context for metrics.
Finally, the combination of internal and external benchmarks can
help set goals. Goals themselves can be an important context for
evaluating the performance and effectiveness of the learning
organization, as well as a process check when initiating
improvement plans.
As you can see, it is crucial to have a strategy in place to
effectively leverage learning metrics. If you use the information
presented here as a guide to setting your strategy, you will be
much better positioned to adapt and scale to meet the needs of your
organization and its stakeholders.