Keeping a results-based evaluation and ROI process on track is one
of biggest implementation challenges for evaluation professionals.
In many ways, it's equivalent to training for a marathon. You start
small with manageable goals, stretch your muscles, change your
habits, celebrate milestone events, and then wait for the training
and preparation to kick in so you can go the distance. Yet the
similarity ends there, since for us "going the distance" represents
just half of the journey. Staying on track and maintaining the
health and integrity of the ROI process in the face of continuous
change is equally critical and an aspect of results-based
evaluation work that is often overlooked and underestimated.
Stages of implementation
To best ensure that your evaluation solutions stand the test of
time, critical stages in ROI process implementation must be
addressed and anticipated.
Stage one: Recognition. An accountability "wake-up
call" is generated and preliminary evaluation actions are taken.
This stage is usually initiated by a single person or small group,
and at this point ROI is not necessarily viewed as a strategic
imperative by senior management. In this stage, you're likely to
- increased demand that the training and workplace learning and
performance improvement function show bottom-line results
- internal complaints or concerns about the value of training and
workplace learning services
- training and workplace learning expenditures being identified
- workplace learning programs have a history of more than one
- increased emphasis on the linkage of training and workplace
learning and performance improvement to strategic directions.
Stage two: Reservation. In this stage, there may
be renewed objections about the impact of new evaluation processes
on time and resources, and business demands will escalate
competition for like resources. The initial support and goodwill
from preliminary successes with the process have waned, and there's
a prevailing sense of impatience and concern that the investment
towards a results-based WLP focus is not worth the effort. It's not
uncommon for the organization to either abandon evaluation efforts
or significantly decrease its original commitment of resource
support. Indicators that your organization is in this stage include
- reduced funding for the ROI process
- fewer mentions of the programs and processes in various company
- a lack of participation and involvement by the management team
- constant shuffling of people involved in the process
- fewer requests for products and services around the
- postponed or canceled review sessions aimed at keeping
results-based processes on track
- complaints about the time or cost of evaluation activities.
Stage three: Renewal. Now an organization starts
to move past its inhibitors and explores how to renew its initial
commitment to invest in expanded evaluation solutions. The need for
integrated evaluation processes and solutions is getting increased
visibility and attention from internal staff, including senior
management. Indicators that your project has entered this stage
- renewed mechanisms for monitoring and addressing evaluation
mindsets, behaviors, and practices
- increased linkage between needs assessment and evaluation
- increased energy and potential chaos around identifying where
further support is needed and how to get it
- some conflict as multiple solutions are explored.
Stage four: Integration. In this stage, enabling
strategies and infrastructures are clearly in place to ensure that
the ROI process is firmly integrated into the "DNA" of training and
development work. This may range from technology-based support
processes to standardized evaluation guidelines, policies, and
procedures. Here, the ROI process is universally and mutually
understood by all who are involved and the following indicators are
- a spirit of continuous improvement, innovation, and
out-of-the-box thinking around performance improvement and
- increased respect for the training and WLP improvement
functions as key players in organizational change initiatives
- a growing training and workplace learning and performance
- strengthened collaborations between workplace learning and
other key functional areas
- workplace learning staff being viewed as internal experts in
organizational performance evaluation
- evaluation mentoring and support of increased internal
- identified and shared best practices.
It's important to note that these implementation stages rarely
occur in a linear fashion. However, the concerns and indicators of
each phase are typical, in varying degrees, as organizations begin
to implement results-based evaluation processes as standard
practice. By understanding these stages, an evaluation professional
can identify critical leverage points and assume roles that will
accelerate organizational movement from one phase to the next. The
key is to keep moving toward integration and avoid prolonged
inertia. Specific actions that facilitate movement are called
Conduct case studies. Applied, targeted case
studies foster internal capabilities in results-based evaluation
and build acceptance for the process. They also show
- which programs have achieved success
- what organizational enablers or barriers occurred in the
process of achieving desired results
- what new program expenditures are justified.
Initiate annual reviews. This review process is
typically for the senior leadership team and is designed to track
targeted improvements from the ROI process, including where
significant deviations from expected results have occurred;
determine what evaluation priorities and potential payoffs exist
for the near future; and determine what is needed to keep
evaluation targets in place for the coming year. These reviews can
also be used to reinforce the role of management in building and
sustaining a results-based culture.
Review staff roles. If workplace learning staff
lose their enthusiasm for the process or fail to complete the tasks
associated with their roles, others can perceive this as a lack of
commitment. This view can be contagious and cause other internal
groups to lose support and commitment, as well. WLP staff must make
sure that evaluation policies are implemented, practices adhered
to, and data delivered in a timely manner. Periodic reviews of
staff roles and how they translate into day-to-day job descriptions
are essential to ensure that all members of the group understand
their responsibilities in making the ROI process work as a
system-wide approach. Also be sure that staff job descriptions
emphasize evaluation skill sets as a core competency.
Implement continuous improvement mechanisms.
Implementing the ROI process and monitoring its progress within the
realm of a single impact study may be a relatively simple task. But
as the process becomes more visible and integrated in the
organization, continuous improvement mechanisms become more
critical in order for the process to remain credible and flexible
over time. Use a scorecard or dashboard approach to tie evaluation
results to organizational performance measures. Publish success
stories. Establish routine meetings where lessons learned and
solution implementations are reviewed and tracked. Develop
capabilities by integrating results-based training into core
business processes and developing communities of practice for those
charged with implementing and supporting evaluation efforts.
Identify and share best practices. With increased
prevalence and application of evaluation methodology, much of the
focus has now turned to best practices for ROI implementation. Best
practices include positioning ROI as a process improvement tool and
not a performance evaluation tool for the WLP staff. In addition,
it is important to pace yourself and conduct ROI impact studies on
a selective basis--usually involving 5 to 10 percent of all
programs and solutions
Putting it all together
Ultimately, the task of sustaining results-based evaluation
processes over time is not the sole responsibility of the training
function. A key challenge is to secure and foster ongoing ROI
support, cooperation, interaction, and dedication from individuals
and groups across all organizational levels. Focus on the following
to assist with these challenges.
- Remember that organizations go through predictable growth
stages in the move from episodic implementation to long-term
integration of the ROI process.
- Look for specific indicators that the ROI process may be off
- Continually renew and refresh commitment to the process across
individual, process, and organizational levels so that it remains
consistent, reliable, and credible in the eyes of stakeholders.
- Develop best practices for capturing organizational responses
and lessons learned during your accountability journey.
- Continually seek best practice examples from professional
associations, training literature, journals, case studies,
colleagues, and the ASTD ROI Network.
Finally, one of the best pieces of advice for any WLP professional
seeking to standardize the ROI process as a way of doing business
is this: Commit to the long haul!
There are no short cuts to building a results-based measurement
culture. It takes time, effort, and a dedicated desire to make a
difference. Keep in mind, however, that the payoff from that
commitment will help transform the strategic role of workplace
learning in your organization; improve efficiency in training
design, development, and delivery; and increase respect, support,
and commitment from the senior executives and major program
sponsors who seek and approve resources for value-added performance
2007 ASTD, Alexandria, VA. All rights reserved.