JMA Management Center (JMAM) is a Japanese training consulting
company that provides workshops, training solutions, and
training-evaluation consulting. Generally, for Level-1 training
evaluation, we collect satisfaction data from participants and use
it to modify the program to increase satisfaction. However, most
Japanese training professionals have only rudimentary instructional
design and evaluation skills and generally do not know how to
improve a program based on reaction data. Therefore, we developed a
three-tiered approach to supplement reaction data with feedback
from instructors and observers and add value to the process. Click
here to see a chart that shows this approach.
The main data from participants tells whether they found the
training content useful and practical. It also tells whether
participants evaluated the training experience as meaningful,
engaging, and worthwhile.
Supplemental data from the instructor tells how he or she reflected
on the course's effectiveness. While the instructor's assessment
provides good information, it is subjective and influenced by his
or her own character and abilities.
To reduce subjectivity, a coordinator for Level-1 evaluation
provides an objective third-party appraisal. A coordinator serves
in an auditor role. Because there are no professional coordinators
in the training market who can evaluate the range of detailed
training programs from JMAM, we selectively recruit other
experienced instructors as coordinators.
JMAM carefully chooses its coordinators based on their training
program experience and their observation, communication, and
problem-solving skills. They also should have professional
knowledge of the program's theme and content. Whenever possible,
coordinators should have some experience in program design and
instruction so that they can better evaluate instructors' abilities
and participants' level of understanding and involvement. The role
of coordinator requires deep insight and creativity.
Before trying out this three-way evaluation approach, JMAM expected
stakeholders in its customer organization to reject the project
because of the higher evaluation costs of the large-scale Level-1
measurement. However, once we explained each of the evaluation
functions, and its logic in assigning them to participants,
instructors, and coordinators, we met no resistance.
This approach is now being piloted for two government projects,
along with a new, expanded Level-1 questionnaire. We believe that
this three-tiered Level-1 evaluation method can be a powerful tool
for both training professionals and their organizations as they
strive to improve the value of their workshops.