JMA Management Center (JMAM) is a Japanese training consulting company that provides workshops, training solutions, and training-evaluation consulting. Generally, for Level-1 training evaluation, we collect satisfaction data from participants and use it to modify the program to increase satisfaction. However, most Japanese training professionals have only rudimentary instructional design and evaluation skills and generally do not know how to improve a program based on reaction data. Therefore, we developed a three-tiered approach to supplement reaction data with feedback from instructors and observers and add value to the process. Click here to see a chart that shows this approach.

Tier 1

The main data from participants tells whether they found the training content useful and practical. It also tells whether participants evaluated the training experience as meaningful, engaging, and worthwhile.

Tier 2

Supplemental data from the instructor tells how he or she reflected on the course's effectiveness. While the instructor's assessment provides good information, it is subjective and influenced by his or her own character and abilities.

Tier 3

To reduce subjectivity, a coordinator for Level-1 evaluation provides an objective third-party appraisal. A coordinator serves in an auditor role. Because there are no professional coordinators in the training market who can evaluate the range of detailed training programs from JMAM, we selectively recruit other experienced instructors as coordinators.

JMAM carefully chooses its coordinators based on their training program experience and their observation, communication, and problem-solving skills. They also should have professional knowledge of the program's theme and content. Whenever possible, coordinators should have some experience in program design and instruction so that they can better evaluate instructors' abilities and participants' level of understanding and involvement. The role of coordinator requires deep insight and creativity.

Before trying out this three-way evaluation approach, JMAM expected stakeholders in its customer organization to reject the project because of the higher evaluation costs of the large-scale Level-1 measurement. However, once we explained each of the evaluation functions, and its logic in assigning them to participants, instructors, and coordinators, we met no resistance.

This approach is now being piloted for two government projects, along with a new, expanded Level-1 questionnaire. We believe that this three-tiered Level-1 evaluation method can be a powerful tool for both training professionals and their organizations as they strive to improve the value of their workshops.