Resources Needed to Make Evaluation Work
By Jack J. Phillips, Patricia Pulliam Phillips, and Toni Krucky Hodges
This excerpt from the new ASTD Press book, Make Training Evaluation Work, outlines the technology (both hardware and software) and time requirements needed to implement evaluation, as well as the budgetary requirements to make it successful.
Technology requirements
Much technology is available to support evaluation execution and reporting needs. The evaluator needs to research and select from the many vendors and programs available, pinpointing the most appropriate technology for the organization. Hardware and software can support almost any phase of the evaluation, including data collection, data management and reporting, and data analysis.
Data collection
Because reaction data are collected for most learning and development programs, you can exploit technology collect, store, and report data. Some organizations want to collect learning and application data regularly from surveys, adding to the need to collect and distribute data.
For small organizations, the technology available in most office suite applications enables the evaluator to organize, tabulate, and store data. For medium-sized organizations, data input can be made more efficient with scanning tools. A variety of scanning hardware and software packages are available, such as Scantron, which allows data to be input, stored, combined, and reported automatically. Many organizations are moving to systems for online input whereby participants in learning and development programs provide feedback information at reaction, learning, and application levels. Data is collected, organized, and presented routinely in a variety of different reports. KnowledgeAdvisors.com offers a software package called Metrics that Matter to aid data collection and reporting.
Much technological help is available for designing, administering, and analyzing surveys and questionnaires. For example, SurveyPro, SurveySays, SurveyTracker, and QuestionMark are just a few of the software packages available to help design and administer this most common data collection tool.
Data management and reporting
For larger organizations, software is needed to help organize and manage all or the learning and development system data. These learning management systems, which essentially manage the complete learning process, often have an evaluation component. Systems are available from SumTotal Systems, Saba, and other suppliers.
Sometimes these systems provide evaluation data in a summary report usually referred to as a training or learning scorecard, which is a summary of all the data collected and organized around the levels of evaluation. Figure 5-3 shows one type of scorecard that presents a variety of data of different programs (Schmidt, 2003). The data are arranged by levels and include several measures monetary values, application measures, business impact measures, and ROI values, if available, are also included. The software available from knowledgeadvisors.com (Metrics that Matter) has the capability to build scorecard from data across programs.
Data analysis
The data analysis required for the evaluation of most learning and performance improvement programs do not require sophisticated statistical analysis, so a robust system may not be necessary. Microsoft Excel, which is included in Microsoft Office Suite, is usually sufficient. This software summarizes and describes the data in terms of the mean, range, and standard deviation, and so forth. It can perform the analysis required to conduct trend line analysis, hypothesis testing, and correlation analysis. Excel can create bar graphs, pie charts, and other acceptable methods for displaying data. Other statistical packages, such as SPSS, SAS, STATPACK, DataDesk, Statview, and MINITAB can conduct additional analyses. The important thing is to keep the analysis simple.
Software programs designed specifically for comprehensive evaluation are now available. Programs start at the beginning point of evaluation and move through the various stages of data collection, analysis, processing, and reporting. In some cases an impact study is generated. Programs available from knowledgeadvisors.com are comprehensive and designed specifically for high-level evaluation, usually involving impact and ROI analysis.
Time requirements
The time requirements for collecting, analyzing, and reporting evaluation data depend on the level of evaluation addressed, the resources available, the expertise of the staff, and the technology used. Using standardized processes for collecting reaction, learning, and application data reduces the time needed for each evaluation. Of course, outsourcing evaluation studies, can also reduce the time requirements, as well as increasing the level of automation used for various tasks related to evaluation data.
With so many potential variations, it is impossible to pinpoint the length of time it takes to conduct an evaluation study. Nevertheless, a rough estimate can provide some insight into time requirements. The table below offers estimates based on the authors’ combined experience. It is important to remember that these estimates vary with the type of study, setting, and methods utilized. As the table reveals, automation drastically reduces the amount of time required to conduct evaluation studies at all levels. Because the variances are smaller, more reliable predications can be developed when automation is used.
The table above also assumes that higher level evaluations include data collected at lower levels. For example, when an impact analysis (level 4) is conducted, data is also collected at levels 1, 2, and 3. It is assumed, for these estimates, that such data is available for input for the higher-level evaluation, providing evidence of the chain of impact and clearly showing the target audience how the participants reacted, learned, applied, and secured results with the program.
Planning for a major study should take no more than a day, and the design of the instruments can vary depending on the number of instruments and method of data collection. Data collection is the most variable part of the analysis. Simple data collection instruments, such as questionnaires, are inexpensive and involve less time to implement. Other instruments, such as observation, interview, and focus groups are more time consuming and can add considerably to the time estimates to this table. Nevertheless, the table is a rough guide of the time required to conduct evaluation studies. The greatest variance occurs with impact and ROI studies for which the setting, type of program, type of data collected, and the variety of instruments used can significantly influence the time required.
This table reveals several important principles about the time for evaluation:
· The processes should be automated to the extent possible. Fortunately, there are software packages available to help make data collection routine and consistent.
· The design time can be very significant and should be automated if possible.
· The use of standardized tools and templates is extremely helpful.
· The programs destined for impact and ROI analysis should be limited to those considered to be significant, critical, expensive, highly visible, and time consuming. This selection process helps manage the resource requirements and keep the commitments low for the overall time required for evaluation.
It takes a considerable amount of time to establish the evaluation system so it becomes routine, automated, and replicable. It may take 6 months to a year to build a system necessary for a comprehensive measurement and evaluation process for a large organization. For example, the figure below presents a timeline for building a system for a major Fortune 500 organization. This organization used the evaluation data to develop scorecards. External consultancy support and automation were critical parts of the process. Such is the case for many large organizations that have large budgets and need to conduct evaluation over several hundred programs.
Cost and budget requirements
Budgeting for evaluation depends on all the issues presented in chapter 4 of the book (purpose, philosophy, strategy, staffing, and so forth). The following table lists the items that should included in the budget.
This table shows the typical cost categories of any process involved in learning, development, and performance improvement. Evaluation should be considered a major component of the performance improvement cycle and should be included in the budgeting process just as design, development, implementation, and initial analysis are. The most significant cost is the value of staff time allocated for evaluation.
Participant costs reflect the cost of the participants’ time required for evaluation activities. For example, if a participant is involved in a focus group, he or she is not able to take customer orders or work on the assembly line. This item is handled in different ways. In some situations, participant time is not a budgeted item and is considered negotiable. But, when the time involved in evaluation is significant and requires time away from work to provide input and data for evaluation issues, the costs are included.
The line item for outside services includes consultants’ fees and external services for evaluation. Fees and licenses include fees for staff to attend programs or other learning activities to build expertise with evaluation and may also include license agreements to use certain technologies or processes for evaluation. Technology includes both hardware and software used to support the evaluation process. This item may be prorated if the hardware or software is used for other functions in the organization.
Organizations with some history of using evaluation likely have data from prior evaluations to determine costs for evaluation budgeting. Those new to evaluation or who are improving the evaluation process significantly may need help with evaluation costs. The table below shows a sample of cost items associated with evaluation. Some of these are based on actual cost, some are estimates based on benchmarking data from evaluation implementations, and others are some guesses.
The total cost of measurement and evaluation as a percentage of the budget, taken from benchmarking, is probably the most valuable item. For comprehensive evaluations, for which data is collected on every program and a few programs are taken to the levels of impact and ROI, the cost can be as much as 3 to 5 percent of the total learning and development (or performance improvement) budget. The budget figure should include all the direct items in a typical budget including staff salaries, office expense, and external charges.
The cost of processing one evaluation questionnaire varies, but many external providers charge about $2 or less per questionnaire for processing and reporting back to the organization. At this price, a program with 25 participants can be summarized and reported to appropriate groups for $50.
The cost of conducting impact studies varies significantly as already shown. The costs shown in this table are based on the authors’ experience. The cost of an impact or ROI study is typically 5 to 10 percent of the total cost of the project. For example, in a major learning and development program that costs an organization $200,000, the evaluation cost should be in the range of $10,000=n=$20,000. These are only rough guides and can vary considerably. The other cost items are offered just to show a sampling of different costs and to provide a bit of humor in what may be considered a boring issue.
Final thoughts
Planning for evaluation takes into account many factors, including expertise and resources. Data collection and analysis must be carefully planned to consider technology and time requirements. A variety of technology options are available to assist in data collection, analysis, and reporting. Using technology drastically reduces the amount of time required for these activities. Budgetary requirements depend upon the decisions made regarding evaluation purpose, role, and strategy. Planning for evaluation costs should be a routine part of the budgeting process.
Regardless of resources available, every organization should be able to establish and implement an evaluation program that provides appropriate data about the success of learning, development, and performance improvement programs.