Data collection and reporting are an unavoidable part of training and development. Whether designing technical or soft skills training, creating a performance baseline, crafting an evaluation plan, or assessing characteristics of the organization at large, data collection is an essential driver for the creation and revision of processes and programs.
As with any skill set, data collection can be done well or poorly. In my experience, I have found that having a data collection plan in place makes a world of difference, as opposed to shooting from the hip and determining the steps as you go through the process. Here are a few points to consider before collecting data.
Why are we collecting data in the first place?
There are a number of items to consider when approaching a data collection effort. The primary one is to ask, “What is the purpose or objective of the data collection effort? Are you creating a training course, job aid, or performance support system? Are you collecting data to provide a performance baseline, measure employee engagement, or assess specific elements or organizational culture or dynamics?” Knowing why helps shape the project scope, target audience for data collection, reporting and collection method, questions, needed response rate, and more. This is also useful to know and relay to participants. If participants understand the use of the data and understand what’s in it for them at the end of the evaluation process, they will be more open to providing data for your use. This is also useful information for the data collection team to establish a common focal point.
What are the goals of the data collection effort?
How will you determine if the data collection effort was successful? For some, response rates for paper or online instruments and number of interviews conducted might be considered a goal. For other data collection initiatives, the results might need to be statistically valid to be useful or considered for analysis. Evaluation goals will change according to the project you are completing. It is essential to determine the goals before starting to collect data to measure success and provide the evaluation team a goal to work toward.
Who is the target audience?
There might be multiple target audiences or stakeholders within the process. One group has the answers and another might have a bit of information and more of a vested interest in the outcome of the data collection effort. Another group could be the target audience for the final report (stakeholders) and another might use the raw data collected and the final report to take a closer look at what is happening in their organization, department, or program. When collecting data, it is essential to know who your target audience is at each stage of the process. The instrument and reports will need to be relevant to the target audience to be useful. A one-size-fits-all approach to this portion of planning could lead to invalid data and irrelevant reports.
How will the data be collected?
The data collection instrument needs to be clearly defined and reviewed before placed in use, regardless of the delivery format. Consider the following questions when reviewing the data collection instrument: “Are the questions too complex? Are the questions too simple? Are the questions easy to understand? Are you asking the right questions?”
The actual data collection methods come into play at this time. What is the best format for collecting the desired data? There are many methods to choose from or combine to meet the desired outcomes of the study, including interviews (face-to-face or telephone), focus groups, surveys (hard copy or online), and observation. Pick the data collection method or combination of methods that meets your needs.
If time allows, conduct a beta test on the data collection instrument across multiple areas that represent the target audience, then resolve any questions and address feedback that arises from the testing group. This will improve the data collection instrument in the end.
When will data collection occur?
Timing for the data collection can be critical. If you are evaluating a course, there are specific time windows that must be considered to measure the course feedback. Kirkpatrick and Phillips address this in their evaluation models. If conducting interviews or focus groups, you should consider the schedules of people involved on both ends, as well as the amount of time needed to conduct interviews or focus groups. If there is limited availability when scheduling the target audience, you might need to expand the timeframe for data collection or pursue an online data collection effort.
These are the basics—the who, what, when, where, and why to think about before starting a data collection process or when setting up a data collection plan. Having these items documented from the start will make for a more organized and focused process and produce more relevant and useful data in the end.
© 2012 ASTD, Alexandria, VA. All rights reserved.