Adaptive and Interactive Assessments with E-Learning
By Nathan A. Thompson
Adaptive and interactive assessments are well-researched and effective methods for making tests more accurate, efficient, and valid. While most attention comes from large tests such as licensure or admissions, they still have much to offer your educational program.
We all know that e-learning has done much for the process of learning—making it more active, interesting, and appealing than conventional printed materials while also providing a higher level of fidelity. Unfortunately, far less attention has been given to electronic assessment, and most of that attention comes from a psychometric perspective rather than a student interface perspective. The assessment process is quite underserved in e-learning when compared to instructional design or delivery, but is just as important. Consider the content retention and effectiveness ratings of a course if there is a long, boring, stressful multiple-choice test at the end, especially if the learner knows that it is coming. The course can be as informative and engaging as possible, but if learners are dreading a two-hour test at the end, retention is not going to be where at the level you desire. Nor will be the satisfaction of your learners.
Attempting to spruce up the assessment by adding gimmicks or other superfluous content is not always a good idea. For example, I once saw training material in which the assessments were delivered in a game show format, complete with a loud salesman-pitch announcer: “Ding, ding ding! That’s corrrrrect! The next question is worth 20 points…” Not only was it incredibly annoying to have to listen to it, but it took twice as long as if the student was simply allowed to read the question and answer. Moreover, the most likely thing that a learner would remember is the irritating voice, not the material.
Two important assessment technologies are available that can be applied to this issue: adaptive testing and interactive testing. While not designed specifically with the intent of making tests more interesting or even (gasp!) fun, both technologies advance the development of tests with a scientific, empirical approach to making the tests better. Adaptive testing (often abbreviated CAT for computerized adaptive testing) was designed with efficiency and precision in mind, so that only half as many items are needed to obtain precise examinee scores. Interactive testing, usually in the form of computerized simulations, attempts to increase the fidelity of the assessment by making it more real-life than multiple-choice questions, and therefore arguably more valid. So while not directly targeted toward making the test more enjoyable from the learner’s perspective, obviously having a test that is only half as long or more closely models real tasks can have some serious benefits.
Computerized Adaptive Testing
The goal of adaptive testing is to dynamically adapt a test to each examinee by giving them questions that are only appropriate for their ability level. Test questions, which typically are referred to as items in the testing industry, that are too easy are a waste of time for both the learner and for the test itself, and increase the public exposure of an item without helping the measurement process. Items that are too difficult, on the other hand, similarly do not help the measurement process, and have the additional drawback of being discouraging to the test taker. Unfortunately, it is difficult (if not impossible) to adequately match learners with appropriate items—it is the proverbial comparison of apples and oranges.
To implement a matching, learners and items must be placed on the same scale. A methodology known as item response theory (IRT) provides this basis for matching people and test questions. IRT is very complex, with stacks of technical books devoted to the subject. If more info is desired, several introductory textbooks exist, such as Item Response Theory for Psychologists by Susan Embretson and Steven Reise, while a brief technical introduction is found at http://edres.org/irt/baker. The immediately relevant aspect of IRT is the substantial advantages gained by this scaling of people and items. This is what enables the matching of examinees with appropriate items that is the foundation of adaptive testing.
This matching provides a distinct advantage over conventional linear tests: test length will be reduced by 50 percent or more without any reduction in precision or reliability. Obviously, this is of great practical importance. By decreasing the amount of time required to assess learners at the end of instruction, not only can the additional time be directed toward feedback or further instruction, but the assessment process will be that much better for the learners. Why bombard learners for two hours with multiple-choice items when the same effective measurement can be had in less than an hour? An additional advantage is the effect that the decreased test length has on item exposure. Because tests are only half as long, the number of exposures of items to the public is also cut in half. If your testing program is high-stakes enough that there is some incentive for learners to cheat, this reduction in exposure has great benefit. Good items can be expensive. Additionally, cheating is also reduced by the fact that nearly every student will receive a different set of items because the test is adapting to the individual as is proceeds along.
Adaptive testing is not without its disadvantages, however. The biggest disadvantage is the required sample size. At the very least, you need data from 100 learners on your items before they can be calibrated with IRT. In some cases, more than 1000 learners are needed. However, this drawback is tempered by the economics of scale involved with adaptive testing. Applying adaptive testing is generally more feasible and rewarding for larger-scale testing; those programs that have the most to benefit from moving to adaptive testing are also those that have the least to worry about when it comes to getting more data. Another drawback is the public relations with those taking the tests. Adaptive testing is justified by 40 years of empirical research, which might be accepted by you, but it is a little more difficult to explain to an angry student who just failed the test. However, this can be alleviated by education of learners concerning the testing process.
So, if adaptive testing is something that might be of interest, how do you go about implementing it? The simplest method would be to outsource it to a vendor. Large-scale testing companies have the capability to administer adaptive tests built into their extensive testing networks. These networks can include hundreds, even thousands, of high-security testing rooms across the globe. Additionally, such companies have the expertise to help you implement adaptive tests using their networks. The drawback to this approach is the substantial cost that it will involve—one reason the economics of scale apply so strongly to adaptive testing. On the other end of the spectrum is the possibility of your organization hiring its own expertise in psychometrics and programming to implement adaptive testing yourself. This might be substantially more financially feasible than complete outsourcing, however, the expertise of psychometricians is in high demand, and finding one to be dedicated to your organization could be quite difficult. Again, the economics of scale are directly applicable.
In between these two approaches is the possibility of using commercially available software for adaptive testing. This will significantly reduce the cost of moving towards adaptive testing by allowing you to administer the tests yourself, but the drawback is that psychometric expertise will still need to be acquired by your organization, either as a consultant or as an employee. An additional approach is to use prepackaged adaptive tests. While this can be substantially cheaper than other approaches, it is not likely that the content of the prepackaged tests aligns well with your instructional content.
Interactive testing
Another important technological advance in testing and assessment is the development of interactive simulation-type tests. Unlike adaptive tests, which seek to make the testing process more efficient, interactive tests might actually take longer than their conventional parallels. The goal of simulation-type tests is to increase the fidelity of the tests, which means trying to make the test closer to the thing that it is actually assessing, rather than oversimplifying it. For example, if you are testing on a complicated cognitive process, such as designing a website, it might make more sense to simulate the creation of a website rather than try to break it down into a few multiple-choice questions. This is most appropriate for tests that are not assessing knowledge, but something of a higher level. Often, knowledge-based tests are better served by multiple-choice or other conventional item types.
An important component of this approach is to be able to break the topic into specific tasks. Often, this is already accomplished by the content itself. If the content being assessed involves a number of specific tasks that learners are being trained on, this will easily determine the goals of the interactive test. An example of this is the field of medical assisting, in which professionals are often required to perform certain tasks with high frequency. Simulative tests can fit quite well with this, as is done by the Joint Commission on Allied Health Personnel in Ophthalmology (www.jcahpo.org). This organization assesses the ability to perform seven common ophthalmic tasks by means of a computerized simulation. Learners are required to turn dials, take readings, and so forth with a computer interface that mimics the instruments used in real life.
This type of computerized assessment fits well with electronic learning for two reasons. First, as previously mentioned, it increases the fidelity of the assessment. Second, it is more interesting and engaging from the point of the test taker. While all tests are arguably part of the instructional process, interactive tests are truly instructional in that they require the learner to actually do what it is they are supposed to be learning. This obviously helps retention of material and is what really separates interactive simulation tests and performance testing in general from tests are they are usually conceptualized.
If interactive tests are so good, why are they not used all the time? There are several reasons. First, not all material lends itself to any type of performance test; as mentioned, the material needs to involve doing something, not simply the assessment of a knowledge base. Also, if a test is made to assess a complicated process, it is necessary to have a method of breaking down the process in a way that can be scored, which is not always feasible. Finally, of course, there is the cost; it is not cheap to develop a computerized simulation that is customized to the process you are trying to assess.
Final word
Adaptive testing and interactive testing are two important advances in the technology of assessment with e-learning. These innovations offer an approach to testing that makes it more efficient and engaging. While not universally applicable to all programs, mostly due to sample requirements, adaptive testing and interactive testing have much to offer programs looking for a way to make their assessments better.
More important, these methods are actually designed to make the measurement itself better, rather than just providing window dressing to the delivery of an assessment. Reducing test time has the very real benefit of freeing more time for instruction or simply letting the learners get back to work. Indeed, innovations such as these can increase learner retention of material in many contexts. And because that’s the ultimate goal of e-learning, we should take advantage of methods that contribute.
Nathan A. Thompson is the vice president of Assessment Systems Corporation, a leading provider of solutions to the testing and assessment industry; nthompson@assess.com.