Augmented Reality Check
By Eva Kaplan-Leiserson

This article is third in a series of articles on technology trends and their applicability to learning. 

Think you can predict the future of learning technology? If you point to simulations and virtual reality, you’re wrong. Or at least half wrong.

Simulations will have their place. They’re getting more and more sophisticated and the popularity of such virtual universes as Second Life and There prove that people can be engaged by, even addicted to, experiences that are purely virtual.  

But the real future of learning is what one developer calls “blended learning on steroids.” Just as training designers now choose among classroom-based training, synchronous online seminars, asynchronous Web-based training, and other options when determining the best delivery mechanism for learning content, in the near future those designing training will choose among training delivered in the real environment, via augmented reality, or in a virtual environment.

(These designations, along with a fourth—augmented virtuality—were laid out on a continuum in 1994 by Paul Milgram, Haruo Takemura, Akira Utsumi, and Fumio Kishino in their paper “Augmented Reality: A Class of Displays on the Reality-Virtuality Continuum.”)

The real environment is self-explanatory. Virtual reality, in the form of computer-based simulations, has been written about quite frequently in learning publications—but much less attention has been paid to the learning potential of augmented reality. This is no doubt because its more complicated technology has matured at a much slower rate. But research and hardware advances have helped propel AR forward, and it is poised to make a very big splash in the learning arena. It is AR, then, that we will examine here in more depth.

A definition and brief history

Augmented reality combines features of a virtual environment with the real world. Most often, the augmentation is visual, with a user sporting an eyepiece connected to a wearable computer and positioning equipment. By tracking where the user’s head is and what he is seeing, the computer is able to overlay graphics and/or text onto his vision.

This type of technology has been under development for more than 30 years, according to Scientific American, at such places as Harvard University, University of North Carolina at Chapel Hill, The University of Utah, the U.S. Air Force’s Armstrong Laboratory, and the NASA Ames Research Center.

The U.S. military has funded much of the AR research over the last decade, not only spending millions of dollars at their facilities but also at universities, says Sonny Kirkley, whose company Information in Place develops augmented reality solutions and has contracts with the Army, Air Force, and Coast Guard. He says that although the university research was not ostensibly for military use, “everybody knew the intent was, how is this going to apply in a military space?”

In the 1990s, the term augmented reality was coined at Boeing, Scientific American  says, when scientists there developed a prototype solution to help workers put together wiring harnesses.

As computer technology improved, augmented reality developed more rapidly. In 2000, a project called MagicBook garnered excitement at the computer graphics SIGGRAPH conference. MagicBook could be read like normal text, but when viewed through a head-mounted display, animated 3D figures acted out the story. In his 2002 article “Augmented Reality and Education: Current Projects and the Potential for Classroom Learning,” Brett Shelton says the MagicBook project sparked people’s interest in the industrial applications of AR.

In the last few years, the definition of augmented reality has expanded as researchers have developed additional technologies. In addition to visual augmentation, AR can also encompass auditory augmentation (a computerized earpiece whispers information into a person’s ear), touch augmentation (also called haptic augmentation) or augmentation via a personal digital assistant (PDA). One researcher is even designing visual augmented reality that works with online learning. We will examine examples and learning applications for the various types of AR technology.

Visual

Because visual AR has been under development for much longer than other types, it is the furthest along in terms of practical application. Rather than being just prototypes, as many of these technologies are, visual AR solutions are actually implemented in some instances. For example, with technicians at Boeing and mechanics at the American Honda Motor Company. Both companies are providing schematics on “heads-up” displays, as they’re called, to repair and maintenance crews for real-time electronic performance support. 

Honda is deploying Nomad Expert Technician Systems from Microvision, hands-free wearable displays that reflect images directly into the user’s eye, in their 12 U.S. training centers. The systems give the entry-level and experienced technicians who come for training access to online vehicle history and repair information without them having to take time away from the car to look at a separate computer display.

Microvision says their technology results in higher-quality work from less experienced technicians, and 30 to 40 percent efficiency gains measured in real-life trials. The devices cost about $4000 but the company says they can be paid back in less than 3 months and that a typical dealership adds $16,107 in gross profit per technician by using the devices.

In addition to Honda, some vocational technology colleges are using Microvision devices, also for automotive repair, as well as the military for maintenance training. We can imagine many other uses for this type of visual augmented performance support for workers who don’t sit at desks and would not normally have access to a computer, but who need to access written material or diagrams. Medical personnel and factory workers are just two groups who could benefit. 

Auditory

Accenture Technology Labs has developed a prototype auditory AR system called the Personal Awareness Assistant, a small wearable computer device that can record and transmit sound and contains two microphones, a small camera, and voice recognition software. One use for it is in networking. When a user says the words, “It’s nice to meet you, Jane,” the system records the name of the person and takes a picture. Then the assistant stores the audio, image, time and date stamp, and location in an address book for later retrieval. So asking, “Who was that woman I met last week at the holiday party?” will bring up the information.

The Personal Awareness Assistant could have further uses in performance support. Kelly Dempski, one of the researchers at Accenture, imagines that the system could, for example, warn workers at a chemical plant that they are entering an area that’s not safe for them because they lack the required safety gear.

Dempski envisions that the system could increase its level of support based on user needs. It could whisper in the ear of a first-day employee and tell her where to report and when. After she’s been on the job for a while, it could provide suggestions on improving the quality of her work.

Accenture also foresees uses for the device for what the Website calls “collective intelligence.” The scenario: A user is in an important business meeting and a new customer asks a question that he doesn’t know the answer to. Via the device, colleagues who are situated elsewhere can give him the answer discreetly.

Haptic

Touch is another sense that can be augmented with learning applications. A paper presented at the 2004 International Symposium on Mixed and Augmented Reality (ISMAR) discussed the development of augmented Standardized Patients (SPs) in medical training.

SPs are traditionally real people who portray patients in order for a medical student to practice such skills as getting a medical history, performing an examination, and so forth. However, it’s often difficult to find people who can realistically simulate various symptoms.

Completely virtual patients accessed via a computer raises a different set of concerns, the papers’ authors state. The medical students must learn not just how to diagnose patients but also how to interact and communicate with them. Human-to-computer interaction doesn’t provide the necessary skills transfer.

Fortunately, there’s a third option, thanks to new technology being tested and prototyped by the Eastern Virginia Medical School and the Virginia Modeling, Analysis, and Simulation Center at Old Dominion University. (Some funding also came from the Naval Health Research Center.)

Researchers rigged a system that tracks the movement of a student’s stethoscope on an augmented SP. When the stethoscope reaches the proper location, it triggers a sound file played over headphones the student wears. The system was developed using mannequins because of the long periods of immobility needed during testing, but it can be used just as easily on real people.

Other training situations that currently use actors, such as rescue worker or other emergency personnel training, could provide additional uses for haptic augmented reality.

Online learning and collaboration

Augmented reality can even be used with online learning. At the United Kingdom’s University Sussex, researcher Fotis Liarokapis developed “an interactive e-learning AR environment” in which users can view and interact with three-dimensional virtual objects aided by online instructors.

Part of an ongoing research project, the Multimedia Augmented Reality Interface for E-Learning (MARIE) uses many of the same technologies as visual augmented reality—the head-mounted display, camera, and computer—to present 3D multimedia information to the learner.

But the objects are part of synchronous e-learning: The instructor guides learners to view various them in sequence as part of his or her learning strategy, much as a traditional instructor might tell learners to turn to a certain illustration in a manual or textbook. However, with MARIE, the learner can interact with the object, rotating and manipulating it. Future development may enable the addition of other forms of augmentation, such as touch and smell, and he is continuing to develop the project in order to integrate it with e-learning platform WebCT.

Similarly, Kelly Dempski talks about a project he’s working on at Accenture that combines videoconferencing with augmented reality for augmented collaboration. Two people in different locations have cameras and display screens set up so they seem to be talking to each other from across a table. Then, the two of them can both interact with shared virtual objects in 3D for collaboration or learning.

Liarokapis says augmented reality tops virtual reality for learning and training “in terms of cost, realism, and human factors. Virtual reality completely replaces the real environment and this makes it much more difficult for users to adapt.” In 5 or 10 years, he says, we will be using AR for “a number of everyday applications,” including training and learning.

Personal Digital Assistants (PDAs)

In most of the previous examples, the augmented reality technology was under development and not in commercial use. For now, the technology is still too complicated and expensive for most companies. That will change as computer hardware gets smaller and less expensive. In the meantime, there is a workaround that has arisen over the past few years: augmented reality via personal digital assistant (PDA). The experience is obviously not as immersive, but it is a viable strategy being used with some success in learning, especially when combined with a game-based approach.

For example, the Massachusetts Institute of Technology’s Education Arcade (a group that promotes the use of computer and video games in education) joined with the Boston Museum of Science last February to create Mystery at the Museum, a “Hi-Tech Who Done It” they tested with pairs of middle school children and adults.

The premise: thieves have stolen a museum artifact and replaced it with a fake. The detective teams, three adult-child pairs, each received a handheld device, which they used in various galleries (along with the museum’s Wi-Fi network) to analyze objects with virtual instruments, interview characters, exchange information with other teams, and more. The children were not only highly engaged but also learned quite a bit. 

College students in MIT’s Environmental Education courses play Environmental Detectives, an AR game in which they investigate a toxic spill and learn about observation, hypothesis testing, data gathering and analysis in a problem-based learning simulation untethered from the desktop.

Can these experiments translate to adult learning? Absolutely. Game-based learning is already being used with adult learners, and combining that with whole-body experiences as augmented reality does can not only make the learning more engaging but also more “sticky.”

The ADL (Advanced Distributed Learning) Academic Co-Lab is working to develop new augmented reality applications through the AR gaming platform developed by Education Arcade. Common elements include “engaging backstory, differentiated character roles, reactive third parties, guided debriefing, synthetic activities, and embedded recall/replay to promote both engagement and learning.”

The future of augmented reality

Where is augmented reality and learning going? For the answer to that question, we can look to the U.S. military, which has a head start on the technology as well as the educational pedagogy.  

The Army is developing the technology not only for performance-support information and graphics to be overlaid on soldiers’ vision—navigation lines or equipment repair instructions, for example—but also for thoroughly virtual characters, buildings, explosions, and so forth. Eventually, soldiers will be able to interact with these virtual items in the real world for sophisticated, realistic, whole-body training.

That kind of technology can cost millions of dollars and is still years off—Sonny Kirkley says at least five years and maybe as many as 10, depending on the rate at which the hardware matures. But the Army is getting ready now. As part of their Future Force Warrior program, they’ve contracted with Information in Place not just to work on the technology side but also to develop the instructional methodologies that will be needed in this new brand of training.

The Army said to Kirkley, “Great that you can make [the technology] work, but do people really learn?” So his company has been working to develop problem-based learning—they call it Problem-Based Embedded Training—and learner supports that will be used not only with the augmented reality training but also with the mix that is developing of augmented reality, virtual reality (simulations), and real-life training. Kirkley calls this “mixed reality” training.

In addition, Information in Place is developing an authoring tool that helps training designers think through the process of developing this new type of blending. “Instructional designers have enough trouble developing face-to-face and Web-based [training],” Kirkely says. “So we’ve been looking at an authoring support tool that focused on the instructional design process [for mixed reality training] from needs analysis to final product.”

Kirkley and his colleagues at Information in Place stress that the focus needs to be on what is effective, not the technology. Just as in selecting Web-based or classroom training, training designers need to look first at the training objectives and then decide which medium will best help learners meet those objectives. And that’s not always the expensive solution with the highest fidelity. In short, augmented reality training is no different than other types in that it needs to be focused on solid instructional design and well thought-out cost-benefit analysis.

Eva Kaplan-Leiserson is associate editor of Learning Circuits.