Using 3-D and Intelligence in WBT
By Janet Faye Johns
A leading researcher describes efforts to develop "intelligent three-dimensional (3-D) practice environments" that coach learners through Web-based mechanical skills training.
A Web browser that uses Virtual Reality Modeling Language (VRML) provides an interactive, 3-D world in which users can learn by discovery and by doing. A Java expert system adds the ability for the practice environment to coach and guide the learning process. Together, they can create intelligent 3-D skills practice environments, ideal for various types of mechanical skills training.
First, some background on our research. Past projects at MITRE led to the development of structured 3-D practice environments that provided a training and problem-solving environment by integrating 3-D animations with multimedia CBT. We extended the problem-solving environments into 3-D "virtual worlds" where the user could freely explore and learn by discovery. This extension enabled us to identify many of the advantages and disadvantages of the technologies currently available. We then began exploring integrating practice environments with the capabilities offered by such artificial intelligence (AI) technologies as expert systems. Expert systems provide an ideal first step toward creating an intelligent practice environment because they're widely used, their advantages are well-understood, and programming an expert system is less complex than other AI approaches.
Adding intelligence should improve learning by creating a practice environment that adapts coaching and feedback to the user's individual needs. Figure 1 shows our instructional strategy for using practice environments.
Figure 1: Instructional Strategy with Practice Environments
Current technologies
What can we realistically do today? Using readily available off-the-shelf products and standards are essential to make 3-D practice environments an affordable training solution. Extensive research and software development is too costly and time-consuming for most training projects. Recent advances in Web technology have paved the way for desktop-based 3-D environments. A Web browser with a VRML plug-in provides dynamic user interactions with 3-D objects in a 3-D world. VRML worlds can be used to provide simple practice environments where the user can learn by discovery with the browsing and exploring capabilities inherent in the technology.
More sophisticated practice environments can be achieved by integrating 3-D VRML worlds with Java, JavaScript, and HTML frames. The integration of these technologies can provide highly interactive practice environments that respond dynamically to the user's interactions, as illustrated in figure 2. Here's a brief summary of how these technologies work together.
- HTML. The VRML world can be displayed as a frame in an HTML page with other frames of related information and user guidance. The HTML frames can have hyperlinks to associated images and other reference material.
- JavaScript. JavaScript can provide a mechanism to display textual information and update the information displayed in the HTML frames. JavaScript can be integrated with VRML worlds through the VRML Script node and executed in response to VRML events. JavaScript can also be used on the HTML pages to update content and graphics.
- Java. Integrating Java with the VRML worlds can provide dynamic assessment, coaching, and feedback capabilities for a practice environment. Java classes can be integrated with VRML worlds through the VRML Script node, and Java methods can be executed in response to VRML events. Java classes and methods integrated with a VRML world can query and update VRML objects and events.
Figure 2: A Java-Enhanced Practice Environment
Figure 2 shows an early prototype practice environment that uses a Java state machine (one that tracks interactions and equipment) instead of an expert system to manage the user's activities. This example teaches the user to mount a measuring device--known as a dial indicator--on the shaft assembly. The screen has three HTML frames. The frame on the left side of the screen contains training guidance, including an objective, a learning approach, and an expected time limit; it can also contain hyperlinks to related material. The VRML world is in the topmost right frame, and dynamic feedback is given in the bottom right frame. Coaching, feedback, and assessment information are also given to the user in JavaScript alert message boxes. Java applets could also provide dynamic data to the user. This approach is adequate for a simple practice environment.
Adding intelligence with an expert system
Table 1 identifies some useful learning-by-doing approaches for mechanical skills training. These learning approaches can be implemented by integrating VRML, VRML Script, JavaScript, Java, and an expert system.
Table 1: Approaches for Learning by Doing
|
Learning Approach |
Practice Environment Features |
|
Construction |
The user performs meaningful tasks, such as constructing a product, from the 3-D components in the virtual world. Instructional data can be displayed in the HTML frame(s) and dynamically updated to provide guidance and assessment related to the user’s actions and current state of the world. |
|
Procedures and sequences |
The user experiences a practice environment that provides guidance and coaching through a sequence of actions in the 3-D world. This learning approach builds on the Construction learning approach and enforces a recommended sequence of actions. |
|
Simulation |
The user experiences a practice environment that responds in a realistic manner to the accumulation of his or her actions and the behavior of the objects in the 3-D world. This learning approach includes construction and sequential activities. |
One off-the-shelf product--Jess--is available for integration with products such as a 3-D practice environment. Written by J. Friedman-Hill of Sandia Corporation, Jess is distributed with sample applets and console applications. We found these easy to modify and integrate with VRML practice environments. Figure 3 shows the configuration of existing products and developed software integrated to create an intelligent practice environment with the capabilities of the Jess expert system. The accompanying Table 2 provides information about each of the interfaces in the configuration.
Figure 3: Configuration of an Intelligent Practice Environment
Table 2: Interfaces in the Intelligent Practice Environment
|
Interface
Number |
Interface Origin |
Interface
Destination |
Purpose of Interface |
|
1 |
Rule base |
Expert system |
Load rules and initial set of facts into the expert system knowledge base. |
|
2 |
Persistent data store |
Expert system |
Load facts and other state data stored from last session. |
|
3 |
User interactions in VRML scene |
VRML Script node |
Access information about user interactions in the practice environment. |
|
4 |
VRML Script node |
Java event handler |
Pass user interaction information and other state information to the Java event handler for analysis. |
|
5 |
Java event handler |
Expert system |
Update knowledge base and assert new facts based on user interactions and state of the objects in the practice environment. |
|
6 |
Expert system |
Persistent data store |
Store facts and state information to be available in another session. |
|
7 |
Expert system |
Java event handler |
Update state data, feedback, and advice in the practice environment. |
|
8 |
Java event handler |
VRML Script node |
Pass updated data from the expert system and Java event handler to the practice environment. |
|
9 |
VRML Script node |
VRML scene |
Update the scene with expert system and Java event handler data. |
A Jess console application modified to include a Feedback button and an Advice button are shown in figure 4. At any time during the lesson, the user can request feedback or advice. A simple knowledge base manages the sequence of practice activities, feedback, and advice for the practice environment. A set of facts define the practice activities, the sequence of activities, advice on accomplishing each activity, and feedback when an activity is successfully accomplished. The knowledge base contains a set of rules that manage the user's practice activities by determining which activities are currently valid and by updating the VRML world, based on the current situation.
Figure 4: A Modified Jess Interface for the Practice Environments
 |
Intelligent 3-D in action
With the objective of making 3-D practice environments an affordable training solution, existing tools and 3-D objects were used to create the sample mechanical skills training environment below. The sample environment trains maintenance technicians how to perform shaft alignment tasks by positioning the shafts of two pieces of coupled rotating machinery to work together smoothly. Shaft alignment is a common maintenance task at electric power plants, processing facilities, and manufacturing plants, and aboard ships. It's an ideal mechanical skills training problem for investigating the effectiveness of 3-D practice environments because performing precision shaft-alignment tasks requires
- significant cognitive and perceptual skills in addition to motor skills in visualizing the orientation of mechanical components
- practice in taking precision measurements
- visualizing the misalignment conditions from the precision-measurement tool readings
- using the perceived misalignment condition to select and use the correct mathematical formulas
- practice correcting the misalignment conditions.
Figures 5 and 6 illustrate the difference between a standard multimedia approach and an intelligent practice environment when training users to correct a horizontal parallel misalignment. Figure 5 shows a multimedia CBT practice session in which the user interacts with the equipment using a mouse to select objects or a click-and-drag approach to place objects in the correct location. The 3-D objects used in the example were created in a computer-aided design (CAD) program to generate animations for the multimedia CBT course. Shareware tools were used to convert the CAD files to VRML files. VRML editing tools and old-fashioned hand editing were used to create a hierarchy of VRML objects in separate files.
Figure 5: Problem-Solving Practice with Multimedia CBT (static version)
Figure 5: Problem-Solving Practice with Multimedia CBT (interactive version--viewable only in Internet Explorer)
| A VRML browser plug-in is required to view this demo. The Authorware web plug-in may be obtained from Macromedia. |
An intelligent practice environment with the capabilities of an expert system is another option (a static version is shown in figure 6). The expert system determines which activities the user can perform at the current time in the 3-D VRML scene. State information from the expert system is used to activate animations and selections in the VRML scene. The expert system console display in the lower left frame of the HTML page gives the user access to advice and feedback. The user requests advice with the Advice button, and automatically receives feedback when he or she finishes one of the active tasks. The user also can access feedback at any time with the Feedback button. The effectiveness of the advice and feedback is dependent on the quality of the knowledge captured in the knowledge base.
Figure 6: An Intelligent Practice Environment
The value of practice environments
Experience through practice is extremely important to the development of basic and advanced skills. A highly interactive practice environment encourages the user to explore complex relationships and increases the development of advanced skills through self-motivated discovery and problem-solving activities. The following elements are essential to an effective practice environment:
- Objective or goal. Understanding the objective for the training is necessary for a positive learning experience. For an effective learning experience, the user needs to understand why he or she is interacting with the 3-D environment.
- Learning approach. The user needs to be given a learning approach for interacting with the 3-D environment.
- Time-limit expectations. The user needs to know how much time he or she is expected to spend interacting with the practice environment.
- Association. The user's learning experience is greatly improved by associating the 3-D environment with related information. The user can learn by associating such related information as images, textual resources, and other data, with practice activities in the VRML world.
- Coaching. Coaching and guidance enhance a user's learning experience while performing complex tasks. Coaching can increase motivation for a novice user by preventing frustrating situations in the freedom of a typical VRML world.
- Feedback and assessment. Timely feedback directly related to the user's interactions increases his or her ability to analyze the practice situation. The user benefits from assessments that help him or her understand whether practice actions are correct.
Ongoing challenges
Use of CAD objects adds realism to the practice environments. However, reuse of CAD objects for multimedia is never easy. The original CAD designer's objectives are often different from those of a training practice environment designer. Inevitably, the objects must be edited. A good approach is to create a hierarchical scene with each object of interest in a separate VRML file. The 3-D scene is created as a composite of all of the individual VRML objects. A user's interactions are easier to monitor and analyze with a hierarchical scene composition.
The bandwidth needed to share 3-D practice environments over the Web or intranets is still an issue. CAD files are huge, and generating VRML files from CAD files produces large VRML files. Tools can be used to reduce the polygon counts and subsequent file size, but this often produces a loss of detail that undermines the quality of the practice environment. Another approach is to compress files for delivery. If they're still prohibitively large, a hybrid delivery approach can be used in which the basic practice environment is served over the Web and the large VRML files are distributed on CDs or other media to the users. If properly designed, many VRML files can be reused in different practice environment lessons without modification.
A multimedia designer is faced with tough design decisions when allocating functions to the different technologies. For example, animations and other object transformations can be developed using VRML alone, or they can be implemented in Java. Often, there isn't a clear criterion for selecting one approach over another. Usually, the final solution contains a mixture of approaches.
Ensuring that the technology works across several platforms and operating systems requires extra effort during development and increases the time required to test products. The lessons learned by Netscape in developing cross-platform components are equally applicable to integrating off-the-shelf technologies to build an intelligent training environment.
The examples described in this paper were crafted as prototypes to learn about the process of adding intelligence to practice environments and to begin understanding the potential advantages of intelligent practice environments. With a well-defined concept for an intelligent practice environment, it should be possible to develop software tools that generate most of the coding, instead of custom-crafting code for each lesson.
Janet Faye Johns is a principal engineer at The MITRE Corporation, where she's responsible for software systems design and development. Johns has been vice chair of the Association for Computing Machinery special interest group for Ada Artificial Intelligence Working Group since 1990.