Ratings Questionmark Perception

Product Information

Product: Questionmark Perception, version 4

Type: online survey tools

Company: Questionmark Corporation

URL: www.questionmark.com

Pricing: $2,995 for one year for 50 users

Demo: www.questionmark.com/us/download

Does Questionmark Have the Answer?

There is a movement in the training and performance industry toward evermore accurate assessment and measurement of human performance in the workplace. Since 1959, thanks to Kirkpatrick's seminal work on summative evaluations that use four levels of performance measure, trainers have had guidelines in how to do it. When training face-to-face in a classroom, the traditional model for evaluating the results of the training may have been the level 1 "smile sheet" or reaction. Some companies are brave enough to move up the Kirkpatrick ladder of evaluation methodology and strive for more meaningful data collection, such as whether the learner can apply the learning back on the job or as a result of the learning, what business impact can be observed.

All this talk about measurement of training means that there must be some tool used to do the measuring. The tools used for assessments must be chosen or created. The tests used for measuring must be chosen or created. The tests (in all fairness and legality) must be valid and reliable. In other words, are you testing what you want to test, and are you always testing for the same thing? In the age of digital technology and online learning there are additional standards to consider: SCORM, AICC, and IMS, to name a few. If you're currently using a learning management system, it may have a testing component included.

Questionmark is an assessment management system. Its job is testing. There are four different types of tests (survey, exam, quiz, and test) in 19 different formats, including drag and drop, multiple choice, hotspots, and embedded Macromedia Flash files. Testing can be administered to the learner offline using hardcopy, CD-ROMs, or downloaded to PDA for later resync to a Web-based server.

More important, what good is testing if you can't compile, compare, and evaluate the data? Another important component of the Questionmark system is its reporting capability. Among the report types are coaching, score list, grade book, transcript, gap, survey, and item analysis. In particular, the item analysis report can help clarify correlation and question validity issues.

Questionmark offers a workflow management system that enables organizations to define and enforce processes and rules to control the development and deployment of questions. There is an audit trail for legally defensible high-stakes exams.

Installation and initial use

Perception is enterprise software designed to be installed on an organization's network. Version 4 has the following system requirements:

• authoring: PC with at least 256Mb of RAM running Windows 2000/XP/2003
• server: Web Server PC with at least 512 Mb of RAM running Windows 2000 Server or Windows 2003 Server. If administering more than 100 assessments a day, you need a separate database server running MS SQL Server or Oracle.

The user interface is not intuitive to the casual user, nor is it meant to be. A 148-page Getting Started with Perception Version 4 online manual walks you through the wizards and dialog boxes used to create or edit topic areas, questions, assessments, and outcomes.

Help/support

The following types of help are available from within the application:

• wizards that guide you through the authoring process
• indexed, searchable, context-sensitive Windows-based help system that includes a glossary
• an authoring best practice guide built into the authoring application.

As mentioned above, the Getting Started guide is available in print and online. User manuals are available on the Questionmark customer support site as searchable web documents and printable PDFs. The PDF versions are also available on a CD. In addition, there is an online knowledge base and best practice guides available to Software Support Plan subscribers.

The web and CD-ROM feature tours that unfortunately are delivered with background music. The audio-based narration would be nice to listen to if not for the very obtrusive background noise.

Questionmark offers both phone and email support. Technical support is free during product evaluation. After purchase, you may choose to subscribe to a software support plan (pricing is a percentage of the license purchase), or users can pay on a per incident basis.

Ease of use

Ease of use is obviously a relative term. If you're an instructional designer, Questionmark Perception is understandable. But the user interface does have a learning curve. Like all software, the more you use it, the easier it is to use.

The user interface displays several different areas based on what feature the user chooses:

• toolbar
• file menu
• navigation pane
• preview/properties pane
• display pane
• holder views pane.

Each area of the screen has a special function that changes as you select a feature. However, the display pane and preview/properties pane always show the contents of the folder view chosen and the test item selected in the display pane.

To create one of the 20 different reports that Perception offers, you can use the Perception Enterprise Manager and choose the Enterprise Reporter menu option. Based on the permissions granted by the administrator, users can run different reports on course statistics, student statistics, or test item statistics.

Best uses/functionalities

The Questionmark Perception system has four major functions:

1. Authoring: Word templates for SMEs, Windows-based authoring manager, browser-based authoring

2. Delivery: standard browsers (such as Netscape, MS Internet Explorer, AOL); Questionmark Secure; Windows CD (no reporting); PDA, hardcopy printout

3. Administration and scheduling: manage participants, manage schedules

4. Reporting (Enterprise Reporter): assessment overview, grade book, question statistics, survey, coaching, gap, item analysis, score list, and transcript; export to ASCII or Excel spreadsheet available.

Developers can author questions using wizards in a Windows environment and then store and edit them in local or shared (remote) repositories. The authoring wizards make it easy to create 20 different types of questions. Designers can add multimedia files including audio, video, Flash animations, and Macromedia Captivate simulations. In addition, it is easy to assemble and publish assessments from the questions created.

In Version 4, a new Authoring Manager replaces Question Manager and Assessment Manager, allowing the user to author and publish questions and assessments. Important enhancements include

• WYSIWYG HTML editing used to author questions
• support for style sheets; Version 4 supports cascading style sheets, which enables easier global updates to existing assessments
• repositories that make it easier and faster to organize, store, secure, and publish assessment content; authors can work directly with local or shared repositories alone or in collaboration with other authors
• workflow management module that enable organizations to define, control, and document how questions move from creation to release
• security; multiple levels of control are now possible through the administration of assigned access rights and multiple levels of access
• content management tools.

The question-by-question format template presents one question at a time. The template creates interesting possibilities, such as tests and assessments that can be delivered to handheld wireless devices.

With the workflow management module, an organization can define and enforce processes and rules to control the development and deployment of questions. Workflow management lets users control and document how questions move through stages (from authoring to approval and then production). If multiple SMEs and designers are working in more than one Perception repository, a tracking system is imperative to successfully completing assessment projects. The module also helps create an audit trail essential for legally defensible high-stakes exams.

Another feature lets developers deliver immediately to a standard browser for low- and medium-stakes assessments. The system can deliver asynchronously via Perception to Go, which enables remote users to synchronize their Windows PCs with a server. The end user can download an assessment, complete it offline, and then upload answers to the testing server. Users can print assessments when a computer id unavailable. For high-stakes exams, Questionmark Secure can limit the use of printing, screen sharing, navigation, and save-to-disk functions.

Finally, Perception server software enables managers to schedule individuals or groups. Users can log in, with a user name and password if desired, to take a test or respond to a survey.

Compatibility

Perception Version 4 uses the original authoring tool for editing files. Questionmark supports open standards, and Perception is compatible with many learning management systems.

• It supports ADL SCORM, IMS Content Packaging, LOM (Learning Object Metadata), IMS QTI, and AICC HACP and HR-XML.
• Question and assessment content are exportable to IMS QTI format.
• Assessments can be packaged into SCORM, AICC, and/or IMS content packages for integration with an LMS.
• Perception Integration Protocol, one of two APIs, includes methods for launching and tracking assessments according to SCORM and AICC standards.

According to Questionmark, Perception is one of only two assessment management systems to demonstrate interoperability with Blackboard, Sakai, WebCT, and Moodle.

Value for the money

The cheapest Perception license is 50 participants for $2,995.00. Other testing options are using an LMS or a testing site such as Prometric. However, the proof is in the pudding, or one might say "in the reports." What good is raw data if you can't sift, sort, mash, and mull through it? One value measurement of Perception or any other assessment system is how it can aggregate and display data for evaluation purposes. The system offers a long list of built-in reports that seem to meet most needs.

To quote a satisfied customer, "Our experience over four years using versions 3 and 4 is that there is plenty of reporting power and flexibility for our assessment data analysis and question item analysis." The top three "likes" of this customer are the ability to customize reports, administer security in a multi-author environment, and design and develop new surveys.

Recommendation

Take a close look at the Questionmark's Perception assessment tool if you're in the business of certification and compliance training. While testing is being administered in most organizations at different levels of the Kirkpatrick model, certification and compliance require a structured, valid, and reliable assessment system. Questionmark Perception might be the answer to your assessment challenges.

Rating