Product: Questionmark Perception, version
Type: online survey tools
Company: Questionmark Corporation
Pricing: $2,995 for one year for 50 users
Does Questionmark Have the Answer?
There is a movement in the training and performance industry toward
evermore accurate assessment and measurement of human performance
in the workplace. Since 1959, thanks to Kirkpatrick's seminal work
on summative evaluations that use four levels of performance
measure, trainers have had guidelines in how to do it. When
training face-to-face in a classroom, the traditional model for
evaluating the results of the training may have been the level 1
"smile sheet" or reaction. Some companies are brave enough to move
up the Kirkpatrick ladder of evaluation methodology and strive for
more meaningful data collection, such as whether the learner can
apply the learning back on the job or as a result of the learning,
what business impact can be observed.
All this talk about measurement of training means that there must
be some tool used to do the measuring. The tools used for
assessments must be chosen or created. The tests used for measuring
must be chosen or created. The tests (in all fairness and legality)
must be valid and reliable. In other words, are you testing what
you want to test, and are you always testing for the same thing? In
the age of digital technology and online learning there are
additional standards to consider: SCORM, AICC, and IMS, to name a
few. If you're currently using a learning management system, it may
have a testing component included.
Questionmark is an assessment management system. Its job is
testing. There are four different types of tests (survey, exam,
quiz, and test) in 19 different formats, including drag and drop,
multiple choice, hotspots, and embedded Macromedia Flash files.
Testing can be administered to the learner offline using hardcopy,
CD-ROMs, or downloaded to PDA for later resync to a Web-based
More important, what good is testing if you can't compile, compare,
and evaluate the data? Another important component of the
Questionmark system is its reporting capability. Among the report
types are coaching, score list, grade book, transcript, gap,
survey, and item analysis. In particular, the item analysis report
can help clarify correlation and question validity issues.
Questionmark offers a workflow management system that enables
organizations to define and enforce processes and rules to control
the development and deployment of questions. There is an audit
trail for legally defensible high-stakes exams.
Installation and initial use
Perception is enterprise software designed to be installed on an
organization's network. Version 4 has the following system
- authoring: PC with at least 256Mb of RAM running Windows
- server: Web Server PC with at least 512 Mb of RAM running
Windows 2000 Server or Windows 2003 Server. If administering more
than 100 assessments a day, you need a separate database server
running MS SQL Server or Oracle.
The user interface is not intuitive to the casual user, nor is it
meant to be. A 148-page Getting Started with Perception Version
4 online manual walks you through the wizards and dialog boxes
used to create or edit topic areas, questions, assessments, and
The following types of help are available from within the
- wizards that guide you through the authoring process
- indexed, searchable, context-sensitive Windows-based help
system that includes a glossary
- an authoring best practice guide built into the authoring
As mentioned above, the Getting Started guide is available
in print and online. User manuals are available on the Questionmark
customer support site as searchable web documents and printable
PDFs. The PDF versions are also available on a CD. In addition,
there is an online knowledge base and best practice guides
available to Software Support Plan subscribers.
The web and CD-ROM feature tours that unfortunately are delivered
with background music. The audio-based narration would be nice to
listen to if not for the very obtrusive background noise.
Questionmark offers both phone and email support. Technical support
is free during product evaluation. After purchase, you may choose
to subscribe to a software support plan (pricing is a percentage of
the license purchase), or users can pay on a per incident basis.
Ease of use
Ease of use is obviously a relative term. If you're an
instructional designer, Questionmark Perception is understandable.
But the user interface does have a learning curve. Like all
software, the more you use it, the easier it is to use.
The user interface displays several different areas based on what
feature the user chooses:
- file menu
- navigation pane
- preview/properties pane
- display pane
- holder views pane.
Each area of the screen has a special function that changes as you
select a feature. However, the display pane and preview/properties
pane always show the contents of the folder view chosen and the
test item selected in the display pane.
To create one of the 20 different reports that Perception offers,
you can use the Perception Enterprise Manager and choose the
Enterprise Reporter menu option. Based on the permissions granted
by the administrator, users can run different reports on course
statistics, student statistics, or test item statistics.
The Questionmark Perception system has four major functions:
1. Authoring: Word templates for SMEs, Windows-based authoring
manager, browser-based authoring
2. Delivery: standard browsers (such as Netscape, MS Internet
Explorer, AOL); Questionmark Secure; Windows CD (no reporting);
PDA, hardcopy printout
3. Administration and scheduling: manage participants, manage
4. Reporting (Enterprise Reporter): assessment overview, grade
book, question statistics, survey, coaching, gap, item analysis,
score list, and transcript; export to ASCII or Excel spreadsheet
Developers can author questions using wizards in a Windows
environment and then store and edit them in local or shared
(remote) repositories. The authoring wizards make it easy to create
20 different types of questions. Designers can add multimedia files
including audio, video, Flash animations, and Macromedia Captivate
simulations. In addition, it is easy to assemble and publish
assessments from the questions created.
In Version 4, a new Authoring Manager replaces Question Manager and
Assessment Manager, allowing the user to author and publish
questions and assessments. Important enhancements include
- WYSIWYG HTML editing used to author questions
- support for style sheets; Version 4 supports cascading style
sheets, which enables easier global updates to existing assessments
- repositories that make it easier and faster to organize, store,
secure, and publish assessment content; authors can work directly
with local or shared repositories alone or in collaboration with
- workflow management module that enable organizations to define,
control, and document how questions move from creation to release
- security; multiple levels of control are now possible through
the administration of assigned access rights and multiple levels of
- content management tools.
The question-by-question format template presents one question at a
time. The template creates interesting possibilities, such as tests
and assessments that can be delivered to handheld wireless devices.
With the workflow management module, an organization can define and
enforce processes and rules to control the development and
deployment of questions. Workflow management lets users control and
document how questions move through stages (from authoring to
approval and then production). If multiple SMEs and designers are
working in more than one Perception repository, a tracking system
is imperative to successfully completing assessment projects. The
module also helps create an audit trail essential for legally
defensible high-stakes exams.
Another feature lets developers deliver immediately to a standard
browser for low- and medium-stakes assessments. The system can
deliver asynchronously via Perception to Go, which enables remote
users to synchronize their Windows PCs with a server. The end user
can download an assessment, complete it offline, and then upload
answers to the testing server. Users can print assessments when a
computer id unavailable. For high-stakes exams, Questionmark Secure
can limit the use of printing, screen sharing, navigation, and
Finally, Perception server software enables managers to schedule
individuals or groups. Users can log in, with a user name and
password if desired, to take a test or respond to a survey.
Perception Version 4 uses the original authoring tool for editing
files. Questionmark supports open standards, and Perception is
compatible with many learning management systems.
- It supports ADL SCORM, IMS Content Packaging, LOM (Learning
Object Metadata), IMS QTI, and AICC HACP and HR-XML.
- Question and assessment content are exportable to IMS QTI
- Assessments can be packaged into SCORM, AICC, and/or IMS
content packages for integration with an LMS.
- Perception Integration Protocol, one of two APIs, includes
methods for launching and tracking assessments according to SCORM
and AICC standards.
According to Questionmark, Perception is one of only two assessment
management systems to demonstrate interoperability with Blackboard,
Sakai, WebCT, and Moodle.
Value for the money
The cheapest Perception license is 50 participants for $2,995.00.
Other testing options are using an LMS or a testing site such as
Prometric. However, the proof is in the pudding, or one might say
"in the reports." What good is raw data if you can't sift, sort,
mash, and mull through it? One value measurement of Perception or
any other assessment system is how it can aggregate and display
data for evaluation purposes. The system offers a long list of
built-in reports that seem to meet most needs.
To quote a satisfied customer, "Our experience over four years
using versions 3 and 4 is that there is plenty of reporting power
and flexibility for our assessment data analysis and question item
analysis." The top three "likes" of this customer are the ability
to customize reports, administer security in a multi-author
environment, and design and develop new surveys.
Take a close look at the Questionmark's Perception assessment tool
if you're in the business of certification and compliance training.
While testing is being administered in most organizations at
different levels of the Kirkpatrick model, certification and
compliance require a structured, valid, and reliable assessment
system. Questionmark Perception might be the answer to your
Installation and initial use
Ease of use
Value for the money