| |
| The Sponsors |
|
|
This Shootout
was held March 1st, 2nd, and 3rd in 2004 in Atlanta, Ga.,
at the World Congress Center. |
Gallery |
 The
E-Assessment Shootout tool place on the Educational Stage,
located in the Expo hall. Members of the audience serve as
judges for the competition.
 The
team from TRA shows how they completed the task using their
tool called "automatic e-Learning."
 We
had good participation from the audience and want to thank
everyone who took the time to rate each product.
The team from Eedo Knowledgeware developing an exam in front
of a live audience – pressure!!!
 After
completing the task, each team debriefed the audience on their
assessment. This is the team from Deakin KM, telling how they
constructed the exam using their product – Knowledge
Presenter.
Bryan Chapman, e-learning analyst at Brandon
Hall Research, served as the referee for the Shootout.
|
Gallery of
Simulations For those of you who
weren't able to join us in person, we've invited the participants
to post the assessment they built here. We'll add more links
as they become available.
| Product (Company) |
Link to Software
Simulation |
| Exam Engine
(Platte Canyon Multimedia Corp) |
|
| ExamBuilder
(ExamBuilder) |
|
| Testcraft
(Ingenious Group) |
|
| ForceTen
(Eedo Knowledgeware) |
|
| Knowledge Presenter
(Deakin KM) |
|
Challenge for Others... If
you have assessment software and for some reason weren't able
to join us for the Shootout, we'd like to challenge you to
download
the nine questions and create your own exam. Once you've created
your version of the exam, post it to your own Web server,
then send the link to bryan@brandon-hall.com.
If the page is free of advertisements and does not require
a log-in to run, we will post the link for others to see.
We hope to make this a page that will serve as a test bed
for what's possible when creating online exams. Send us your
link today!
| Product (Company) |
Link to Software
Simulation |
| The Quiz Show |
|
|
|
|
| The Participants |
|
Background |
| Assessment is a critical part
of any learning strategy, yet we sometimes take online assessment
tools for granted. Nearly every authoring tool, LMS, LCMS, etc.,
has some form of built-in test creation and delivery mechanism.
The goal of this Shootout was to focus on which systems do the
best job of (1) making it easy for relatively novice developers
to create tests, exams, quizzes, etc.; (2) to see how robust
and dynamic an online assessment can be (e.g., How do learners
move through the questions? What options are available for feedback?
How is performance data tallied and displayed?); and (3) to
find out which systems do the best job of analyzing and reporting
data on the back end (post assessment).
|
| In this live format, assessment
vendors were given a CD containing raw material for nine test
questions. Within a few minutes, their task was to take the
raw material and turn it into a fully functional online exam.
Each team then had the opportunity to debrief the audience on
what they built. The audience, in turn, rated each system. |
 |
The competition was run over
the course of three days, with three new teams competing each
day. The audience was able to observe the full development
on large screen to make it clear that each team was doing
original work on-site. As the event began, each team received
a CD-ROM containing one Microsoft Word file and four graphics.
The Word file contained the content for each of the nine questions
they had to build. (You can download
a zip file containing the actual Word document that each team
received, as well as the four graphics). The CD-ROM, along
with a printed copy of the exam questions, were presented
to each team in a sealed envelope that they were only allowed
to open in front of the live audience.
Here is a list of the question types found in the document,
along with an explanation of why the question type was included:
| Question Type |
Purpose and Approaches |
| #1 Multiple-choice question |
Standard question type. We decided to start
the task with a simple question type that is common to
nearly every assessment tool. However, right from the
start, the audience could see variations in the forms
and wizards used to create a simple question. |
#2 Multiple-select, multiple-choice question |
Some assessment systems limit developers
by not allowing a question to have more than one correct
answer. This was a test to make sure systems didn't restrict
users. |
| #3 Multiple-choice, with graphic |
The question required visual recognition
to make a positive identification. The question tests
the system's capability to support graphics as part of
a question. |
| #4 True/false |
Standard question type. |
| #5 Fill-in-the-blank (or equivalent) |
This question is available in most assessment
system, however, there is often some differences in the
way they handle things like case matching, spelling tolerances,
etc. We designed the question to have 2 different words
slots to fill in, so that it would be difficult to replicate
as a multiple choice question with one answer. |
| #6 Matching (or equivalent) |
Instructionally, this question was written
as a classification exercise. The focus was not to have
the competitors create a specific type of matching question,
rather, we wanted to see the variance of different approaches.
Some teams produced matching similar to the old paper-and-pencil
matching format, others used a drag-and-drop interface,
yet others created the assessment item as a series of
two-item multiple choice questions. It was fun to watch
the differences in their systems. |
| # 7 Hot spot (or equivalent) |
Again, we were not looking for a specific
format for the question. The idea is that the learner
must visually recognize a specific control on the back
of the clock (our subject matter for the competition).
Some teams had the learner click directly on the graphic
(ala hot spot), others opened the picture of the clock
controls in a paint package and labeled the controls from
A-D, then wrote the question as a visual, multiple-choice
question. |
| #8 Labeling |
From a learner perspective, we wanted to
know if they could recognize the parts of the clock. We
called this the "labeling" question, however,
the teams could use any format desired to complete the
task. Some teams used click-and-drag, allowing the learner
to pick up the label and drop it on the part of the clock,
others touched up the graphic to show the part of the
clock, then used a matching question to have them match
the name to its corresponding part. Again, it was interesting
to see the different approaches. |
| #9 Author's choice |
We decided to allow for one open-formatted
question to see if the assessment product vendors had
any creative question types they would like to share with
the audience. |
This was not the first time that the teams had seen the content.
One month ahead of the competition they were given a practice
copy of the exam, which had the exact same question types
and subject matter so they could practice for the competition.
However, the content was altered for the final exam to ensure
that all work was done originally, on-site.
Once each team was ready, the competition began. A large
clock showed the audience how much time each team spent in
the development process. The completion times were recorded
for each team. Completion times did not factor into the final
score.
The first team that finished in each round had the opportunity
to make the first debrief presentation. Each team had a total
of eight minutes to explain to the audience how they approached
the task. During the debrief session, each team was instructed
to answer the following areas:
- A brief demonstration of how test question items were
created. You will probably not have enough time to walk
through each question type, so pick a few types that illustrate
the process a developer would use to create a test question,
set parameters for the exam, and the publishing process.
- Show the output of the assessment. Run through the nine
questions created and show the audience how it works. Describe
any characteristics that would help the audience understand
the flexibility and range of the system, such as randomizing
test questions, randomizing answers, feedback, opportunities
to review performance, score cards, etc. We understand that
not all systems will have the same features in this area.
Remember that audience will rate your system on the robustness
of the output, so focus on what makes your system unique.
- Show how the system collected performance data. The last
objective of the debrief is to show the audience how the
system collected data, reporting flexibility, how someone
might use this data to assess learner performance and/or
test accuracy.
Following the debrief from each team, the audience evaluated
each system according to the following categories:
On a scale of 1 to 10 (with 10 being the highest rating)
how would you rate this product in the following areas?:
| Ease of development
|
10 9 8 7
6 5
4
3
2 1 |
| Robustness of the assessment
output |
10 9 8 7
6 5
4
3
2 1 |
| Data collection, reporting,
assessment statistics |
10 9 8 7
6 5
4
3
2 1 |
The top three finishers in each category are shown, as well
as overall winners for the competition. Teams could earn a
total of 10 points in each category, and the overall winner
was based on adding the individual category scores together
for a total possible score of 30 points. Since the audience
varied in size in each day of the competition the scores were
averaged in each category.
We'd like to thank all of the teams that participated in
the Shootout. It was a great opportunity to see nine different
teams tackle the same business use case using the same content.
The audience commented to us how much they learned about how
to create, deploy, and track performance data for online assessments.
|
Overall Winners |
|
|
Product |
Company |
Score |
|
1st Place |
Testcraft |
Ingenious Group |
24.03 |
|
2nd Place |
ForceTen |
Eedo Knowledgeware |
23.87 |
|
3rd Place |
Knowledge
Presenter |
Deakin KM |
23.54 |
30 points possible, with 10 points possible in each category. |
Ease of Use |
|
|
Product |
Company |
Score |
|
1st Place |
ForceTen |
Eedo Knowledgeware |
8.30 |
|
2nd Place |
Testcraft |
Ingenious Group |
8.24 |
|
3rd Place |
Knowledge
Presenter |
Deakin KM |
8.23 |
10 points possible. |
Robustness of Assessment Output |
|
|
Product |
Company |
Score |
|
1st Place |
Knowledge
Presenter |
Deakin KM |
8.06 |
|
2nd Place |
ForceTen |
Eedo Knowledgeware |
7.87 |
|
3rd Place |
Rapid
Exam |
XStream Software |
7.83 |
10 points possible. |
Data Collection, Reporting, Assessment Statistics |
|
|
Product |
Company |
Score |
|
1st Place |
ExamBuilder |
ExamBuilder |
8.21 |
|
2nd Place |
Testcraft |
Ingenious Group |
8.06 |
|
3rd Place |
Rapid
Exam |
XStream Software |
7.74 |
10 points possible. |
|
|
